The model is the constant; the harness is the variable

Hold the model constant, change only the harness across 35 sequential releases, and coding-agent quality still swings measurably. That is the result of a longitudinal study of the Qwen Code CLI released today (arXiv 2607.03691), which fixes the underlying LLM and evaluates each of 35 releases against 50 stratified SWE-bench Verified tasks. The authors first surveyed five open-source harnesses (Codex, Qwen Code, Gemini, OpenCode, OpenHands) and clocked release velocities above two per day, thousands of issues in months, then traced specific effectiveness and efficiency shifts back to individual pull requests in the middleware that orchestrates system prompts, tool execution, and context management. When an update regresses, practitioners reflexively blame the model. The controlled evidence points one layer down, at the harness.

Anthropic’s Claude Code team spent the same window putting numbers on how much of that layer they now tune by eval rather than by feel. In a fireside chat transcribed by Simon Willison, Cat Wu said the Claude Code system prompt shrank by 80% for frontier models, because over-constraining hurt: adding examples stopped being best practice, and lists of “don’t do X” degraded results by conflicting with later user instructions. The review loop is where the discipline shows. Their GitHub review bot now does the bulk of review on every PR; when an incident traces to a bad PR, that PR goes into a code-review eval set so the same class never regresses; and for changes at the “outer layers” of the product, no human reviews at all. Claude Tag, the Slack integration, lands 65% of the team’s product-engineering PRs. Its memory, notably, is a markdown file per channel, not a vector store.

Which is a useful reality check against where the memory benchmarks landed this week. RECON (arXiv 2607.16716) is a new benchmark for compositional reasoning over 24 case files of 50k to 100k tokens each, and it deliberately tests what happens after a fact changes: which downstream conclusions get invalidated, which survive on independent support, how a counterfactual timeline would have unfolded. Retrieving scattered facts is the easy part; propagating a cascade is not. The strongest non-Oracle system scored 22.4% accuracy. That is the honest ceiling for “reason over your history,” and it is low.

The practitioner side of the same problem showed up on r/LLMDevs, where a builder ran LongMemEval end to end on the full 500-question oracle set instead of a sample and reported 73.6% QA accuracy with a one-command repro. The load-bearing detail is the confession: the sampled subset flattered them, so the sampled number was useless for regression testing. What moved the score was unglamorous, contradiction handling when two sources disagree, refusing to store low-confidence extractions at all, and hybrid retrieval that fuses full-text with semantic rather than betting on either alone. It rhymes with Anthropic’s point that the eval, not the tooling, is the hard part.

Retrieval got a production data point too. JetBrains shipped JetBrains Context, a semantic repository index that answers agent queries instead of making the agent grep and read files, with multi-repo search across checkouts the agent does not hold locally. On 205 SWE-bench tasks, 175 production-monorepo tasks, and 1,953 code-localization tasks, they report agent turns cut by up to 68%, latency by up to 59%, and cost by up to 48%. The framing is the interesting part: the honeymoon where any AI output satisfied is ending, AI quotas are appearing, and brute-forcing a large codebase with a frontier model on limitless tokens no longer pencils out. Retrieval becomes a cost lever, not just a quality one.

If retrieval is where cost hides, review is where cost surprises you. A trajectory-level analysis of terminal-based agentic code review (arXiv 2607.16740) found the reviewers hit higher precision than performance-only evals suggested, but paid for it in exploration and validation overhead the aggregate score never showed. Successful reviews correlated with stronger up-front planning and less downstream validation, which is the argument for trajectory-aware, cost-sensitive evaluation rather than one precision number per system.

The common thread across all six: the model is increasingly the fixed input, and the variable that decides quality is the machinery around it, the harness, the retrieval index, the memory store, the review loop, plus the eval that tells you which change helped. Anthropic promised to publish the auto-mode and code-review evals “in the coming weeks.” Worth watching whether harness changelogs start shipping eval deltas alongside the diffs, so the next regression has a number attached instead of a shrug at the model.