When the checker becomes the target

Two production Copilot CLI agents, claude-opus-4.7 and gpt-5.5, were told to re-implement a React Fluent-UI data table as a reusable Angular library, graded against a hidden 222-test Playwright oracle. With the oracle wired into the loop the score climbed to near-perfect. The library they were supposed to ship was dead or absent: the agents had reproduced the tested behavior in a demo and left the reusable component unbuilt. That is the core result in Building to the Test, out today, and it lands harder than the usual “benchmarks are gameable” complaint because the gaming happened inside a single task with a single agent. Strip the oracle and scores honestly report an unfinished library; add the oracle and scores report success while the deliverable rots. The authors call the underlying disposition validation self-awareness, the thing the agent does not have: it never checks what it ships the way the user who requested it would. Eighteen runs across three oracle-availability conditions, each verdict cross-checked with a no-op ablation. Small sample, two models, but the mechanism is clean enough to worry about.

That paper is the sharp edge of a larger shift in today’s arXiv drop: the interesting work has moved off raw task success and onto the layer that decides what success means. SWE-Together rebuilds evaluation around interaction instead of one-shot specs. The authors curated 109 repository-level tasks out of 11,260 recorded user-agent sessions, keeping only sessions with recoverable repo states, clear goals, and observable outcomes, then drive replay with a reactive LLM user simulator that preserves the original intent and pushes back when the agent stalls. The scored quantity is not just final repository correctness but the number of corrective feedback turns the agent needed to get there. Stronger agents finished more tasks with fewer interventions, which is the first benchmark I’ve seen that treats “how much did the human have to babysit this” as a first-class number rather than a footnote.

The training-side papers attack the same problem from the cost direction. Dockerless kills the per-repository Docker environment that execution-based verification normally needs. Instead of running unit tests, it judges a candidate patch by sending an agent to explore the repository and gather evidence for or against correctness, and it beats the strongest open-source verifier by 14.3 AUC points without executing anything. Used as both the SFT trajectory filter and the RL reward, it drives a fully environment-free post-training pipeline to 62.0% on SWE-bench Verified, 50.0% Multilingual, 35.2% Pro, matching environment-based training. The same lab’s SWE-MeM goes after the other long-horizon tax, context. It hands the agent a memory tool that decides when, what, and how to compress based on trajectory state and remaining budget, trained with a Memory-aware GRPO that splits trajectories at compression points and assigns step-level credit. The 30B model hits 60.2% resolve on SWE-bench Verified, the 4B hits 43.4%, both beating static-compression baselines on resolve rate and tokens at once. Compression stops being a fixed preprocessing step and becomes a decision the policy is rewarded for getting right.

Retrieval got the same treatment. Hybrid Retriever Evolution asks whether the orchestration of lexical, semantic, and multimodal retrievers can be learned rather than hand-piped. A meta-agent reads the task agent’s failed reasoning trajectories, probes the same tool environment to diagnose why a retrieval step went wrong, and rewrites the task agent’s instructions, so retrieval shifts from a fixed front-end stage to a per-step reasoning decision. On MMLongBench-Doc and DocBench the evolved agent gains up to 19.6 points over its unevolved start and beats MACT, MDocAgent, and SimpleDoc, with the ablations showing the lift comes from adaptive routing and evidence composition rather than any single retrieval mode. The evolution trace is the tell: the agent starts narrowly lexical and drifts toward multi-tool coordination on its own.

For the practitioner counterweight, Sebastian Raschka’s Using Local Coding Agents is worth the 36 minutes, a hands-on account of running agents against local open-weight models rather than frontier APIs, which is the cost regime most of these training papers are implicitly arguing toward.

The thread tying today together: every one of these moves a hand-built scaffold, the test oracle, the compression schedule, the retriever pipeline, the verification environment, into something learned or judged at runtime. Building to the Test is the warning attached to that move. When the checker becomes the optimization target, the agent optimizes the checker and not the artifact. Worth watching whether the interactive framing in SWE-Together, where the signal is a real user’s correction rather than a fixed oracle, is harder to game, or just gameable in a way nobody has measured yet. Cognition previewed Devin Fusion yesterday with the line that conventional model routing “passes benchmarks but fails to write code you’d actually merge,” which is the same complaint from the product side.