A stale CLAUDE.md beats the code in 90% of coding-agent choices

Ninety point one percent of convention choices followed the project’s instruction file over the code it described. That number comes from a paper posted yesterday that ran Claude Code and Codex CLI 320 times on 40 Python projects built so the two disagree (Xu and Wu, arXiv 2610.30571). The CLAUDE.md or AGENTS.md asks for one convention, say print, while the existing code uses another, say logging, and the agent is asked to add a feature. Claude Code pointed out the conflict in 79 of 80 runs and followed the stale file anyway. Codex CLI rewrote the existing code to match the file in 20 of 80 runs, and the project’s own tests then failed in 17 of those. One sentence in the user-level instruction file giving existing code precedence moved code-following choices from 4.2% to 91.4%, stopped the rewrites, and took the test pass rate from 89.4% to 100%. The authors’ explanation is about channels: the instruction file reaches the model as an instruction, the code reaches it as tool output, and models are trained to rank tool output lower. If you run agents across repos whose instruction files were written once and never revisited, add that sentence today; if you build a harness, ask why the precedence rule has to live in user config at all.

What an agent is told versus what it is graded on is also the subject of Fei Ding’s audit of test-graded coding benchmarks (arXiv 2610.30635). Ding names a specification-oracle gap: the task is described in prose, but the grading tests come from upstream and can depend on a constant’s name, a module path, or an exact message the description never states. A static audit across five languages classifies every name-, path-, and literal-level dependency of every scored test, at 89.2% precision and 92.9% recall on held-out data. On SWE-Milestone, 25 of 98 graded milestones contain a test that depends on an unstated, non-inferable name or path. Build units with such a dependency fail in 80.6% of runs against 48.4% for comparable units without one, and withholding a single hidden name from the reference solution stops its unit’s tests from compiling in 126 of 128 cases. The leaderboard ranking survives removing the affected tests, but scores shift within repositories toward agents that happen to reproduce upstream names. For anyone building evals, the cross-benchmark result matters most: a third or more of tasks are affected where specifications were written from release notes or LLM summaries, and almost none where tasks were screened or written for their tests. The audit, corrected filter lists, and specification addenda are public.

When a coding agent fails, what should the harness hand it next? Lyu et al. (arXiv 2610.28834) hold the failed state fixed and use Banzhaf attribution to measure the marginal contribution of computation, information, and test feedback, across six models, 194 function-level failures from MBPP+ and HumanEval+, and 126 repository-level failures from SWE-bench Verified. Task-relevant specifications consistently improve repair, public-test feedback gives small gains, and retrieved code gives little to none. Extra computation is worth less as models get stronger, while information keeps its value. The finding that should change harness design is about preference: three of four frontier models choose feedback in 95 to 100% of cases, and that choice earns lower repair gains than picking a resource uniformly at random. Read beside the instruction-file paper, it describes agents that are poorly calibrated about which inputs to trust and which to ask for, and a harness that lets the model choose its own recovery path inherits that miscalibration.

Business rules have the same specification problem as benchmark tests. AgenTest (Quan et al., arXiv 2610.28116) mines the rules a tool-using agent is supposed to obey from system prompts, tool definitions, and domain knowledge, rewrites them as Given-When-Then specifications, builds an environment state and simulated user that satisfy each Given, and checks the Then over messages, tool calls, and state changes. On tau2-bench’s airline, retail, and telecom agents under two models, it produced 551 refined specifications and caught violations of 89 and 121 distinct ones, 3.4x and 2.5x as many as all baselines combined. It was the only method that found agents improperly reading or modifying another user’s data, and about 60% of bug-triggering runs could not be detected from the final state alone. For agents operating on governed data, that last figure is the argument for checking the full trajectory against explicit rules, since an end-state diff misses most of what went wrong.

On the analytics side, Peihao Yuan’s InfoQ article (AI Changes the Data Interface, Not the Query Engine) walks through a disaster-recovery analytics workflow built on a determinism split: the agent generates constrained SQL or Python, DuckDB computes reproducible facts from Parquet, and the agent interprets the result. It is an architecture argument rather than a measurement, with four signals for deciding when an embedded engine is enough and when a managed lakehouse earns its cost. The boundary it draws is the useful part, because the model chooses what to compute while the engine owns the numbers, so every figure in an answer can be replayed.

Muffin (Kuo et al., arXiv 2610.31355) goes after a coordination failure in multi-agent reasoning, where critical intermediate findings never reach the agent that needs them. Instead of synchronous debate with full-trace sharing, an orchestrator routes queries by reasoning demand, sends the hard ones to a multi-agent setting, and gives those agents an asynchronous shared memory buffer for intermediate findings while each agent’s private reasoning stays local. It posts the best accuracy in 17 of 20 benchmark settings and, against multi-agent debate, gains 19.2 accuracy points on 77.4% fewer tokens. The benchmarks are target-driven reasoning tasks, so whether a shared findings buffer holds up for parallel coding agents editing one repository remains untested.

Manju Rajashekhar’s InfoQ talk (Actors for Agents: Patterns for Production AI) makes the runtime case for treating agents as distributed systems: one owner per piece of state, message passing, write-through state, and supervision boundaries so that failures, continuations, and mutable execution graphs have somewhere defined to land. Muffin’s buffer and an actor’s mailbox answer the same question of who owns an intermediate result and who may read it.

Three things to watch. Harness vendors could ship code precedence as a default, or make the ranking of instruction files against tool output configurable per project, instead of leaving it to a sentence in user config. Benchmark maintainers could run the specification-oracle audit before the next leaderboard round and stop writing task specs from release notes. And the resource-attribution paper invites a direct follow-up: give a frontier agent its menu of recovery resources with their measured value attached, and see whether it still reaches for test feedback.

---

Source freshness: the code-intel-copilot mirror reported its last sync as 2026-09-01 (about 74,760 minutes stale), although the candidate pool held items dated through 2026-10-23. Every item here was checked against its own published date.