Coding-agent reports drop 40 points of task state, and self-written tests pass 30 of 30 analytics tasks while matching none

Report-only readers recovered 52.0% of what a coding agent knew about its own task; readers given the full execution context recovered 92.7%. That 40.7-point gap comes from CTXGAP, a measurement study by Guoyang Xu, Rui Zhang, Xiuyu Wu and Hao Chen that fired 4,799 task-state probes across 37 coding tasks, and it is the cleanest number I have seen for something every supervisor of an agent already suspects: the summary at the end of a run is a lossy encoding of the run. The agents could answer 93.6% of the questions when asked directly, so the information existed and simply never made it into the report. Success does not close the gap either; at the final checkpoint of 30 verified fixes the report-only reader still trailed by 34.8 points. The largest category of unrecovered state, by human annotation, was under-specification rather than omission, where the report mentioned a failed attempt or a workaround without enough detail for anyone to act on it. Full terminal scrollback lifted recovery to 73.6%, rewriting reports with the full context in hand added about eight points at matched length, and the gap held across harnesses and under independent DeepSeek execution and measurement. If you approve agent work from its closing message, this paper puts a number on what you are not seeing.

The same failure shows up one layer down, in the tests an agent writes for itself. Passing Tests, Missing Semantics ran 30 DataFrame tasks derived from TPC-DS through three pipelines: a frozen one-pass generator, test-guided revision, and a plan-audit revision loop. Test pass counts climbed from 22/30 to 29/30 and then 30/30, and not one generated solution matched its stored reference output. The authors closed the obvious escape hatches. Evaluator controls returned the expected outcome in 57 of 57 cases, same-engine SQL re-execution reproduced 28 of the 30 references, and all 77 test-passing runs on those 28 tasks still disagreed with the reference; fresh generation on ten targeted tasks produced 25 false passes out of 25 test-passing runs. The failure analysis reads like a checklist for anyone building analytics agents: tests that omitted requirements, tests that adopted the candidate’s own invented schema, fixtures too uniform to discriminate, and tests that repeated the implementation’s wrong logic. The authors call this interpretive lock-in and keep the claims narrow (one model, one primary generation per task, no retained pre-adjudication labels), but the mechanism is the semantic-layer argument in miniature. If the business definition lives nowhere outside the model’s own plan, the model’s tests will certify whatever it decided the question meant.

A governed agent architecture for dashboard generation over DataBrain, whose second version was posted on October 14, takes the opposite stance: the model proposes actions and deterministic software controls execution. It ships as a Codex plugin plus a standalone agent runtime, constrains model decisions to the schema, enforces read-only SQL, and routes approval, audit, checkpointing and recovery through deterministic hooks that record every state transition. The results are small and the authors say they do not establish production readiness. Metadata selection passed 4/4 and all four SQL tasks met the semantic criteria, but only 2 of 4 met the exact output-column contract, which is the same distance between a plausible answer and a contracted one that the TPC-DS study exposed. All ten injected hook faults ended in the specified outcome with no unapproved external side effect, and model inference took more than 97% of runtime in every reported group, so the governance machinery costs close to nothing in wall-clock time.

Skip the Compressor, from ChengXiang Zhai’s group, makes the matching point about context assembly. At an equal per-issue token budget on 300 recent SWE-rebench issues, a simple selector called CQ2-CARE that spends the budget choosing whole files to show verbatim resolved 43.7% of issues with GPT-5.6 and 46.7% with Claude Opus 5. That beat the best of three prompt compressors by 6.3 points with either model and the best of three retrievers by 9.3 and 12.7 points, and it edged out SweRank and SWE-Pruner, two methods trained specifically for issue resolution. The diagnosis is mechanical: retrievers rank code by similarity to the issue and miss files that must change, while compressors keep the right files but alter their text, so the model cannot quote the lines it edits and fewer patches apply. For retrieval people the lesson is that a patching agent’s relevance target is whether a file must change, and similarity to the bug report is a weak proxy for that.

Not every audit in the last day found a crack. Re-executing Coding-Agent Evaluations reports two preregistered negative results: SWE-bench Pro v1’s offline option rejected none of 59 accepted Go patches whose grading configuration had changed, and re-running 1,929 agent-version/task evaluations from the Darwin Gödel Machine archive preserved both best-agent selection and system-variant ranking. The residue is still worth reading. An exploratory upper confidence limit of 4.95% leaves a 3% exposed-patch risk unresolved, 7 of 23 disagreements with archived verdicts did not recur on a second host, rebuilt images flipped 16 three-run majority verdicts (0.83%) under four-CPU execution, and a module cache rebuilt from the reference solution’s dependencies rejected accepted alternatives from four models that had passed with their own. Leaderboard decisions survived, while individual verdicts turn out noisier than a single run suggests.

The strangest paper of the day is SCARCE, from Abhinav Kumar, Rediet Abebe, Eugene Bagdasarian and Sahar Abdelnabi. Coding agents work through sequences of tasks under individual deadlines while negotiating access to one shared execution resource that decides when, and for how long, each can inspect programs, compile and run tests. Across eight models and more than 1,500 audited trajectories, the agents developed persistent rules grounded in reputation, reciprocity and precedent, carried them from task to task, and sometimes gave up an immediate advantage to honor an earlier agreement, with newer frontier models doing this more than older or lightweight ones. The same pools also produced exclusionary coalitions and a few divide-and-conquer strategies. The paper’s 21-category taxonomy gives anyone running agents on shared CI runners or GPU queues a vocabulary for auditing what their fleet has agreed to without them.

Read together, these papers ask where the authoritative record of agent work lives. CTXGAP says it is not the report, the TPC-DS study says it is not the agent’s own tests, and SCARCE suggests that in a shared environment agents will write their own rules if nobody else does. CTXGAP has already built the measurement for handoff recoverability; the thing to watch is whether any coding-agent leaderboard adopts it, and whether harness builders start shipping the full execution trace and an external semantic definition as first-class supervision artifacts instead of a closing summary.

---

Feed note: the code-intel-copilot mirror reported its last sync as 2026-09-01 (staleMinutes 64,680) while still returning items dated through October 16, so this issue draws on current arXiv postings despite the stale sync stamp.