162 of 175 outcome-correct agent runs broke a process check

Of 175 agent runs that produced the right final number, 162 took a wrong step on the way there. That is the headline from an IBM evaluation of two enterprise agent skills, posted in the last day, and it is the cleanest measurement I have seen of how little a final-output check tells you about an agent that writes to a database.

The setup is 240 trials of a revenue-apportionment skill and a productivity-apportionment skill, crossed over two specification files, two harnesses (Claude Code and Codex) and three models (GPT-5.6-Sol, Claude Opus 4.8, Claude Sonnet 4.6). The evaluator computes ground truth through its own API calls, then checks tool selection, arguments, ordering and database integrity against templates whose expected values resolve at evaluation time. 92.6% of the outcome-correct runs still broke at least one of those checks, and when the authors widened “outcome” to seven final-state checks, 151 of 164 passing runs still violated a trajectory check. The common roots are concrete: wrong arguments when staging utilization data in 75 of 120 revenue runs, a skipped database readback in 24. A dependency graph collapses a mean of 6.34 failed checks per run into 2.65 roots, which is the part a team would feel day to day. Two cautions from the paper’s own limitations section: the invariants are authored policy, so a flagged trajectory is not automatically a defect, and the LLM judge is Sonnet 4.6, which is also one of the models under test.

The same paper has a result that should worry anyone planning a harness migration. On the revenue skill, GPT-5.6-Sol barely notices the condensed spec under Claude Code (a 1.2 point gap the other way) and loses 12.9 points under Codex; Sonnet 4.6 does the reverse, 6.3 points on Claude Code and 0.4 on Codex. A spec validated on one harness says little about the other.

Agents Are Systems, Not Models makes the companion argument from the eval side. Across five configuration axes and 18,240 released trajectories on four scientific tasks, about 54% of outcome variance came from rerunning the same configuration. Among the axes, the information handed to the agent mattered more than time budget or backbone model. The finding I would act on first: telling the agent to verify its answer barely changed its verification behavior, while giving it a dedicated verification tool changed it substantially.

Two training papers pull in the same direction, moving behavior out of the prompt and into the policy. AutoCompact trains Qwen3-Coder-30B to decide when to compact its own context, what to keep, and how to continue, using 1,052 judge-corrected trajectories followed by RL on task success alone. It reaches 39.6% on SWE-bench Verified against a 30.4% base, and the gain holds with a 256K window that never overflows, so stale exploration hurts well before the limit. Length-triggered compaction, the design Claude Code and Codex ship today, scored 1.6 points below doing nothing. Cross-Benchmark Transfer from RL on Agentic Coding Tasks post-trains Kimi K2.7 Code on 1,700 expert-built tasks with a reward that drops to zero if any existing test breaks. One epoch on a rank-32 LoRA moved Terminal-Bench 2.1 from 67.4 to 82.0 and DeepSWE from 31.0 to 43.4, with trajectories 24 to 35% shorter and gains on benchmarks released after the training data was collected.

On coordination, a study of 944 issues across 21 open-source multi-agent projects finds orchestration and execution the most common category, with workflow, tool-integration and memory problems as the leading causes. One practitioner’s numbers show what that looks like up close. The author of Médula, a kernel that gates writes from parallel Claude Code agents, published the raw data: a fast decider that escalated 13% of calibration pairs to the slow path escalated 61% of real write requests, so the decision layer cost half of an all-Sonnet setup instead of a sliver. The sample is one to five runs per mode with thresholds fitted on the measured pairs, and the author says so.

For the semantic-grounding thread, JoinGR treats the join graph as the retrieval space for text-to-SQL, so tables never named in the question become reachable through foreign keys from ones that are. The abstract reports parity on BIRD and Spider and a recall improvement on the enterprise benchmark BEAVER without giving the figure.

What to watch: whether anyone reruns the IBM process checks with human severity labels, because “92.6% deviated” and “92.6% were harmful” are different claims, and only the first has been measured.

Feed note: the mirror’s status endpoint reported its last sync as 2026-09-01, but items ingested through the morning of 2026-10-03 were present and this issue draws only on those.