Tool calls spent reading code predict agent accuracy on SWE-bench Verified better than lines edited

Lines of code edited is the usual stand-in for how hard a repository task is, and Code Understanding is a Bottleneck for Coding Agents, dated October 7, reports that it is the weaker predictor. Balepur, Tomlinson, and Schnabel built CABRA, a set of 6,840 tasks generated from scratch as call-graph transformations, with a size parameter that scales difficulty along four axes: function traversal, search, runtime resolution, and instruction following. Eight bare LLMs lose accuracy as tasks grow, while six coding agents stay near-perfect because they hand the work to tools like grep. The larger tasks draw more tool calls for reading and analysis, and in a separate study on SWE-bench Verified those call counts predict agent accuracy better than lines edited do. When the authors extend CABRA to a task that requires analyzing divergent logic across two classes, agent accuracy finally drops. Patch size is the worse of the two signals for guessing which tickets an agent will get wrong, and the authors argue for pairing SWE-bench-style tasks with controlled diagnostics so a score can say which ability failed.

If comprehension is the constraint, repository history is one place to get more of it. Chronos, from Friday’s arXiv batch, distills merged pull requests into structured experience cards and links them in a typed graph of code-level, developer-intent, and organizational relations. Semantic search picks entry cards, then weighted multi-hop expansion pulls in connected changes. Across six LLM backbones the full workflow lifts SWE-Agent’s mean resolution rate on SWE-Bench Verified from 69.2% to 72.9%, with 79.8% on MiniMax M2.5, and with that same backbone it moves SWE-Bench Pro from 48.3% to 51.7% and FEA-Bench Lite from 41.0% to 43.5%. The number I would carry around comes from a human evaluation on 100 tasks: with ten cards retrieved per task, graph-grounded retrieval raises the mean count of useful cards from 1.24 to 2.87 over flat semantic retrieval. That more than doubles the hit rate, and it still leaves roughly seven of ten retrieved cards as context the agent has to read past for a gain of two and a half to four points.

Chronos ends with a selection step, in which two agents each develop a patch and an “evolution steward” consults history to choose between them. Who makes that call is the whole subject of Humanize: Judgement Engineering for Agentic Coding, dated October 6, from a group spanning NVIDIA, UCLA, MIT, and Tsinghua. A human approves a plan contract, a builder agent implements it in rounds, and a reviewer agent from a different vendor decides completion. Deterministic hooks route work between the roles and enforce 72 mechanical gates, so no model picks the next step. The authors argue that alternating builder and reviewer samples jointly from two models, so a defect survives only if both miss it. I would treat that as a ceiling on the benefit, since two frontier models tend to share blind spots and their misses are correlated. The evidence is 118 public postmortems of real loops, which the paper itself labels observational. Independent review catches unsupported builder claims, and stopping is the weak point: in reports that separate rounds by phase, two thirds of rounds occurred after the implementation had been accepted. The applications include a 567-file gem5 build-system migration under upstream review, 672/672 on PutnamBench, and first place on Lean-Eval’s leaderboard at 251/303.

SpecGuard, also dated October 6, moves the judgment to before the agent has done anything. Param Biyani at MATS and Krishnamurthy Dvijotham at Google DeepMind target tasks whose description and tests cannot both be satisfied, the setup in which agents edit tests or hard-code expected outputs and rarely flag the conflict. Given only the task description and the codebase, SpecGuard autoformalizes the intended behavior into a Lean 4 specification, formalizes the tests independently, and asks the Lean kernel whether any implementation could satisfy both. When none can, the output is a machine-checked certificate. On conflicted SWE-bench tasks it detects up to 72.8% of conflicts and formally certifies up to 51.1%, with a conflict miss rate nearly five times lower than model-based judgment. The certificate covers the two formal statements. Whether the Lean specification matches what the issue author meant still rests on the autoformalization step, which the kernel cannot vouch for, so I read this as a strong filter for broken tasks and a weaker claim about intent.

Humanize’s gates and SpecGuard’s kernel both inspect something that arrives in front of them. Chase W. Hughes’s DevOps.com piece from October 9, The Agent Failure Your Approval Gate Can’t Catch, is about the unattended run that terminates cleanly, logs success, and produces no artifact, so that nothing reaches the gate, the review queue, or the feedback loop. More tracing does not help, because it yields what he calls “a perfectly instrumented trace of nothing.” His fix derives run status from the state of the world: declare the expected effect as a precondition of success, alert on runs that finish implausibly fast, and trigger verification from the expectation that work was due, on a schedule or contract the agent does not control. “A reviewer invoked by the run inherits the run’s silence: no run, no invocation, no review.” He is also direct about the price, which is paying to confirm work you already paid to have done.

The pool held no direct hit on semantic layers or governed analytics for a second day. Two things to watch from here: whether anyone runs the controlled comparison Humanize lacks, with rounds after acceptance as the metric, and how SpecGuard’s certification rate holds up on conflicts found in live repositories, where nobody has labeled the task as broken in advance. CABRA leaves a third question open, which is whether reading effort can be estimated before a run and used to decide which tickets an agent gets at all.

Feed note: the mirror’s status endpoint reported a last sync of September 1, 2026, although items ingested through October 11 were present. October 11 is a Sunday, so the newest arXiv batch in the pool is Friday’s; the three papers dated October 6 and 7 entered the pool on October 9.