A code graph cut agent cost 36%; the completeness gap stayed open

A Java interface change came in 36% cheaper when the coding agent could ask a code graph “who implements this?” instead of grepping for a name. That figure comes from a controlled comparison Sonar’s engineering team ran, written up by Manish Kapur for Turing Post on August 27: six tasks across four languages, real previously-merged open-source commits as ground truth, prompts phrased the way a developer actually asks with no file names or line numbers, ten runs per side, and every run required to pass the real build and test suite before it counted. Cost fell on all six tasks. A related package rename landed 20% cheaper, a Python compiler change 20%, a C# return-type change 20% on the typical run, a Java argument-order fix 15%, a TypeScript change 5%. Where finding the code was never the bottleneck, the numbers moved a few percent in either direction and nothing regressed.

The taxonomy underneath those numbers is worth more than the percentages. Kapur splits text search into three distinct failure modes: a flood of matches the agent must open and discard one by one, a match on the wrong symbol where two methods share a name or a local shadows a field, and a site that shares no text with the query at all because the relationship is structural rather than textual. The first two burn tokens. The third one ships defects, since a missed site in a rename breaks the build immediately while a missed site in a behavior change compiles cleanly and passes tests nobody wrote for a dependency nobody knew existed. The graph in question rebuilds without a compiler or language server, a few seconds for roughly 1,000 files and about a millisecond per edit, which is the whole point given that code mid-refactor usually does not compile. Every one of the six wins involved a change that had to land identically across all implementations of a shared interface or base class.

That is the same question showing up in a very different setting. SimVerity (Zhan, Zhang, Li, Haddadi, arXiv:2608.25067) asks how much evidence a simulated agent pass provides about physical deployment, and answers it by replaying matched smart-home scenarios against independently qualified physical witnesses. An advanced simulator cleared all 240 light trials. A camera caught 42 sub-second failures that settled-state checks could not see, because completion, reported state, observable effect, and settled outcome all diverged inside the same execution. The finding that should worry anyone running a two-judge setup: a second qualified simulator added no independent cross-check at all, never disagreeing on any overlapping case, so only physical measurement exposed the blind spot the two shared. The team also learned a risk profile from measured trials, locked it before evaluation, and used it to predict false clearance on a path it had never physically measured, beating a property-blind baseline across all eleven held-out sessions. One more number from that paper deserves attention from anyone tuning a harness: swapping a single agent loop’s model-client and serving configuration moved its scenario-matching share from a 52-88% range to 100%.

Structured output has the same shape of problem, measured more finely. Structure-Content Decomposition (Zhang, Wu, Wang, Li, arXiv:2608.25358) separates placement errors, where the right value lands in the wrong position, from value errors, where the wrong value lands in the intended position. Evaluating six models from 7B up to frontier on nested JSON and table tasks, structural fidelity degrades earlier and more sharply than content accuracy as nesting depth grows. At the highest complexity DeepSeek-V4-Flash with reasoning misplaces 35% of the values it correctly recalled; Qwen2.5-7B misplaces 74%. If your agent framework validates JSON against a schema and then trusts the fields, you are catching value errors and missing most of the placement ones, since a misplaced value is still type-correct. The authors turn the decomposition into verifiable rewards via GRPO and lift JSON value-placement accuracy from 26% to 63% with generalization to held-out schemas.

Two tools landed in the last day aimed squarely at the gap between “it ran” and “here is why it went wrong.” llmmas-otel (Seyedghorban, Klimov, van Deursen, Panichella, Kulahcioglu Ozkan, arXiv:2608.24271) pairs OpenTelemetry distributed tracing with targeted fault injection for LLM multi-agent systems, instrumenting workflow phases, agent steps, inter-agent messages, tool calls, and model invocations so a baseline run and a deliberately faulted run can be compared through aligned traces. Fault injection at chosen interaction points is the part most agent observability stacks skip, and it is the only way to find out whether a recovery path works before production finds out for you. On the retrieval side, Telem is a Show HN from a former IR practitioner: one gateway routing agent web search across Exa, Parallel, Tavily, Brave, Linkup, SerpAPI and others in a homogeneous response format, plus per-search tracing with an evaluator scoring relevance and diversity. The author’s motivating observation, after reading trajectories instead of final answers, is that a run can be doomed at minute one by a stale search provider and still churn for ten more minutes before returning nonsense, with nothing in the logs saying so.

Model-Based Agentic Software Engineering (Davis, Kalu, Peng, Patil, arXiv:2608.25174) is the theory paper for all of this. Its framing: coding agents make implementation abundant relative to engineering judgment, so the scarce work moves to choosing abstractions, producing evidence, and deciding which obligations govern acceptance. MAGE externalizes the smallest purposeful representation that answers a given engineering question, then gives settled obligations proportionate authority through constraints, sensors, validators, and gates, while leaving uncertain intent deliberately open. It was developed from a longitudinal case and refined against six independently reported industrial accounts. Read next to the Sonar study, it names the mechanism: a code graph is an externalized representation that stops the agent from reconstructing structure from text on every run.

Also from August 28: Terminal-Bench-Science surfaced on Hacker News, a benchmark for evaluating AI agents on scientific research workflows. Only the announcement is in the feed so far. Worth watching whether its tasks carry real verifiers or just settled-state checks, which, per SimVerity, is precisely the distinction that decides what a passing score means.

---

Note: the upstream item feed’s most recent ingest is 2026-08-28, so this issue draws on items published August 26 through August 28.