Checks that do not check: 5.3% of LLM call sites meet their contract

Of 496 LLM call sites in 40 popular open-source agentic systems, 82% have checks before and after the call, and 5.3% satisfy the full contract the surrounding code needs from them. That gap is the finding of When Code Becomes an LLM Call, from Weng, Sarro, Xie and colleagues, posted to arXiv yesterday. The study covers 13.4 million lines across four languages, traces where every model output flows, derives an expected contract (pre-conditions, prompt constraints, post-conditions) for each call site without reading its implementation, then compares that with what the code enforces. 48% of model outputs reach control flow or execution, and 73% of those never pass through a person. Programs enforce 78% of their constraints in code, but of the constraints stated in the prompt, 67% are only requested from the model and never verified. Ten recurring patterns explain 99% of the gaps. 38 of the mechanisms behind those gaps fire at runtime, and a small code fix that enforces the expected constraint stops all 38. The line worth carrying around: the presence of checks says little about their adequacy. If your harness has a post-condition on tool-selection output, the question is whether it checks the property the downstream code assumes, not whether a check exists.

The second paper of the day is about what agents do when the outcome cannot tell their choices apart. Over-Defense in Coding Agents Begins at Silent Forks, from Luo and colleagues, names the turns where every available next action leads to the same outcome, and shows that outcome rewards, including the recent methods that concentrate credit on decisive turns, cannot see them. At those forks frontier models take the disproportionate branch, meaning defensive actions beyond what the task requires, in up to 91% of decisive actions under implicit budgets. Putting a behavioral checker into the trajectory reward does not help much, because the checker only reaches the forks through fully proportionate trajectories; 74% of GRPO groups end up with identical rewards and nothing changes. Their fix, ForkDPO, contrasts a proportionate and a disproportionate action at the same prefix, at silent forks on both sides of the required behavior. On unseen repositories it lowers a 4B model’s disproportionate-behavior rate from 70.3% to 58.9%, against 67.7% for DPO applied at randomly chosen turns. The general claim is that behavioral divergence is a learning signal for how agents act, separate from whether they finish, and it lands on the same point as the contracts paper from the other side: completion is not the property you care about.

Two eval items argue the benchmark task shape is wrong, in different ways. JetBrains Research analyzed 4,782 agent sessions in their IDEs, about 14,000 developer messages, and compared them with SWE-Bench Pro. Bug fixes and new features make up under half of what developers ask for; the rest is questions about the codebase, planning, review, refactoring and command execution. SWE-Bench Pro, labeled with the same classifier, is 55% features, 39% bug fixes, 6% refactoring, and every task arrives as a single polished message. 50.9% of real sessions are one message; 33% run three or more. Their SWE-TaskFlow rewrites each issue into three ordered requests delivered one at a time, adds repository questions with executable proof scripts, and picks sequences to match the target distribution via a Jensen-Shannon alignment score (59.6 for pure splitting, 77.8 after optimizing question placement). The result that matters: splitting an issue into three messages made every model 2.0 to 2.6 times more expensive per task while resolve rates moved 0 to 5 points, inside run-to-run spread. A Concat control, the same three rewritten parts pasted into one message, lands next to Single, so the cost comes from sequential delivery rather than the rewording. Progressive disclosure reliably changes effort and only weakly changes outcome, and today’s leaderboards price neither.

PACE, from Liu and colleagues, goes after the other gap: whether an agent fulfills a whole product requirements document inside a mature enterprise repository, rather than whether a patch passes hidden tests. 43 enterprise tasks, 750 expert-audited Code Points, evidence-based judging that reaches 95.2% agreement with expert consensus. Across 11 model-scaffold configurations most scores fall between 52.1 and 78.1, and the best complete-delivery rate, from GPT-5.6-Sol, is 14.0%. Incomplete implementation and missing integration are the principal bottlenecks; producing the individual components does not get you a delivery. The useful operational detail is that the diagnosis fed back into harness work: successive harness releases raised mean performance from 59.4 to 63.0 with the same model and task suite.

On the analytics side, TEAL, from Yang, Ozcan, Halevy and colleagues and posted today, starts from the fact that a coding agent answering questions over a data lake discovers, cleans and joins the relevant data and then throws that work away before the next question. TEAL evolves a library of reusable tools and usage skills for a given Worker agent with no labels: an Explorer proposes tools from the lake itself, a Critic proposes them from Worker trajectories, and a change is kept only if the Worker adopts the tool on a rerun without extra steps and any changed answer is backed by trajectory evidence. Under a 15-step budget it lifts Sonnet-5 by 9.3 points on KramaBench and 21.7 on LakeQA. The counterintuitive result is that a tighter budget produces better feedback for evolution: a LakeQA library evolved at 20 steps and deployed unchanged at 30 scores 68.3, above the 59.6 baseline and above a library evolved directly at 30 steps (64.2). A library evolved on synthetic questions, their TEAL-Q variant, performs about as well as one evolved on real ones. For the governance charter the interesting object is the evolved tool library itself. It is a machine-written semantic layer over the lake, and the paper does not ask who owns those definitions, how they drift, or whether two Workers evolve conflicting joins for the same business question. That is the next paper someone should write.

The Princeton group (Chen, Rabanser, Kapoor and Narayanan, with coauthors at Stanford, the UK AI Security Institute and Brookings) published Emerging Trends in AI Agent Evaluation, a synthesis of 33 trends across methods, research agenda and the evaluation ecosystem. Their framing matches the day’s papers: evaluators must scrutinize the whole trajectory rather than the final outcome; attributing a score change to one component of the agent is hard; and the cost keeps climbing, with several agents on ExploitBench costing tens of thousands of dollars in credits to evaluate. They also note that post-incident analysis of the OpenAI and Hugging Face episode relied on other agents to flag harmful actions, a tool that itself needs evaluation.

What to watch: whether the contracts paper’s ten gap patterns turn into lint rules that framework maintainers adopt, and whether anyone starts reporting cost per task next to resolve rate once SWE-TaskFlow-style splits spread to other benchmarks.

Feed note: the item mirror reported its last sync as 2026-09-01 at the time of writing, although the pool contained items published through 2026-10-21; the status metadata looks stale rather than the data.