Regenerated tests cut coding-agent resolve rates from 50.6% to 33.2%

About a third of the coding-agent trials that current benchmarks mark as solved do not do what the task asked. That is the headline number from TestJack (Yao, Wang, Sen, Chen, Ray, Song), which audits agentic coding benchmarks by generating new tests per trial rather than per task. For each patch, the framework writes tests aimed at prompt requirements the patch might violate, keeps only tests the ground-truth patch also passes, and re-examines failures so every confirmed miss ships with a replayable test. Across six frontier backends and five benchmarks including DeepSWE and SWE Marathon, 34.4% of trials judged correct violate task requirements, and the aggregate resolve rate drops from 50.6% to 33.2%. The argument underneath the number is that static test augmentation, which strengthens a task’s tests once before any trial exists, cannot see how trials actually fail, and that evaluators have to adapt as fast as the models optimizing against them. A cheaper variant audits a random sample of trials in depth and reuses the generated tests across the task, which is the version most teams will actually run.

A second evals paper, from JetBrains Research, reaches a related conclusion from the opposite direction. Coding-Agent Benchmarks Should Match Their Users’ Task Flows collects 4,782 real agent sessions from JetBrains IDEs and studies the third of them with at least three user messages. Those sessions do not look like issue-derived benchmark tasks: requests span code questions, planning, review, refactoring and execution, and users switch between types mid-session. Three public interaction corpora show markedly different task-flow distributions, so the authors argue no single interaction protocol is universally realistic and a benchmark should name its target use case and calibrate to measurements from it. Their SWE-TaskFlow transform keeps the verified tasks and tests from an issue benchmark but steers the interaction toward a target flow through prompt splitting and verifiable repository QA. The pilot on 700 SWE-Bench Pro tasks is the part to remember: solving sequentially in several steps roughly doubles agent cost with no stable change in resolve rate. The interaction protocol is a cost axis your leaderboard is not reporting.

On the semantic-governance side, No Trace, No Claim: Two Contracts for Database Agents (Xiaofei Zhang) is the sharpest statement I have read of the gap between what an agent executed and what it tells the user. The proposal is two enforceable contracts: a plan contract that fixes what the agent may execute and reference, and an evidence contract that records the belief state, completeness and provenance under which a result supports a claim. Instantiated in TGMS, a bi-temporal graph system where the LLM plans over a fixed temporal operator interface, a static verifier checks plans before execution and a claim verifier checks typed claims against content-addressed traces. Live runs surfaced two failures that input-only schemas and value-only grounding both miss: nonexistent result fields, and page-local counts reported as complete-result counts. Completeness propagation catches the second in all 15 controlled cases and misses all 15 when disabled. Before claim gating, 21 of 220 answers contained an unsupported gated claim; after gating, none of 199 did, at the cost of 21 answers. Two caveats the author states plainly: on Bitcoin-OTC, direct SQL over the same bi-temporal store matched TGMS, so the fixed operator interface has no universal accuracy edge (TGMS reached 0.408 typed-answer accuracy on CollegeMsg versus 0.064 to 0.284 for baselines), and neither contract guarantees the agent understood user intent. Provenance and faithfulness are enforceable in the data layer; intent is not.

ERP Event Logs: A Canonical Form for KPI Time-Series Extraction, from SAP (Hadian, Rebmann, Korkmaz, Berg, Brackmann), approaches metric grounding from the schema end. An operational KPI in an ERP lives across hundreds of normalized tables and needs a bespoke join per metric. The authors show that a (case, activity, timestamp) event log derived from the same tables collapses that heterogeneity into one flat structure, after which volume, duration and rate KPI families reduce to reusable operations following a one-time mapping. They build standardized logs from hundreds of S/4HANA customers under a shared activity ontology, extract KPIs uniformly for order-to-cash and procure-to-pay, and characterize forecastability: volumes are easiest, then durations, then rates, consistently across both processes. Chronos, a pretrained time-series foundation model, is competitive with ARIMA and ETS on rate and duration series while classical models keep the edge on volumes. For anyone building a metrics layer an agent will query, the canonical intermediate representation does the semantic work, not the query generator.

When Sub-Agents Work in Parallel (Li, Li, Jiang, Lou; Nanjing and UIUC) is the largest controlled study so far of dynamic concurrency, the mode where Codex, Claude Code and Kimi Code decide mid-run whether to spawn sub-agents. The setup pairs Codex with GPT-5.4, Claude Code with Claude Opus 5 and Kimi Code with Kimi K3, runs 354 tasks from four long-horizon benchmarks (RepoZero, ProgramBench, NL2Repo, LoopsBench) plus 100 SWE-bench Verified tasks, and compares concurrency on versus off: 2,124 executions, 11.2 billion tokens, $20,780 at list price. Concurrency does not consistently improve success; resolution effects range from an 11.9 point drop to a 2.3 point gain, while token use runs 1.41 to 3.31 times the sequential baseline. The failure taxonomy (four categories, 13 subcategories, 28 patterns, annotated at kappa 0.77) covers orchestration, execution governance, global context management, and shared state and merge. One trace is representative: 17.5 minutes into a 40-minute budget the main agent fans out 22 subcommands to 20 sub-agents with a two-hour sub-agent timeout, two finish, the rest are aborted at the deadline, and the main agent never reaches integration. The program passes 92 of 440 tests. All trajectories and annotations are released.

Google’s FlowAgent (Ziftci, Greene, Liu, Dalloro, Dini; accepted at ASE 2026) is the production counterweight: a pre-submit repair agent for TAP test failures, surfaced in Critique and Cider. The design constraint comes from a 30-day analysis of developer behavior in September 2025: median time from failure notification to the developer’s next edit is 27.48 minutes, p25 is 6.72. FlowAgent runs a fine-tuned Gemini 2.5 Pro at temperature 0.1 in a ReAct generate-and-validate loop, skips flaky tests and changes over 100 files or 1,000 lines up front, and drops obsolete fixes after the fact. Median wall-clock to a fix is 9.85 minutes, p90 25.83, inside the window. The funnel: triggered on 2,567,829 changes with failing tests, 1,785,955 skipped by filters, 781,874 attempted, 421,818 fixes produced, 126,310 dropped as obsolete, 295,508 posted, 65,069 previewed, 28,554 applied. Manual evaluation on 195 failures across 37 teams scored 67.18% correct. Each execution averages 21.37 tool calls and 424,149 tokens, 643 billion tokens to date. The IDE surface converts better than code review: 59.69% of previewed fixes were applied in Cider versus 37.84% in Critique.

Last, a retrieval paper that runs its ablation properly. Does Document Structure Help Dense Retrieval? (Kuehlkamp, Saboia Moreira, Rund) tests four common RAG treatments (structure-aligned chunking, LLM-generated chunk context, heading-path metadata, two-stage hierarchical retrieval) under one protocol, with matched chunk sizes and a placebo: heading paths that are structurally valid but shuffled across documents, so any gain from merely prepending tokens is separated from the gain from correct content. On 200 Wikipedia Featured Articles (951 queries) and 1,585 QASPER papers (4,303 questions), scored with coverage-aware nDCG and tested via document-clustered bootstrap with Holm correction, real heading paths beat contextualized fixed windows by +0.022 and +0.012 cov-nDCG@10 and beat the placebo by +0.010 and +0.016. Naive two-stage hierarchical retrieval hurts (-0.033, -0.015), traced to first-stage section recall. Effect sizes are small (dz 0.06 to 0.11) but significant and consistent. If your chunker prepends headings, the content is what helps; if you added a hierarchical first stage, measure its section recall before trusting it.

What to watch: whether DeepSWE, SWE Marathon and the rest adopt per-trial evaluator regeneration or something like it, because a 17-point gap between reported and audited resolve rates is large enough to reorder leaderboards. On the data side, the plan-contract plus evidence-contract framing from TGMS and the canonical-event-log framing from SAP are the same idea at two layers: put the semantics in a verifiable intermediate form and keep the model out of the part that has to be exact.

Feed note: the copilot’s mirror-status endpoint reported a last sync of 2026-09-01, while the item store carries timestamps through 2026-10-09. The staleness counter appears wrong, not the data.