Every new agent benchmark today argues about what caused the failure

Turn off tiered refresh in ChurnBench’s synthetic enterprise fabric and freshness errors jump from 4 to 45 at a 28-day cache age, while staying identical at one day. Leave the schedule running and cache age stops predicting staleness at all: across cache ages of 1, 14, and 28 days the counts were 7, 4, and 4. ChurnBench, out of arXiv on 10 September from Vivek Kumar Singh and Preeti Priyam, generates a four-source enterprise data fabric as a timeline rather than a snapshot, writes every change to an append-only ground-truth ledger, and computes gold answers from that ledger instead of the live stores. The payoff is a distinction retrieval benchmarks have not been able to draw. An answer that was true when its data was retrieved and false when it was graded gets labeled a freshness error, separate from a reasoning error, with ground truth resolved at both timestamps for every case reported. If you run agents over business data that changes under them, the variable worth sweeping is TTL configuration against each entity’s rate of change, not how wide your drift window is.

Four of the day’s papers share that move: take a failure everyone already logs, and split it into the two causes that call for different fixes. Ecdysis, from groups at the Chinese Academy of Sciences and the Beijing Institute of Technology, applies it to self-evolving agent harnesses. Existing harness-evolution loops re-run the agent, read the failure, patch the harness, repeat, which is slow and drifts toward the failure patterns it happened to observe. The authors argue the real bottleneck is attribution: an observed failure can reflect a quirk of the current model or a systematic deficiency in the harness, and optimizing against individual failures buys the former by default. Ecdysis aggregates failures across a batch of task instances first, promotes the patterns that recur across tasks, and refines modification specifications through a multi-role diagnosis step before touching harness code. They report up to 1.84x faster harness training than prior evolution methods with 18.56% better reasoning accuracy in the resulting harness, plus lower inference-time token consumption and better transfer across models. The generalization claim is the interesting one, since a harness that only works for the model it was tuned against is a maintenance liability rather than an asset.

BenchShield runs the same logic against the benchmark itself, asking whether a reward came from solving the task or from working the scoring path. It grounds detection in a finite lifecycle model of an evaluation’s reward-relevant events, then pairs a phase-aware static taint analysis that surfaces reward-hacking paths before a run with a runtime counterpart that attributes what the agent actually did from infrastructure-side evidence. The evaluation is the part worth noting: 456 human-adjudicated trajectories drawn from more than 31,000 public agent runs across three benchmarks, with full-chain recall against an agentic hackability scanner rising from 23-94% to 77-100%, same-vector coverage from 16-56% to 43-78%, per-task cost down as much as 65%, and 96% accuracy detecting reward hacking at runtime. The 23% floor on the baseline is the figure that should worry anyone shipping an eval harness, because it means a scanner-style defense can miss three quarters of the exploit chains on some benchmarks while reporting clean.

Jason Hickey’s SaltBench attacks a quieter version of the same problem, and its headline result cuts against the method it tests. The protocol measures how a machine referee, a proof kernel or a withheld test suite, changes the way a coding agent works, with every outcome decided outside the agent’s own toolchain, the agent walled off from the network and the harness, and the wall itself probed before any scored run so isolation is observed rather than assumed. Across five systems components written in Rust under a pinned Verus toolchain, the arm instructed to write a specification and verify against it cost more on every component; no premium exceeded 2.8879x under either reading of the declared set, and the three cheapest sat below 1.4x. The registered sign test reached no verdict at k = 4 (3 of 4, p = 0.3125), which the paper reports as a non-result instead of a trend. The pattern that does emerge is scale-dependent: the premium runs near 1 on the smallest components and rises with size.

Microsoft’s Grounding Agent Memory carries the strongest production numbers of the batch. A post-task curator that only sees completed trajectories preserves its own errors, overgeneralizes from partial evidence, and keeps stale facts, so the authors give the existing asynchronous curator least-privilege, read-only world tools to check, scope, and refresh a candidate memory before it is written. No retraining, and the task agent, retriever, memory representation, and production write authority all stay as they are. In a production-like GitHub Copilot harness on CLBench database exploration, probing lifts pass rate from 39% to 73% and pass-discounted reward from 8.60 to 22.60 while cutting queries per question from 8.8 to 4.7 and task-agent cost from $3.38 to $1.68. Across six worlds of 90 adapted APEX consulting tasks, all 18 memory-versus-baseline mean reward comparisons came out positive and task-agent tool calls fell 16-75%, holding on both Sonnet 4.6 and Opus 4.7 without schema drift. Cheaper and more accurate at once is rare enough to be worth reproducing; the mechanism is simply that a memory checked against the live environment replaces several rounds of the task agent rediscovering the same structure.

On the data side, Generating a Consistent Enterprise describes a generator with no real dataset at either end. Given an industry, company size, business model, and a seed, it produces a fictional company whose workforce, customers, deals, tickets, calls, chats, and documents agree with one another, then projects one entity graph into the native formats of 66 business products so the same customer resolves under a single identity in the CRM, the support desk, and the call system. Since there is no real counterpart to measure distance against, quality is graded reference-free by a five-axis scorecard of 28 statistical checks, an adversarial detector hunting the marks of synthetic generation, and a classifier test against an independently shuffled copy. Because the instruments predate the tuning, the progression means something: over 23 generated companies mean realism went from 60.3 to 99.1, the weakest company from 41.1 to 94.9, and a detector that first flagged 55.2% of all records now flags none, with scores holding on a seed never used in development. A companion generator builds relational databases from a list of business questions, forcing qualifying rows, adding controlled near misses, and computing exact labels from the finished tables, which is a credible answer to the shortage of governed evaluation data for enterprise analytics agents.

The one practitioner piece worth reading alongside these is CodeRabbit’s argument for a review gate in an agent-run software factory, published 10 September. Their rule is that review depth should follow the cost of error, with tightly bounded changes moving through automated checks while anything touching security, architecture, customer data, or a public interface keeps a visible gate. The framing lands because it names what passing checks does not cover: code can satisfy every stated requirement and still install a bad boundary or an unwanted dependency. The same gap the four benchmarks above are trying to instrument, sitting one layer up in the pipeline.

What to watch over the next few days is whether anyone runs SaltBench-style attribution on a specification-first workflow at larger component sizes, since the premium it found rises with size and its registered test at five components reached no verdict. The semantic-governance lane stayed quiet in this window; the closest thing to it is the synthetic enterprise generator above, and governed-definition work from late August has not yet drawn a follow-up.

---

Feed note: the copilot mirror reports its last hourly sync at 2026-09-01 (staleMinutes 17160) because the prod-to-local sync has been failing; the daily local ingest is current, and every item above was published or announced on 10-11 September 2026.