Root-cause localization tops out at 24% on 145-step agent traces

1,140 failed agent trajectories, median length 145 steps, and the strongest baseline names the exact step where things went wrong 13.2% of the time. That number comes from LongRCA Bench, posted to cs.SE yesterday, and it is the sharpest thing in the last day’s reading because it separates two questions most teams still collapse into one. The benchmark collects real failures across five domains with no injected errors, then attaches independently scored human labels for both the responsible role and the earliest decisive root-cause step. The authors’ training-free method, Root-Cause Trajectory Attribution, retrieves candidate error steps from segment summaries and traces them back to earlier handoff instructions; on the same backbone and scoring protocol it reaches 51.1% responsible-role accuracy and 24.1% exact root-step accuracy. Half the time you can say which sub-agent broke the run, and a quarter of the time you can say where. If your postmortem tooling reports one score, it is almost certainly reporting the easier one.

The same pattern showed up three more times in the last thirty-six hours: a benchmark that stops scoring the outcome and starts scoring a specific mechanism, and the mechanism turns out to be much weaker than the outcome suggested. StateMemBench does it for memory. Existing memory benchmarks are recall-shaped, so an agent that retrieves a fact it was told six sessions ago scores well even if that fact was revised four sessions ago. StateMemBench’s 234 multi-session scenarios use closed-pool grading that scores whether an answer reflects the current state, the superseded state, or neither, which isolates state-tracking failures by construction. Memory systems, retrieval-augmented baselines, and long-context baselines all do poorly. StateMem, which tracks supersession and relational dependencies explicitly, lifts current-state accuracy from 0.205 to 0.363 on DeepSeek-V4-Flash and from 0.149 to 0.233 on Qwen-3.5-9B. Dropped in as a single-call wrapper over six existing memory and retrieval backends it adds 32 to 67 points, and a length- and cost-matched control attributes 15 to 32 of those points to the state structure rather than the extra context. That control is the reason to take the result seriously.

Adjacent question, same window: not whether the agent tracks a revision, but whether it should have written the fact down at all. MCB frames this as the memory-clarification boundary, scoring whether interaction-derived information gets persisted, used only in the current context, re-verified, or clarified with the user, across 140 scenarios with non-author labeling at 97.1% agreement and Cohen’s kappa 0.962. Across Claude and Qwen models, verifying changing facts is far more reliable than asking the user to resolve ambiguity: bare Qwen asks on 0 of 12 clarification items while verifying 12 of 18 freshness items. Few-shot prompting moves accuracy from 0.557 to 0.771, and clarification recall stays at 0.333. A policy prompt cuts erroneous persistence from 0.243 to 0.100. The load-bearing result for anyone designing agent memory is the last one: label-tool agreement is 57% for each Claude model and 23% for Qwen, so models frequently state the correct decision and then call a different tool. Evaluating stated decisions alone overstates memory competence by a wide margin.

Retrieval got the same treatment in cs.IR. SameCapRisk-Bench starts from the observation that a retrieved agent skill is not just a document; it ships instructions, scripts, resource bindings, and execution assumptions. So the interesting retrieval failure is not irrelevance, it is finding the right capability family and returning the wrong same-capability sibling, one that differs on an execution-controlling contract like a required resource, a precondition, a procedure, or an output artifact. The benchmark pairs 1,190 skill-risk units with 1,686 query cases, including 496 hard role-flip units where the same two skills swap helpful and risky roles depending on the query. Public routers SkillRouter, SkillRet, and R3-Skill post Recall@3 between 0.848 and 0.888 while placing the marked risky sibling in the top three between 0.346 and 0.372 of the time. A score-and-cluster pipeline built from public components pushes that exposure rate down to 0.128 to 0.182 at Recall@3 of 0.713 to 0.776, which is a real trade and worth making explicit rather than hiding behind a single recall number.

The economics of running all this measurement got a useful result too. Task-CoEvolve targets harness optimization, the loop that iteratively rewrites harness code against validation performance without touching model weights. The usual setup re-runs the entire validation set every iteration, including tasks the harness has long since saturated or long since failed. Task-CoEvolve samples toward tasks where candidate harnesses disagree, on the argument that those sit near the capability frontier and carry the discriminating signal, then reconstructs a full-set score from the sampled subset by correcting for sampling probability so iterations remain comparable. On online text classification and Terminal-Bench 2.1 it matches full-set search’s final performance with 80% fewer evaluations. Code is promised at github.com/Agent4Science-UTokyo/Task-CoEvolve.

Two production data points from the same window round it out. AWS released aws-bench, an open-source benchmark that runs agents against real AWS resources in disposable accounts rather than mocks, covering tasks like diagnosing misconfigurations and provisioning infrastructure, with automated verifiers doing the scoring. Provisioning against live APIs makes the eval slower and costlier, and it also makes the failure modes real. And LinkedIn described the multi-agent code review platform it built after concluding that neither human-only review nor an off-the-shelf reviewer bolted onto GitHub handles its PR volume; the write-up treats code review as production infrastructure, with the design work concentrated on encoding organizational coding context and suppressing hallucinated or low-signal comments.

Worth watching over the next few weeks: whether any of these mechanism-level numbers start appearing next to pass rates in model and agent releases. Exact-root-step accuracy, current-state accuracy, and harmful sibling rate are all cheap to report once the benchmark exists, and all three are exactly the numbers a vendor has no incentive to publish unaided.