Corrected pass@k drops agent scores from 0.98 to 0.12

A synthetic multi-rollout benchmark reported agent scores of 0.96 to 0.98. The same estimator, applied the way Chen et al. defined it in 2021, returns 0.00 to 0.12 on the same data. Beyond Pass@k, posted today by Jiajun Jiang, Sharon Zheng, Natan Vidra, and Spurthi Setty, traces that entire gap to one implementation choice: benchmarks set n to the number of unit tests in a single submission rather than the number of independent rollout attempts, which conflates test-suite size with attempt independence. The authors prove the error by counterexample and propose reliability@k, the identical estimator with n as independent rollouts and c as fully-passing rollouts per task-agent pair. If you were hoping to approximate that cheaply, the single-rollout proxy correlates with repeated runs at Spearman rho of 0.417, which is not enough to rank anything. A 5-task SWE-bench Verified pilot shows the same shape outside the synthetic setting: macro-averaged hidden-test pass rate of 0.80 against strict task resolution of 0.20.

They also propose a security-adjusted reliability@k that counts only rollouts which are both functionally correct and free of high-severity insecure patterns, then report that in a live-API test with three agents the adjustment changed no rankings under their scanner and threshold. Publishing a proposed metric that did not move anything is the right call, and it frames the question the rest of today’s papers keep running into: how much of a measured difference comes from the model, and how much from the system wrapped around it.

DDBench answers a version of that with an experiment design worth copying. Yibo Yan and colleagues mined 60 historical bugs from 13 open-source distributed systems, sorted them into three difficulty tiers, and ran ten models under two matched conditions: symptom plus repository, or symptom plus repository plus a bounded debugging context of logs, traces, runtime state, and targeted code-investigation notes. Frontier models cluster in the high 70s on SWE-bench Verified; on DDBench their pass rates span 61 percentage points, and pairwise bootstrap separates 9 of the 15 top-tier model pairs at p below 0.05 on the hardest case set. Bounded context lifts aggregate pass rate by 18.1 points, and the lift is asymmetric: weaker models convert it into pass rate, stronger models convert it into efficiency. The caveat lands harder than the headline, because even faithful debugging context sometimes misled models, which makes curation part of the system rather than a free input.

Demystifying Agent Skills, out yesterday, normalizes 8,135 trial records and 238 unique labels from 240 open-coded records into a taxonomy of three categories and twelve skill-use modes. Skills beat Workflow Memory by 6.06 points in matched comparisons, and the mechanism is not the one the framing usually implies: procedural anchoring accounts for 65.7% of skill cases against 4.5% for explicit knowledge injection. Skills stabilize what the agent does next rather than supplying facts it lacked, which matches what anyone who has written a long procedural CLAUDE.md section has felt without being able to measure. The failure mode is retrieval. As candidate pools grow from 5 skills to 100, actual-use precision falls from 29.6% to 3.3%. Confusable distractors hurt offline identification while downstream success stays stable, and exact ground-truth invocation turns out to be neither sufficient nor necessary for the task to succeed.

Skill2Query attacks that bottleneck head-on by parsing each skill document into a Skill Knowledge Graph and generating pseudo-queries in three stages, style mimicking, template generation, and parameter filling, so the generated query carries capability grounding and parameter consistency instead of topical similarity alone. Across nearly 30,000 skills the team produced 700,000 pseudo-queries and measured an average Recall@1 gain of 6.70 points across sparse, dense, and skill-routing retrieval on TheoremQA, LogicBench, ToolQA, and CHAMP, with the retrieval gains carrying through to end-to-end agent task success across multiple LLM backends. Code is at github.com/MatZaharia/Skill2Query.

The sharpest result today is about where the stochastic part sits. Direct text-to-SQL asks a model to interpret a business question and construct the relational query in one move, and in enterprise schemas that query can execute cleanly while using the wrong relationship role or the wrong aggregation grain, which is the failure that survives every smoke test. Bounded Semantic Planning and Deterministic Compilation keeps a multi-turn planner for grounding phrases and selecting from governed, question-specific options, then implements graph traversal, role predicates, grain lowering, SQL construction, and the checks in code. On a 38-question adjudicated comparison set with three runs each, semantic path compilation was correct on every run for 37 questions (97.4%) against 21 (55.3%) for the DDL-to-SQL baseline, with paired discordance of 16 to zero and a two-sided exact McNemar p of 3.05e-5. Across 114 run outcomes SPC produced one refusal and zero adjudicated wrong-but-executed runs, while the baseline produced 29 wrong runs plus seven more the judge flagged as data-only coincidences. Yi Ai is explicit that this is an end-to-end systems result and not a causal claim about compilation, since SPC receives governed semantic artifacts the baseline never sees.

If the retrieval layer is doing that much of the work, keeping it current becomes an operating cost rather than a build-time one. Cost Scales with Change, Not Corpus Size argues the maintenance bill tracks churn instead of corpus size and puts numbers on a controlled pilot: dimension 256, rank 32, a corpus grown from 3,000 to 9,000 documents across 50 update events. Incremental low-rank updates ran 33.7 times cheaper per update and 23.8 times cheaper cumulatively than full re-SVD, while the incremental subspace tracked full recomputation to within floating-point precision, with maximum principal-angle drift below 1e-11 degrees and recall@10 of 1.0. For the case teams actually dread, swapping the embedding model, an orthogonal Procrustes virtual axis update recovered 0.95 mean cosine to truly re-embedded vectors by re-embedding roughly 10% of the corpus.

On the practitioner side, MARGINAL landed Claude Code support yesterday with a governance stance most supervisor tools skip. It watches trajectories through native hooks and records repeated actions, unchanged workspace state, weak progress, redundant verification, outcomes, and its own governance overhead into a local Decision Ledger with deterministic reason codes and decision hashes. The integration is observe-only: it does not block tools, rewrite arguments, or inject text back into the session, and the hook fails open if it breaks. The author calls the principle Earned Enforcement, meaning a supervisor does not get permission to interrupt an agent merely because it detected a loop; it has to first demonstrate that its interventions are reliable. The next piece he is building is counterfactual evaluation and intervention regret, which is exactly the measurement that would settle whether the layer belongs in the loop.

Every result above is a measurement claim about a system, not about a model. Watch for the first public leaderboard that reports reliability@k with rollout counts printed next to the score, and for the first agent supervisor that publishes intervention regret before shipping in blocking mode. Those two numbers would say more about the state of agent reliability than another point on SWE-bench Verified.