The pass@k in your harness is not the pass@k in the paper

A synthetic multi-rollout benchmark this week reported scores of 0.96 to 0.98 that, recomputed correctly, were actually 0.00 to 0.12. The gap is not a modeling result. It is an arithmetic error in how the field’s most-cited agent metric gets implemented, and it has been sitting inside benchmark harnesses long enough that nobody checks it anymore.

The pass@k most harnesses compute is not the pass@k in the paper

Jiang, Zheng, Vidra, and Setty spend Beyond Pass@k on a single operational claim: current implementations of the Chen et al. (2021) estimator set n to the number of unit tests in one submission rather than the number of independent rollout attempts. Those are different quantities. One measures how wide your test suite is, the other measures whether the agent can hit the target twice in a row, and conflating them turns a per-attempt reliability estimate into something closer to a per-assertion pass rate. They prove the substitution wrong by counterexample and propose reliability@k, which is the same estimator with n = independent rollouts and c = fully-passing rollouts per (task, agent) pair. On their synthetic benchmark the misapplied version inflates absolute scores by 0.85 to 0.97. A cheap single-rollout proxy does not rescue you either: rank correlation against the repeated-run ground truth came in at Spearman ρ = 0.417, which is roughly a coin flip dressed as a leaderboard. Their 5-task SWE-bench Verified pilot shows the shape of the problem in a real repository, where macro-averaged hidden-test pass rate was 0.80 while strict task resolution was 0.20.

Microsoft’s Thinkingbox arrives at the same conclusion from the environment side rather than the estimator side. Thinkingbox-bench is 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support, run inside isolated MCP-compatible tool sessions where grading happens against terminal backend state rather than against the agent’s final message. The strongest model tested reached 65.36% pass@1 and 25.25% pass^20. That second number is the one that matters for anyone shipping: pass^20 asks whether all twenty independent attempts succeeded, which is the closest thing a benchmark has to asking whether you could put this in front of customers. The failure texture is worse than the headline. Many failed trials terminated cleanly and executed valid state-changing actions, so the signals an observability stack would naturally alert on, malformed tool calls and hard errors, were absent in exactly the runs that produced wrong persistent state. If your production monitoring is watching tool-call validity, it is watching the wrong layer.

Read those two together and a third result gets sharper. In A Jagged Frontier, Najib Mahmud and colleagues at Colorado State, Microsoft, UIUC, and CMU rewrite the codebase around a SWE-bench instance into a semantically equivalent form (control-flow rewrites, dead-code injection, identifier renaming) and measure paired resolve rates across two scaffolds (mini-SWE agent, OpenCode) and four models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, Qwen 3.6-27B). Degradation is modest in aggregate, up to 6.7 percentage points in the worst configurations, statistically significant in 6 of 16 model×scaffold×dataset cells. The finding with teeth is that robustness rankings do not survive a scaffold change: Qwen is among the most robust under mini-SWE agent on SWE-bench Verified and the most brittle under OpenCode. You cannot pick a model for robustness and then swap harnesses, because the robustness you measured belonged to the pair, not the model. The simpler scaffold was the more robust one, which is an uncomfortable result for anyone whose reliability story is “add another layer.”

Al-Haque and Johnson give the difficulty axis some structure in What Makes Software Issue Resolution Tasks Difficult for Agents?, mining CoderForge-Preview, currently the largest open dataset of coding-agent trajectories, for static features of the patch, repository, and prompt. Task outcome is predictable from those static features at AUC = 0.863, driven mostly by patch fragmentation and repository scale, with prompt linguistic features showing up as top contributors only in the mid-difficulty band. Pre-hoc difficulty estimation from structure alone changes what a benchmark can be: instead of reporting one saturated number, you can stratify and report where on the difficulty curve an agent stops working, which is the question anyone doing capacity planning actually has.

The plumbing to make any of this portable is still missing, and the Evaluation Context Protocol is an early attempt at it. Wattamwar, Anandani, and Kakirwar propose a small vendor-neutral JSON-RPC interface over which an agent exposes user-visible output, the tool calls it made, and evaluator-safe audit context, so programmatic checks run uniformly across frameworks and CI. The reference implementation ships adapters for LangChain, LlamaIndex, CrewAI, and PydanticAI. The authors are explicit that this is work in progress with the empirical validation still ahead, which is the right posture; the interesting bet is that the evaluation surface, not the grader, is the thing worth standardizing first.

Benchmarks are moving into regimes where source code is not enough

Frontier models cluster in the high 70s on SWE-bench Verified, and the response this week is a set of benchmarks built where that number stops predicting anything. DDBench from Yan, Wang, He, Liang, Wang, Sun, and Park is 60 historical bugs mined from 13 open-source distributed systems across three difficulty tiers, and its design is a controlled experiment rather than a leaderboard. Every case runs twice: symptom-only, where the agent gets the bug symptom and the repository, and context-augmented, where it also receives a bounded debugging context of logs, traces, runtime state, and targeted investigation notes. Ten LLMs spread 61 percentage points in pass rate, and pairwise bootstrap separates 9 of 15 top-tier model pairs at p < 0.05 on the hardest case set, which means distributed debugging still has headroom to discriminate models that single-process benchmarks now rank identically. Bounded debugging context lifts aggregate pass rate by 18.1 points, and the lift is asymmetric in a way worth internalizing: weaker models convert context into pass rate, stronger models convert it into efficiency. Faithful context can also mislead, so context curation is itself a design surface with failure modes, not a free win.

ORCA closes the same loop from the operations side. Gao, Tian, Li, Zhang, and Jacobsen point out the mismatch that automated program repair starts from issue reports, localized code, or failing tests, while microservice failures are diagnosed from telemetry. ORCA distills paired failure and reference telemetry into a fault signature, uses the signature to localize candidate code and deployment-configuration sites, and has repair-graph agents plus an exploration agent produce unified-diff candidates. The part I would steal is the Telemetry-Grounded Patch Verifier, which separates patch validity, syntactic and semantic correctness, test-oracle integrity, and telemetry replay into distinct verdicts rather than one pass/fail. On a 575-case benchmark it leads all evaluated baselines on cost-effectiveness, and the authors note that telemetry-grounded verification catches repair outcomes that issue-only or test-only evaluation misses. Config changes counting as first-class repair locations is the other quiet correction here, since a meaningful share of production incidents are not code bugs at all.

SWE-bench Science targets a domain where wrong code corrupts evidence rather than behavior, and it deliberately reports why agents fail instead of only how often. Scientific software sits inside the instrument, so a silent numerical regression propagates into published conclusions with no crash to alert on. That failure mode is the same one Thinkingbox surfaced (clean termination, wrong state) transplanted into a setting where the downstream cost is a retraction.

FM-Bench runs the long-horizon axis about as far as it goes: an agent manages a football club for 20 in-game seasons against competing agents in an environment that responds to its choices. Bounded tasks are close to solved; cumulative-consequence decision-making where your earlier moves change the state distribution you later face is essentially unmeasured. Whether football management transfers to enterprise workflow is an open question, but the shape (competing agents, compounding consequences, no episode boundary to reset at) is closer to a real multi-agent deployment than a single-turn tool call is.

Memory and context are a routing problem, not a storage problem

Huang and a large collaboration run a controlled harness over memory substrates in Harness the Memory: dense and sparse indices, text records, structural stores, hierarchical stores, refinement-based memories, parametric updates, and activation-compatible context mechanisms, measured across three backbone models, four benchmark suites, and 26 performance and efficiency metrics. No substrate dominates. Broad retrieval helps long-context factual QA and actively hurts sequential decision-making, where pulling in more context shifts attention away from action-critical state. Scale adds a second axis, since substrates that behave well at moderate history lengths get costly or brittle at longer horizons. Their conclusion is that substrate routing belongs in the architecture: pick the store per operating regime rather than per project. Anyone who standardized on one vector DB for both their RAG path and their agent-state path should read the QA-versus-decision-making split carefully.

Ontology-Grounded Project Memory for Coding Agents attacks the adjacent problem. When agents author most of the new code, change velocity outruns anyone’s ability to track why the changes happened. MOOSEDev captures architectural decisions, lessons, constraints, and rationales as structured ontology-grounded memory rather than as accumulated prose. That framing is a direct response to the CLAUDE.md-keeps-growing failure we covered last week: an append-only markdown file has no schema, no retrieval story, and no way to expire a constraint that stopped being true.

The empirical grounding for all of this is in From Agent Behaviour to Agent-Friendly Documentation, where Gao and colleagues instrument 557 agentic coding sessions from SWE-chat covering 94,813 development events, plus a second public dataset, to answer which documents agents actually consult, when in the session they consult them, and what happens afterward. Documentation strategy has been running on vibes; teams write agent-facing docs by imagining what an agent would want. Behavioral traces let you write them against evidence instead.

Orchestration converges on fewer agents and harder governance

Adversarial Review stakes out a middle position that matches what practitioners keep rediscovering. Role-separated multi-agent teams show diminishing returns as agent count grows on repository-level tasks; the fashionable correction, treating agents as passive subagent tools, throws away agent interaction entirely. The paper asks whether a subagent paradigm can support minimal agentic cooperation, structured disagreement between a small number of grounded reviewers, without the coordination overhead of a large team. Structured disagreement as the mechanism, rather than consensus or voting, is the right instinct: a reviewer prompted to refute finds things a reviewer prompted to assess does not.

Specification Portability Across LLM Development Agents tests whether spec-driven development actually survives an agent swap, using Oracle-to-PostgreSQL migration as the controlled transformation. The first stage ran a specification-first pipeline over 1,006 PL/SQL files, regenerating 623 of them, with 380 generated scripts executing successfully against PostgreSQL 16. Those numbers are worth staring at: roughly 62% regenerated, and of the whole corpus only about 38% produced scripts that ran. The second stage checks cross-agent compatibility, which is the load-bearing claim behind every “write specs, not prompts” argument. If specifications are portable, they are infrastructure. If they are agent-specific, they are prompts with better formatting.

On the commercial side, LangChain’s dcode is being positioned by DevOps.com as a governance product rather than a coding product, aimed at enterprises hitting the wall where the coding agent is capable enough but the code it touches is too sensitive to hand over without an audit trail. The pattern is now unmistakable across this week’s items: the differentiator moved from generation quality to what surrounds generation, permissions, provenance, verification, and evidence you can show an auditor.

The through-line is that four independent groups converged on the same complaint from four directions. The estimator conflates tests with attempts. The benchmark grades the message instead of the state. The robustness ranking belongs to the harness, not the model. The failures that hurt terminate cleanly. What I want to see next is whether anyone reruns a published leaderboard under reliability@k with real rollout counts and publishes the delta. That single number, applied to a benchmark people already trust, would do more to reset expectations than another benchmark will.