65.36% pass@1, 25.25% pass^20: the second number agent benchmarks aren't printing

Microsoft’s Thinkingbox reports two numbers for the same models on the same 507 tasks: the strongest reaches 65.36% pass@1 and 25.25% pass^20. The first number is the one that lands on a leaderboard. The second is the one that describes a workflow you actually run, because pass^20 requires the agent to succeed on all twenty independent attempts, and a business process that works two times in three is not a business process. Thinkingbox is a sandbox for tool-agent-user interaction with isolated MCP-compatible tool sessions, full execution traces, and outcome checks that read the terminal backend state rather than the transcript. The 507 policy-conditioned workflows span retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support, and each attempt is graded by task-specific executable checks that accept any valid trajectory while rejecting wrong, missing, or extra effects.

The gap is the headline, but the shape of the failures is the more useful finding. Many failed trials terminate cleanly and perform valid state-changing actions along the way. Nothing errors. The agent finishes, reports, and leaves the backend in a state that no check accepts. That kills the most common cheap proxy in agent evaluation, which is grading the response or the tool call instead of the world.

A second paper from the last day attacks the same blind spot from the runtime side. Outcome Monitors starts from the observation that a timed-out tool call is the easy case: the agent sees it and routes around it. The hard case is a cached error page or a negative price arriving in exactly the expected schema and getting consumed as fact. Panthi and Abdelfattah mine outcome contracts from task-disjoint traces or derive them from public schemas, then, on a violation, preserve the result and issue a nonbinding receipt naming the violated property and the public recovery tools available. In frozen, prespecified evaluations with injected failures, ToolMaze completion goes from 10.9% to 28.1% across four models in two provider families and replicates in a third; tau-bench retail improves by 14.0 and 12.0 points on two tiers. The ablations are what make it credible: stripping the recovery-tool list out of the receipt erases the gain, restoring it brings the gain back, and varying diagnostic detail or timing produces no detectable difference. So the active ingredient is telling the model what it can call next, not telling it what went wrong. The honest limit is stated too. On a suite transcribed from a published incident taxonomy, detection outside the mined contract vocabulary falls to 46%.

Both papers are describing a failure class that has an old name, and a third argues we should just use it. The position paper Multi-Agent Systems Should Prioritize Concurrency Control maps the reliability drop that appears when you add agents onto classical concurrency anomalies: stale reads, lost updates, inconsistent outcomes. The mechanism is specific to this workload. A long inference window is a long window in which the shared state an agent read at the top of its turn drifts under it, so the read-modify-write races that a database would serialize away instead surface as what teams write up as coordination or communication breakdown. The prescription is conflict detection, isolation guarantees, and structured access to shared resources as framework primitives rather than as prompt discipline.

Mark Fussell makes the infrastructure version of the argument in What the Microservices Era Can Teach Us About AI, and the sharpest part is his insistence that agent durability is not the durability we already have. Existing durable execution engines were built for payment flows and order processing: high concurrency, millisecond steps, thousands of short workflows multiplexed per worker. An agent step might be a 45-second model call, or a pause of several hours waiting on a human, during which the process should be free to die and resume later because nobody cares whether the resume costs seventeen milliseconds or two seconds. Replay does not restore the step either, since the step is non-deterministic by construction. His name for what the layer has to become, agentic durable execution, bundles recovery, per-step identity, and per-step visibility into one runtime property on the theory that in a non-deterministic system, proving what an agent did and being able to see it are the same requirement.

If reliability is the theme, the measurement layer under it got audited in the same window, and it did not come out well. Credit Without Ground Truth builds causal ground truth in ALFWorld by executed replay, re-sampling the policy’s own alternatives at each decision point and rolling forward to see what the outcome actually does. Against that, none of the step-level credit signals used to train LLM agents beats chance at identifying which steps mattered: not LLM-judge scores, not outcome-conditioned logprob ratios, not the policy’s own confidence. The diagnosis is precise. Implicit credit tracks the policy’s fluency at median rank correlation +0.75, replicating at +0.70 in a second model family, while conditioning on the outcome adds no causal information at all (partial correlation -0.004 for Qwen). Causal contribution is sparse to begin with, at 30.5% of decision points where ground truth is defined, and measurability is model-dependent, with the fraction of points lacking a policy-supported counterfactual differing by a factor of two between two similar-scale policies. In a seven-arm pre-registered training experiment, no arm reliably beat the untrained policy, and the apparent instrument signature in the checkpoints was fully explained by training dose, since sparser credit retains fewer examples and spreads optimizer steps across an order of magnitude. Compare credit rules without matching effective sample size and you are measuring dose.

Retrieval got the same treatment. BrowseComp-Plus_CM keeps the BrowseComp-Plus questions but relocates their evidence out of a 100K-document corpus that was assembled from the benchmark’s own supporting documents plus mined hard negatives, and into ClimbMix, NVIDIA’s 400B-token, 553M-document pre-training mixture built without reference to any benchmark. The projection pipeline decomposes each question into atomic reasoning hops and keeps a question only when automatic verification, an independent agent, and human review all confirm every hop is grounded in the new corpus, which takes 830 test questions down to 57 fully grounded ones with question-level relevance judgments. The effect of moving the haystack is the number to remember: the strongest agent loses five points of answer accuracy, but its evidence recall collapses from 84.3% to 21.4% while it issues 63% more search calls. Answer accuracy was hiding almost all of the retrieval difficulty.

One more, on what agents read while doing any of this. An empirical study of coding-agent documentation behaviour instruments 557 agentic sessions from SWE-chat (94,813 development events, 3,033 documentation interactions) and 33,097 agentic pull requests from AIDev. Instruction files and working notes account for 60.5% of all documentation interactions, against 10.6% for classical technical documentation and 1.3% for API references. Consultation is self-initiated 70.2% of the time and failure-driven only 7.5%, and it is associated with less immediate testing, not more (adjusted OR 0.39, CI 0.25 to 0.60). In multi-commit pull requests that change both, code is touched before docs 4.7 times more often. The authors conclude that actionability and verifiability, the two properties everyone assumes make documentation agent-friendly, lack consistent behavioural support in the traces.

Worth watching: whether pass^k starts appearing next to pass@1 in benchmark tables, and whether outcome contracts get pushed down into tool schemas where an MCP server can declare them, rather than mined after the fact from traces the way both monitoring papers above had to.