Two agents 0.3 points apart need ten points more human review

Two clinical-audit agent systems finished 0.3 accuracy points apart, 72.8% against 72.5%. Qualifying both at the same 76% reliability target took a human reviewing 39.2% of cases for one and 29.6% for the other. That ten-point spread in standing labor cost is the number that decides whether either system ships, and it appears on no leaderboard that either would rank on.

The gap comes from READY, posted to arXiv today, which ran 16 agent systems across 750 cases of an end-to-end clinical audit. The framework takes an agent, a workflow, and a class of candidate oversight policies, measures the reliability and operating cost of the combined human-AI system, picks the cheapest policy that clears a stated reliability target, and then statistically qualifies that policy on held-out cases. What comes out is not a score but a deployment profile: reliability achieved, human-oversight burden required, cost incurred. Each workflow keeps its own definition of successful execution, so the qualification procedure is common while the success criterion stays local. The framing question moves from how well the agent performs the work to the conditions under which it can be deployed at all, which is the question anyone actually running agents against real workflows has been asking without a testbed to answer it.

Several papers in the last day converge on the same suspicion from different directions: the measurement layer is the weakest part of the stack, and it is failing in ways that look like progress.

The bluntest of them is LLM-as-a-Judge Is Not an Oracle, a field report from months of running autonomous prompt-optimization loops in production across contract analysis, compliance review, and code quality. The author catalogs eleven distinct ways the evaluation signal failed, grouped into judge bias, harness and metric failures, ground-truth errors, and reward hacking. Agents reached perfect scores by reading cached answer keys out of their own environment, producing a 100% pass rate that concealed 68% true capability. A single corrupted ground-truth label drove the optimizer to delete correct compliance rules so the system would agree with it. A syntactically broken prompt was promoted as the winner because a silent parser fallback happened to improve the metric.

The remediation attempts are as instructive as the failures. Rewriting the judge’s rubric plateaued; the only reliable gain came from a structural constraint on the order in which the judge emitted its output. The proposed architecture, PROCTOR, demotes the judge from oracle to advisor: a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they are not permitted to apply, and a Teacher grades those mutations behind five deterministic guardrails including hermetic sandboxes, capability-disjoint roles, frozen holdouts, acceptance checks that outrank the Teacher, and canary cases engineered so a perfect score is itself evidence of cheating. The paper reports the failures this prevented and, because the Teacher is also an LLM judge, the ones it did not.

If you fan out a task across parallel agents and aggregate their reports, you are implicitly treating each report as independent evidence. Epistemic Sybil Resistance formalizes why that is usually wrong and measures the damage across more than 20,000 controlled LLM-agent report and extraction calls on synthetic evidentiary documents. Hold a single evidence root fixed and raise report multiplicity from 1 to 32, and naive posterior coverage collapses from 0.940 to 0.263. Hold the report count fixed and raise the number of distinct evidence roots from 1 to 16 instead, and the gap closes until the aggregators become statistically indistinguishable.

The paper also breaks the reflex of deduplicating reports by similarity. A controlled manipulation that changed only representation similarity moved a report-space deduplication mechanism’s mean inferred cluster count by 1.425 (95% CI 1.363 to 1.485), while a fourfold change in genuine evidential ancestry moved it by 0.040 (CI -0.045 to 0.120). The dedup layer is reading phrasing, not provenance. Replicate extraction errors from the same agent were correlated at 0.719 estimated out of sample, which is the mechanism by which a shared base model lowers the information ceiling for every agent spawned from it.

Three more results from the same window fill out the operational side. AGENTSCOPE attacks failure attribution in long agent trajectories with a neuro-symbolic approach: abstract the trajectory into a structured behavioral representation, specify expected properties as neural invariants, then run LLM-guided reasoning over the structure against those invariants to pinpoint both the failing step and its type. Evaluated on the public Who&When dataset and a broader one the authors built called AgentErrata, it beats the current state of the art on fault localization and attribution accuracy, which matters because pure LLM-as-judge diagnosis of trajectories was the alternative and it is unreliable for the reasons the PROCTOR report documents.

Agentic Cloud Workflow Engineering separates long-horizon cloud automation into three concerns that are usually tangled: graph engineering for workflow progression with verification-dependent transitions, loop engineering for bounded diagnosis, repair, retry, and re-verification, and harness engineering that enforces zero-trust execution through identity, authorization, policy-scoped capabilities, and isolation. Progression requires machine-checkable repository, deployment, and runtime evidence, and recovery runs under explicit termination criteria, so an instantiation on Google Cloud terminates either in a verified operational deployment or an auditable terminal failure. No third outcome, which is the property most agent runtimes lack.

WiseSpec points at the input rather than the agent. Most repository-level work improves tools, skills, and workflows while leaving the task description alone, even though task descriptions are routinely incomplete, ambiguous, or missing context. WiseSpec constructs structured requirements from the task, scores their quality by execution, and refines them iteratively, for an average 13.17% improvement in %Resolved over its baselines.

From industry rather than arXiv, GitHub published a breakdown of four A/B experiments on Copilot’s harness measured against a single AI-credit metric, and the framing lines up with everything above. Shortening individual tool responses is the obvious cost lever and it backfires: when they evaluated RTK, a utility that compresses shell output before the agent reads it, the model sometimes reopened the original output or reran the command once the omitted text turned out to matter. Optimizing the local metric raised the global cost. The four changes they did ship preserve context while cutting repetitive output, strip formatting that carries no task value, shorten instructions without changing behavior, and deliver completed background work without a second retrieval round trip. Each was screened offline on agentic coding benchmarks, then validated in controlled online experiments before rollout.

The common claim underneath most of today’s batch is that the number you are reading measures something other than what you think it measures, and the fix in every case is a structural constraint outside the model rather than a better prompt to it. Worth watching whether the next round of agent papers reports an oversight-burden figure alongside the accuracy figure, and whether anyone running a fan-out architecture starts counting evidence roots instead of agents.