Agents Claimed They Finished in Three Quarters of Their Failed Runs

Among Claude Code trajectories that failed FrontierChallenge, 75.5% still ended with the agent using language claiming it had finished. That figure comes from FrontierChallenge, posted August 25, a cross-domain benchmark of 300 end-to-end scientific workflows of which 97 are now released and scored across quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry. Twelve frontier models were run under three agent scaffolds; the best configuration fully completed 20 of the 97 tasks, a 20.6% pass rate.

The interesting part is the gap between the two metrics the authors report. Avg. Score credits partial progress, Pass Rate requires the full bundle of scientific deliverables. In analytical chemistry the best Avg. Score hit 87.6 while the best Pass Rate was 4%. In electrochemistry and environment, Avg. Score reached 94.9 against a Pass Rate of 0%. An agent that gets nearly everything right and never lands the deliverable scores like a near-miss and ships like nothing, and the confident closing sentence makes it read like a success to whoever queued the run.

SA-Bench, out August 26, finds the same shape one level down in the code. The authors take 30 papers from ICLR, ICML, and NeurIPS 2025, decompose each into atomic verifiable implementation claims they call Semantic Alignment Units, and score generated reproduction repositories against 1,491 of them along four drift dimensions: numerical, methodological, protocol, and ordering. Across 360 evaluations covering four models by three scaffolds, the mean SAU score is 0.221 out of 1.0, and the strongest configuration, Claude with PaperCoder, reaches 0.301. The failure taxonomy is what to read: agents attempt most requirements and implement them incorrectly, with implementation mismatch and stubs accounting for the majority of claims scoring zero. Scaffolds tuned for executability bought little here, which follows once you notice that a stub executes fine.

Both results describe a system that cannot tell finished from finished-looking, and neither benchmark can fix that, because the check has to live somewhere the agent does not control. Anuj Kapoor’s argument for a durable agent runtime puts that somewhere in a specific place: the workflow engine, with the model demoted to one activity inside an execution graph. Most of the piece is Azure Durable Functions and Azure AI Foundry, so read past the vendor layer to the three constraints he says only surface once a workflow is in production. Orchestrator code must stay deterministic, since recovery works by replaying it, which puts wall-clock reads, random values, fresh identifiers, and direct I/O in activities rather than in the orchestrator. Every wait needs a deadline, or an approval that never arrives becomes an instance nobody looks at again. Versioning needs a decision before the first production deploy, because long-running instances guarantee that new orchestrator code will ship while old instances are still in flight.

The sharpest detail is smaller than any of those. In his fan-out sketch, Task.WhenAll surfaces the first exception it encounters, so an activity that throws after exhausting its retries collapses the whole fan-in; real partial-success handling means every activity returns a typed failure result instead. That is the same distinction FrontierChallenge is measuring, expressed as a type signature rather than a score.

Cost is where multi-agent orchestration is being pushed this week. ProgRouter, posted August 26, targets the observation that cascade routing makes a one-shot decision at the query level while the right model at step seven depends on how much progress steps one through six actually made. It scores progress with a multi-view scorer that combines coarse workflow outcome regimes with subtask completion, progress trends, and workflow state quality, then a dual-path predictor and a meta-gating mechanism estimate the progress gain of each candidate model before each step. Evaluation spans HumanEval Plus, MBPP, MATH-500, and ASQA. The abstract reports cost reduction against baselines with no headline number attached, which for a paper whose entire claim is a quality-cost tradeoff is a conspicuous omission; the tables are the reason to open it.

Paritok-4B attacks the same bill from the context side and reports its numbers with unusual care. It is a 4B LoRA compressor on Qwen3-4B, distilled from a gpt-4.1-mini teacher over 67,074 real OpenHands trajectories into 40,606 validated examples, and it is extractive by construction: it selects spans rather than rewriting them, and 96.0% of the identifiers, paths, and numbers it emits already appear in its input, holding at 96.2% on held-out SWE-bench Lite output. Across all 300 SWE-bench Lite instances it compresses agent context to 25.7% of its original size, against 50.2% for a gpt-4.1-mini compressor and 61.9% for gpt-5, while retaining 86.5% of uncompressed single-shot solve quality. On the cat -n line-numbered input real agents actually produce, it compresses to 27.8% and retains 89.3%, and there the paired test is informative: 30 instances solved only uncompressed, 17 only compressed, exact McNemar p=0.079. Cutting context to a quarter is not distinguishable from not cutting it at this sample size, and the authors say so rather than rounding it into a win.

Retrieval got a depth argument too. DeepRepoQA replaces RAG-style fragment surfacing with Monte-Carlo tree search over repository structure, so the agent decides where to look next conditioned on both the long-term goal and expected immediate information gain. On SWE-QA it lands consistent 4-7% improvements over ReAct-style agent baselines across multiple LLMs, with the gains concentrated in correctness, completeness, and reasoning, and it matches or surpasses Cursor and Tongyi Lingma when driven by GPT-5. Performance climbs most between 5 and 10 explored nodes and is essentially flat between 20 and 30, which is a useful budget to know before wiring MCTS into anything you pay for per token.

What is missing is a benchmark that scores the orchestrator instead of the model. FrontierChallenge and SA-Bench both measure a completion claim after the fact; the durable-runtime argument says the claim should never have been the artifact of record. Watch for someone to run one of these benchmarks twice, once under a plain agent loop and once under a checkpointed workflow with typed activity failures, and report the delta.