When Agents Fail Quietly: Scientific Repair, Long Horizons, and the Evaluation Gap

Even Claude Code running Opus-5 at max settings clears less than half its tasks on a new benchmark for scientific software repair. That’s the headline out of SWE-bench Science (arxiv.org/abs/2608.19799), 119 tasks pulled from 98 GitHub repositories across 20 scientific domains, split into issue-driven fixes, expert-exploratory work, and engineering-integration tasks. The best agent tested still fails on more than half. The authors trace four recurring failure modes: gaps in scientific knowledge, exploration that never gets past the surface, repairs that don’t cover the whole system, and an inability to generalize domain knowledge to new cases. The more interesting finding sits in the ablation: feeding agents explicit scientific guidance helps when it’s well-grounded, cutting token spend and improving repair rate, but poorly aligned guidance anchors the agent on the wrong fix and doesn’t reliably help even when it sounds authoritative. Scientific code isn’t just harder SWE-bench, it’s a different failure surface, and bolting domain hints onto a coding agent is not automatically net positive.

That gap between benchmark score and operational trust is also what motivates the Evaluation Context Protocol (arxiv.org/abs/2608.19263), an early-stage, vendor-neutral spec for evaluating agentic systems. ECP defines a small JSON-RPC interface an agent exposes: user-visible output, the tool calls it made, and audit context a grader can use, with reference adapters already built for LangChain, LlamaIndex, CrewAI, and PydanticAI. The pitch is that every framework currently rolls its own eval harness, which makes cross-framework comparison close to meaningless and lets the “confidently wrong” failure mode, an agent that’s fluent and completely incorrect, slip past evaluators tuned to a single stack. The authors are explicit that ECP is not a finished standard; the method set and grader families are expected to change as it gets exercised against more systems. Worth tracking less for the protocol itself than for the admission baked into it: nobody has a portable way to say “this agent is reliable” across tooling boundaries yet.

On the practitioner side, a DevOps.com piece on production-grade AI eval systems (devops.com) makes the operational stakes concrete. The author’s support agent invented a refund policy and a SKU, answered in 1.2 seconds, and every SRE dashboard stayed green because nothing about a wrong-but-fluent answer trips a 4xx. The fix described is a four-layer evaluator stack: cheap deterministic checks on every trace, sampled LLM judges above that, and human-in-the-loop domain checks at the top, with guardrails kept in a separate code path from evaluators because a three-second guardrail is a broken product but a three-second evaluator sampling 5% of traffic is fine. The most actionable line: never let an LLM generate both the eval question and the eval answer, since that produces a dataset that scores 99% and means nothing. Real evals, the piece argues, only come from clustering your own production failures and promoting them into a regression suite.

On the multi-agent side, Adversarial Review (arxiv.org/abs/2608.18167) pushes against the assumption that more agents means better code review. The setup is three roles: a main coding agent, a reviewer, and a critic that audits the reviewer’s judgment through structured disagreement before any edit lands. On LiveCodeBench it beats a five-agent baseline while using only three agents. On SWE-PRBench, the naive version exposes a false-consensus failure, agents agreeing with each other without enough evidence to justify it, but adding one prompt iteration that forces explicit disagreement fixes it and produces the best F1 among tested methods. The conclusion cuts against the current bias toward larger agent teams: what matters is that disagreement is minimal, structured, and evidence-grounded, not that there are more voices in the room.

A companion survey, Self-Evolving Coding Agents (arxiv.org/abs/2608.03392), maps a different reliability lever: agents that update their own memory, tools, workflow topology, or context based on prior coding interactions rather than staying static after deployment. The taxonomy is organized around what evolves, when it evolves, and what code-specific signal drives it, executable feedback, repository-level context, coding trajectories. The authors are upfront about the tradeoff: the same feedback loops that make software engineering a natural fit for self-evolution also introduce benchmark overfitting, reversibility problems, and safety questions once an agent is rewriting its own operating parameters mid-deployment.

Long-horizon reliability gets a stress test in FM-Bench (arxiv.org/abs/2608.18423), which runs an LLM agent as a football club manager for 20 in-game years across roughly 340 to 400 decision points and 26 tools. All 15 tested frontier models finish every run while scripted baselines die out in most of theirs, but the ranking only settles late in the horizon and neither model scale, price, nor vendor predicts it. What separates the top performers is managerial behavior: reducing slow-payoff investment near the deadline, keeping cash deployed instead of idle, opening contract renewals early, none of which correlates with token spend. Self-managed memory fails in two opposite modes across the field, an archive that only accumulates or a plan that gets rewritten every season with nothing carried forward. It’s a cleaner illustration than most of why single-task benchmarks miss the reliability question entirely: the interesting failures only show up when the horizon is long enough for early decisions to compound.

Watch whether ECP or a similar cross-framework eval contract gets real adoption outside its own reference implementation, and whether anyone runs FM-Bench-style long-horizon evaluation against a coding agent instead of a football club.