The harness moves agent scores as much as the model does

Hand a coding agent a REST API and a hundred sequential change requests, and it breaks around turn five or six. That number comes from StaminaBench (arXiv 2606.19613), Amazon Science’s new benchmark for something the fraction-of-tasks-solved leaderboards ignore: how many consecutive change requests an agent survives before it ships a bug. Six harnesses, seven open-source models, twenty scenarios of a hundred turns each, codebases growing to 6,000 lines, tests generated programmatically so no model grades its own homework. Every model tested cracked inside six turns. Two interventions helped, and neither was a bigger model. Feeding test results back and letting the agent retry raised the passed-turn count up to 12x. Swapping harnesses opened a 6x gap between the best and worst rig around the same strong model, while weak models failed under every harness.

That result sets up the week. Most of what landed in the last seven days locates agent reliability somewhere other than the model weights, and our measurement tools mostly point at the wrong layer.

Benchmarks are scoring the wrong layer

StaminaBench’s 6x harness gap is the empirical version of an argument made directly in “Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering” (arXiv 2606.17799). A coding agent in practice is not a model. It is a system harness, a composite of models, harnesses, contexts, environments, and feedback signals, any one of which can move a benchmark score by a margin comparable to the gap between adjacent model generations. The authors name three symptoms: scores conflate the model with the rest of the harness; grading against one reference solution penalizes equally valid alternatives; and without component-level signal, an end-to-end number is nearly impossible to iterate on. If a harness swap is worth a model generation, a single leaderboard column hides more than it reports.

The same blind spot shows up in how we count tests. “All Smoke, No Alarm” (arXiv 2606.18168) studied 86,156 test-file patches from 33,596 agent-authored pull requests across 2,807 repositories, spanning OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code. 80.2% of those test patches carry weak or no explicit oracle signal: they run code without asserting anything about its behavior. Strong oracles do correlate with merging (odds ratio 1.28, p < 0.001) after adjusting for agent, PR size, repository popularity, task type, and language. The backdrop is a prior count of more than 932,000 agent-authored PRs across more than 116,000 repos. A green test count is not a verified change, and quality gates keyed on test-file presence overstate how much verification actually happened.

Control belongs to the program, not the model

“LLM-as-Code: Agentic Programming for Agent Harness” (arXiv 2606.15874) makes the architectural case bluntly. Token explosion, control-flow hallucination, and unreliable completion are not implementation bugs; they are the consequence of handing deterministic work, the looping and branching and sequencing, to a probabilistic system. A better prompt or a stronger model cannot fix that by construction. The proposal puts the program in charge of control flow and treats the model as an adaptive component invoked only where reasoning or generation is genuinely required, with no authority to alter the execution path. Context for each call is assembled from the execution history as a directed acyclic graph, so context length scales with call depth rather than accumulating across every step. A computer-use case study shows it stabilizing long visual operation sequences.

Scale, more than cleverness, is what breaks orchestration. “Autonomous Event-Driven Multi-Agent Orchestration for Enterprise AI at Scale” (arXiv 2606.20058) compared DAG Plan-and-Execute against ReAct across 208 production-derived scenarios at three sizes: fewer than 10 agents, 20 to 80, and 200. The dominant variable was scale, not task complexity. Both architectures held at small scale and degraded as agent-discovery noise became the bottleneck, with simple tasks degrading more sharply than complex ones. DAG planning bought precision and structured parallelism at small scale but its overhead hurt at 200 agents, while ReAct degraded more gracefully by absorbing failures incrementally. A Task Manager adding priority inference, related-event merging, and preemption cut high-priority queue latency by 14 to 75% and improved related-event correctness by more than 20 points at enterprise scale.

Once several agents share state, the problem is concurrency, and it has a database-shaped solution. “Verified Detection and Prevention of Concurrency Anomalies in Multi-Agent LLM Systems” (arXiv 2606.17182) models shared memory stores, vector indices, and tool registries as long-running read-generate-write operations and formalizes four anomalies in TLA+: stale-generation, phantom-tool, causal-cascade, and tool-effect reordering, structural analogues of classical isolation anomalies. Detectors are backed by 274 Verus proof obligations, and the paper finds the bugs in shipping code: a silent lost update in ByteDance’s deer-flow, fixed as a verified refinement, and tool-effect reordering in LangGraph’s ToolNode, removed with a commit-order sequencer. Prevention twins moved from 1000/1000 anomalies to 0/1000.

Retrieval is moving back out of the solver’s context

Repository exploration is where coding agents burn budget, and when the same model explores and solves, every exploratory read pollutes the solver’s context. FastContext (arXiv 2606.14066, from Microsoft) separates the two jobs: a dedicated exploration subagent, built on specialized models from 4B to 30B parameters, issues parallel searches and returns only file paths and line ranges as focused context. Integrated into Mini-SWE-Agent across SWE-bench Multilingual, SWE-bench Pro, and SWE-QA, it raised end-to-end resolution by up to 5.5% while cutting the coding agent’s token consumption by up to 60%.

For text RAG, the recurring failure is chunk boundaries that ignore document structure. SCAR (arXiv 2606.16661) expands neighboring chunks adaptively, weighing query-neighbor relevance against a structural continuity penalty, with an expansion threshold tied to each chunk’s own relevance so the decision rule stays roughly scale-invariant across embedding models. On four corpora, an RFC, GDPR, a 10-K, and a merger agreement, with 320 queries of which 160 were boundary-fragmented, SCAR reached 92.8% recall on the fragmented set using 7.84 chunks, 22.9% fewer than static windowing’s 10.16, and trimmed context tokens by 27.1% without hurting generation faithfulness.

The field is reaching the same conclusion from the other extreme. A developer on r/vibecoding described tearing out RAG entirely on a decade-old Java monolith and dumping the whole repo into context, after a year of chunk-embed-top-k kept losing cross-file context. SCAR’s careful expansion and the brute-force dump are two reactions to one problem: chunk boundaries don’t respect code or document structure, and retrieval that fragments evidence is worse than no retrieval at all.

Memory’s real failure mode is forgetting, not recall

Almost every memory benchmark measures recall, but “Control-Plane Placement Shapes Forgetting” (arXiv 2606.15903) argues production failures are mostly forgetting failures: the system cannot supersede, release, or purge a fact cleanly. Across 13 system configurations on a 385-case adversarial surface, where the LLM sits in the pipeline decides what it can fix. Deterministic primitives handle lexical and temporal cases but collapse on canonicalization, scoring 5% on identifier obfuscation and 0% cross-lingual. An inscribe-time LLM recovers canonicalization fully but does nothing for intent-aware deletion. A mutation-time hook recovers intent-aware deletion at 78 to 85% and lifts almost every category at once to 91.7-93.2% overall, for $0.17 per 385-case run. The ForgetEval suite and adapters are released under MIT.

The unit of storage matters too. AtomMem (arXiv 2606.19847) extracts high-value atomic facts from long interactions instead of storing coarse summaries, organizes them into hierarchical event structures and temporal profiles, and activates an associative graph at retrieval to reconnect fragments. It reports state-of-the-art results on LoCoMo. The atomic-fact framing matches what production agent-memory systems are converging on: one fact per record, updated in place, beats blobby session dumps that drift and contradict.

Memory does not have to stay private to one agent. “Multi-Agent Transactive Memory” (arXiv 2606.19911) notes that a population of agents keeps re-discovering solutions because each agent’s trajectory is discarded after a single use. MATM treats agent-generated trajectories the way a search engine treats human documents: index them, retrieve them across the population. Producers contribute trajectories to a shared store and consumers retrieve them. On ALFWorld and WebArena, retrieving prior trajectories improved task performance and reduced interaction steps with no coordination or joint training.

Reviewing what the agent hands back, and finding what it hid

Does an explicit contract make agent work better, or just more reviewable? “Software Delegation Contracts” (arXiv 2606.17099) ran 64 agent executions across two model tiers and three conditions, an issue-style prompt, an explicit delegation contract, and a contract plus a required evidence bundle, then collected 192 blind model-based reviews. Contracts did not change correctness: all 64 runs passed hidden acceptance tests with zero scope violations regardless of condition. They bought reviewability. Evidence sufficiency rose by 0.83 on a five-point scale (p < 0.0001), reviewer ambiguity dropped, and changed-file lists, known-limitations sections, and residual-risk notes appeared mostly when the contract demanded them. The cost was 13% more tokens and 38% more wall-clock, worse for the weaker model tier. Reviewability is a property you specify for, not one you get for free.

Addy Osmani’s Agentic Code Review is the practitioner companion to that result, a long field guide to pointing agents at diffs instead of blank files and to where automated review earns trust versus where it rubber-stamps. Read it next to the 80.2% weak-oracle number: an agent reviewer that doesn’t check for real assertions inherits exactly the blind spot it was deployed to catch.

The scariest failures never reach review at all. “When Errors Become Narratives” (arXiv 2606.14589) is a longitudinal postmortem of one production personal-assistant runtime: roughly 40 scheduled jobs, 8 LLM providers, a tool-governance proxy, a knowledge-base memory plane, 4,286 unit tests, and 827 governance checks. Over eight weeks the team documented 22 incidents, and one meta-pattern, an error whose signal never reaches a human in actionable form, showed up at least 28 times. The most dangerous class in their taxonomy is unique to LLM systems: the model transforms a failure into fluent, plausible narrative and hands it to the user. They call it fail-plausible, and the observer is not just blind, it is convincingly lied to by the failure itself. Three findings stand out: about 70% of silent failures were caught by a human looking at output, not by tests or audits; a retrospective audit prevented 0% of incidents ahead of time but blocked 87% from recurring, so audits are regression engines, not oracles; and the longest-lived failures, from 13 hours to 60 days, lived in the seams between components, where no test runs.

What to watch

Every result this week put reliability somewhere other than the weights: in the harness behind StaminaBench’s 6x spread, in the control flow that LLM-as-Code pulls back into the program, at the chunk boundary SCAR repairs, on the mutation path Control-Plane Forgetting exposes, in the seams between components where silent failures hide. When a harness swap can swing a score as far as a model generation, a single model-comparison number stops meaning much. Watch for whoever publishes the first component-level benchmark, separating model from harness from context from feedback, because the lab that can attribute a score to its parts is the one that can actually improve it.