Prompt wording multiplies agent cost 7x; harness choice multiplies it 30x

Asking a large reasoning model to “develop and compare several approaches” raised its reasoning-token spend by 2.4x to 7.4x across every model tested, and bought nothing in correctness. The number comes from a preregistered two-harness benchmark by Sarel Weinberger and Amir Hozez covering six large reasoning models, 24 deterministic coding tasks with hidden evaluators, and 4,643 valid runs across screening, stress, holdout, replication, and cross-provider arms. Generic “think deeply” cues raised deliberation 1.6x to 2.2x on their own. The one prompt shape that came out cost-neutral, and sometimes cut reasoning in half, was a bounded template naming scope, acceptance criteria, and a stop condition.

The harness effect is bigger than any prompt effect they measured. Identical model-task-prompt triples cost 5x to 30x more per success under Claude Code than under pi, which the authors trace to larger static prefixes and more turns rather than to anything the model is doing differently. They add a finding worth pinning above the cost dashboard: provider-side caching lowers the billed number without changing agent behavior, so it is a discount and not an efficiency gain. Misleading architectural hints cost far more than merely irrelevant prose, which says the expensive failure is a wrong lead the model chases down, not a long context it skims past.

If cost localizes to the harness, failure attribution should too, and a taxonomy posted in the same window makes that the entire premise. Harsh Raj and seven co-authors assign 41 failure modes to an edge between two components (model, harness, user, tool, memory, environment) plus a fault side saying where the repair belongs. They call it the repair-assignment problem: the same visible failure may call for post-training, scaffolding and tool-integration work, environment redesign, or benchmark repair, and an outcome-level pass/fail label cannot tell you which of the four you are looking at. They checked whether the categories are shared structure or annotator taste by running four frontier models as independent judges against human labels; the strongest hit Cohen’s κ of 0.76.

Microsoft pushed the same shift into the benchmark harness itself with LoopsBench, which frames the move as harness engineering giving way to loop engineering. Each of its 112 tasks, spanning 8 languages and 9 domains, is a dependency DAG over separately testable development units with source-evidenced prerequisite edges, and the flow-aware runtime releases tests along the ready frontier while keeping completed nodes as standing regression obligations. An agent that breaks unit three while building unit seven gets scored for it. The strongest configuration they ran, Opus-4.7 with Claude Code and outer continuation, resolved 25.00% of tasks; recorded plans recovered only part of the prerequisite DAG the sources implied, and regression events stayed visible across every loop profile. Tasks, tests, and more than 5,300 development units are open at microsoft/Loopsbench.

SWE-Touch goes after an assumption baked into every repo-level benchmark we have, which is that the agent has the workspace to itself. Yuqiao Tan and colleagues mine task-critical regions from multiple repair trajectories, use a separate User Patch Generator to build validated Counter-Edits (plausible edits to task-relevant code that conflict with completing the task), and inject them with a contextual user message at the moment the agent reaches that code. Across nine coding models on SWE-bench Verified, average resolve rate falls 7.7 percentage points, and the drop persists on the longer-horizon SWE-Bench Pro and DeepSWE tasks. The trajectory analysis is the useful part: agents either retain the conflicting code or overwrite it, in both cases without re-inspecting the repository or running targeted tests against the changed behavior. Autonomous skill and workspace state awareness turn out to be separate capabilities.

The retrieval-side version of that gap got measured this week too. Qi Liu, Jiaxin Mao, Fengbin Zhu, and Tat-Seng Chua ran a trajectory-level diagnosis of long-horizon search agents with human document-level relevance judgments, holding the retrieval model and evaluation harness fixed across six agents on BrowseComp-Plus and validating on BrowseComp with an open-web API. They split failures into retrieval gaps, where the necessary evidence is never found, and utilization gaps, where it is found and then not used correctly. Search effort and answer quality are only weakly aligned; accuracy tracks cumulative retrieval recall much better than it tracks query count or context consumed. Useful evidence usually shows up early and the agents keep going anyway, trailing a long tail of low-yield steps, while the best performers still issue exploratory reformulations but far fewer redundant ones. What they want built is a stopping criterion grounded in whether sufficient supporting evidence has been retrieved, which is the same missing primitive LoopsBench exposes from the other end.

Practitioners running this in production are converging on layered measurement. In a DevOps.com piece on agentic SLOs, Shahid Ali Khan of TestMu AI describes adding a behavioral correctness layer on top of retained infrastructure metrics: semantic similarity against a baseline corpus, model confidence thresholds, and outcome variance over rolling windows, where variance past threshold is an SLO breach even when the service is technically up. Ihor Zakutynskyi of FORMA arrived independently at three layers, with inference reliability (token-level latency, response variance, fallback frequency) sitting between infrastructure and behavior. Ronak Desai of Ciroos offers the four dimensions that are cheapest to instrument first: latency per logical conclusion rather than time-to-first-byte, step count as a reliability signal (“It should not take an agent 20 steps to solve a two-step problem”), pass@k distributional correctness in place of binary tests, and low-confidence actions in high-stakes contexts treated as first-class events that trigger review. Everyone quoted keeps error budgets separate per layer so incidents route to the team that owns the fix, and everyone concedes there is no 99.9%-availability equivalent for agent correctness, so thresholds get set empirically from production data.

Five of these six artifacts are measuring the loop rather than the model. The thing to watch is whether cost-per-success profiles start shipping next to resolve rates on new benchmarks, since a 25% resolve rate at a 30x harness premium is a different result than the same number at baseline, and whether anyone lands a stopping criterion general enough to work for both a search trajectory and a build loop.