Adding one file-edit tool took an agent from 28% to 50.7%

A developer trying to reproduce Qwen’s reported 53.5% on SWE-bench Pro got roughly 28% running the model behind a bash-only agent. Adding a single str_replace file-edit tool took the same model to 50.7%. One line in the wrapper was worth more than most of the distance between model generations, and that anecdote anchors The Harness-Maxxing Trap, Pascal Biese’s argument that harness engineering has become the highest-leverage discipline in agent work and, for exactly that reason, the easiest thing to mistake for a finished deployment.

The supporting numbers are not subtle. LangChain froze the model, changed only the wrapper, and moved their coding agent from 52.8% to 66.5% on Terminal-Bench 2.0 using a middleware hook that forces a verification pass before exit, a startup step that maps the directory so the agent stops burning turns on discovery, and a loop detector. The most common failure they were patching was an agent writing code, re-reading it, deciding it looked fine, and never running a test. Same benchmark, same effort level: GPT-5.5 scores 83.1% in Codex and 78.0% in Terminus 2. Five points of apparent model capability that were never in the model.

Biese’s three risks each carry a procurement consequence, which is why they are worth naming precisely. A published score belongs to the model-harness-effort triple rather than the model, so selecting on a leaderboard number means buying a wrapper you do not have, tuned against a benchmark you do not run, then wiring the model into a harness of your own where none of the tuning transfers. A maxed harness is scar tissue: every component encodes one model’s specific failure, which is why OpenAI’s harness team describes its best components as designed to be deleted. And the harness runs inside the agent’s process, the wrong place for a cap that has to hold when a second team reaches the same agent through an internal API and bypasses the wrapper entirely. Gartner’s projection that more than 40% of agentic AI projects get canceled by the end of 2027 cites unclear business value, weak governance, and the cost of integrating with legacy systems. A better wrapper touches none of those.

Two papers landed in the same window putting numbers on how much structure in the interface is worth, and both point the same way: the weaker the model, the more the scaffolding buys you.

Architecture as Capability Equalizer for Coding Agents ran 90 multi-turn agent trials across six models from Anthropic, OpenAI, and Google, feeding each one of five informationally equivalent architecture specifications: informal prose, Mermaid diagrams with constraints and ADRs, OpenAPI, C4/Structurizr DSL, and TypeScript interface contracts with ArchUnit-style rules. On Sonnet 4.6 and GPT-5 the format barely registers, a quality spread of 0.17 to 0.92 points. On weaker models the spread runs 0.83 to 2.42, and the code-proximate formats recover most of the capability gap; TypeScript contracts triple the weakest model’s API route coverage, 33% to 100%. Two findings underneath the headline matter more than the headline. Mid-tier models consumed more tokens than frontier models for worse output, because they fall into compilation debugging loops the stronger models avoid, so the cheap model is not reliably the cheap option. Self-validation rates collapse from 100% on Sonnet to 0% on Gemini Flash, which means “have the agent check its own work” is not a portable design decision.

From SQL Generation to Tool Selection makes the same trade explicit at the tool layer and gives it a name. Bartolomeo Bogliolo’s Domain-Oriented Tooling Pattern replaces the generic “execute this SQL” tool that most MCP database servers expose with a small set of domain-aligned tools whose parameterized queries hold schema navigation, joins, and business rules on the server side. The model stops synthesizing SQL and starts classifying intent. Across 609 completed cells over the Sakila database, seventeen customer-facing tasks, four local models from 3B to 8B, temperature 0, three repetitions per cell: the verticalized pack scores a pooled mean of 0.939 against 0.666 for raw SQL execution and 0.605 for a thin generic tool pack. The smallest model goes from 0.583 to 0.929, matching or beating every larger configuration while cutting cost per correct answer by an order of magnitude. Bogliolo calls it Model Demotion. It is the architecture paper’s result arriving through a different door, and both are arguments for spending engineering budget on the interface instead of the model tier.

Where that logic runs out is scheduling. PeakBench points out that agent benchmarks mostly grade tool selection, argument generation, and end-to-end success under serial execution, which hides a failure mode every production agent meets: serial calls are safe and slow, resource-agnostic parallel calls are fast and prone to avoidable resource overflow. The benchmark is a set of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles, and its real contribution is attribution. A failure could come from incorrect dependency planning, from poor resource-constrained scheduling, or from both, so the evaluation splits logical planning from physical scheduling and scores each with its own metrics. Strong logical planning does not reliably produce safe or efficient execution under resource constraints, and simply exposing resource information to the agent reduces overflows and improves utilization.

Memory is the other place a wrapper accumulates its own bugs. MemGuard treats verifier output as persistent lifecycle metadata rather than a one-shot admission filter, motivated by two failure modes: unreliable admission, where failed trajectories, accidental successes, and misleading observations enter memory because they look relevant and mislead later decisions, and memory drift, where a long-running bank accumulates duplicate, stale, and conflicting records that retrieval alone cannot repair. Multi-criteria score-token verification becomes reward, confidence, label, and uncertainty descriptors attached to every candidate before activation, then reused during retrieval, conflict resolution, summarization, and archival. Evaluated on Terminal-Bench 2.0, SWE-bench Verified, WebArena, and Mind2Web across four backbones and five seeds under matched runtime budgets, it takes the best success metric and lowest average steps in all 16 backbone-benchmark settings, gaining 7.9 success-rate points on WebArena over ReasoningBank and 2.4 to 3.5 points on the terminal and software-engineering benchmarks.

Retrieval decides more of a coding agent’s ceiling than most teams measure, and CORE-Bench is scoped for the version of the problem agents actually have. Docstring-to-function matching is not it. The agent has to navigate a concrete repository state, locate the relevant files and functions, gather supporting context, and filter in-repository distractors that look nearly identical to the target. CORE-Bench grades three levels, code understanding, issue-to-edit localization, and broader context retrieval, over more than 180K queries and 106K broader-context relevance labels built from curated code-search tasks and SWE-bench-series instances. Representative embedding models drop sharply moving from traditional code search into the agentic setting, and plain supervised fine-tuning recovers much of the loss, which suggests the gap is closer to a training-data problem than an architecture one.

The commercial version of the same bet shipped in this window. TrueFoundry open-sourced TrueForge, an enterprise agent harness it claims completes tasks 30 to 75% cheaper than Claude’s managed agents. Run that claim through Biese’s first risk before it reaches a procurement deck: the number is a model-harness-effort triple, measured on TrueFoundry’s tasks inside TrueFoundry’s wrapper.

The measurement nobody has published is the transfer study. Every result above is within-harness, and the interesting question is what fraction of a tuned harness’s gain survives a model swap six months later. OpenAI’s “designed to be deleted” framing implies the answer is low. If it is low enough, the right way to report a harness result is as a depreciating asset with a version attached, and no leaderboard is set up to do that.