Same Model, Different Tools: Up to 28x Difference in Agent Cost and Reliability

A repository-issue-fixing agent given a Python CodeAct interface instead of a bash-only one reached the same outcomes with 41.6% fewer steps and 56.3% fewer tokens, running the identical underlying model. That number comes from a controlled study out of a team spanning Purdue and industry co-authors (The Devil Is in the Interface), which ran six tool architectures across three model actors and 11,700 trajectories on repo-level bug fixing, holding the underlying capabilities constant and varying only how they’re organized and exposed. Structured low-level interfaces improved run-to-run consistency by up to 4.7x over a bash-only baseline. Natural-language search tools broadened exploration enough to lift access to relevant files by more than 11%. Lightweight scratchpad tools, the kind that let an agent jot intermediate reasoning, moved almost nothing. Tool shape, not tool count, is doing the work.

A second paper this week makes the same point from the cost side. The Scaffolding Matters More Than the Interface set out to settle a specific disagreement, whether MCP or a plain CLI is cheaper for an agent to drive, by running one fixed git task across seven scaffoldings and five models, verifying completion against actual repository state rather than the agent’s self-report. The MCP-versus-CLI question turned out to be the wrong one: thirteen paired ratios spanned 0.43x to 29x, and scaffolding choice swamped interface choice, with two scaffolds that ship no MCP support at all coming in 5x to 28x cheaper on CLI-only runs than the five that support MCP. A local 27B model’s cost varied 139x depending on which scaffold drove it, while completing the task under every one of them. The two interfaces did separate on the cost of failure, 12.9% of MCP spend bought no completed work against 2.2% on CLI, but not on how often failure happened. Agents also routinely ignored the interface they’d been assigned, which means any benchmark that doesn’t verify actual tool calls is measuring a blend it can’t see.

PyCharm’s team found a narrower version of the same effect and turned it into a shipped feature. Their Agent Environment Coordinator doesn’t make the agent smarter, it just answers a question the agent had no way to ask before: which interpreter and package manager does this project actually use. Across 28 Python environment tasks and six models, giving agents that answer took average success from 68% to 98%. Sonnet 5 went from 73% to 100%, Opus 5 from 94% to 100%, Codex from 80% to 100%, and the weakest baseline improved the most, evidence the models weren’t incapable, they were guessing. Three papers this week, three different labs, one conclusion: what an agent can see is doing more of the work than which model is reading it.

That has a governance corollary once agents start merging code instead of just proposing it. In The Agent Proposes, the Pipeline Disposes, a DevOps.com writer describes catching a regression only because a change-budget gate flagged a protected path, after the diff had already looked clean enough to approve. Their fix borrows straight from software supply-chain security: treat every agent-authored change as an artifact from an untrusted producer, and require a signed, machine-checkable provenance record (which model, which prompt, which tools, which tests) before the change is even eligible for human review. Deny by default, and let the agent earn its way in through staged autonomy levels rather than a confident-looking diff. On Reddit, a builder shipped a smaller, more targeted version of the same instinct as an open-source tool: research-graph validates artifacts passed between a producer and reviewer agent against schemas, chains them with SHA-256, and enforces that the reviewer isn’t secretly the producer, exiting nonzero when the chain doesn’t hold together. Both projects land on the same fix for the same failure mode: a plausible-looking multi-agent pipeline needs a control that lives outside the agents’ own reasoning, because “the diff looked fine” was never actually a check.

The harder-to-automate half of that problem is knowing when a cheaper model is good enough to swap in, and a CTO’s writeup on benchmarking against your own production data is worth reading for the harness bugs alone. Their setup replays logged production requests through a candidate model with the exact original settings, runs free structural checks first, then a blind, randomized, cross-provider LLM judge. Two bugs nearly invalidated the results: their runner initially swallowed truncated judge responses silently (Claude’s extended-reasoning tokens ate the response budget on the hardest rows), and the candidate model, DeepSeek V4 Flash, degraded silently under a tight reasoning budget rather than erroring. Fixed, the harness ran 314 comparisons across 16 jobs and moved 14 of them to DeepSeek V4 Flash, cutting token costs roughly 91% on those routes, while leaving two jobs that lost blind, twice, on Gemini. The lesson generalizes past this one team: your eval harness needs its own eval, or a silent failure inside the judge quietly becomes a clean-looking pass rate.

On the retrieval side, a new benchmark (arXiv:2608.12812) puts numbers on a question every RAG team eventually asks informally: which vector database, actually. Across seven systems and six datasets totaling more than 4 million vectors, FAISS wins raw throughput at 866 QPS on SIFT1M but ships no database features, Weaviate clears 99% recall out of the box, Qdrant holds the best latency among full databases at a 4.55ms median, and LanceDB trades recall for much faster index builds. None of these are surprises to anyone who’s benchmarked their own stack, but a reproducible cross-system number to point at is rarer than it should be.

Put together, this is a week where the model barely shows up in any of the results. What kept moving the numbers was the interface, the scaffold, the environment context, the provenance record, the eval harness, the database underneath. Watch for whether that pattern holds once these interface-layer fixes get standardized rather than reinvented per lab, per IDE, per Reddit post.