Committed AI config tracks half the complexity growth after coding-agent adoption

Across 441 repositories, 73.8% of the files teams commit to configure their AI coding tools were written once and never modified again. That number comes from RAMP, the Repository AI Maturity Profile introduced in A Few Pages of Markdown, and it sits underneath the paper’s more consequential result. RAMP scores a repository on a four-level cumulative scale, running from behavioral rules and coding standards, through named agent definitions, up to multi-agent orchestration, with observed practice clustered in the first three levels. Independent human annotators reproduced the repository-level labels on 97% of a held-out sample, so the instrument at least holds still when you look at it twice.

The authors then re-estimated an existing agent-adoption panel within each maturity stratum. Agents raised commit throughput by 28 to 38 percent no matter where a repository sat on the scale, and that part is now unsurprising. Quality diverged instead. Among agent-first repositories, where the contrast is identified, the ones carrying no committed AI configuration showed roughly twice the increase in cognitive complexity, 53% against 27%, and 1.7 times the increase in static-analysis warnings. Maturity here is observational, and the authors say so directly: correlated engineering discipline or model capability could account for part of the gap, which is why they release RAMP as a reusable instrument and frame the result as hypothesis-generating. The set-and-forget statistic is the piece worth carrying around. A rules file written once in March and never revisited is a standing claim about a model and a codebase that have both moved since.

A benchmark score belongs to the model and the wrapper jointly

Pascal Biese opens The Harness-Maxxing Trap with a reproduction attempt: a developer trying to hit Qwen’s reported 53.5% on SWE-bench Pro measured roughly 28% with a bash-only agent, then added a single str_replace file-edit tool and landed at 50.7%. One tool moved the number 23 points. The rest of the piece is the version of that story with receipts. LangChain held the model fixed, changed only the wrapper, and moved its coding agent from 52.8% to 66.5% on Terminal-Bench 2.0, from outside the top thirty into the top five, on the strength of a pre-exit verification hook, a startup directory map, and a loop detector. The same GPT-5.5 scores 83.1% in Codex and 78.0% in Terminus 2 at the same effort level, which is five points of apparent model capability that live entirely in the scaffolding.

Biese’s taxonomy is the useful export. An outer harness is built from guides that shape behavior before the agent acts, sensors that observe after it acts, and caps that the model cannot argue past. Guides advise, sensors report, and only a cap enforced by permissions or infrastructure outside the agent process stops anything. His three failure modes follow from that: a published score describes a model-harness-effort triple that an enterprise does not own, a heavily tuned harness is scar tissue for one model’s specific habits and expires with it, and the harness sits inside the agent’s blast radius, so it cannot address the things Gartner cites when it projects that more than 40% of agentic AI projects will be cancelled by the end of 2027. Unclear success criteria, governance that does not match autonomy level, identity and audit, integration with systems the agent does not own: none of those are harness problems, and a maxed harness can make several worse by producing something that looks reliable in the lab.

The practitioner-side version of the same point ran on r/ClaudeCode, where one developer compared Opus 4.8 and Opus 5 across 25 tasks replayed from merged work in their own repository, one attempt per cell at medium reasoning. The strict test score tied at 9 of 25 each, the same eight tasks plus one unique pass apiece, and a separate equivalence check (did the patch make the same behavioral change as the merged one) landed 12 to 11. Underneath the tie, the two models spent nearly identical interaction budgets on opposite halves of the work. Opus 5 ran more shell commands on 18 of 25 tasks and more test commands on 15, going looking for the file that owned the requested behavior rather than patching the nearest consumer; Opus 4.8 left the smaller patch footprint on 20 of 25, staying closer to the merged change. Cost came out within noise, with Opus 5 about 1.4% cheaper on the typical task while using 4.3% more tokens and running 3.7% longer. The author is candid that this is n=1 on one repository and that the eval scores the artifact, not the experience of reading walls of agent prose to get it.

Two papers put numbers on how much the prompt side of the wrapper is worth. Jakub Smékal’s Can your AI agent be cheaper? ran 2,700 agentic coding runs with Kimi K3 at three thinking efforts and found that stripping a full task specification down to a bare user story raises token spend by 29.7%, with prompt sensitivity ranging from 13% to 115% depending on the task, and run-to-run variance unmoved by any prompt change. A simple predictor prices a full distribution of specification and effort configurations from one cheap probe on an unseen task to within 36%. Arquimedes Canedo’s Architecture as Capability Equalizer for Coding Agents holds information content fixed and varies only the format: informal prose, Mermaid diagrams with constraints and ADRs, OpenAPI, C4/Structurizr DSL, and TypeScript interface contracts with ArchUnit-style rules, across six models from three vendor families and 90 multi-turn trials. On Sonnet 4.6 and GPT-5 the quality spread across formats is 0.17 to 0.92 and format barely registers; on weaker models the spread runs 0.83 to 2.42, and TypeScript contracts take the weakest model’s API route coverage from 33% to 100%. Self-validation rates collapse from 100% on Sonnet to 0% on Gemini Flash, and mid-tier models can burn more tokens than frontier models for worse output once they fall into compilation debugging loops. Structured specification buys the most exactly where the model is cheapest.

Three benchmarks that check whether the work happened

SWE Refactor Bench names the hack that whole-repository migration benchmarks have been paying out on. If you grade only behavioral correctness, an agent can copy the original implementation forward, keep every test green, and never perform the migration; the authors call this Blindness. Their protocol runs three stages: a Migration Audit that verifies the migration occurred, a fixed behavioral test suite, and an Agentic Verification stage where six independent coding agents generate targeted tests hunting for hidden behavioral differences. Across 520 runs spanning 8 frontier models and 26 model-effort configurations on 20 whole-repository migrations, 28 runs pass all three stages. That is 5.4%. Thirteen of the twenty tasks received no accepted solution at all, and the best model, claude-opus-5, scored 47.0 out of 100. The two abilities separate cleanly: a few runs preserve behavior by skipping the migration and die at the audit, while most attempt the migration and break behavior. Among the 340 runs that clear the audit, 58% reach 99% of the fixed checks and only 26% reach 100%, which is the shape of a system that gets you most of the way and then hands you the last mile. Category matters too, with 31.4 on build toolchain rewrites against 5.6 on language rewrites.

PeakBench goes after a dimension most tool-use benchmarks skip entirely: scheduling. Agent benchmarks generally test tool selection, argument generation, and end-to-end success under mostly serial execution, which hides the choice between running serially (safe, slow) and running resource-agnostic parallel calls (fast, prone to avoidable resource overflows). PeakBench supplies executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles, then splits evaluation into logical planning and physical scheduling so a failure can be attributed to one or the other. Strong logical planning does not reliably survive contact with resource constraints, and simply exposing resource information to the agent reduces avoidable overflows. Anyone fanning out subagents against a fixed pool of CPU, memory, or API quota is running this experiment already, without the metrics.

SABER moves coding-agent safety evaluation from the prompt to the filesystem. Rather than asking whether a model refuses an unsafe instruction, it drops models into realistic agent-style projects and reads safety off the final environment state after a sequence of actions, then categorizes violations by cause so models get distinct safety profiles instead of a single number. The best-performing model still records a harmful safety-violation rate above 54%. Refusal training evidently does not transfer to a stateful workspace where the damage is a sequence of individually reasonable actions.

Durable execution shows up as the agent runtime

Anuj Kapoor’s The Missing Runtime for Long-Running AI Agents is the clearest writeup this week of an argument that has been circling for months: the workflow is the application and the model is one activity inside a larger execution graph. The framing is concrete rather than architectural fan-fiction. A stateless service holds progress in process memory, so a host failure discards completed work and the retry restarts from step one, while a durable runtime checkpoints each completed activity and lets a replacement worker replay history and continue. He builds the example on Azure Durable Functions with Azure AI Foundry as the model layer, fanning out enrichment, retrieval, and precedent agents in parallel and fanning in a synthesized recommendation, with a human review event raced against a durable timer so an approval that never arrives escalates instead of parking an instance forever.

The three constraints he flags are the ones that only surface after a workflow has been in production for a while. Orchestrator code has to stay deterministic because recovery works by replaying it, so wall-clock reads, random values, fresh identifiers, and direct I/O belong in activities. Every wait needs a deadline, and the timer has to be cancelled on the success path or the instance sits in Running long after the orchestrator function returned. Versioning needs a decision before the first production deploy, because long-running instances guarantee that new orchestrator code ships while old instances are still in flight. There is a fourth, subtler one: Task.WhenAll surfaces the first exception it encounters, so genuine partial-success handling means each activity returns a typed failure result rather than throwing once its retries are exhausted. Scoped credentials per agent close it out, since one over-permissioned super-agent turns a single poisoned input into estate-wide exposure.

Two papers attack the cost and memory sides of the same runtime. ProgRouter argues that cascade routing makes one-shot, query-level model choices while a multi-step workflow’s right model at each step depends on evolving task progress, remaining difficulty, and budget. It scores progress with a multi-view estimator combining coarse workflow outcome regimes with fine-grained signals on subtask completion, progress trends, and workflow state quality, then predicts progress gain per candidate model through a dual-path predictor and an adaptive meta-gate, routing step by step under time and cost budgets. Evaluated across HumanEval Plus, MBPP, MATH-500, and ASQA, it cuts operating cost against cascade-routing baselines while holding task performance. MemGuard targets the two ways agent memory rots: unreliable admission, where failed trajectories and accidental successes enter memory because they look relevant, and drift, where long-running banks accumulate duplicate, stale, and conflicting records that retrieval cannot repair. The move is to stop treating verifier output as a one-shot filter and persist it as lifecycle metadata, converting multi-criteria score-token verification into reward, confidence, label, and uncertainty descriptors attached to every candidate before activation and reused during retrieval, conflict resolution, summarization, and archival. Across Terminal-Bench 2.0, SWE-Bench Verified, WebArena, and Mind2Web on four backbones, against four memory baselines plus a verifier-only control at matched runtime budgets and five seeds, it takes the best success metric and lowest average steps in all 16 backbone-benchmark settings, with a 7.9-point gain on WebArena and 2.4 to 3.5 points on the terminal and software-engineering benchmarks.

Chunking and re-ranking outweighed the choice of model

The most useful ablation study of the week is buried in a practitioner writeup. Abhimanyu Garg’s team built a RAG system for incident diagnosis and evaluated it on 2,400 annotated incident scenarios drawn from a midsized financial services organization’s real incidents, DeathStarBench with injected faults, and synthetic cascading-failure cases. Headline numbers are respectable: 87.3% root-cause identification accuracy against 71.8% for BM25 plus the same LLM, mean diagnosis time down 59%, and P1 incidents going from 48.2 minutes of manual search to 19.8. The ablation is the part worth acting on. Removing cross-encoder re-ranking cost 7.7 accuracy points. Removing the type-aware chunking strategy cost 10.9 points, the single largest hit they measured, larger than swapping the underlying frontier model, which they validated across GPT-4-Turbo, Claude 3 Opus, and Llama 3.1 70B and found roughly interchangeable. Their chunking splits three ways by document type: fixed-window for long-form runbooks and postmortems, boundary-aware for tickets whose summary/timeline/resolution structure a fixed window chops through, and sentence-level for short alert annotations where overlap only adds noise. SRE satisfaction came back at 4.6 out of 5, attributed to click-through provenance rather than raw accuracy, and end-to-end p95 held at 3.5 seconds with vector search at 142ms p95 over 10,000+ documents. They are upfront that the corpus leans financial services and web architectures, that ground truth was human-assigned, and that they never independently measured the hallucination rate of the reasoning layer. A reference implementation is on GitHub under MIT.

Two more retrieval items are worth a look. DeepRepoQA treats repository question answering as a search problem rather than a similarity problem, running Monte-Carlo Tree Search over repository structure so agents navigate and inspect code across multi-hop dependencies rather than retrieving surface-level snippets, with gains over strong baselines on SWE-QA. On the community side, Memoria V4.5 posted a LongMemEval-S retrieval run over 500 questions: 82.60% Recall@1, 92.00% Recall@3, 97.40% Recall@10, with a 209.6ms average query broken down as 50.2ms embedding, 42.0ms retrieval, and 0.06ms ranking, all on a 4GB CPU-only machine with no cross-encoder in the path. It is retrieval-only, not end-to-end answer accuracy, and the author is explicitly asking people to break it, which is a better posture than most benchmark posts arrive with.

What to watch

The interesting tension this week is between two directions of travel. SWE Refactor Bench, PeakBench, and SABER all move evaluation off the model and onto the final state of the environment, which is the right direction and also the expensive one. Meanwhile RAMP, the token-spend study, and the specification-format experiment all say that the cheap, boring, committed artifacts around the agent carry a large share of the outcome. The open question is whether anyone publishes a study that holds the harness fixed across model generations and measures how fast the tuning decays. OpenAI’s framing that harness components should be designed to be deleted implies a half-life; nobody has measured it. Worth watching for: a durable-execution benchmark with fault injection rather than an architecture writeup, and the first eval that scores the review surface an agent leaves behind alongside whether its tests passed.