Reasoning effort buys reliability, testing tools buy cost

Raising reasoning effort from High to xHigh took first-try perfect runs from 28 percent to 89 percent. That number comes from an observational study posted yesterday in which 90 independent agent runs built the same application, a real-time retrospective board, from one detailed spec, scored against a fixed 14-criterion rubric. The extra effort cost 9 to 29 percent more and cut corrective prompts about five-fold. The counterpoint is what didn’t help: a browser-based testing tool raised cost by 42 to 68 percent without improving functional score or reliability, even on criteria visible in the interface the tool could inspect. A design-oriented system prompt raised visual quality from 3.0 to 4.5 out of 5, and a one-paragraph paraphrase of its directive reproduced the whole lift, so the value was in the instruction, not the elaborate prompt package. Mehta’s conclusion is worth pinning above every harness config: most first-run failures come from weak reasoning, and the fix is a stronger model or more effort, not another checking tool.

Where autonomy actively goes wrong is the subject of UnderSpecBench, which measures what coding agents do when instructions are benign but underspecified. The team built 69 DevOps task families grounded in documented incidents and CVEs, varied each along intent clarity, target certainty, and blast radius, and evaluated the resulting 2,208 prompt variants with deterministic side-effect oracles across Claude Code, Codex, and OpenCode. Underspecification doesn’t mainly make agents fail; it makes them guess. Between 55.8 and 67.8 percent of runs violated at least one action boundary, target ambiguity sharply degraded action quality, and cues about blast radius barely reduced the agents’ propensity to act. Completion-centric benchmarks would count many of those runs as successes. Read next to the reasoning-effort study, the pair sketches a useful division: reliability failures respond to more thinking, while safety failures need the harness and the instructions to carry state the model won’t infer.

The memory papers in this window converge on one lesson from two directions. A-TMA names a failure mode most memory systems currently paper over: ghost memory, where old facts, current facts, and the transition between them coexist in the bank, get retrieved together, and mislead the answer model. The fix is an overlay that keeps superseded and transition records but labels them by state, plus a conflict-heavy benchmark, LoCoMo Temporal Plus, to measure the failure directly; layered on Graphiti it improved conflict accuracy by 0.240 absolute and raised temporal F1 on LoCoMo from 0.0295 to 0.1705. Those base numbers are strikingly low, and that is the real finding: final QA accuracy hides where in the bank-retrieval-answer pipeline the failure lives. AutoMem treats memory management itself as a trainable skill. File-system operations become first-class memory actions, a strong LLM iteratively revises the memory schema by reviewing full trajectories, and the agent’s own good memory decisions become training signal. Optimizing memory alone, with task behavior untouched, improved a base agent 2x to 4x on Crafter, MiniHack, and NetHack, enough to make a 32B open-weight model competitive with Claude Opus 4.5 on those long-horizon games.

ctx is the practitioner’s version of the same insight, shipped as a Rust CLI. Months of full-fidelity agent transcripts already sit on your machine; ctx ingests them into SQLite and serves ranked text search, fully local, no graph database or hosted memory service. The authors run a history-research subagent that briefs the main agent before it touches an area, and their motivating example is exactly the kind of detail memory papers abstract away: test runs failing because a runner’s disk was full, agents repeatedly misdiagnosing it as a regression, and one history search surfacing the known workaround.

Simon Willison closed the day by one-shotting a CLI coding agent on top of his LLM Python library with two prompts and a red/green TDD instruction, shipping it to PyPI as llm-coding-agent 0.1a0. The agent implemented six tools and a Python API he never asked for. The artifact matters less than the demonstration: the harness layer has gotten thin enough that a well-specified afternoon prompt produces a working one.

What to watch: whether UnderSpecBench’s clarification-versus-action taxonomy shows up in vendor evals, and whether anyone replicates the reasoning-effort result across harnesses. If it holds, a lot of agent tooling budgets are pointed at the wrong layer.