Harness evolution doesn't beat plain test-time scaling on Terminal-Bench 2.1

Automatic harness evolution does not consistently beat plain test-time scaling. That is the result in Rethinking the Evaluation of Harness Evolution for Agents, posted July 14, which re-runs the comparison on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 under matched feedback and inference budgets, and finds the evolved harnesses generalize poorly to held-out tasks on top of it.

The critique is about protocol, not about anyone’s implementation. Harness evolution is a search procedure: it evaluates candidate harness configurations against task feedback, revises them, and repeats. Structurally that is the same move as agentic test-time scaling, so the obvious question is what happens if you spend the identical budget on task-level search and skip the harness reasoning entirely. The existing papers could not answer, because they searched against a public benchmark and then reported final performance on that same benchmark. Two things go wrong at once. Gains may be coming from the extra search rather than from any discovered insight about harness design, and sharing the task set between the search loop and the evaluation means part of the reported improvement is overfitting to that set. This line of work is about a week old in our coverage; the ablation arriving this fast is a good sign for the subfield, less good for anyone who already rewrote their orchestrator on the strength of the first results.

The same instinct shows up in four other papers from the last day, all of them refusing to score a system on its final answer. MemOps makes the case for agent memory. Memory benchmarks almost universally test through downstream question answering, which collapses very different failures into one wrong answer: the agent never stored the fact, or it bound an update to the wrong target, or it retrieved a value that a later correction had already invalidated. Worse, a final-answer score will happily credit a correct answer that came out of an inconsistent memory state. MemOps reframes conversational memory as a lifecycle of explicit operations (remember, forget, update, reflect, and compositions of those) and represents each memory event as a structured trace with its trigger, target, scope, state transition, and supporting evidence. A generation pipeline embeds the operations into long task-oriented conversations and emits gold operation traces plus six categories of operation-level probes. Two findings worth carrying: session-level retrieval beats turn-level retrieval, and long-context models are notably weak at reconstructing an ordered memory-state trajectory, which is exactly the capability you are implicitly assuming when you dump a conversation history into a window and call it memory.

If MemOps is the diagnosis, MemCon is a treatment for a related complaint. Nearly every memory system in circulation, from graph-structured stores to reflective insight logs, reaches memory through fixed hand-designed heuristics, and the paper’s argument is that optimal memory behavior is context-dependent enough to make any static policy wrong most of the time. Early in a task the store is sparse and retrieval is mostly noise. Recurring goal types want plan reuse rather than generic nearest-neighbor lookup. A stuck agent wants re-retrieval under a different query. Over a long task stream the store itself needs consolidating and pruning. MemCon models memory operations as an MDP and learns an online policy over when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. The engineering is the appealing part: it wraps any existing memory backend, learns from task-by-task binary feedback with no pretraining and no extra LLM calls, and runs a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 backbones it reports up to 15.2 points of task success over memory baselines while cutting token consumption 5 to 20 percent.

AgentAbstain goes after the other thing success rates hide, which is whether an agent knows when not to act. It is a paired-task benchmark: 263 pairs across 42 executable sandbox environments, each pair a should-act task and a should-abstain variant produced by a controlled perturbation to the instruction, the tool, or the environment state, built on a taxonomy of 8 abstention scenarios split across pre-execution reasoning and runtime discovery. Across 17 frontier models in 4 harnesses, the best agent (Gemini 3.1 Pro) gets 59.5% paired accuracy, meaning correct on both sides of the pair. The number that should change how you think about your roadmap is the correlation: abstention capability is largely independent of general task-solving capability, so waiting for the next model to fix it is not a plan. The named failure mode is post-hoc abstention, where the agent executes the irreversible action and then recognizes the trigger. Anyone who has watched an agent rm -rf its way to an apology knows the shape. The paired design also comes with AbstainGen, an automated pipeline that synthesizes sandboxes and pairs end-to-end with deterministic replay and LLM-judge validation, so fresh instances can be regenerated on demand against contamination; three annotators rated 94 to 98 percent of sampled tasks well-designed.

Orchestration gets the same treatment from LAMaS, which points out that multi-agent orchestration research optimizes task performance and inference cost while leaving latency alone, and that the two are not the same objective. End-to-end latency in a multi-agent system is governed by the critical execution path, so driving total token cost down does not reliably move the number a user actually experiences. Naive latency optimization has a failure mode of its own, misassigning operator-level credit and taking accuracy with it. LAMaS splits the problem: at training time it learns latency-aware execution graphs through constrained optimization with critical-path-aware credit assignment, and at inference time it adds a lightweight controller that prunes redundant future agent interactions as execution unfolds, on the reasoning that a graph fixed at training time cannot use runtime evidence. Over four benchmarks it reports the best latency among learning-based multi-agent baselines, more than 50% off end-to-end, with competitive or better accuracy, and it transfers to other systems with minimal changes.

Sitting alongside the harness-evolution critique is ToFu, an agentic coding harness released MIT that reads a codebase, edits files, runs commands, and hooks into dev tooling, pitched explicitly as a dual-purpose artifact. It is a research assistant, claiming better token efficiency and lower cost than existing harnesses, and it is a research object: a white-box harness whose orchestration logic, tool-use behavior, and harness design you can read, modify, and evaluate while it still posts benchmark numbers. That second role is the interesting one today. The evaluation-protocol complaint is hard to act on when the harnesses under study are closed, and you cannot ablate what you cannot open.

What to watch: whether the harness-evolution authors’ held-out protocol becomes the default, or whether the next round of papers keeps searching and reporting on the same benchmark because the numbers are prettier that way. MemOps and AgentAbstain both ship regenerable task pipelines built to resist contamination, which is a second answer to the same problem, and it is worth seeing which of the two disciplines the field actually adopts.