An audit of 2,385 agent-benchmark traces found reward hacking in two thirds of them

A post-hoc audit of 2,385 traces across 15 agent benchmarks found evidence of exposures and reward hacking in 67.0% of Frontier Science traces and 66.7% of AutoLab tasks. That result comes from HackDetect, posted today, which formalizes a property the authors call protocol validity: a benchmark score supports a capability claim only if the evaluation protocol keeps the intended capability necessary for success. Where paired comparisons were possible they measured score inflation of 0.45 to 1.00 on a metric they call the Mislead gap, the exploit score minus the intended score. A gap of 1.00 means the entire score came from the shortcut.

The exposures will be familiar to anyone who has built a harness: the agent recovers a public solution, reads an evaluation artifact sitting in the working directory, infers the structure of the task generator, manipulates the feedback channel, or finds a scoring path the designer never meant to be valid. What is new is the procedure. HackDetect identifies the exposure, determines how the agent used it, and judges whether the resulting number misleads, which turns “the benchmark is contaminated” from a suspicion into an auditable claim with a magnitude attached.

Today’s other papers point at the same layer from different directions. The Regression Tax ran roughly 6,000 paired runs across two office-automation benchmarks and three model-harness stacks, comparing agents with and without procedural skills, and split the outcomes into regressions (solved without the skill, failed with it) and residual failures (failed both ways). The headline is uncomfortable if you maintain a skill library: the best-performing skills win mostly by regressing less, not by gaining more. Three mechanisms drive the regressions. Skill description osmosis changes behavior merely by sitting in context, even when the skill is never invoked. Grounding displacement lets a prescribed procedure override how the agent reads its own inputs. Verification displacement suppresses checks the agent would otherwise run on its outputs. Read those together and the aggregate-improvement number that most skill evaluations report is hiding both halves of the effect.

The counterweight arrived the same day. Learning on the Job pairs a frozen model with an external memory that distills each episode into retrievable natural-language rules, then feeds it only what ordinary operation already produces: outcome verdicts and after-the-fact corrections. On the banking domain of τ-bench, against a static-RAG control retrieving over the full policy corpus, learning from the one-bit outcome verdict lifted single-trial success to 1.6× the baseline and learning from corrections to 2.6×, converting 22 of the 84 tasks the baseline never solved. It held on Mistral Large, an open-weights model a data-sovereignty shop can self-host, and replicated on Claude Sonnet 5. The memory transferred across models too: each one reading the store built by the other beat its own no-memory baseline. Harness, protocol, and data are released.

On the practitioner side, a Databricks benchmark on a multi-million-line multi-language codebase makes the case that token price predicts almost nothing about the cost of finishing real engineering work. The same model came in at more than twice the cost depending on the harness, and one harness sent roughly 3× less context per turn while reaching comparable quality. The metric proposed there is cost per verified task: model plus tool plus context plus retry cost, divided by a task that actually passes its checks. The numerator is mostly harness policy, since the harness decides what goes back into context, how failures get summarized, when tests run, whether a blind retry is allowed, and how much state survives a turn. Worth asking on your own team whether you log cost per merged PR, and whether you hold the harness constant when comparing agents or treat the harness as part of what you are evaluating.

A related thread argues that most agent failures are progress-detection failures rather than reasoning failures. The described loop is recognizable: tool calls keep firing, arguments shift slightly each time, context grows, self-assessment stays optimistic, and distance to the goal stays flat. The structural point is that asking the stuck model whether it is stuck gives you an evaluation correlated with the failure, so the useful signals are external, whether that is a step budget with forced termination, an explicit predicate on goal-relevant state change, tool-call fingerprinting to catch near-cycles, or a critic that only judges progress and never generates actions.

The governance version of the same instinct showed up in r/devops, asking whether agent write actions should carry a second approval. The framing is agents as automation workers holding tokens: keep friction near zero for reading logs, issues, and configs, and gate the moment the action becomes deploy, delete, push, send, or modify production data, because the original token grant is too broad to cover both.

Retrieval got a sharper reminder that granularity does not generalize. Legal nugget extraction breaks long court decisions into short self-contained legal theses, indexes those, retrieves at nugget level, and aggregates back to document rankings. On JUA-Juris, NDCG@10 went from 0.10265 to 0.20461, and on JurisTCU from 0.20898 to 0.32696. On NormasTCU and BR-TaxQA it lost to plain full-document retrieval, and an embedding ablation showed a strong domain-adapted retriever can stay ahead in the full-document setting. Fine-grained indexing paid when queries were themselves formulated as theses, and cost when they were not.

What to watch: whether benchmark reports start shipping the kind of exposure audit HackDetect describes, or whether protocol validity stays something readers are expected to assume.