Two API flags tripled OpenAI's ARC-AGI-3 score, and GPT-5.6 took 20% off its own serving cost

GPT-5.6 Sol wrote GPU kernel improvements that took 20% off OpenAI’s own production serving costs, and a better speculative-decoding setup worth another 15% in token-generation efficiency. OpenAI published the numbers yesterday in How GPT-5.6 fuses frontier intelligence with frontier efficiency, framing it as the model being applied, after deployment, to making itself cheaper to run. Simon Willison did the arithmetic out loud on the timeline and landed on “presumably that’s billions of dollars a month in savings at this point.” Augment Code responded on the product side the same evening, making GPT-5.6 Sol the default in Cosmos after benchmarking eight models in eight weeks on long-horizon software tasks, on the grounds that it is the most token-efficient model clearing their quality bar. Cost per useful unit of work, not score per task, is what the tooling vendors are now selecting on.

The second OpenAI post from the same day is the one practitioners should actually read twice. How enabling two settings tripled our scores on the ARC-AGI-3 benchmark is a postmortem on a bad harness rather than a model win. ARC-AGI-3 tests whether a model can learn unfamiliar 2D games with no instructions, and OpenAI’s standard harness was throwing away GPT-5.6 Sol’s reasoning after every move and dropping earlier actions as the context filled, so the model restarted from scratch each turn. Moving to the Responses API with retained reasoning and context compaction raised the public-set score 188% while using 6x fewer output tokens. OpenAI’s own conclusion is unusually blunt for a lab blog: evals rarely measure models in isolation, they also measure a bundle of less visible choices about API settings, harness design, and prompting. If you have been comparing models on your own eval and getting muddy results, the harness is now a first-order suspect, and “we used the Chat Completions endpoint” is a confound worth writing down.

That confound has a matching paper in the arXiv drop. Position: Evaluation Scores Are Perishable Knowledge Claims argues that modern eval methodology, which blends automated metrics, LLM-as-judge ratings, human assessment, and benchmark-suite results, breaks when those signals get averaged together: aggregate confidence ends up substantially exceeding the reliability of the weakest input signal, which the authors call trust inflation. Combine it with the ARC-AGI-3 result and you get a fairly uncomfortable pair of claims. A score is contingent on plumbing choices most write-ups omit, and the standard fix of averaging several noisy signals makes the number look more solid than it is.

Which brings up the question the whole week keeps circling: who decides the work is done. Pascal Biese’s Goal Engineering, or: Are We There Yet? is the sharpest mechanical read on this I have seen, because it goes to the implementations instead of the vocabulary. Claude Code’s /goal is a wrapper around a session-scoped prompt-based Stop hook: you state a completion condition, and after every turn the condition plus the transcript go to a small fast model, Haiku by default, which returns yes or no with a reason, and a no sends the agent back to work. The worker does not get to declare victory. What it gives up is durability, one goal per session, no pause and resume, and turn or time bounds that live inside the condition as English text rather than as a counter that trips. Codex took the opposite half: /goal there is a persistent object in the runtime with a real set, view, pause, resume, clear lifecycle, budget accounting in tokens and seconds, and state stored outside the conversation so it survives compaction. But the model marks its own goal complete. Claude Code separates judgment and leaves control loose; Codex separates control and leaves judgment with the entity that did the work.

The operational consequence is worth internalizing even if you never type /goal. Claude Code’s evaluator cannot run commands or read files; it judges your condition against whatever landed in the transcript. So “the tests pass” is not a claim about your test suite, it is a claim about whether test output appeared in the record where a blind judge could read it. Writing conditions is really writing an evidence specification, and the corollary is Goodhart in a lanyard: a condition made only of machine checks selects for the cheapest artifact that clears them.

The same argument showed up from the other direction in r/LLMDevs, where a thread on agent ROI makes the case that runtime and output volume are the easy metrics and neither tells you whether the result was safe to use. For consequential work the real cost includes evidence collection, human review, regression testing, rollback preparation, and cleaning up downstream effects. A fast agent looks cheap right up until you price the proof.

Anthropic’s side of the field had a rougher day. Boris Cherny’s advice to delete your CLAUDE.md, hooks, and skills every six months, circulating from an Instagram clip, set off a detailed pushback thread on r/ClaudeCode, and the objection is not sentimental attachment to config. The poster’s complaint is that Opus 5 already ignores instruction surfaces that are supposed to be load-bearing: a skill invoked from hooks and CLAUDE.md that gets skipped, PRs that keep getting pointed at main instead of stacked as asked, and answers that arrive in long corporate-legal paragraphs. Stripping the instruction layer is a reasonable experiment when the model reliably follows what remains, and reports like this one are the reason people are unwilling to run it. The subreddit spent the rest of the morning on a Claude outage and elevated-error megathread, which did not improve the mood.

On the shipping side, GitHub moved Copilot code review support for agent skills and MCP servers to general availability for all Pro, Pro+, Business, and Enterprise users. Review bots that can call your MCP servers and load your skill files are a different proposition from ones working off the diff alone, and it lands in the middle of a running argument about review load that Gergely Orosz and The Pragmatic Engineer have been documenting for two weeks.

Figma published how its security team built an agent for internal systems, and it is one of the more concrete agent-in-production writeups going: alert triage, forensic investigation, queries against their security data lake, code written to fix what it finds, and persistent memory of what it learned. Worth reading next to the Copilot change, since both are cases of an agent that gets tool access to the systems it reasons about rather than a summarizer sitting outside them.

Finally, a genuinely nasty result for anyone who thinks reproducible builds close the supply-chain question. Trusting-Trust Attack against an Entire Linux Distribution through Binary Manipulation constructs Ken Thompson’s self-reproducing compiler backdoor around GNU strip, an ordinary build utility that neither compiles nor links, and scales it to a whole distribution. The threat model most people carry, that trusting-trust is a compiler problem, is too narrow by a wide margin.

What to watch: whether anyone publishing an agent benchmark this month starts disclosing harness settings alongside the score, now that OpenAI has demonstrated a 188% swing from two flags. The Pacing the Frontier letter crossing 1,200 signatories will get the headlines, but the eval-validity question is the one that changes what you can trust in your own repo.