Microsoft measured a 24% PR lift from CLI coding agents

Microsoft put a number on CLI coding agents: engineers who adopted Claude Code or GitHub Copilot CLI merged roughly 24% more pull requests than they otherwise would have, and the lift held across the full four-month observation window. The study, from Emerson Murphy-Hill, Jenna Butler, and Alexandra Savelieva, covers tens of thousands of engineers in Microsoft’s early-2026 rollout, which makes it one of the largest adoption datasets published so far. Two findings matter more than the headline. First use spread primarily through social networks; engineers tried the tools because peers visibly used them, not because a mandate arrived. And retention tracked engineers’ prior coding activity rather than demographics, so the people who kept the tools were the ones already shipping. The authors flag the measurement problem themselves: a merged PR is a proxy, not delivered value. That candor is worth as much as the number.

The same instinct, measure the thing instead of asserting it, runs through a new steerability paper from Thomas Winninger. The argument: the controls we’ve used for decades to manage large human engineering teams, access control, network policy, coding conventions enforced by tooling, transfer directly to coding agents and cost fewer tokens than elaborate agentic scaffolding. The experiment is small but pointed. A Gemma 4 e4b reviewer inspects a Python codebase seeded with 11 backdoors. Unconstrained, with no tools, it catches 54.5%. Give it a constrained substrate plus a roughly 200-line docs CLI and recall rises to 90.9%, with the substrate and the tools contributing independently. Oversight, on this evidence, is less about smarter reviewers and more about environments that make bad changes easy to see.

Dan Luu published long notes on AI coding that argue the review-centric workflow is the wrong frame entirely. His position: LLMs are mediocre at writing tests by default but excellent at executing well-directed fuzzing and randomized testing, and fuzzing beats asking a model to audit code on latency to first bug, total bugs found, and false-positive rate. He grounds it in his time at Centaur, the x86 CPU company, which ran 1:1 QA-to-developer staffing, spent over half its effort on testing, skipped code review by default, and shipped fewer than one significant user-visible bug a year. The sharpest section for anyone running evals is on variance: a single optimization task showed a 7.5% standard deviation in performance for one frontier model configuration, which makes single-run benchmark comparisons close to meaningless. He ran 50 trials per condition before trusting a cost difference. Almost nobody evaluating coding agents does that.

On the retrieval side, Sewon Min’s group asks whether language models can actually retrieve in-context at million-token scale, and the answer is a qualified yes with a mechanism attached. Their BlockSearch, a 0.6B-parameter LM retriever, length-generalizes to 10x its training regime and then collapses, and the collapse has a name: attention dilution. As the corpus grows, irrelevant documents dominate the softmax denominator, so the gold document loses normalized attention mass even when its pre-softmax score stays high. Length-aware softmax adjustments plus document-level sparse attention fix it well enough that the model matches dense retrieval on MS MARCO and NQ at a million tokens, beats the concurrent MSA model at 7x smaller, and scores 3x higher than dense retrieval on LIMIT, where similarity is defined by something other than embedding proximity. In-context retrieval is starting to look like a real alternative to vector stores rather than a curiosity.

Two agent-memory papers close the loop, and both report their costs plainly. AgenticSTS treats memory as a contract about what each future decision is allowed to see: every decision gets a fresh prompt assembled by typed retrieval, no raw cross-decision transcript, so the prompt stays bounded at any horizon and each memory layer can be ablated in isolation. The testbed is Slay the Spire 2, where a public benchmark of frontier models reports zero wins at the lowest difficulty and humans win 16% of the time. Adding a triggered strategic-skill layer lifted wins from 3/10 to 6/10, and the authors state outright that at this sample size the result is directional, Fisher exact p of about 0.37, while releasing all 298 trajectories. ContextSniper, AntTrail’s code-memory layer for repository-level repair, cuts total token use 51.5% for OpenClaw and 38.9% for Claude Code on SWE-bench Lite, and reports that resolution rates slipped from 26% to 24% and from 32% to 30%. Context compression buys real money and costs a little capability; a paper willing to print both halves of that trade is more useful than most leaderboard claims.

The pattern across the day is a field getting more careful about its own numbers: a PR-count study that questions PR counts, a memory paper that publishes its p-value, a compression system that reports its regression. Watch whether the discussion on Luu’s post, which just hit Hacker News, pushes more teams toward the 50-runs-per-condition standard he applied, and whether attention dilution becomes the framing for the next round of long-context retrieval work.