Digest

Two hundred and six notes. That is what came out of the AI Engineer World’s Fair 2026 once I pointed an agent pipeline at the recordings: three full mainstage days and an eighty-talk online track, pulled down as auto-generated captions, split talk by talk, and summarized into a vault of markdown files. This episode is what survived the compression. Not a recap of everything, because nobody needs forty-five minutes of talk titles, but the takeaways that matter if you spend your days the way I do, building harnesses around models, tuning retrieval systems, and watching token meters.

One disclosure before anything else. These are automated field notes. The transcripts came from YouTube’s auto-captions, which mangle proper nouns with real enthusiasm, so every name and number in this episode should carry an implicit “approximately.” The synthesis is mine, or more precisely, mine and the agents I run. That provenance is the subject of half the conference, and we will get to that.

The through-line first, because it genuinely was a through-line and not a theme I imposed afterward. Across four days and over two hundred talks, almost nobody argued about model quality. Nobody stood on stage comparing benchmark scores between frontier labs. The arguments were about everything around the model: the harness, the loop, retrieval, evals, verification, memory, and the shape of the organization that wraps them. The stated bottleneck, in talk after talk, was reliability, not capability. The models are assumed. The engineering is contested. If you wanted one sentence for the whole event, a speaker from Introspection supplied it: the loop is the product.

So let’s start where the leverage is, with the harness.

The single most useful number of the conference came from a talk by Etsy. They built a benchmark called HarnessBench, a hundred and six tasks, and did the experiment everyone talks about and nobody runs: hold the model completely fixed and vary only the harness around it. Scores ranged from fifty-two percent to seventy-six percent. Twenty-plus points of swing with not a single model weight changed. And the effect was larger for weaker models, which is the part with strategic teeth. The speaker’s argument was that the industry keeps saying the models are so good you can keep the harness simple, and that this is exactly backwards, because it makes you permanently dependent on the largest proprietary models, when a well-built harness would let smaller, local, open-weight models do the same work. Whether or not you buy the whole thesis, the measurement stands. A meaningful fraction of what gets marketed as model progress is harness progress. And harness progress is something you can own.

Anthropic’s platform team gave the sharpest talk in this family, with a title I have been repeating all week: tokens should have jobs. Their observation is that when an agent underperforms, the default lever is a bigger budget, and a bigger budget treats every token as interchangeable execution. Instead, they split the budget into roles. Some tokens execute. Other tokens do a different job, and the combination is what they call a strategy. They demonstrated three. Advise, where a second agent validates each step the executor takes. Grade, where a rubric judge scores each attempt and the executor loops until it passes. And dream, my favorite, where an agent reads the transcripts afterward and writes learnings to memory, so the next run starts smarter.

The benchmark story is worth hearing in full. On a financial-analysis task, one-shot execution scored fifteen percent using about thirty-nine thousand tokens. Raise the budget to six hundred thousand and pure execution climbs to seventy-six percent. Same budget, but with the advise strategy, eighty-nine percent. And when they rescored everything on perfect-run pass rate, which is the standard that actually matters in a domain where anything below a hundred percent is a failure, plain execution managed about forty-two percent while the best strategies reached seventy-five. The framing that generalizes is their cost arithmetic: true cost is budget divided by pass rate. A cheap approach you have to run three times costs more than an expensive one that lands on the first try. Tokens are not fungible. Give them jobs.

The DSPy maintainers made the same case at the level of software structure. Their pitch is program, don’t prompt: treat an AI task as a function with a declared contract, typed inputs and outputs plus instructions, which they call a signature. Keep hard constraints in code. Define what good looks like with evals. Then let optimizers rework everything inside the contract while the contract holds still. New techniques drop in as roughly one line. Their enterprise example: Shopify cut costs by a factor of five hundred and fifty by downshifting to a much cheaper model, and could do it safely because nothing outside the contract had to change. Their claim about what has stayed constant since 2022 is a good filter for tooling decisions: specs, code, and evals. Everything else has churned.

And then there was the workflow version of the harness argument, which showed up in two keynotes. Peter Steinberger of OpenClaw declared the twenty-terminal workflow dead. If you are juggling ten parallel agent sessions by hand, you have made yourself the scheduler, the router, and the memory. His replacement is one persistent manager agent that delegates to workers, coordinates them, wakes on triggers, and surfaces only finished pull requests to the human. The deeper point was his bottleneck analysis. First the constraint was tokens, then it was compute, and now it is human attention, the one resource you cannot provision more of. Deciding where attention goes is now the core skill.

Garry Tan’s keynote drew the same picture at company scale: agents as a managed workforce of markdown skill files, with a resolver table for an org chart and trigger evals for performance reviews. His signature advice fits on an index card. Never do one-off work. If you did a task once, turn it into a skill so the next occurrence is free. And his line about where computation belongs is the one I would tattoo on every harness design doc: judgment and vague intent live in latent space, steered by markdown; storage, logic, and anything resembling a large batch job live in deterministic code; and most bugs come from putting work on the wrong side of that line.

From the harness to the bill, because the second theme of the conference was money, and specifically the discovery that everyone has been optimizing the wrong side of the ledger.

Tesco sent an engineer named Rajkumar Sakthivel to deliver one measurement and one fix. The measurement: on their project, a typical AI coding query sent forty-five thousand tokens of context when about five thousand actually mattered. Roughly ninety percent of AI coding cost is input. Ten percent is output. So compressing your outputs by seventy-five percent saves you about eight percent of the bill, while cutting input by ninety-four percent saves about sixty-one percent. Prompt instructions like “only show relevant code” cannot help, because the cost has already been incurred by the time the model reads the instruction. Fix the input. That is where the money goes.

Their fix is a local code index sitting between the codebase and every AI tool. Parse the code into semantic units, functions and classes rather than files. Run two searches in parallel, semantic for meaning and keyword for exact names, because each one alone misses about a quarter of relevant results and together they miss about a tenth. Shrink what you send to names and descriptions. And gate by relevance, which turned out to be the hard part. Asking an LLM to judge relevance added two to three seconds per query. Fixed thresholds were too crude. What won was an embarrassingly simple weighted formula, half semantic score, thirty percent keyword, twenty percent recency, with an adaptive threshold, running in less than half a millisecond. On their public benchmark the result was eighty-three thousand tokens per question down to about five thousand. Ninety-four percent less, at roughly ninety percent retrieval accuracy. They named the limits themselves, which made the talk more credible, not less: the ninety-four percent is measured against naive full-file reading, real tools are already smarter than that, and on a large codebase with mixed-purpose files their recall dropped to near zero. I run a token-optimizing proxy and a code-graph index on my own machines, and everything in that talk matched my own meter readings. The win is never a smarter answer. The win is sending less.

Artificial Analysis brought the macro view of the same subject, and their headline sounds like a paradox until you look at it: token prices are falling five to ten x per year, and cost per task is rising anyway. The frontier keeps expanding what we ask. A simple question costs a fraction of a cent; an agentic task now routinely exceeds twenty dollars. And because most agentic tokens are input tokens, the cache-hit price, which carries discounts of eighty to ninety-nine percent depending on the provider, matters more than the list price everyone compares. Their model-selection heuristic is the practical takeaway. For tasks with a quality ceiling, pick the cheapest model that clears the bar. For tasks without a ceiling, pay for intelligence. And two models at identical list prices can differ substantially in real cost purely through token efficiency, which no pricing page will ever show you.

The third talk in this cluster was Salesforce on the agent tooling layer, and it contained the conference’s most quotable statistic about context. Fifty MCP tool schemas cost fifteen to twenty thousand tokens. That is around sixty percent of a working context window, spent before the agent has read a single line of your code. Their taxonomy is clean: a CLI is how to execute, MCP is what’s available, and a skill is how to do a task, a runbook that loads only what a job needs, on demand. But the part of the talk that earned applause was the defense of the command line. Fifty years battle-tested. Readable. Composable through pipes. And reproducible, because you can copy the exact command that failed at two in the morning and run it again. Their heuristic: if an engineer could do it from a terminal, the agent probably can too. I recently built a literature-search interface for my own agents as a CLI rather than an MCP server for exactly these reasons, and the context savings were immediate. And their security rule deserves to be carved somewhere permanent: enforce isolation in your infrastructure, never in your prompts. Prompts can be injected. Infrastructure cannot.

Which brings us to retrieval, the theme closest to my own work, and the place where the conference delivered its most precise diagnosis.

Mixedbread ran the experiment that frames everything. Take a benchmark with a fixed corpus, hand the model the correct documents directly, and measure. That oracle setup scores ninety-three percent on BrowseComp Plus. Now let the same class of model find the documents itself with its default tools, and it drops about nine points. The reasoning did not get worse. The retrieval failed. Their conclusion is the sentence I came home repeating: what limits agents is access to the right knowledge, not the reasoning applied to it.

Their analysis of why agents write bad search queries is a small masterpiece of failure taxonomy. Coding agents inherit grep habits, so they search with keywords and regular expressions. Models imitate human web-search behavior, which is also keyword-shaped. And the retrieval benchmarks everyone trains against are built from short entity-style queries, what the speaker called caveman queries, which structurally reward exactly that behavior. One of their fixes costs nothing and you can adopt it today: when you prompt an agent to search, do not say “write a search query.” Say “write one concise sentence describing what you want to find.” That single reframe dodges the keyword reflex. With a four-tool search harness and a small fine-tuned model on top, they recovered to within about three points of the oracle ceiling.

Jina’s talk supplied the reframe that ties retrieval to the compute story. Test-time compute means spending more inference to get better answers, and the famous datapoint is Noam Brown’s poker bot, where twenty seconds of thinking matched a hundred-thousand-fold scale-up in model size. Their observation: building a search pipeline, embeddings plus reranker plus query expansion, is already test-time compute. You are assembling computation at inference time to buy relevance you did not train into the model. And then they did the most WF2026 thing imaginable: they let an agent redesign the pipeline itself, overnight, with no human in the loop. A proposer model mutated one Python program per generation over a frozen encoder it could call but never retrain. An evaluator scored each attempt. A memory file carried scores, lineage, and a one-line lesson per program forward. A hundred and forty-four programs later, two findings survived scrutiny. Cheap structural recombinations transferred to new tasks. Raw more-compute variants did not generalize. And the memory that made the search work also compounded its biases across every descendant program, which the speaker flagged with the most important warning label in this whole genre: the loop optimizes the metric you gave it, not the one you meant.

Two more retrieval results deserve their moment. StarlightSearch located a dead zone in most agent stacks: your observability captures every tool call, your evals judge pass or fail, and none of that signal ever reaches the agent’s context, so the agent cannot learn why yesterday failed. Their fix weights memory retrieval by a learned utility score, similarity to the current task multiplied by whether that memory historically helped or hurt. The agent stops accumulating trivia and starts accumulating lessons, things like “check the settlement status before issuing a refund.” Policy compliance on tau-bench went from sixty-six to seventy-six percent with utility-weighted memory, and eighty once repeated lessons were compiled into skills. They also admitted cold start is unsolved, which I note because speakers who named their own limitations were rarer on those stages than they should be. And Sakana ran a controlled comparison of recall mechanisms on local hardware, a Mac Studio running a twenty-seven-billion-parameter model, and found that a ranked ledger of per-turn decisions beat vector RAG as the memory mechanism, with the corollary that when a task fits in the context window, memory adds cost and no capability at all.

The substrate arguments rounded out the theme. Neo4j argued that enterprises fail at agent scale because every team re-wires and re-trust-checks the same hundred data sources, and proposed a shared semantic layer: a business ontology in human vocabulary, a technical ontology mapping each concept to its system of record, and execution traces that score data-source choices so the substrate learns across all agents at once. Thin agents on a smart substrate. cognee showed the research version, a biomedical knowledge graph built for Bayer whose embeddings predict which research directions deserve a scientist’s attention. And an Ogilvy engineer demonstrated that none of this requires a platform budget: PDF-to-markdown conversion on a plain CPU, Postgres for vectors, hybrid retrieval fusing semantic and keyword scores with reciprocal-rank fusion, guardrails in code rather than prompts, all driven by a model small enough to fit on a phone. I can vouch for that pattern from the maintenance side, because the retrieval lanes on my own site fuse BM25 and embeddings exactly that way, and when ranking goes wrong, every piece is inspectable.

Now, evals, because the loudest single theme of the conference was that evaluation is becoming infrastructure.

Three numbers from Arize’s keynote set the scene. They run more than a hundred million evals a month. The average team on their platform runs about twelve eval jobs. Their top teams run more than three thousand eight hundred distinct evaluators. That distribution says the leaders are playing a different game entirely, and the game has a name that half a dozen speakers used independently: evals are the new CI. As agents produce more work than humans can review, evaluation stops being a gate you pass before shipping and becomes an always-on production service.

Meta traced the shift. Benchmarks ask whether the model produced the right answer. Agentic systems force a different question, whether the system behaved correctly, through planning, tool use, and recovery, which means evaluating realistic multi-step scenarios in simulated environments rather than single prompts. And their warning was about silent drift: agent traces are the distributed tracing of this era, and the failure you should fear does not throw an exception, it just degrades.

The architecture that kept reappearing is a three-layer judge stack. Deterministic rules at the bottom, catching what rules can catch, like whether the task finished within six tool calls or whether a secret leaked into output. LLM-as-a-judge in the middle, scoring against fixed rubrics, which works while trajectories resemble each other. And the new layer on top, agent-as-a-judge, which Arize shipped as a long-running agent called Signal that reads live production traces, finds patterns rubric judges miss, and, because it holds the full analysis in its own context, can open a pull request proposing the fix. Weights and Biases showed what the nightly cadence looks like for their research agent: about two hundred tasks defined in YAML, each judged by an LLM and rule checks together, run every night like a test suite, where a seventy-three versus seventy-two percent delta is enough evidence to promote a change. Tasks are the unit tests of the model era.

Sonar delivered the firmest line on verification as its own discipline. Every model has biases, so a system that verifies with the model that generated is auditing itself. Their production numbers for verification baked into the loop, with multiple models checking each other: forty-four percent fewer AI-derived outages. Amazon’s AGI group made the historical observation that coding fell to agents first precisely because code is verifiable, and extending agents anywhere else means extending verification there too. And a talk by Aditya Mani supplied the governance principle that stuck with me: own the verdict. An agent can gather all the evidence you want, but accountability for the judgment does not transfer to it. The humans in the successful systems calibrate the judges and own the final call. That is a smaller job than hand-writing every eval, done from a higher altitude.

Before the headline act, a word about the frame the whole conference was staged inside, because “software factory” was on the marquee and the fights about it were better than the pitches.

The optimists’ picture is coherent. A software factory is the entire loop run autonomously, signals in from users and telemetry, triage, spec, implementation, verification, ship, monitor, improve, and around it Factory described validation contracts at every stage with model routing that they said saves about a quarter of their spend, Warp talked about factories that improve their own tooling, and Google’s Antigravity pitch was scale itself, the idea that you stop thinking in units of engineers and start thinking in units of loops. swyx opened the whole event with a hierarchy of loops, the inner loop of a coding session nested inside the outer loop of a product nested inside the outermost loop of a company learning what to build, and his claim was that whoever operates the highest loop wins the era. Graphite brought data suggesting agent-written code is now roughly at parity with human-written code on their review metrics, with the pointed corollary that generation was never the expensive part and review is where the whole system now queues.

Notion’s contribution was the strategy line people quoted at dinner: your supplier is your competitor. Every company building on frontier models is buying capability from labs that are simultaneously building products over the same capability, so the durable position is betting on the frontier itself rather than any single lab, staying model-portable, and owning the layers the labs cannot see, your data, your evals, your users’ trust.

And then there was the Great Loops Debate, an actual staged debate about whether autonomous loops are the inevitable core unit of software engineering or a hype cycle outrunning its discipline, and it distilled the conference’s central tension better than any keynote. The pro side argued that everything reliable in computing is already a control loop, thermostats, autoscalers, reconciliation loops in Kubernetes, and agentic loops are just the newest member of a proven family. The skeptical side argued that those classical loops work precisely because their sensors are deterministic and their error signals are trustworthy, and an agentic loop with an LLM judge in the sensor position is a control system built on a sensor that hallucinates. Nobody won, which was the correct outcome, because the disagreement is empirical and the data is still arriving.

Mike Krieger’s fireside added the piece that reframed the economics for me. Writing code, he said, was never the limiting part of building products, and the teams adapting best are the ones who noticed that the constraint has moved to deciding what to build and knowing when it is right, which are judgment problems, not generation problems. Hearing that next to Steinberger’s attention argument and the review-debt findings, the shape of the year becomes visible: every scarce resource in this industry is migrating toward the same place, human judgment applied at the right altitude.

Which leaves the headline act. Autoresearch. Agents improving the systems that run agents.

The results are real, so let’s state them plainly. Weco’s agent set seven records on OpenAI’s Parameter Golf benchmark against the best human’s three, with a better H-index, using no more than four percent of the total compute spent. Recursive dot com reported beating NVIDIA’s best CUDA kernels and improving well-known training speedrun records within days of trying. GEPA, the reflective optimizer out of Berkeley, produced the sample-efficiency result of the year: reinforcement learning collapses a rich rollout into a single scalar reward, while GEPA has a model reflect on the whole trace in text and edit a prompt instead, and one round of reflection on three data points matched twice the gains that policy optimization achieved after twenty-five thousand rollouts. On an undocumented AMD chip, a GEPA loop took kernel utilization from four percent to thirty percent, and along the way discovered that a header file the vendor shipped simply did not work. Databricks reported a ninety-fold cost reduction using the same tool. Anything expressible as scorable text is now an optimization surface, prompts, kernels, harnesses, even the policies that schedule your cloud jobs.

And then Prime Intellect, who run an open benchmark exactly so outsiders can verify claims like these, added the asterisk I consider the most important sentence of the conference. Their record-setting agents produced no truly novel optimizers. The wins were recombinations, plus-one compositions of known techniques applied with superhuman patience and no fatigue. Records without invention. That distinction is worth remembering when the phrase “recursive self-improvement” starts appearing in press releases, and open third-party benchmarks deserve support, because they are the only reason we know the difference.

The skeptics earned their stage time too, and the talk I have thought about most since came from HumanLayer. They actually ran the lights-off software factory, nobody reads the code, keep the queue full, in July of 2025. It broke. Unsolvable issues, outages, accumulating slop. The diagnosis is structural. RL rewards are binary, the test passed or it did not, and there is no reward channel for architecture, so models learn precisely the hacks that make tests pass, the needless exception handler, the type cast, the commented-out test. Verifying maintainability is orders of magnitude harder than verifying correctness, because bad architecture sends you the bill months later, far beyond any reward horizon. Which is why models have improved enormously on greenfield work while brownfield codebases still degrade after three to six months of agent maintenance. As the speaker put it, if a model knew what good code looked like, it would just write it. Their fix is thirty minutes of model-assisted planning up front, architecture contracts and typed design before any code, so that a human can still read every line at review time. Their colleague gave the constructive companion talk, loops built like control systems, with a sensor the agent cannot disable, a committed baseline, and at most one small reviewable pull request per day. The contrast with a bash loop producing forty-thousand-line pull requests is the entire debate in one image.

I want to close with the quietest talk of the conference, because it is the one I could verify from my own chair. Two builders, Pauline and Luis, described their personal research OS: plain markdown files, sources and comparisons and implementations, that both humans and agents read, extend, and reuse across projects. They chose files over NotebookLM because you do not own NotebookLM and it is not agent-native, and over RAG infrastructure because a vector pipeline is not something you can open and edit. Knowledge compounds because nothing is trapped inside a session.

The episode you just heard is an existence proof of that talk. My conference notes live in an Obsidian vault that syncs to a headless Linux machine, where agents read the vault as ordinary files. The two hundred and six talk notes were written by agents from caption transcripts. The essays on my site were distilled from those notes with a style guide enforced along the way, and this transcript came out of the same pipeline, a day after the vault was set up. The loop that the conference kept describing, capture, distill, verify, publish, is the loop that produced the thing you are listening to.

So, what to actually adopt. Give your tokens jobs, because equal budgets do not produce equal results. Fix the input side of your bill before you touch the output side. When your agent searches badly, change what you ask for, one concise sentence describing what it wants to find. Weight your agent’s memory by outcomes, not similarity alone. Run your evals nightly like a test suite, and let a human calibrate the judges rather than hand-write them. Run the outer loop on everything cheap to verify. And keep a human reading the code everywhere verification runs out, because that boundary, between what the loop can check and what it cannot, is where all the interesting engineering lives now. The models will keep getting better without your help. The harness will not.