Daily digest

Agent failures concentrate at unverified boundaries

Aug 4, 2026 · 🎧 12 min

agentic codingevalsmulti agent orchestrationagent reliabilityinformation retrieval

The first production-scale characterization of GitHub Copilot traces (95T tokens, 13M sessions) shows coding-agent workloads breaking chatbot-era serving assumptions. Meanwhile a repo-QA study measures the grep-by-subagent pattern losing to plain semantic search, with 41.8% of its failures silent at the planner-subagent hand-off, and three more items from the last day land on the same point: agents fail at boundaries where verification is missing.

Highlights

  • Semantic search beat grep-by-subagent on SWE-QA, 65.2% to 46.2%, at less than half the cost per correct answer; 41.8% of subagent failures were silent losses at the planner hand-off
  • GitHub Copilot production traces: KV cache hit rates average 90% within a turn and fall to 55% across turn boundaries, with an idle-time predictor capturing 86-90% of user idle
  • Attack success in multi-agent pipelines tracks pipeline structure, not model capability, across GPT-5-mini, Claude Sonnet 4.5, and Kimi K2.5

Fifty-five percent. That is the KV cache hit rate at turn boundaries in GitHub Copilot’s production agentic workload, down from 90% within a turn, according to the first production-scale characterization of coding-agent traces, posted in the last day from Microsoft and GitHub researchers. They sampled Copilot traces from June 2026: 3.2 million users, 13 million sessions, 761 million LLM calls, 95 trillion tokens. The workload shape is nothing like a chatbot’s. Sparse user-initiated turns each unfold into an autonomous loop of LLM calls almost always coupled with tool execution, then the user goes idle for minutes at the turn boundary while cache entries rot. Model switches and context compaction invalidate the cache almost entirely. Their lightweight idle-time predictor captures 86 to 90% of total idle time, which is a signal a scheduler can actually act on: evict, prefetch, or reschedule during the gap instead of holding capacity for a user who is reading a diff. If you serve agents on your own stack, this paper is the empirical argument that chatbot-era serving assumptions no longer describe the traffic.

The sharper result of the day cuts against current fashion in agent design. An empirical study of repository-level code question answering compared semantic search over a pre-built vector index against deep agentic search, the grep-by-subagent pattern that Claude Code, Codex, and Antigravity have all adopted to keep exploration out of the main context window. On SWE-QA, semantic search answered 65.2% of questions correctly against 46.2% for the subagent pattern, and produced each correct answer at less than half the cost. The failure taxonomy is the part worth reading. The single largest share of deep agentic search failures, 41.8%, occurred at the hand-off between the planner and its subagent, and these were usually silent: the run ends in a fluent, confident answer that is wrong. Context isolation solves context rot and simultaneously creates an information boundary where losses do not announce themselves. For read-only questions over a repository that can be indexed, the authors conclude retrieval was the stronger and cheaper option. Subagent isolation earns its keep when the task mutates state or exceeds any index; treating it as the default for every lookup now has a measured cost.

A practitioner ran into the same boundary problem from the other direction. The developer of PViz, a dependency-context tool, reran his controlled evaluation after revising the analyzer: 56 new scored sessions over 28 tasks across repositories in 7 languages, with all transcripts and scorecards published. Bundle-assisted investigation scored 196/196 on substantive correctness, edging the raw-exploration baseline at 192/196; bundle-only access collapsed to 133/196. The instructive failure: in one language the bundle reported no dependency cycles, the bundle-only session confidently agreed, and a real bidirectional cycle existed through a wildcard import the analyzer had not resolved. The session with source access caught it by reading the files. In 4 of 7 languages, source reads surfaced a task-relevant error in the bundle itself. His conclusion matches the SWE-QA paper’s almost word for word: structured context directs investigation, but only when the agent can verify the structure rather than trust it. A representation that is internally coherent, confidently interpreted, and wrong is worse than no representation.

The security version of this argument arrived on arXiv the same day. Multi-agent pipelines pass intermediate outputs between agents as trusted input, and the authors name the missing primitive: boundary verification, explicit validation of content, identity, execution intent, and state integrity as data crosses inter-agent boundaries. Working from annotated traces on GAIA and SWE-Bench plus controlled experiments across GPT-5-mini, Claude Sonnet 4.5, and Kimi K2.5, they show attack success aligns with pipeline structure rather than model capability. Content injection, agent impersonation, plan deviation, memory poisoning: these are architectural properties of unverified hand-offs. A stronger model behind the same unguarded boundary inherits the same vulnerability.

Two shorter items round out the day. A systematic mapping of agentic technical debt argues that autonomous reasoning, tool orchestration, and persistent memory produce debt with different root causes and manifestations than classic ML technical debt; useful as a vocabulary for what accumulates in agent systems between rewrites. And a field report from r/ClaudeCode supplies the week’s cleanest anecdote of a silent boundary failure: a Friday competitor-monitoring agent scraped a promotional table instead of the pricing table after a minor layout change, produced a normal-looking report, and manufactured an 80% price cut that never happened. Nothing crashed; the agent found a valid table. The author’s fix is worth stealing: let the agent discover the workflow once, freeze it as a reusable command, and rerun the frozen artifact, so behavior stays constant until the site genuinely changes.

Every item in the last day lands on the same joint: agent systems fail at boundaries where verification is missing, whether the boundary sits between planner and subagent, agent and graph index, agent and agent, or agent and web page. The models keep improving on either side of those joints; the joints themselves do not. Watch for whether the major coding agents grow verification primitives at the subagent hand-off, because 41.8% silent failure at that seam is now a published number their designers have to answer.

In this issue

← All digests