Real prompts are 88% of usage and 7% of benchmark problems

Eighty-eight percent of real user requests to a coding agent carry a problem statement and little else. In SWE-bench Verified and Pro, prompts of that shape are 7% of the problem set. RealSWE, posted to cs.SE today, measures the gap by holding the task and the gold patch fixed and varying only what information the request carries and how it is written: 381 task families, six information categories, four dimensions of linguistic style. Across seven contemporary models, realistic inputs cut resolution rates by 6.4 percentage points on average and changed model rankings. The ablation is the part worth acting on. Stating the desired behavior and the motivation moves the number; environment information and reproduction steps add tokens without measurable benefit; linguistic style, the thing people fuss over when they tell you to write formal prompts, has small and model-dependent effects. Real prompts are also the ones that omit desired behavior, so the cheapest available accuracy gain is a sentence most users do not write.

That is one of four papers in the last day arguing that outcome scores hide the thing you actually need to know. The sharpest is What Does an Agentic Software Engineering Benchmark Measure? from Shayanfar, Gallaba and Hassan, which proposes a Spread-Novelty-Centrality profile for repository-level tasks and applies it to five widely used benchmarks and 14,922 trajectories across two model families at three scales. Every pair of benchmarks separates on at least two SNC axes, and the separations trace back to specific curation decisions, which means a category label like “bug fix” tells you close to nothing about the engineering work a suite demands. The behavioral finding is stranger and more useful: agents write larger patches than the human gold solution when the problem statement withholds hints, and smaller ones where curation inflated the gold patch. Success also has a family-specific signature. Claude resolves tasks by matching the scope of the gold solution, with its parity share on files climbing from 0.17 at the smallest scale to 0.54 at the largest; Qwen resolves by exceeding gold scope at every scale. Editing too little marks failure for both.

If trajectories carry that much signal, they should also make evaluation cheaper. PTA-IRT takes exactly that position. Existing subset-selection methods for estimating full-benchmark performance fit historical pass/fail matrices and discard how the agent got there. Duan and colleagues feed execution trajectories, explored context, attempted edits, and solving paths, in as privileged information during calibration-subset selection and ability estimation, and beat prior IRT baselines on both score and ranking recovery across four SWE benchmarks under low calibration budgets. Running a full agentic suite is expensive enough that a better sampling rule is worth more than another leaderboard entry.

The raw material for this line of work just got much larger. AgentLogs is a scan of 1,812,362 popular public GitHub repositories that pulled 307,416 Copilot cloud-agent tasks and 549,239 sessions from 35,810 of them, with 64,255,174 log entries recording prompts, intermediate reasoning, tool calls, and token usage step by step. Existing datasets stop at agent-authored pull requests, which is the outcome. This is the process, at a scale where failure-mode taxonomies and cost analyses stop being anecdotes about your own repo.

Handoff Debt picks up the cost question from the other end. Benchmarks evaluate one uninterrupted agent; real tasks get interrupted, reassigned, and resumed from someone else’s partial state. KC and Budathoki interrupt an agent at deterministic points, freeze the repository, and hand the successor one of four views: repository state only, raw trace, summary notes, or structured notes. From 75 source tasks they generate 181 handoff-point tasks and 724 takeover runs per successor model. Context-bearing handoffs cut median agent events by 20-59% and cumulative prompt tokens by 42-63% against a repository-only takeover, across three successor models. Solved rates barely move and the effect is model-dependent; the efficiency gain is consistent. The recommendation follows directly: report what a task costs the next agent to resume, not only whether it was solved.

Tooling is converging on the same problem from the product side. VS Code 1.135 ships an Agent Host built on a new Agent Host Protocol that runs agent harnesses in a dedicated process, so several VS Code windows can attach to one session, and its Sessions list now surfaces recently updated agent sessions started outside the editor, including Copilot and Claude sessions from other applications, behind chat.agentSessions.showExternal. That is the handoff-debt problem treated as a session-continuity feature. The release also adds an experimental /rubber-duck command that puts a second, complementary model on the primary agent’s plans, code, and tests looking for gaps the first one missed.

One item from the governance side, because it is the same argument applied to facts rather than trajectories. MAGG treats knowledge-graph construction as a governance problem: a domain classifier induces entity and relation types from document content instead of a fixed schema, candidate triples are assigned to domain owners, reviewed against supporting evidence, admitted by an explicit decision, and stored with audit metadata, and the same ownership structure routes queries to domain-specific graph experts at answer time. Strict triple F1 improves 47% over flat insertion on SciERC, mapped triple F1 by 51%, and a blinded review of 120 triples finds governed-only triples more often source-supported than flat-only ones. On MuSiQue it beats Microsoft GraphRAG by 9.0 exact-match points. Recording who owns a fact and why it was admitted is not overhead added to retrieval; here it is what makes the retrieval more accurate.

Worth watching: whether anyone runs the SNC profile against AgentLogs. The profile characterizes task demands from curated benchmarks; AgentLogs holds 64 million log entries from tasks nobody curated. The distance between those two distributions is the number that would tell us how far benchmark scores travel.