A grader worth 29 points: auditing agent evals, conflicting skills, and routing data agents by ownership

A reward function in an agent benchmark was modifying the environment’s state before the post-run scorer read it. When an oracle-integrity audit replayed 4,320 episodes against the corrected state, database-pass labels fell from 4,194 to 2,883 and measured success went from 94.3% to 65.5%. No model, prompt, or task changed between those two numbers; all 29 points came from the grader.

Most of the strongest results this week measured the measurement: graders that mutate state, tests that agree with wrong code, agent reports that drop much of what happened, and a single harness setting that moves attack success by 66 points.

Graders, tests, and agent reports each overstated what happened

The audit’s method is simple enough to borrow. It defines five state-integrity invariants and three known-answer controls, then injects seven classes of oracle defect across two task domains plus a synthetic one; every class was caught, and the repaired oracle raised no false positives. A second experiment over 3,240 episodes varied the agent’s policy documents, tool descriptions, and tool error strings, and found tool descriptions had the largest effect on trajectory reliability (a 9.2-point drop) and error strings the smallest (5.3 points). The language effect reversed sign in the second domain, so the authors bound the result to the tested configuration. The limit they report is the most useful sentence in the paper for anyone maintaining a benchmark: the audit could not validate a second, independently built benchmark because its recorded API-call log was not enough to reconstruct the reference terminal state. If a harness cannot replay its end state from its own logs, nobody can check its scores.

Passing Tests, Missing Semantics carries the same problem into generated code. On 30 DataFrame tasks derived from TPC-DS, test-guided revision lifted test passes from 22/30 to 29/30, and a plan-audit loop reached 30/30, yet no generated solution matched its stored reference output. Evaluator controls returned the expected outcome in 57 of 57 cases, and all 77 test-passing runs on tasks with reproducible references still disagreed with the reference. The failure analysis is a catalogue of how self-written tests go wrong: they omit stated requirements, adopt the schema the candidate invented, use fixtures that cannot tell right from wrong, or restate the implementation’s incorrect logic. The authors call it interpretive lock-in and keep the claim diagnostic (one model, one primary generation per task). For analytics agents the lesson is direct: a misread business question survives planning, implementation, and testing intact, because every stage inherits the same reading.

CTXGAP measures what a supervisor loses by reading the agent’s report instead of its transcript. Across 37 coding tasks and 4,799 probes about task state, readers with the full execution context recovered 92.7% of answers, while readers with only the user-facing reports recovered 52.0%, a 40.7-point gap. At the final checkpoints of 30 fixes that were verified correct, the gap was still 34.8 points. The largest category of lost information was state the agent did mention, but without enough detail to act on. Full terminal scrollback raised recovery to 73.6%, and rewriting reports with the context in view gained about eight points at matched length, so the problem is what the agent chooses to say rather than how much room it has. The gap held across harnesses and with DeepSeek doing both execution and measurement.

SWE-Maintain asks whether code an agent wrote makes the next task harder. It runs 100 three-step maintenance episodes from 50 Python repositories twice: once from the developers’ real patches, once from the agent’s own earlier changes. Across five coding systems, two backbone models, and 1,996 paired events, starting from agent history cut later resolution by 5.6 points, and by 14.0 points when success also required every earlier test to pass. Two-thirds of the agent-history starting repositories already failed earlier tests, and only 1.9% of events that began in that state ended with all accumulated tests green. When the inherited code passed its earlier tests, the disadvantage vanished (+2.7 points, 95% CI from -0.5 to +6.1). On this evidence the downstream cost of agent code is the broken tests it leaves behind, which argues for gating agent merges on the full prior suite rather than the task’s own tests.

The harness decides which instructions run and what an attacker can reach

One Skill Too Many mined 20,947 repositories for agent skills that do the same job and found 822,109 candidate pairs. After an LLM judged a stratified sample of 3,754, the authors ran 312 confirmed pairs on three models: 6,368 runs, 169,294 tool calls, 542 agent-hours. Nearly one in four installed skills sits next to a similar one, and 37% of judged skills live in copied collections. The similar skill took one run in five without lowering task completion, and runs that opened the wrong skill first lost more than a third of the core functions only the intended skill carried, such as a ban on touching git. Install location decides which skill wins while listing order barely matters, and the final reply named the skill actually used in 0.9% of substituted runs. Conflicts were settled at the first skill read, almost always before any file changed, so a pre-tool hook at that read restored fidelity. A benchmark that checks only whether the task passed will never see any of this.

HarnessSecurity-Bench tested nine security mechanisms across Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code, and GitHub Copilot, with GLM-5.2 held fixed as the model: 2,500 trials, 81,155 tool calls, more than 2.2 billion tokens. Turning on auto-approve increased task utility and raised attack success from 29.2% to 95.6%. Network isolation and read-only mode cut attack effects at a substantial utility cost; command allowlisting cost a little utility, and denylisting gained some. A review of 400 harness-mechanism cells found about half of the confirmed mechanisms are opt-in, with large evidence gaps in the closed-source harnesses. The task-level cases matter most for anyone writing policy: an allowed tool often leaves a blocked operation reachable through another execution path, so every denylist entry is a reachability claim that needs its own test.

TurnCost moves the question of where coding-agent money goes. Across 600 public trajectories from two scaffolds and two models, median tool observations were 118 to 286 tokens; the giant build logs that motivate compression sit near the 99th percentile. Because 98.6% of input tokens came from the prefix cache, flat input pricing overstated loop cost by about 7x. Turn count explained 84% of the variance in log trajectory cost against 4% for mean observation size, and delegating turns to a cheaper model raised cost in half the settings tested, close to the paper’s closed-form break-even. The authors stop short of showing that turns can be removed at fixed success, but the measurement points cost work at planning and loop termination rather than at observation compression.

Netflix published the other side of that ledger. Its platform teams connected JetBrains’ headless code-intelligence engine to Claude Code and Codex because agents navigating Java and Kotlin had to guess which JARs mattered, unzip and decompile them, and grep an incomplete view. JetBrains ships the engine as an LSP server; Netflix built the daemon and CLI that keep each project’s index warm between an agent’s many small calls, distributed as GraalVM executables to avoid JVM startup. Tested on an internal Spring Boot 4 service and the open-source DGS Framework with Opus 5, Sonnet 5, and GPT-5.6 Sol at medium and high effort, the index made agents typically 20 to 30% more efficient (the average of token and time savings), about 65% in the best case, with correctness holding or improving. The largest gains went to the strongest model at high effort, which is also where tokens cost most. Read next to TurnCost, the mechanism is plausible: one precise “find implementations” call replaces a chain of unzip, grep, and read turns.

Orchestrators now choose their own topology, and peers still don’t check each other

At AI Engineer New York, Anthropic’s CJ Avilla and Harrison Stall walked through dynamic workflows, or “flows,” in Claude Managed Agents. A lead agent plans the work and then writes an orchestration program that launches sub-agents, collects their results, and routes information between phases, replacing the coordinator-plus-roster configuration developers previously wrote by hand. The Q&A said more than the pitch. The model does not know its budget while running, so developers set a session budget instead; asked how Claude decides how many agents to spawn, Stall answered “evals and lower defaults.” Sub-agents share a filesystem but cannot be told through the API to message each other, and the payload the lead sends each sub-agent is inspectable in the Console graph. On the stock-watchlist example, flows cost “much more” than a single agent and found more of the details, which is the trade every team adopting this will have to price.

A²-Failures studies what happens when the agents in a system answer to different principals, such as a buyer’s agent negotiating with a seller’s. It draws a 20-mode taxonomy from more than 2,500 recorded agent-to-agent runs over 488 scenarios in eight families, including legal, CRM, security, and commerce. Three behaviors held across harnesses and models: agents matched tasks to peers whose names appeared in the prompt instead of searching for available peers, agents never fact-checked each other unless told to, and scoring only the final output credited the lead agent while hiding coordination and verification breakdowns underneath. The run logs are public. Flows-style orchestration inherits the second and third risks directly, since the lead now designs the topology and the evaluation usually sees only the end product.

Text-to-SQL now starts with finding the database and its owner

DSABench drops the assumption that the agent is handed the right database. Its 815 human-reviewed tasks, drawn from Spider, BIRD, and LiveSQLBench, sit in a catalog; 618 need several databases, and 105 admit more than one valid set of sources. Across seven model backends, the best score for choosing the right set of sources was 60.55%, and the best for also getting the cross-source dependencies right was 34.66%. With execution required, median end-to-end success was 6.69% to 7.98% across four backends, rising to 11.44% to 13.25% with one clarifying question; the best configuration reached 16.99%. On ambiguous tasks, one clarification improved exact source recovery by 39.8 to 43.2 points, which makes a single question to the user worth more than most routing tricks. Correct local SQL is not the bottleneck in an enterprise catalog; carrying the right information across sources is.

Fan Out or Relay? adds the governance constraint DSABench leaves out: in most organizations, owners query their own tables and never hand them over. A coordinator can fan the question out to every owner at once, or relay it from owner to owner while reusing returned rows. Fan-out wins in 12 of 13 settings on questions one owner can answer, but answers at most 4.3% of questions that join two owners’ tables. On BIRD two-table joins it scores 32 to 44% when one owner holds both tables and 1 to 4% otherwise, so ownership, not query shape, decides the right protocol; hiding returned rows from an otherwise identical relay planner cuts two-owner accuracy from 23.2% to 5.1%. The proposed fix uses only public metadata: draft a schema-only SQL query, count the owners whose tables it joins, and route before any private row is read. That rule was never significantly worse than the best fixed protocol, and beat all of them on PeopleJoin by up to 9.3 points on held-out organizations. Ownership metadata is a routing signal, which is a strong argument for keeping it in the semantic layer next to the metric definitions.

The intent side shows up in a deployment study from a CFO unit’s data lake. Business users wrote prompts in finance vocabulary with no schema references, so the authors placed a semantic mediation layer in front of the assistant and compared two designs: mapping terms to schema elements by semantic similarity, and mapping them through a knowledge graph built automatically from the data lake. In a simulation on real-world data, with LLM-judge scores validated by humans, the knowledge-graph mediator beat the similarity one on precision, recall, F1, and preservation of user intent. The abstract gives no effect sizes, so treat it as a design result rather than a benchmark number.

Netflix also described the definitions layer that kind of mediation depends on. DataJunction began as a fix for its experimentation platform, where onboarding one metric took weeks of tribal knowledge. The team chose it over Airbnb’s Minerva and an internal framework because it parsed SQL into a connected graph of metrics and dimensions and generated SQL back out of it; cubes serve as the consumption interface, materialized into Druid, and every dashboard metric links back to its definition. Three Netflix engineers and a few outside contributors built it as a side project for more than two years. The agent angle is one sentence, and it is the right one: with definitions in a graph, Netflix’s internal LLMs “can deliver accountable information about the business metrics as well as explanation of how they are defined and used across Netflix.”

Fixing search drift in multi-hop RAG exposes a sufficiency problem

ExecutionRAG separates two causes that retrieval metrics lump together: the passage was unreachable, or the agent never issued the query that would have found it. Holding retriever, corpus, and generator fixed, the authors add a training-free register of open sub-questions and the evidence gathered for each. Because the register changes search guidance but not access, every gold passage it recovers proves the original retriever could have found it. On 1,000 MuSiQue questions, wrong answers missing required evidence fell from 382 to 234 with all-MiniLM-L6-v2 and from 383 to 274 with Qwen3-Embedding-0.6B. Total errors did not fall in proportion, because premature answers and abstentions rose as missing evidence fell. Once search stays on track, the hard part becomes deciding whether the evidence in hand is enough, the same judgment the CTXGAP and oracle-audit papers found failing in coding agents.

What to watch

Two numbers from this week need follow-up data. The 0.9% rate at which agents name the skill they actually ran is a gap a single field in the final reply would close, and it will be telling whether any harness ships that field before the next skills study. The second is DSABench’s ceiling near 17% end to end: every result in it was measured against raw catalogs, and nobody has yet run the same multi-database tasks against a governed semantic layer with ownership and definitions attached, which is the comparison that would show whether that layer moves agent accuracy or mainly moves accountability.

---

Sourcing note: items come from the code-intel-copilot feed for 2026-10-12 through 2026-10-19. The feed’s mirror reported its last production sync on 2026-09-01, but the local ingest carried items dated through 2026-10-19, so the window was fully populated.