Agent Reliability This Week Comes Down to Grounding, Not Guessing

Two harnesses, one model, 792 graded runs, and the headline number is a wash: -1.25 percentage points for Claude Opus 4.8 running its vendor-native harness against a neutral one (48.8% vs 50.0%), +1.25 points the other way for GPT-5.5 (55.6% vs 54.4%). A new contamination-controlled study, Harness or Model? (arXiv:2609.11987, Sep 14), set out to settle whether vendor-tuned harnesses beat generic ones on the same underlying model. The averages hide the interesting part: on the Opus split, the native harness lost by 9.0 points on the 61 repository-repair tasks and won by 23.7 points on the 19 timed contest tasks (label-permutation p = 0.003). Task shape decided the outcome, not vendor pairing. The paper also found 22 of 81 runs that hit the wall-clock ceiling had already produced a passing patch, and that re-priced token spend put the neutral harness at 1.3 to 1.6 times the cost per solved task on Opus and 1.2 times on GPT, with billing attribution murky enough on 58 Anthropic-side runs that the Opus ratio could land anywhere from 0.7 to 2.3. If your team is choosing a harness based on which vendor ships the model, this is the first paper with the paired data to say that reasoning doesn’t hold up once you split by task type.

The fix for agent failure keeps being the same fix: stop trusting, start checking

Three papers this week land on the same structural answer to three different failure modes, and it’s worth naming the pattern before the details. An agent that can be tricked, an agent that grades its own work, and an agent that’s asked to guess at something it can’t see: all three get fixed by moving the judgment outside the model, into something that can be checked.

Authority Is Not a String (arXiv:2609.08371, Sep 8) targets prompt injection in coding agents by refusing to let the model decide what counts as a trustworthy instruction at all. Coding agents carry ambient authority: if a tool can name a file, it can act on that file, and an injected instruction in a repo or in tool output can ride that authority to do things the user never asked for. CapScope, implemented on the Pi coding agent, derives a task-wide authority ceiling from trusted input before the agent ever reads repository content, then assigns each sub-agent a separate, typed capability set stored outside the model’s context. Every tool call gets checked against what that specific agent actually holds. Across 300 runs spanning five Python repair tasks, five injection surfaces, and four authorization conditions, the injected effect fired in 33 to 47 of 75 runs under ambient-authority and global-policy baselines. Under CapScope it fired in 3 of 75, while completion held steady at 68 of 75 repairs against 68-72 for the baselines. The mechanism doesn’t ask the model to spot the attack; it makes the attack structurally unable to reach a capability the agent was never given.

That gap between “looks safe” and “is safe” showed up as a live incident, too. DeepSeek Harness shipped a now-patched flaw (CVE-2026-82533, CVSS 9.4) that let an agent call the harness’s own unauthenticated API and elevate its session to “danger-full-access” with approval set to “never.” OX Security researchers Nir Zadok and Moshe Siman Tov Bustan found that a single unauthenticated command was enough, no API key or model call required, to escape the sandbox and pull stored conversation history. It’s the same shape as CapScope’s threat model, minus the fix: authority that a string, not a capability check, was gating.

ExecCritic (arXiv:2609.09133, Sep 9) applies the same logic to test quality. When one trajectory writes both a repair and the test that grades it, their errors can agree, producing false confidence in a patch that hasn’t actually fixed the bug. ExecCritic splits the roles: an independently trained Test agent writes repository-native tests, a fail-closed harness freezes them before the Repair agent ever sees them, and the Repair agent revises code without touching the tests. On SWE-bench Verified, tests from the base Test agent actually dragged resolved rate down, from 61.2% with no test feedback to 57.3%, while tests from a stronger model (GPT-5.6-sol) pushed it to 65.3%. That’s the strongest evidence yet that “give the agent execution feedback” only helps when the feedback is trustworthy, and that trustworthiness has to come from somewhere other than the agent that wrote the patch. Role-specific training closed most of the gap: the trained Test agent’s success at matching gold tests rose from 22.2% to 62.2%, and composing both trained roles reached 72.6% resolved, an 11.4-point gain over the no-test baseline.

Zoom out from code to security operations and the pattern repeats structurally in Why AI Agents Shouldn’t Guess at Vulnerability Exploitability (DevOps.com, Sep 10). Exploitability isn’t a property of a CVE record; it depends on whether a vulnerable package is actually reachable through a live dependency chain, deployed artifact, and running environment, evidence that lives scattered across registries, SBOMs, and call graphs the model was never shown. A fluent, confident-sounding prioritized list built from scanner output alone is still a guess, and the failure mode is dangerous precisely because a guess and a grounded answer look identical in the output. The proposed fix is the one running through every paper above: build the graph of actual relationships first, then let the model reason over verified structure instead of inferring it.

Text-to-SQL stops pretending enterprise schemas are flat

A Cost-Aware Agentic Architecture for NL-to-SQL over Nested Enterprise Schemas (arXiv:2609.04641, Sep 7) opens by pointing out something obvious in hindsight: academic text-to-SQL benchmarks measure flat, well-normalized schemas, while production enterprise schemas are graph-like, semi-structured, and deeply nested. The paper ships two things. First, the DevRev NL2SQL benchmark, 900 execution-verified queries against nested-type, link-graph schemas, scored with a new Semantic Depth Score that rates analytical reasoning depth independent of schema shape. Second, a cost-aware, single-generation agentic architecture whose schema-selection, metadata-retrieval, and error-repair stages are built for that nested regime specifically. On the new benchmark it hits 91.7% answer correctness, 54.6 points over the next-best baseline, and stays competitive with leading systems on the public Spider 2.0 Snowflake benchmark at a single-generation cost point. The margin is the story: existing systems tuned on flat-schema benchmarks apparently fall apart on the nested structure that most real enterprise data warehouses actually have.

Evaluation itself needed the same reality check. SQLMorph (arXiv:2609.08950, Sep 8) argues public text-to-SQL benchmarks don’t capture enterprise schema complexity, and private eval sets are too costly and nondeterministic to reproduce. Its fix is query mutation: Join Query Expansion systematically adds valid joins to raise structural complexity, and Textual Query Augmentation applies controlled linguistic perturbations to test robustness. Applied to state-of-the-art systems, JQE exposed accuracy degradation that grows with join count, and TQA found that heavy abbreviation in the input question can cut accuracy by up to 17%. The paper also replaces binary execution accuracy with Execution Precision and Execution Recall, an F1-style pair that separates over-prediction from under-prediction instead of collapsing both into a single pass/fail bit. Read together with the DevRev paper, the message for anyone building or buying an NL2SQL layer is the same: test it against your actual schema’s join depth and your actual users’ phrasing, not a benchmark’s.

On the product side, Alteryx announced governed enterprise analytics access for AI agents this week, the first mainstream BI vendor move I’ve seen this quarter to frame “governed” as the selling point for agent-facing analytics rather than raw query access. Worth watching whether the governance claims survive contact with nested real-world schemas the way the research above suggests they need to.

A2A gets its first real security audit, and it doesn’t look good

The Agent2Agent protocol, now under Linux Foundation governance, is becoming the default way autonomous agents discover, authenticate with, and delegate tasks to each other across organizational boundaries, the horizontal layer that complements MCP’s tool integration. A2ABreak (arXiv:2609.10871, Sep 9) is the first systematic security analysis of that protocol, and it found 11 new vulnerabilities that require no implementation bug at all, just spec-compliant behavior from an adversarial agent. The method is itself notable: rather than manual spec review, the authors used an LLM to extract a verified finite-state machine from the natural-language specification (37 states, 76 transitions, drawn from 929 formalized statements), then reasoned systematically over that formal model to find protocol-level holes. Findings include cross-client context injection through unprotected context identifiers, credential harvesting via multi-hop identity loss across delegation chains, and data exfiltration through rogue agents that advertise capability claims nobody attested to. Against independent expert review, A2ABreak hit 73.3% precision and 84.6% F1; a zero-shot LLM baseline given the same specification found zero confirmed vulnerabilities. Formal grounding, not a model reading the spec and vibing, was what actually surfaced the holes, the same lesson as the vulnerability-exploitability piece above applied to protocol design instead of production infrastructure.

Retrieval that can’t lie, and retrieval that can’t leak

Two papers attack opposite failure modes in the same retrieval pipeline. Guaranteeing Faithful Evidence Extraction in Speculative RAG (arXiv:2609.10046, Sep 9-14) tackles hallucinated quotes: hybrid and semi-extractive RAG systems don’t actually guarantee that a quoted span came verbatim from the retrieved document, which is a serious problem in domains like aircraft maintenance where an answer has to exactly match certified documentation. The authors measured existing hybrid methods dropping below 40% exact extraction accuracy in technical domains. Their fix, Constrained Hybrid Decoding, repurposes speculative decoding’s architecture, normally optimized purely for inference speed, to enforce hard constraints that restrict generation to continuous spans that actually exist in the retrieved context, achieving near-perfect extraction faithfulness at a measured cost to fluency-oriented metrics.

Shadow Queries for Private Retrieval in Vector Databases (arXiv:2609.04767, Sep 7) goes after the opposite end of the pipe: embedding inversion attacks that reconstruct source text from stored document embeddings in cloud vector databases. Instead of storing the document’s own embedding, SHAQ generates diverse shadow queries that capture different semantic facets of the document and stores those in its place, decoupling the stored vector from the source text it’s protecting. Across IR benchmarks it drove reconstructable-token recovery down to 0.2104, beat baseline defenses by up to 19.5 percentage points on tokens defended, and still hit 0.7967 MAP@10 with up to a 5.5% utility gain over prior defenses. It’s a genuinely different move than the usual noise-injection or embedding-scaling defenses, which tend to trade privacy for retrieval quality more or less linearly; decoupling the stored representation from the protected text sidesteps that trade-off.

An InfoQ talk from Cassie Shum frames why this matters architecturally: knowledge graphs, she argues, are the foundation agentic systems need once you move past basic RAG, with four concrete patterns (context bundling, decision provenance, code as truth, agent visibility) that let a system show its work instead of just returning an answer.

What to watch

Amazon published a production Bedrock AgentCore case study walking a retailer from zero-shot demand forecasting straight through to an automated purchase order, explicitly framed against the operational cost of maintaining thousands of per-SKU classical models. And an InfoQ talk from Columbia’s Zhou Yu makes the case that agents stall at the demo stage because teams lack simulation-driven testing, proposing synthetic user personas and trajectory entropy as the missing CI/CD layer for multi-turn agents. Between those two and the grounding theme running through this issue, the open question for next week is whether “ground it before you trust it” starts showing up as a named architectural layer in production agent stacks, or stays scattered across five different papers solving five instances of the same problem.

---

This issue was compiled from a code-intel-copilot mirror running 18,690 minutes (about 13 days) behind live ingest. Source items are individually dated and verified against their original publication; the delay affects only how recently the mirror itself was refreshed, not the freshness of the items covered.