Daily digest

Hidden facts sink enterprise agents, and the harness sets the coding-agent bill

Sep 25, 2026 · 🎧 12 min

evalssemantic governanceagentic codingmulti agent orchestrationinformation retrieval

Twelve agents answered the hardest hidden-knowledge questions in Era by Eon correctly once in 84 attempts, and two text-to-SQL papers show set-based accuracy and unrestricted access both overstate correctness. On cost, a harness router recovers 14 to 21% of coding-agent spend and a no-rewrite compactor cuts long-session cost by up to half.

Highlights

  • Era by Eon hidden-knowledge questions: best agent 18 of 24; the hardest record-disambiguation questions solved once in 84 attempts.
  • Set-based execution accuracy hides 5.8 to 6.8 points of multiplicity errors on BIRD-Dev.
  • Text-to-SQL systems with high unrestricted scores degrade sharply under role-based access control (SIGMOD 2027).
  • Cache-aware harness routing recovers 14 to 21% of spend in an emulated 10,000-seat enterprise, $3.3M to $5.0M a year.
  • CliffCompaction never rewrites or re-compacts content and cuts long-session cost by up to 50%.

Across 84 attempts by twelve agents, two questions in the new Era by Eon hidden-knowledge suite were answered correctly exactly once. Both asked the agent to pick one record out of several near-identical ones, such as which of three renewal offers a customer actually signed. That number is the sharpest thing in the feed today, and it sets the tone for the rest: the gap between an agent that runs code and an agent that knows what the business means is still wide, and most of the useful papers from the last day are about measuring it or paying less for it.

The Era by Eon hidden-knowledge paper starts from an awkward result on its own earlier benchmark. When questions state the rules for their answers and agents can execute code, the four strongest models score 22 to 25 of 27, so the benchmark stops separating them. The authors add eight templates where the answer depends on a fact no question or document states. The sales system says a customer dropped a purchase because of timing; a recorded call has the customer blaming an outage. Code computes an exact answer for each generated company without a model in the loop. The best agent gets 18 of 24 attempts right, and four of six models manage at most 6 of 24 regardless of which agent program wraps them. For anyone building a semantic layer, this is the case it has to handle: the governed definition says one thing, and the evidence of what happened lives in a different system.

Two text-to-SQL papers make the same argument from the execution side. ModularSQL points out that set-based execution accuracy collapses duplicate rows, so a query with a missing DISTINCT, an inflated aggregate, or a join fan-out can still be scored correct. On BIRD-Dev (1,532 executable questions) the gap between set and multiset accuracy runs 5.81 to 6.79 points across three backbones under DeepEye-SQL, and 5.22 points on released DAIL-SQL with GPT-4 predictions. Their runtime guardrail probes executed results for multiplicity anomalies and patches only flagged queries, lifting multiset accuracy from 65.86% to 67.75% for $0.0076 of total LLM cost. The gain is small; the finding about how the metric hides wrong-grain answers is what matters for anyone reporting “execution accuracy” to a business owner. A revised version of Benchmarking Text-to-SQL under Role-Based Access Control, now accepted at SIGMOD 2027, adds plausible user roles and access policies to existing benchmarks and separates SQL utility from policy compliance. Systems with high unrestricted scores, open-weight models especially, degrade sharply once RBAC applies, mostly through violations. Both papers argue the leaderboard number overstates correctness once real grain and real permissions are in play.

On the coding side, Control the Harness, Control the Cost treats the harness (Claude Code, Codex and the rest) as the thing that actually sets an enterprise’s bill, because it picks the model, the context, the cache behavior and the subagents. The authors build a router with a calibrated classifier over a bring-your-own taxonomy of agentic requests, and they only switch models where no running conversation has to rebuild its prompt cache: session start, side lanes, subagent launch. Repricing about 10,000 real sessions turns up a crossover where the most expensive model costs less than the next tier on long tool-heavy sessions. In an emulated 10,000-seat enterprise the router recovers 14 to 21% of spend at Anthropic list prices from 21 September 2026, or $3.3M to $5.0M a year. The paper also maps risks across twenty harnesses and proposes a control plane an enterprise can run before deciding whether to own the harness outright.

CliffCompaction, with Tim Dettmers among its authors, attacks cost from inside the session. Its compactor only truncates or drops content and never rewrites it, and it never compacts a previous compaction, so drift cannot compound. That discipline cuts cost by up to 50% under a bounded context while holding or improving Terminal-Bench scores, and on KernelBench it reaches 3.58x CUDA speedups after 400 steps. Under parallel test-time scaling, Kimi K2.6 with the method matches Opus 4.7 at lower cost. It ships as a scaffold-agnostic API proxy for Claude Code and Codex, which makes it cheap to try.

For orchestration, the sparse multi-agent debate study finds that letting each agent debate two distinct, freshly sampled peers each round beats more elaborate learned topologies on the accuracy-cost trade-off, and that a lightweight stopping rule cuts inference cost further. The authors’ request is modest and correct: any learned topology controller should be benchmarked against random routing plus early stopping before its complexity is accepted.

In retrieval, Seek replaces single-pass ranking with test-time exploration: an LLM writes pseudo-passages from accumulated relevance feedback, a retriever pulls fresh candidates, and an assessor grades them for the next round. Training-free, it reaches 37.4 average nDCG@10 on BRIGHT with GPT-4.1, 37% above the strongest baseline, and an 82% relative gain over BM25 with Qwen2.5-7B.

The question to watch is whether anyone connects the two halves of today’s feed. Era by Eon and the RBAC benchmark show agents failing on meaning and permissions; the harness and compaction papers show the cost levers sit outside the model. A router that knows a request touches governed metrics, and sends it to a model and context tuned for that, would be the first system to price semantic correctness directly.


Feed note: the item mirror reported its last sync as 2026-09-01 (about 574 hours stale), though it returned items dated through 2026-09-25. Some late-window items may be missing.

In this issue

← All digests