A production text-to-SQL judge agreed with its human graders at kappa 0.04

The LLM-as-judge at the end of a production text-to-SQL pipeline, a gpt-4o-mini prompt that had been grading query faithfulness in deployment, agreed with two-author gold labels at Cohen’s kappa 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot check. In the enriched set it flagged 77.1% of the queries the humans had marked faithful. Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline traces most of those over-flags to a single mechanism the authors name grade-hallucination, and the repair turns out to be cheaper than the audit. A self-hosted Qwen3.6-27B replacement reaches kappa 0.72, in the same range as Claude Opus 4.7 at 0.71, at roughly 1/300 the per-call cost; the head-to-head is underpowered at n=96, which the authors concede does not matter for the deployment decision. Ensembling is not free either. Pairing the weak judge with a strong one lowers agreement, while three strong judges under unanimity routing reach kappa 0.79 with 89.7% auto-coverage. The out-of-domain number is the one for anyone who benchmarks against BIRD: applied to BIRD-financial, the same audit protocol flags 25.5% of the expert-authored gold SQL as candidate gold errors. Code and a pre-registration are in the synca-audit repo.

CARGO goes after a different judge failure, one specific to agents that operate over live entities such as support cases, assets, and accounts. The closest available reference answer usually applies the correct procedure to a different entity, so a literal reference-based judge marks every differing identifier, date, and status as an error or a hallucination. The authors call this reference-instance divergence and measure it on CARGO-Bench, 246 items with ground truth by construction, two judge models, 7,872 judgments: the standard judge penalizes 100% of correct entity-transplanted answers and has a discrimination index near zero, and supplying the live facts without reframing the rubric changes nothing. CARGO treats retrieved references as procedural exemplars, grounds factual checks in the live instance’s observed context, assigns each claim one of supported, contradicted, or unverifiable, penalizes only contradictions, and gates the whole evaluation on retrieval confidence, which turns production grading into selective prediction. False penalties drop to 0 of 50 while contradiction recall holds at 50 of 50 and 49 of 50, the discrimination index rises to 0.58 with a 0.48 to 0.68 interval, and a rubric-swap control attributes most of the gain to context-grounded dimension definitions. The paper also reports its own blind spot: the leniency that protects entity values suppresses detection of procedural corruptions to 20% recall, a post-hoc fix does not close the gap, and an LLM-as-annotator study with written guidelines and adjudication shows the same hole.

The CI version of the problem arrived in a practitioner thread on r/LLMDevs. An unchanged coding agent went from 92% to 88% task success between runs with no code change, so a fixed threshold would have blocked a clean PR. Then the author deliberately degraded the agent, success fell from 92% to 52%, and the first statistical rule passed it, because the damage was concentrated in a couple of tasks rather than spread across the suite. The design that came out of it separates two questions, whether reliability got broadly worse and whether a capability that used to work reliably collapsed, and treats each PR as an experiment: repeated runs, practical effect thresholds, multiplicity correction across capabilities, and an explicit insufficient-evidence state. The tool is open-sourced as AgentSeism, and the thread is asking how other teams decide a score drop is real enough to block a merge.

Silent Success is the same failure one stage later, at the release gate. A blocking quality gate in a production pipeline reported PASS on a run in which one subgate had executed neither of its two checks and another had run six of its eight. Both keys that decided the run asked whether a violation had been observed, and both computed the answer from a population already stripped of the cases that failed to run, so absent data read as no violation and the builds stayed green for two weeks. The remedy was a third value, pass, violate, and unable to determine, with the shortfall pushed into the exit status; follow-up work on the same specifications then found a detector firing with a margin of 0.000177 percentage points. The paper collects eight more instances of the same form: five from the same engagement, two in open-source projects (a gateway whose configured cache TTL was declared but never applied on the write path, an inference server that credited free cache slots on every step for releases that returned nothing), and one the author committed while writing the paper, using tooling built to prevent exactly that. Seven of the nine are answerable by a single query, command, or comparison, and the author offers that convergence as the falsifiable claim.

Beyond Approved Actions moves the check into the agent runtime. Its opening case is an approved database update that succeeds and also leaves an unapproved notification behind: execution produced a persistent effect beyond the requested change, and current safeguards either approve the action up front or record its aftermath, so the unapproved outcome gets accepted as success and propagates into later steps. EffectMatch collects persistent changes inside a controlled execution boundary, compares them with what the application approved for the current state and execution, and lets that comparison govern both commit and dependent execution. On 206 public business tasks it preserved every clean execution and prevented every tested incorrect commit; six 20-run ablations each exposed the failure their removed mechanism was there to catch, and 80 task-topology cases kept handoffs truthful and blocked invalid continuation.

A negative result on agentic coding that built its eval before trusting it: Compact Documentation for Coding Agents asks whether natural-language documentation helps an agent resolve real repository issues. The authors first construct a roundtrip benchmark that scores a code description by whether code regenerated from it passes the original tests, show that completeness rather than length drives fidelity, and use the benchmark as an optimization signal to find a description-writing prompt that reaches full fidelity and generalizes to unseen files. Then, across two model families and ten repositories, with a positive control confirming the evaluation can detect a genuine improvement, they find that when the source is present neither static compact documentation nor retrieved context beats the issue alone. They characterize the boundary at which documentation does help. For anyone maintaining a per-repository agent doc, the control run is what makes the negative worth acting on.

Joe Reis’s Weekend Windup #47 names the organizational form of all this. The vendor label at Big Data London this year is context, every booth sells a context layer or a context graph, and the pitch is ontology in a box: an LLM reads the Slack threads, emails, and internal docs and generates the business domain model. His test case is the customer definition. Sales means an account with an executed deal, Finance means a distinct entity generating recognized revenue, Support means anyone who opened a ticket, and all three can sit in one warehouse under an agent-built semantic layer with an agent pulling the metrics, and someone still has to decide which definition applies to a given decision and who has the authority to say so. Parsing corporate chatter automates and scales the existing confusion, and the swamp moves from the data lake to the semantic layer. Set that next to the 25.5% candidate gold-SQL errors in BIRD-financial from the first paper and the two stories are one story: the definitions agents are graded against were never audited either.

What to watch: the judge audit ships a pre-registration and CARGO ships a preregistered protocol for expert agreement, risk-coverage, and cost on production traffic, so both teams have committed to numbers that can come back worse. The pattern across the day is where the third state lives. The release gate needed unable to determine, CARGO needed unverifiable, the CI tool needed insufficient evidence, and EffectMatch needed a compare-before-commit boundary. None of those pipelines had one until someone counted what the verdict layer had skipped, and the next question is which of these ships as a default in an agent framework rather than as a paper.

---

Source note: the code-intel mirror reported its last sync as 2026-09-01 (about 38,760 minutes stale), but items dated through 2026-09-28 were present and used here; the arXiv entries are from the 2026-09-28 announcement batch.