Correct agent patches still break 43% of repo policies, and an audit puts BIRD's label error near 37%

Coding agents that produce functionally correct patches still violate 43.1% of the project policies that apply to them. The number comes from SWE-CC, posted to arXiv today, which converted the contributor documentation of 12 open-source repositories into 823 machine-checkable policies covering style, git usage, and testing workflow, then ran 500 contribution tasks extended from SWE-bench Verified across four LLMs and two agent scaffolds. Each policy gets a deterministic checker, and the audit inspects the agent’s runtime behavior as well as the final deliverable. Nearly half of the violations happen during intermediate execution steps, where a benchmark that only reads the final diff sees nothing. A merge gate that runs the test suite and stops there is grading the artifact and skipping the process a maintainer would reject the PR for.

The obvious response is to write the rules down where the agent reads them, and a study of Agent Skill revisions measures what that buys. Working from 2,608 first/last revision pairs across 3,159 SKILL.md files, the authors find that 55% of revisions change a rule or procedure. An added rule raises the rate at which four sandboxed agents take the required action by 0.23 on average, and raises blind-judged final correctness by 0.10 for the three agents that were judged. The gain comes mainly from rules that name a command or path the old Skill never mentioned, which argues for writing “run this exact command” over describing intent. Lazy loading eats into it. When the Skill body loads only if the agent decides it needs it, the agents keep about 51% of the action gain, and loading the body raises episode tokens by 50% on average.

Rules also live in the harness, and the harness is thinly tested. Complex Agents, Shallow Tests finds that LLM-dependent harness code, the code that processes model output and governs what the agent does next, has less than half of its lines and branches covered by the projects’ existing tests. The authors’ generator, HarnessTester, generates tests with explicit support for the agent-harness contract and found 122 bugs in widely used agentic systems including OpenClaw, 88 of them previously unknown and 69 confirmed by developers.

A traced multi-agent proof-search system shows what one bug of that kind costs. Across three runs totaling 186 hours and $5,694, the three-model verifier required unanimity and counted a parse failure or API error as a non-approval. In 10 of 12 verification events one member returned nothing parseable, so acceptance was arithmetically impossible and no error surfaced. When the ensemble did function, one verifier approved three attempts that GPT rejected, each claiming to resolve the open problem, so a single-verifier design would have announced a solution three times. And because every review ended as a refutation, the lemma extractor, which mines reviews as well as proofs, pulled 24 of 93 lemmas (26%) from rejected arguments with the refutational context removed. The authors’ prescription is to distinguish abstention from rejection and to keep provenance and polarity attached to anything that becomes future context.

Retrieval tools have their own abstention problem. Search Engines Never Say No removes gold passages from a 21M-passage BM25 index and has the tool return a one-sentence refusal in place of irrelevant top-k results. For Qwen3-8B and 32B, an unannounced refusal moves abstention on unanswerable questions from 23% to 97% on average and beats a system-prompt instruction by 51 points; Claude Haiku 4.5 goes from 28% to 57%. Claude Sonnet 5.5 and Opus 5.5 mostly answer from memory (abstention up 2 points) and respond to a system-prompt directive instead (up 16), while Search-R1 ignores the refusal and fabricates retrievals. Those figures assume an oracle that knows when the index has a hole. Realistic miss detectors, from a score-based predictor to an LLM grounding judge, fall well short of it, so for agents that comply the limiting component is the detector inside the tool.

Benchmarks carry the same weakness in their answer keys. SALUS audits NL-to-SQL benchmarks by treating SQL from multiple LLM agents as weak labels and learning per-agent trustworthiness from features of the gold query. It scores F1 0.9194 on 298 manually verified BIRD dev tasks and estimates annotation error rates of roughly 37% on the BIRD dev set and 27% on Spider’s. A schema-linking or semantic-grounding result reported as a few points of execution accuracy on BIRD dev sits well inside that error band, and some share of the “wrong join” and “wrong column” counts in those papers may be models disagreeing with a wrong gold query.

GitHub’s ReviewBench, published yesterday, spends most of its write-up on label validation. The corpus is 219 PRs from 187 public repositories in 19 languages, sampled to match distributions from 103.9 million GitHub PRs, with a golden set assembled from human reviewers, author follow-up commits, static analysis, and several frontier models. Senior engineers who had not built the dataset re-labeled every finding and agreed with it 96.6% of the time. GitHub also reports that the benchmark called the direction of a production A/B test on a multi-model ensemble reviewer, including a predicted 227% increase in critical comments against 262% observed. It is a vendor grading its own benchmark, but the benchmark is open and the rubric and judge are published.

Two things to watch from here: whether SWE-CC-style process checkers move out of scoring and into harnesses as runtime gates, and whether anyone reruns recent text-to-SQL leaderboards on a SALUS-cleaned BIRD to report which published gains survive.

Feed note: the mirror’s status check reported a last sync of September 1, but items ingested as late as 08:05 UTC on October 6 were present, so this issue draws on current data.