The judge grading your agent run has a leniency bias

Every vision-language model that OSReward tested as a judge of computer-use agent trajectories shares the same defect: it marks failed runs as successes. The benchmark, posted to arXiv yesterday, collects trajectories from a spread of agent backbones running human-verified instructions across platforms, then labels each one through multi-stage human annotation to get a ground-truth verdict. Against that reference, the frontier models are not ideal judges and the direction of their error is consistent. Leniency, not noise.

That matters more than a normal benchmark result because trajectory judgment is load-bearing in three places at once: evaluation, training-data curation, and reinforcement learning. A lenient judge inflates your success rate and seeds your RL corpus with plausible-looking failures, and the number still moves in the direction you were hoping for, which is what makes it worse than an obviously broken judge. The paper’s second finding is the operational one. The judges reliable enough to trust cost too much to run at scale, and the affordable open models trail badly. To close that, the authors released OS-Shepherd-100K, a corpus of reasoning-annotated trajectory judgments, and trained OS-Shepherd at 9B and 35B, which they report matching commercial judges at 30-60% lower cost.

The cheapest way out of judge calibration is to not need a judge. A team working on COBOL-to-Java migration published a method that does exactly that: stand up both runtimes, the COBOL source and the generated Java target, each instrumented with mocks and run off-mainframe on commodity hardware, then compare them under deterministic parity checks. The agentic part is the search. Their “Locksmith Loop” iterates witness search over input mocks to drive execution into unvisited branches, applies parity-preserving mutations, and when it hits a routing boundary an analyzer names the Locked Paragraph, the specific condition blocking deeper exploration. Across three case studies from 430 to 4,114 source lines, coverage went past the plateau that plain input search reaches: near-complete on the two open-source programs, 91.90% branch coverage on the internal production-like one. In every accepted test case the generated Java matched the COBOL reference. Legacy migration is an unusually friendly setting for this, since the old system is the oracle, but the shape generalizes to anything with a reference implementation.

Two papers in the last day went after the other half of the problem, which is what the agent reads before it acts. MRCoder starts from a complaint anyone who has instrumented a repo-level RAG pipeline will recognize: retrieval pulls in redundant context that interferes with the model’s use of the relevant parts, and the usual fixes trade quality for compute or the reverse. Their approach is Map-Reduce. In the map phase a lightweight draft model generates drafts over partitioned contexts, and a structure-aware selector keeps context based on API consistency and logical similarity to those drafts; the reduce phase aggregates what survived, with parallel verification speeding up decoding. On CoderEval and DevEval with Qwen2.5-Coder and DeepSeek-Coder backbones, accuracy improved over strong baselines while token consumption dropped 30-50% and inference time dropped by up to 52%. Selection driven by a cheap draft of the answer beats selection driven by similarity to the question.

EMBL AI Librarian makes the same argument at the interface layer. Europe PMC has over 40M indexed records and an interface built for humans: keyword syntax in, whole papers out. Every agent that wants a fact from it has to learn the syntax, fire several searches, and read full texts to find the evidence. Librarian replaces that with natural language in, evidence out, with a single LLM planning complementary subqueries against the live Europe PMC search engine, then reading the selected papers and locating the passage that answers the question. On ScholarQABench it improves Citation F1 by more than 16 points over recent baselines, and on the open-form LitQA2 benchmark a GPT-5.4 agent scores about 8 points higher grounded in Librarian than with web search. A purpose-built retrieval layer beating a general one is not news; a gap of sixteen points in citation F1 is wide enough to argue that every large corpus needs an agent-facing front door rather than expecting agents to drive the human interface.

The practitioner side of the day produced the sharpest artifact. At YC Startup School on Monday, Boris Cherny told Claude Code users to delete their CLAUDE.md, skills, and hooks every six months and watch what the model does, and to push harder with Opus 5 by deleting all of it. Supporting evidence: Claude Code shipped Opus 5 with more than 80% of its system prompt removed and no regression, because most of the prompt was correcting behaviors the model now gets right. One user sorted all 79 rules in a global CLAUDE.md, roughly 9.5k tokens loaded into every session, against a single test: would a strictly smarter model work this out from the repo on its own? Seventeen were model patches, the category the advice targets. Thirty-seven were environment facts no model upgrade can discover, including a zsh shell where ${PIPESTATUS[0]} reads empty so a build that exited 0 gets reported as failed, and a repo under iCloud sync where a cold test run took four minutes instead of one second. Twenty-five were preferences with no derivable right answer. So 22% of the file was the target, and the delete-everything version of the experiment would have thrown out the machine’s documentation along with it.

The methodological point in that post is better than the taxonomy. “Delete it and see what happens” is a test that mostly cannot fail, because a rule that only fires on a deploy, a migration, or a release will look dead during a week with none of those. The fix costs one line per rule: write down what each rule was for before deleting, so “seems fine” becomes checkable against whether you exercised that path at all. A related Ask HN thread posted the same day asks the adjacent question from scratch, whether skills are architecturally different from an AGENTS.md pointing at folders of markdown, and it is telling that the answers are not obvious to people who use these tools daily.

Watch whether the next round of agent benchmarks reports judge calibration alongside pass rates. OSReward gives everyone a way to check, and a benchmark whose verdicts come from an uncalibrated VLM is reporting two numbers fused into one.