Skill Lift: Measuring Whether an Agent Skill Actually Helps

Across 947 paired trials drawn from 58 production skills and four harnesses, the mean composite gain from attaching a skill to an agent was 0.2134, with a 95% paired-case confidence interval of [0.1967, 0.2301]. Positive in 72.8% of cases. That leaves better than one case in four where the skill did nothing or made things worse, which is roughly the number anyone maintaining a skills directory has suspected and could not previously cite.

The number comes from ACES, a repository-native evaluation framework out of a group including Rama Akkiraju and Jean-Francois Puget, released today. The design is blunt in a useful way: run the same task twice under a fixed model, sandbox, workspace, and scorer, once with the target skill loaded and once without, normalize both trajectories into a common interchange format, and report the delta as Skill Lift. The comparison it draws against existing practice is what should change a repo review meeting. Static gates that scan a skill for structure, style, and security correlate with LLM-judge quality at Spearman ρ = 0.14. Those two things are measuring nearly unrelated properties, so a clean lint on a SKILL.md tells you approximately nothing about whether the skill helps an agent finish work. The largest observed gains landed in skill execution, behavior check, and skill efficiency, which are exactly the discovery-and-routing behaviors a document scan cannot see. An implementation ships as NVIDIA SkillEvaluator.

If skills are the artifact under test, the harness is the thing that decides whether the test means anything, and a second paper today argues enterprises should treat it as infrastructure rather than as a developer tool. George Juraj Salapa assembles a case from prior work: harnesses suffice at the task level and beat more elaborate agent architectures on enterprise work, harness choice explains more variance in agent benchmark results than model choice does, and the barrier to adoption inside large organizations is governance rather than capability. The proposed architecture keeps one unmodified harness as the backbone so the code is byte-identical across every deployment, which reduces reviewing a new automation to reading its instructions file. Backends get a generic request tool plus a scoped credential instead of a hand-written method per integration; authorization sits outside the harness so the same artifact runs as a cron backbone, a chat surface, and a terminal tool; registration falls out of pushing code. Whether that survives contact with a real compliance function is untested, but the diagnosis is right that the review burden, not the build burden, is what caps how many of these things an organization can run.

The portability assumption underneath spec-driven development took a harder hit. A team at Lviv Polytechnic ran Oracle-to-PostgreSQL migration as a controlled transformation task, first regenerating 623 of 1,006 PL/SQL files through a specification-first pipeline (380 of the generated scripts executed cleanly against PostgreSQL 16), then handing specifications written by one agent to a different agent across a 1,802-script dataset covering Amazon Kiro, Google Gemini, and GitHub Copilot, with Claude Code and Cursor in the single-agent stage. Feeding Gemini a Kiro-origin specification produced Token F1 of 0.035, SQL syntax validity of 2.33%, and AST mean similarity of 0.015. That is not degradation, it is collapse, and it replicated. Rewriting the specification recovered much of the loss; compression did not help universally; retrieval-augmented ingestion was the only strategy that appeared on the per-agent Pareto frontier for both Gemini and Copilot. The practical read: a spec is not a neutral artifact you can hand between agents the way you hand a design doc between engineers, and any multi-agent SDD pipeline needs a rewrite or retrieval step at each handoff boundary.

On the retrieval side, Clarify-Then-Search puts a number on something deep-search systems mostly hand-wave. Built from real Baidu query data, it pairs 518 intent queries with underspecified versions, archives evidence once per intent query via WebDancer to build a static weighted-nugget gold reference, then lets a clarifier ask k ∈ {1,2,3} questions of a closed-book user simulator that answers only from what the intent query states and otherwise returns unknown. A closed-book rewriter reformulates, WebDancer runs, and scoring is weighted nugget recall with partial credit. Every model tested beat the no-interaction baseline at a single question, and larger budgets generally helped further: GPT-5.2 led at k=1, ERNIE-4.5-Turbo-128K took the top spot at k=3. The failure diagnostic is the takeaway for anyone building a clarifying agent. Systems over-ask region and location questions that the user’s actual intent never contained, burning the clarification budget on turns that return unknown.

Two practitioner notes round out the day. Simon Willison reframes the verification problem as confidently instructing an agent and then confidently verifying the result, with line-by-line reading as one option among several rather than the standard: “Eyeballing every line of code has never been the most effective” path to confidence. And Cloudflare shipped Kitesurf, a browser engine built for automated workloads that runs its components in isolated WebAssembly/Rust environments on Workers while speaking the Chrome DevTools Protocol. Browser automation has been the least reliable limb of most agent stacks; a purpose-built engine with a familiar protocol surface is a more interesting bet than another headless Chrome wrapper.

What to watch: whether Skill Lift, or something like it, gets adopted as a gate in the skill registries that are now shipping in every major agent product. The ρ = 0.14 result says the gates currently guarding those registries are measuring the wrong thing.