Agent skills encode preconditions, and agents violate them up to 70% of the time

Xuan Chen and colleagues scanned more than 5,000 public agent skills and found that 70% of them contain at least one logical relation: a precondition that gates an action, a constraint on how an allowed action may be performed, or a fallback specifying what to do after something fails. Then they checked whether agents honor those relations. Running Codex and Claude Code across six LLM backbones on an 86-case executable benchmark distilled from the high-confidence, locally testable relations, SLBench measured unsafe rates as high as 70%, and the violations were not cosmetic: privacy leaks, unsafe configuration changes, cleanup steps that silently never ran. The human audit splits blame between agent capability gaps and low-salience skill text, which is a polite way of saying that half the problem is that nobody writes SKILL.md as if it were a specification. SLGuard, the authors’ inference-time scaffold, cuts violations by 63% on targeted cases, which says the failure is shallow rather than fundamental.

That result landed in the same day as two papers arguing, from opposite directions, that skills have become an artifact class with real supply-chain properties. Jialun Cao and co-authors ran what they call the first large-scale empirical study of SE skills across public repositories and marketplaces, characterizing them by the software-engineering activity each one encapsulates rather than by the prompt text. Their coverage map across the development lifecycle is lopsided in a predictable way: the activities that package cleanly into a skill are the ones with low context requirements, and the high-context activities that would actually benefit most from encapsulation are the ones nobody has found a mechanism for. Reuse in software has always moved through this ladder, from source to libraries to components to services, and each rung took a decade to grow the registries, versioning, and provenance that made it safe. Skills are on rung five with roughly none of that infrastructure and, per SLBench, a 70% violation rate on the rules they do encode.

Yubo Li’s lifecycle survey is the structural counterpart, built from a 124-paper audit set spanning 2023 to 2026. The framing worth stealing: a skill library is not a prompt collection, it is a lifecycle-managed, verified, evolving artifact store, and the eight stages it names (evidence acquisition, proposal, verification and admission, storage, retrieval and composition, maintenance, distillation, governance) map almost exactly onto what a package registry does. The evidence-graded findings are the useful part. Admission and repair matter repeatedly across systems. Verifier quality materially determines whether skill-aware RL works at all. Flat retrieval degrades as the library grows, which is the finding most likely to bite anyone who has been accumulating skills into a single directory and assuming the agent will pick correctly at 200 entries. The survey’s closing complaint is that current benchmarks under-report library trajectories, usage-utility gaps, and safety surfaces, which is precisely the hole SLBench just started filling.

The benchmark work in this window is converging on a related idea: the pass bit is not the measurement. BackendForge makes the point with unusual clarity. It gives an agent a visible specification and an OpenAPI contract, requires a Dockerized service as output, and evaluates only through black-box HTTP calls against the deployed thing. To keep the oracle honest without smuggling in hidden requirements, a test agent and a code agent co-evolve the oracle and the reference implementation. GPT-5.5 passes 55.4% of the 56 tasks under the base oracle and 28.6% under the final one. The same model, the same code, half the score, because the tests got serious. Anyone quoting a resolve rate without describing oracle strength is quoting a number about the tests.

Which makes the timing of Beyond Test Presence almost too neat. Preet Jhanglani and co-authors ran AST-level static analysis over 204,673 test artifacts from the AIDev dataset, 24,941 human-authored against 179,732 agent-generated, scoring assertion strength, edge-case coverage, and flakiness potential. The result inverts the usual assumption. Agents beat humans on boundary testing, with roughly double the variety of boundary checks (0.62 against 0.32) and more null-safety testing (13.40% against 8.3%). Humans hold a thin lead on assertion strength, 88.1% strong assertions against 85.37%. The cost shows up in stability: agent-generated tests carry a flakiness candidate rate of 0.41 against 0.30, driven by file I/O and non-deterministic logic. The agents are thorough and environmentally unaware. They test the edges of your function and then reach for the filesystem to do it.

Underneath all of this sits retrieval, and CORE-Bench finally measures it in the setting that matters. Fuwei Zhang and colleagues separate three levels (code understanding, issue-to-edit localization, and broader context retrieval) across 180K queries and 106K broader-context relevance labels built from curated search tasks and SWE-bench-series instances. Embedding models that look fine on docstring-to-function matching drop sharply when the task becomes navigating a concrete repository state and filtering in-repository distractors, which is the only retrieval problem an agent ever actually has. Plain supervised fine-tuning recovers a lot of it, so the gap is a training-target mismatch rather than a ceiling.

Three benchmarks in one window all measuring the thing behind the pass rate, and two surveys saying the skill ecosystem needs admission control it does not have. The question worth watching is which arrives first: a skill registry that verifies before it admits, or the first widely-reported incident traced to a skill whose precondition an agent read and ignored.