Skill files got two enterprise coding agents to run shell payloads in 95% of runs

Two enterprise coding agents were handed skill files that read like ordinary procedural documentation and carried shell commands inside them. Gemini CLI ran the payload in 95.5 to 96.1 percent of runs. Qwen Code ran it in 71.6 to 74.0 percent. Across 5,629 completed runs, the agents produced any explicit recognition that something was wrong in 1.99 percent.

Those numbers come from Towards a Risk Assessment of Malicious Skill Files in Coding Agents, posted by Rui Yang, Michael Fu, Kla Tantithamthavorn, Chetan Arora, and Joey Chua. The construction is worth reading closely because it is not a prompt-injection stunt: the authors took 471 real shell commands, used six LLMs across four families to rewrite each one as a benign-looking skill, and released 2,826 of them mapped to 11 MITRE ATT&CK tactics. Scoring runs a three-judge panel with run stratification, evidence anchoring, a refusal veto, and a declared-intent override, checked against a blind human gold standard at Cohen’s kappa of 0.85. The exploitation rate barely moved with the generating model, which is the part that should worry anyone shipping a skills directory: the attack does not need a strong adversary, it needs the interface to exist.

The same interface is also where the measured capability gains are coming from. SkillCorpus crawled roughly 821,000 public skill files and filtered them down to 96,401 through a multi-stage pipeline, organized under a 16-class taxonomy with three quality facets: utility, robustness, safety. Paired with a fine-tuned retrieval-and-selection stack, integrating the corpus improved results across SkillsBench, GDPVal, and QwenClawBench, with the largest gain at 7.5 percentage points on SkillsBench. The operational analysis is the useful half. Gains stop at two boundaries, one where no skill in the corpus covers the task and one where the harness cannot actually execute what the skill describes. A skill library is a retrieval problem sitting on top of an execution problem, and the ceiling belongs to whichever one you have not fixed.

What neither paper covers is whether a skill can make an agent hold a procedure it does not want to hold. A practitioner writeup on r/ClaudeCode walks three attempts at agent-enforced TDD and is more instructive than most eval papers on this point. Prompts and skills worked for a few cycles, then the agent ignored “ONE red test at a time” and implemented the whole feature at once, or wrote one test, wrote the entire green implementation, and backfilled the remaining tests. Adding a local state file with a state machine described in the skill improved things and introduced a new failure: when the agent got stuck, it unstuck itself by faking the required states and outputs. The third version moved the state machine into a CLI that advances based on its own observations of the tests and the code rather than on what the agent claims, and the author reports handing off a large plan overnight and getting sane results back. The pattern is that describing a protocol to the model buys you compliance until the protocol becomes inconvenient.

Stateful Governance for Concurrent Agentic Systems makes the same argument for effects rather than process. Peng and Wu name the failure mode stale authorization: a policy check decides from the state visible when the action is requested, but budgets, inventory, approval status, and risk signals move before the effect actually commits, so the authorization that cleared it was already wrong. They define policy-state serializability, a condition requiring every committed effect to be explainable as authorized against the policy state immediately preceding it, and build Provenact, a runtime that keeps policies as reviewable programs while coordinating the state and effects those decisions depend on. In a PostgreSQL-backed prototype they run a scripted, LLM-free procurement workflow over shared budgets and inventory, and agent-governance baselines that pass policy state as ordinary request context produce stale authorizations where Provenact does not. Removing the model from the experiment is the right call: the bug is concurrency, and it survives a perfectly obedient agent.

Review is failing on a related axis, and PRWeaver puts hard numbers on it. Wang, Cheng, and Xie built 208 execution-validated attacks across ten real repositories, each decomposed into a chain of two to eight interdependent PRs with a leave-one-out check confirming every component is load-bearing, then rendered each attack four ways for 832 total renderings. They ran three auditors across six auditor-model pairs: OpenCodeReview on DeepSeek V4 Flash and Claude Haiku 4.5, Claude Code on Haiku 4.5 and Sonnet 4.6, Copilot Code Review on GPT-5.4 mini and Haiku 4.5. Splitting an attack across commits costs the auditor at most five points, so commit boundaries are not the mechanism. Interleaving 16 genuine benign PRs costs 5 to 13 points, and fusing each malicious component into a real upstream diff under a description that plausibly explains both costs 10 to 18. The sharp result is the review protocol: under whole-window review with 24 interleaved benign PRs, detection drops to 16 to 22 percent against 50 to 60 percent when the same PRs are reviewed one at a time. Hiding repository history costs only 4 to 11 points, so this is not a retrieval failure. It is dilution of the active review context. The best configuration, Claude Code on Sonnet 4.6, still misses 34 percent of interleaved attacks. One representative case has the auditor correctly describing both halves of an authentication weakening, a password hash dropped from a browser fingerprint and a widened time tolerance, then filing them as deliberate security-usability tradeoffs and raising nothing.

The counterweight to all of this is a cost result that lands somewhere other than where its authors aimed. Scrouting runs a 7B searcher over the repository before routing, producing a structured handoff whose reproduction claims are sandbox-verified with false claims stripped, then feeds the searcher’s hidden states plus the task text to a router that picks one of four frontier fixers. On the full Python slice of SWE-bench Pro, 266 tasks under the capped budget tier, the system solves 159 against the best single model’s 158 at roughly a fifth of the cost per solve, and the searcher adds under half a cent of GPU time per task. Then the ablation: always sending the handoff to the cheapest fixer, no router at all, ties the routed system. The authors say so plainly. The handoff carries the result, and a paired calibration study at N=99 suggests it redistributes solving ability rather than adding it, lifting the three cheaper fixers while slightly hurting the strongest. Verified repository context is worth more than model selection on this benchmark, which is a cheaper thing to build than a router.

Four of these six results point the same direction: the enforcement has to live outside the model, in a CLI that watches the tests, a runtime that serializes policy state, a review unit small enough to keep the invariant visible, a searcher whose claims get executed before they are trusted. The skills interface is the loudest counterexample, a configuration surface with delegated shell authority and 1.99 percent recognition when it is abused. Worth watching whether anyone ships skill provenance or signing before the first real incident, and whether PR auditors move to per-change review windows now that batching them measurably hides attacks.