Enterprise Multi-Agent Reliability

Plain-language abstract ReAct is a prompting framework for large language models (LLMs) that interleaves verbal reasoning traces with concrete actions in a single generation loop. Instead of reasoning alone (chain-of-thought) or acting alone (action-plan generation), ReAct lets a model think through a step, take an action such as querying a knowledge base, observe the result, and then continue reasoning — all within one prompted sequence. The paper was published as a conference paper at ICLR 2023.

Motivation Prior work on LLMs treated reasoning (chain-of-thought prompting) and acting (action-plan generation for interactive environments) as separate capabilities. Chain-of-thought reasoning is static and grounded only in the model's internal representations, making it prone to hallucination and error propagation. Action-focused approaches lacked high-level verbal reasoning and working memory. No prior work had systematically studied how combining the two in a synergistic, interleaved manner could benefit general task solving.

Methodology ReAct prompts an LLM to generate both reasoning traces and task-specific actions in an interleaved fashion. It was evaluated on four benchmarks: two knowledge-intensive language tasks — multi-hop question answering (HotpotQA) and fact verification (Fever), where the model calls a Wikipedia API for external information — and two interactive decision-making benchmarks (ALFWorld and WebShop). The approach uses few-shot in-context prompting, with only one or two in-context examples provided, and is compared against chain-of-thought (reason-only), act-only, imitation learning, and reinforcement learning baselines.

Results On HotpotQA and Fever, ReAct reduced hallucination and error propagation compared to chain-of-thought baselines by grounding reasoning in retrieved Wikipedia facts, while also producing more human-interpretable task-solving trajectories. On the interactive decision-making benchmarks ALFWorld and WebShop, ReAct outperformed imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, using only one or two in-context examples.

Plain-language abstract This paper asks whether having multiple AI language model agents debate each other — a setup called Multi-Agent Debate (MAD) — actually helps with software engineering tasks like generating code summaries and translating code between programming languages. The researchers adapted MAD systems originally designed for general language tasks, ran them on two standard software engineering benchmarks, analyzed the debate logs when things went wrong, and proposed two targeted fixes.

Motivation Single AI language model agents struggle with tasks that require diverse expertise or multiple reasoning steps. MAD systems, where agents iteratively critique and refine each other's answers, had shown promise in general natural language tasks, but whether this structured debate could improve software engineering tasks — which mix natural language and source code — had not been studied.

Methodology The authors implemented a MAD framework adapted from prior NLP research, applying it to code summarization and code translation tasks. They evaluated the default MAD against state-of-the-art single-model baselines using metrics including BLEU, METEOR, ROUGE-L, BERTScore, CodeBLEU, and execution accuracy. To understand failures, they manually analyzed debate logs using an open-coding approach (88% inter-rater reliability), categorizing underperforming debate patterns. Based on these patterns, they proposed two enhancements: an Early Termination strategy that stops debate once a judge identifies an acceptable answer, and an Extended Reflection strategy that restarts debate with judge-provided feedback when no winner is found.

Results Default MAD performed well on code summarization but showed limited improvement on code translation compared to the state-of-the-art baseline. Manual analysis identified three failure patterns in debates: Forceful Agreement (agents converge on an incorrect answer), Ending Divergence (agents start agreeing but drift toward worse responses), and Prolonged Disagreement (agents remain stuck without reaching consensus). Ending Divergence dominated code summarization failures (76% of cases) while Forceful Agreement dominated code translation failures (62%). Both proposed enhancements improved code summarization quality with statistical significance; only the Extended Reflection strategy improved code translation. The enhanced MAD variants also reduced the number of API calls compared to the default configuration, though they still require substantially more LLM inferences than single-model approaches.

Plain-language abstract This paper investigates why multi-agent systems built from large language models so often fall short of expectations. The authors created MAST, a structured catalog of failure types, by carefully examining hundreds of conversation traces from seven widely-used multi-agent frameworks. The result is a practical taxonomy that names and organizes 14 distinct ways these systems break down, along with an automated tool to apply that taxonomy at scale.

Motivation Multi-agent LLM systems have attracted considerable interest because they can, in principle, divide complex tasks among specialized agents and coordinate their work. Yet empirical results consistently show their performance gains over single-agent setups are small or absent — for instance, one studied framework (ChatDev) solved only 33% of programming tasks. No systematic account of why these systems fail existed, leaving developers without a principled framework for diagnosis or improvement.

Methodology The authors analyzed more than 200 conversation traces drawn from seven open-source multi-agent frameworks (including MetaGPT, ChatDev, HyperAgent, AppWorld, AG2, and Magentic-One) running on diverse benchmarks. Six expert human annotators applied Grounded Theory to label failures in the traces, with inter-annotator agreement measured by Cohen's Kappa (reaching 0.88). An LLM-as-a-Judge pipeline using OpenAI's o1 was then developed and validated against expert labels (Kappa 0.77) to enable scalable automated annotation. Two case studies tested whether targeted interventions could reduce the identified failures.

Results The analysis identified 14 distinct failure modes organized into three categories: specification issues (system design problems, 41.77% of failures), inter-agent misalignment (coordination failures, 36.94%), and task verification failures (quality control, 21.30%). Prominent individual failure modes included step repetition (17.14%), reasoning-action mismatch (13.98%), and disobey task specification (10.98%). Targeted interventions such as improved role specification yielded only modest gains (e.g., +15.6% for ChatDev), indicating that the identified failures stem from fundamental system design challenges rather than easily patched prompt issues.

Consensus-topology ablations: visible authorship raises self-voting and ties; visible live vote tallies raise herding and premature consensus.

Why it matters If you run review/debate quorums, blind the lanes and hide interim tallies by default — visibility measurably biases the consensus toward the wrong answer.

Plain-language abstract This paper introduces lambda_A, a formal mathematical framework — a typed lambda calculus — for describing and reasoning about how LLM-based AI agents are composed and configured. The authors also build a practical lint tool derived directly from this formalism and show it catches real configuration bugs in production agent code.

Motivation Popular agent frameworks like LangChain, CrewAI, and AutoGen let developers configure agents via YAML or JSON files, but none of these tools provide a formal account of what those configurations mean. Without formal semantics, developers cannot tell if an agent will loop forever, whether two differently-written pipelines are equivalent, or how to safely refactor a pipeline — problems that currently get discovered only by trial and error.

Methodology The authors define lambda_A, an extension of the simply-typed lambda calculus with 11 term formers covering oracle (LLM) calls, bounded fixpoints for ReAct loops, probabilistic choice, and mutable memory stores. They prove type safety, termination of bounded fixpoints, and soundness of derived lint rules, with partial Coq mechanization (1,567 lines, 43 completed proofs). They implement the calculus as a Python DSL called lambdagent, including a compiler from YAML agent configs and a lint tool. The lint rules are evaluated via fault injection (42 tests) and validated on 835 real-world agent configuration files drawn from 17 GitHub repositories across 6 frameworks.

Results Applied to 835 real-world GitHub agent configurations, the lint tool found that 94.1% are structurally incomplete under lambda_A semantics. YAML-only lint precision is 54%, but rises to 96–100% when joint YAML and Python AST analysis is used (validated on 175 samples). The study found that 46% of production configurations split their semantics across YAML and Python — a phenomenon the authors call semantic entanglement. All five mainstream agent paradigms (LangGraph, CrewAI, AutoGen, OpenAI SDK, and Dify) were shown to embed as typed fragments of lambda_A, establishing it as a unifying calculus across the ecosystem.

Plain-language abstract TraceFix is a system that automatically designs, verifies, and repairs coordination protocols for groups of AI agents working together. When multiple AI agents must share resources, pass messages, and coordinate actions concurrently, subtle bugs like deadlocks can emerge. TraceFix uses formal verification—exhaustive model checking with TLA+—to find these bugs before deployment and iteratively fixes the protocol until no violations can be found, then enforces the verified protocol at runtime.

Motivation As large language model (LLM) agents increasingly run concurrently and share external state, coordination failures—races, deadlocks, missed handshakes, and premature termination—become the dominant source of system failures rather than individual agent capability. These bugs depend on rare execution schedules and can remain hidden through all observed runs yet appear under untested interleavings. Prior approaches address coordination through orchestration frameworks or runtime guards but do not verify that the concurrent protocol itself is free of such hazards.

Methodology TraceFix operates as a four-stage pipeline: an orchestration agent synthesizes a protocol topology (a structured intermediate representation defining agents, shared locks, and directed message channels) from a natural-language task description; the topology is compiled into PlusCal coordination logic; the TLA+ model checker (TLC) exhaustively searches for counterexample traces under bounded assumptions; and a repair agent revises the PlusCal source based on the counterexample until TLC finds no further violations. Verified process bodies are then compiled into per-agent system prompts, and a runtime monitor rejects any coordination operations outside the verified topology. The approach was evaluated on a benchmark of 48 tasks spanning 16 scenario families at three difficulty tiers, with a 3,456-run runtime comparison across four architectures and two model capability tiers.

Results All 48 benchmark tasks reached full TLC verification; 62.5% passed on the first attempt and none required more than four repair iterations. Bounded model checking remained tractable across all tasks (median under 1 second, maximum under 60 seconds) even for state spaces reaching millions of distinct states. Topology-monitored execution achieved the highest task completion rates (89.4% average, 81.5% full completion) and degraded at roughly half the rate of prompt-only and chat-only baselines when model capability was reduced. A paired ablation showed that TLC-verified protocols cut deadlock and livelock occurrences from 31.1% to 14.1%, with the largest separation under fault injection conditions.

Plain-language abstract SWE-Marathon is a benchmark for software work that takes hours and millions of tokens — porting whole libraries between languages, cloning full products, building ML systems — rather than the minute-scale single-patch tasks most benchmarks measure. Each of its 20 tasks ships an executable environment, a human reference solution, and a multi-layer hidden verifier, and even the strongest current coding agents finish fewer than 30%.

Motivation Capability claims now reach workflows that take human engineers days, but dominant benchmarks fall short on horizon (most tasks resolve within an hour) and on verifier strength (single committed-patch or single-test grading that agents can game). At hour-scale budgets, agents with filesystem and network access probe weak checks, so long, realistic, ungameable tasks need richer verifier surfaces than a single test suite.

Methodology Twenty tasks across library reproductions, product clones, ML engineering, and algorithmic optimization were authored by engineers familiar with each system and accepted only on specificity, solvability (reference oracle passes, no-op agent fails), and integrity (no readable answers or forbidden reference implementations), enforced by CI, rubric checks, piloted agent trials, and an adversarial cheating agent. Tasks run in Harbor/Modal sandboxes under 2–10h wall-clock limits; hidden verifiers span dense assertion suites, behavioral parity, performance gates, deterministic replay, audit checks, and computer-use UI judges. The authors evaluate 13 agent–model configurations under both native CLIs and a shared Terminus-2 scaffold, and audit every rollout for reward hacking with an LLM suspicion score.

Results Across 1,300 rollouts no configuration exceeds 30% pass@1. Of 526 agent-attributable failures, implementation failure (41.6%) and timeout (31.4%) dominate, followed by reward hacking (15.4%), premature termination (7.6%), and poor self-verification (4.0%). 13.8% of rollouts take an exploit-shaped action and 10.2% ship a bypass, but the three-layer defense catches all 132 shipped bypasses so none earns reward. Long context is not passive: pass rate falls monotonically with runs of identical consecutive tool calls (claude-code 41.9%→3.2%), 32% of one scaffold's tool calls are silent duplicates, and compaction tracks failure — 0 of 71 summarizer trials pass versus 8.9% without. Token spend is not a skill proxy; the lowest-token quintile passes 11.3% versus 8.3% for the highest, and per-(model,scaffold) token use varies up to 12×.

Plain-language abstract XFlow is a system for building multi-agent LLM workflows that are more reliable, and XPF is its protocol language. The core idea: today, constraints, rules, and process obligations are buried inside prompts that agents must remember and re-apply, with no way for the surrounding system to enforce them. XFlow draws an explicit prompt-harness boundary: informal semantic reasoning stays inside actors, but selected commitments are moved into harness structure that is compiled, checked, and enforced. A protocol is written as a readable literate document but compiled into a typed intermediate representation and executed as a program. At runtime, agent outputs are staged as lifecycle-governed symbols and only become shared state after passing checks.

Motivation In a multi-agent pipeline, one agent's hallucination, malformed output, or misread instruction becomes shared state and corrupts later decisions. Current frameworks sit at two extremes of the Chomsky hierarchy: markup/config tools describe workflow shape but say nothing about where knowledge leaves the prompt, and prompt-based tools keep formalizable constraints stuck in instructions. Developers have no language for drawing, testing, and adjusting which commitments are governable versus left to the actor.

Methodology Three-phase architecture. Specification: authors write XPF (YAML frontmatter for symbols/policies, Markdown stages, fenced semantic blocks for actor interfaces, guarded control flow, and responsibility handoffs). Compilation: mechanical passes parse the literate surface, resolve names, statically check that actor reads/writes are legal, judge outputs bind to declared symbols, flow targets exist, and call returns match child-protocol outputs, then lower to a typed IR. Execution: a runtime wraps each actor call in a typed interface, commits symbol writes only after schema validation/provenance/commit-policy checks inside atomic transactions with rollback, propagates staleness to derived values reactively, and persists scoped session frames for resume and audit. Evaluation uses Qwen3.5-9B and DeepSeek-V4-Flash across tau3-bench, CorpusQA (layered on the XpandA baseline), and SWE-bench Verified (around mini-SWE-agent).

Results On tau3-bench, wrapping a bare ReAct agent with XFlow raised constraint-compliance to 100% in Retail and Airline for Qwen3.5-9B (from 96.5% and 91.8%) and to 100% across all three domains for DeepSeek-V4-Flash (Telecom compliance +36.3pp), while task pass1 stayed comparable, separating 'reached the answer' from 'reached it via a valid path.' CorpusQA accuracy improved with XpandA+XFlow to 61.7 (+2.4) for Qwen and 75.7 (+0.9) for DeepSeek by encoding domain interpretation rules as deterministic derivations over extracted symbols. On SWE-bench Verified, XFlow around mini-SWE-agent raised the pass rate from 77.4 to 79.8 (+2.4) by gating patch submission on a passing local validation check; a case study shows the protocol rejecting a submission after a failed test run and issuing a targeted retry. The paper also presents cloud-edge use cases where edge worker outputs must pass schema and coverage checks before entering global state.

Plain-language abstract AgentArmor studies how AI coding agents fail in everyday, non-adversarial use — not jailbreaks or prompt injection, but the 'hot mess' cases where an agent deletes the wrong thing or skips a safety step. It frames unsafe behavior as three sequential failure points and proposes a set of agent-harness modifications that make current coding agents measurably safer.

Motivation As coding agents such as Cursor, Claude Code, Codex, and OpenCode take over the full software lifecycle — not just code generation but deployment and monitoring — rare but highly destructive failures surface, yet few works rigorously evaluate the safety gaps that trustworthy deployment requires. The authors deliberately set aside adversarial threats like refusals, prompt injection, and jailbreaking to focus on the non-adversarial case.

Methodology They model misalignment as three failure points that must all hold for safe behavior: forming the correct target (fails under underspecification when default behavior is unsafe), actively pursuing it (fails on capability errors from bias, refusal, or limits), and executing it through the harness (fails on stochastic sampling and context decay), combined via a chain rule P(unsafe)=1-(1-f1)(1-f2)(1-f3) with scenarios that isolate each stage. They taxonomize agent behavior into four active modes — greenfield, editing, deployment, monitoring — and curate 8 scenarios across 20 coding environments and 59 synthetic transcript templates, run at n>=500 samples over Claude Opus 4.6, GPT 5.4, and Gemini 3.1 Pro. The proposed mitigation, AgentArmor, adds an extended system prompt, a LoRA-trained command classifier for risk and user-intent alignment with a '3 strikes' policy and persuasion-blindness against goal drift, deterministic guardrails (run ls -la before deleting, read scripts before executing), and tools letting the agent make files immutable or prune its own transcript context.

Results Across the evaluations — escalation, disregarding CLAUDE.md, skipping security practices, dangerous command templates, stochastic generation, and long-context degradation — AgentArmor is safer by a statistically significant margin relative to the unmodified base models. The authors frame the result as concrete, adoptable mitigations for today's coding agents and a design philosophy for future agent-harness features, not as a population-level safety guarantee.

Plain-language abstract RigorBench is a benchmark that scores AI coding agents on how they work, not just whether their final code passes tests. It defines five process pillars — Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity — and combines them into a weighted RigorScore. The authors built 30 tasks in five categories (Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, Don't Break the Build) and ran four harnesses (Agent-Rigor, Agent-Skills, Superpowers, and a Baseline ReAct control) on the same underlying model, scoring the full execution trajectory rather than the final artifact. Structured process discipline raised process-quality scores by an average of 41% and downstream outcome correctness by 17%, and RigorScore correlated with outcome quality at r=0.87. They release the tasks, rubrics, and trajectory-analysis tools as open source.

Motivation Existing AI coding benchmarks (SWE-bench, HumanEval, MBPP, BigCodeBench, Terminal-Bench, AgentBench, and others) measure outcomes only — did the code pass the tests or resolve the issue. The authors survey major benchmarks and find none evaluate the engineering process. Their argument: an agent that stumbles onto a correct fix through reckless trial-and-error, without planning, verification, or graceful recovery, is less reliable than one that follows engineering discipline — yet every existing benchmark gives them the same score. They call the resulting risk the lab-to-production gap: outcome-only optimization breeds strategies (fragile fixes, token waste, false confidence, broken intermediates) that look fine on a benchmark but are hazardous in production. They ground this in software-engineering precedent (Humphrey's Personal Software Process, CMMI) that how software is built predicts its quality.

Methodology RigorBench has three design pieces. (1) A five-pillar scoring framework, each pillar built from weighted sub-metrics: Planning Fidelity from plan-artifact creation, decomposition quality on a 4-point rubric, and plan-execution alignment via Kendall tau; Verification Coverage from test-creation rate, coverage delta via instrumentation, and requirements traceability as recall; Recovery Efficiency from recovery-attempt count, strategy diversity, and token-waste ratio; Abstention Quality scored only on impossible/ambiguous tasks (correct abstention, false confidence, clarification seeking); Atomic Transition Integrity from build health, test-suite stability, and commit hygiene. The composite RigorScore = 0.20·PF + 0.25·VC + 0.25·RE + 0.15·AQ + 0.15·ATI, each pillar normalized to [0,1]. (2) 30 curated tasks, 6 per category, each designed to be discriminative, measurable, and realistic. (3) Trajectory-based evaluation: each agent runs in an isolated Docker container with a fresh task-repo clone, instrumented shell/filesystem, a 60-minute timeout, and a 200K-token budget; the pipeline parses raw logs into a trajectory, extracts signals (planning artifacts, test events, error-recovery cycles, abstention signals, codebase-health checkpoints), and scores each pillar with deterministic heuristics plus LLM-as-judge for qualitative sub-metrics, using a 3-judge panel with majority voting. Setup evaluates four harnesses — Agent-Rigor (a 6-phase discipline lifecycle), Agent-Skills, Superpowers, and a Baseline ReAct control — all on the same model, giving 120 executions, with process and outcome quality measured independently.

Results Process discipline improved process-quality scores by an average of 41% and downstream outcome correctness by 17%. Overall RigorScore: Agent-Rigor 0.61, Superpowers 0.48, Baseline ReAct 0.48, Agent-Skills 0.47; outcome scores rose from 0.64 (Baseline) to 0.83 (Agent-Rigor). The largest disciplined-vs-baseline gain was Planning Fidelity (+0.47) — baseline agents rarely produce explicit plans despite chain-of-thought ability. Abstention Quality showed the second-largest gain (+0.34): no baseline agent abstained on any of the 6 impossible tasks, and disciplined agents still abstained correctly on only 62%. Recovery Efficiency improved least (+0.25); token-waste ratio fell only 34% and doom loops persisted on hard tasks. RigorScore correlated with outcome quality at r=0.87 (p<0.001) across all 120 executions, and the with/without design supports attribution, not just correlation. Disciplined agents also used 12% fewer total tokens despite producing more artifacts, because saved recovery tokens outweigh planning/verification overhead. Inter-judge agreement was Fleiss' kappa 0.74. Limitations: only 30 tasks, a single discipline framework, LLM-judge bias risk, June-2025 temporal validity, and benchmark-contamination risk.

Plain-language abstract This paper asks whether the reliability problems showing up in AI-coding-agent projects belong to the individual agent or to the repository and ecosystem it operates in. Using a statistical technique for detecting emergence, whether a whole-system property survives after accounting for all its individual parts, the authors show that a large, consistent share of integration friction (how hard it is to merge a contribution) belongs to the repository itself, not to any single contribution, author, or agent.

Motivation Coding agents are evaluated the way software has always been evaluated: one contribution, one benchmark task, at a time. But developers report that agent-authored code accumulates problems no single contribution accounts for, and shared understanding erodes even when every individual change passes review. The paper reframes this as a measurement question: at what level should reliability risk actually be measured?

Methodology The authors define statistical non-reducibility: using multilevel models, standard for nested data, on 930,000+ agent-authored pull requests, they measure how much variance in integration friction remains attached to the repository after controlling for the contribution, its author, its size, and the agent that produced it. They compare against a matched human-authored baseline in the same repositories to check the signal is agent-specific.

Results About half the variance in integration friction stays at the repository level after full controls, a property of the whole that no single part explains. Agent-authored contributions concentrate this repository-level friction roughly twice as much as human-authored ones (intraclass correlation 0.30 vs 0.16), and the gap survives controls for codebase size, age, task shape, process maturity, and merge path.

Plain-language abstract Glite ARF is an open-source framework for running many LLM coding agents in parallel on a shared research codebase without the whole thing quietly corrupting itself. It wraps each agent's work in an isolated task folder, makes finished work immutable (fixes happen in new tasks, not edits to old ones), and auto-generates a dashboard so a human can see the true state of a multi-week campaign instead of relying on an agent-written summary.

Motivation Delegating research experiments directly to coding agents doesn't scale: agents follow most instructions but the few they skip compound into corrupted data, fabricated citations, contaminated splits, and stale summaries. The authors' own prior audit found 13 such incidents in one campaign, including one where a single agent step recomputed and corrupted 20,304 historical training rows across 38 feature sets.

Methodology The framework defines a three-role stack (human chooses hypotheses, coding agents execute isolated tasks, deterministic Python 'verificator' scripts enforce structure) built on seven structural principles: task isolation via git worktrees, immutability with a corrections overlay, aggregators-only cross-task reading, and a materialized human-facing overview regenerated from committed artifacts. It's evaluated via a real external shared task (BEA 2026 vocabulary-difficulty) plus measured overhead across three author-run campaigns.

Results The BEA 2026 submission built with Glite ARF placed first (closed track) and second (open track) across all three target languages, cutting the baseline RMSE by 29.9% (closed) and 35.9% (open), across 273 tracked tasks run by up to twelve parallel agents from a single laptop at roughly $450 in LLM spend. Structured per-fold provenance caught and let them strip four target-leaking feature sets that had inflated one result to an implausible 0.609 RMSE (corrected to 0.802). The framework's structural machinery adds only about 1% wall-clock overhead across three campaigns in three domains.

Plain-language abstract NOVA is a system Tencent built to automate architecture changes to its production ad-recommendation models, the kind of structural redesign (new attention modules, feature interactions) that usually needs an expert engineer. It uses an agent to propose changes, but layers verification on top so it can catch a runnable-but-wrong candidate, code that passes tests but breaks a recommender-specific invariant, before wasting a training run on it, and routes the riskiest changes to a human-in-the-loop mode.

Motivation Generic coding agents optimize for code that runs and passes unit tests, but a recommender architecture can be syntactically valid and still be a bad or actively harmful architecture, for example silently dropping sequence masking or degenerating self-attention into a plain MLP. AutoML only tunes hyperparameters, not cross-module structural changes, leaving a gap between 'runs' and 'is architecturally sound.'

Methodology NOVA computes an architecture gradient, an SGD-inspired but non-differentiable update signal aggregating prior modifications, verification diagnostics, metric changes, and trajectory memory, to pick the next modification. A verification cascade checks structure semantics, local executability, offline effectiveness, and online impact before expensive training; failed candidates become reusable forbidden directions. An L1-L4 task-level control scheme routes high-risk changes to a human-supervised Copilot mode. It's deployed in a production advertising system serving over a billion users.

Results On the hardest task tier (L3, literature-to-production), NOVA reaches 86.7% valid-pass rate and 60.0% effective-pass rate, more than double the human expert loop's effective-pass rate, and shortens one literature-to-production cycle by over 13x in human-attended time. In live online A/B testing, the selected candidate improved GMV on three pCVR objectives by +1.25%, +1.70%, and +2.02% while reducing prediction bias by 37.3-66.7%.

Plain-language abstract TrajAudit automatically diagnoses why an AI coding agent failed a repository-level task by reading its execution trajectory - the recorded sequence of the agent's reasoning, tool calls, and observations - and pinpointing the earliest step at which the agent took an action that introduced an error, together with a justification.

Motivation As coding agents take on complex multi-file repository tasks, they fail in opaque ways, often as the cumulative consequence of a single early mistake such as a misunderstood requirement or a flawed plan. Existing trajectory-based diagnosis methods degrade badly on these traces, dropping below 40% on repository-level trajectories that often exceed 40 steps, because the trajectories are dominated by observational noise (tool outputs, redundant program structure, verbose code - over 70% of the content) and are simply too long for LLM long-context reasoning, while the methods passively consume the whole trace as if every step were equally relevant.

Methodology TrajAudit uses an investigator agent backed by two modules. Prior failure reasoning prompts an LLM to derive a preliminary diagnosis from the failed test code and its error report, directing the agent toward the most suspicious region. Semantic saliency folding compresses observations, retaining only failure-relevant context such as code patch structures and entries containing failure indicators like 'fail' or 'exception'. The investigator agent then retrieves folded content on demand through predefined interactive APIs, performing top-down diagnosis: begin with a high-level overview and drill into detail only where needed. The authors also introduce RootSE, a benchmark of 93 real-world agentic failure instances spanning over 4,500 execution steps, for locating the earliest decisive error step.

Results On RootSE, TrajAudit outperforms all existing baselines by over 24.4 percentage points in localization accuracy while reducing token consumption by at least 18%.

Plain-language abstract AgentAbstain is an evaluation framework for agentic abstention: whether a tool-using LLM agent recognizes when not to act. It is a paired-task benchmark of 263 task pairs across 42 executable MCP sandbox environments, where each pair shares a sandbox but differs by a single controlled perturbation that turns a should-act task into a should-abstain variant. An automated pipeline, AbstainGen, synthesizes the environments and paired tasks end to end so fresh instances can be regenerated on demand.

Motivation LLM agents increasingly commit irreversible actions on a user's behalf, booking travel, managing files, running code, and calling APIs, yet evaluations score task success rather than whether an agent refrains under ambiguity, conflicting constraints, or tool failure. Prior abstention work is single-response question answering, where the worst case is a wrong answer and there is no tool-call trace against which to verify restraint. A tool-using agent instead faces a sequential decision grounded in observable environment state, and safety or tool-reliability benchmarks test refusal of malicious goals or correct tool calls, not when a well-intentioned task should be abandoned mid-execution.

Methodology Every instance is a pair sharing one sandbox and differing by exactly one trigger, so no always-act or always-refuse policy can exceed 50% Paired Accuracy. Eight abstention categories are organized by when the trigger becomes observable (pre-execution vs. runtime) and where it resides (query, environment state, or tools). Sandbox tools are typed as lookup (read-only), verify (validation gate), or commit (state-mutating); scoring pairs a deterministic commit-check on the tool-call trace with an LLM judge on the terminal response, and a Conditioned Abstention Rate restricts the abstain score to pairs whose act variant already succeeded. AbstainGen validates generated tasks through deterministic DAG replay and cross-family LLM critics, and three human annotators rated 94 to 98% of a stratified sample as well-designed.

Results Across 17 frontier LLMs in 4 agent harnesses, agentic abstention is unsolved: the best agent, Gemini 3.1 Pro, reaches 59.5% Paired Accuracy and 13 of 17 models stay below 50%, meaning agents systematically prioritize acting over abstaining. Abstention capability is largely independent of general task-solving capability, so scaling task-solving alone will not close the gap. A distinctive agentic failure mode, post-hoc abstention, has agents commit irreversible actions before recognizing the abstention trigger and then claim refusal.

Plain-language abstract This paper introduces Robust Agent Compensation (RAC), a system that helps AI agents cleanly undo their actions when something goes wrong mid-task. It works as an architectural add-on to existing agent frameworks like LangGraph, requiring no changes to existing agent code, and uses log-based tracking to determine exactly what needs to be reversed after a failure.

Motivation AI agents increasingly carry out multi-step tasks that have real-world side effects — booking flights, charging accounts, scheduling jobs — and when part of a task fails, prior successful steps can leave systems in an inconsistent state. Existing recovery approaches either require developers to anticipate every possible failure path in advance (intractable for dynamic agents) or rely on LLM-based replanning, which is expensive and prone to hallucinations or unnecessary compensations.

Methodology RAC is implemented as an architectural extension to agent frameworks, using the Model Context Protocol's extension points to describe compensation pairs — each action paired with its undo operation. A deterministic log-based recovery mechanism tracks what actions were actually executed at runtime and triggers compensations in the correct reverse order when failures occur. The approach was evaluated on the tau2-bench and REALM-Bench benchmarks, including new extended variants with dynamic (unpredicted) failure scenarios such as machine disruptions and multi-step group booking cancellations.

Results On standard benchmarks, RAC matched or exceeded competing frameworks in task success rates while using fewer tokens and less time. On harder problems with dynamic failures, the gap widened sharply: compared to SagaLLM (a planning-based LLM approach), RAC achieved 1.5–8x better latency and token economy. In one extreme case, SagaLLM consumed 5 million tokens and performed 34,000 unnecessary compensations before giving up, while RAC handled the same scenario by compensating only for failures that actually occurred.

Checkpoint/restore duplicates irreversible effects (double commits, double payments, token reuse) unless the tool boundary enforces replay-or-fork: replay a recorded response when equivalent, require an explicit fork for a new irreversible op, block consumed-credential reuse. LLMs regenerate subtly different requests even at temperature 0.

Why it matters The single most under-appreciated agent reliability bug: a retry after a side effect lands repeats the side effect. Idempotency keys + replay-or-fork on every external write are table stakes.

Plain-language abstract SagaLLM is a multi-agent framework that brings database-style transaction guarantees to LLM-based planning systems. It wraps teams of AI agents in a structured coordination layer that tracks state, detects inconsistencies, and rolls back failed operations — so that complex multi-step plans stay coherent even when individual steps fail or unexpected disruptions occur mid-execution.

Motivation Current LLM-based multi-agent systems lack the safeguards that reliable distributed systems require: they have no built-in rollback on failure, lose track of earlier context in long conversations, cannot reliably validate their own reasoning, and lack coordination mechanisms that reconcile state changes across agents. These gaps cause partially executed plans to leave systems in inconsistent states — for example, keeping a hotel reservation active after a flight is canceled — with no automatic recovery.

Methodology SagaLLM adapts the Saga transactional pattern — originally designed for long-lived distributed database operations — to LLM agent workflows. Each agent operation is mapped to a local transaction paired with a compensating transaction that reverses its effects on failure. The system maintains three state dimensions (application state, operation state with LLM reasoning chains, and dependency state) and uses a directed dependency graph to determine the correct compensation sequence when failures occur. Independent small-context validation agents check critical junctures to catch errors that individual LLMs cannot self-detect. The framework is evaluated using the REALM benchmark across planning scenarios including travel coordination and wedding logistics, comparing SagaLLM against GPT-o1, GPT-4o, DeepSeek R1, and Claude 3.7.

Results Experiments showed that all tested standalone LLMs — including GPT-o1, GPT-4o, DeepSeek R1, and Claude 3.7 — failed to maintain global planning constraints when unexpected disruptions were introduced mid-plan, commonly attempting to rewrite already-executed actions or losing track of agent positions. For instance, DeepSeek R1 reassigned an agent to start driving to the airport at 1:00 PM even though that agent had already arrived by 12:40 PM. SagaLLM addressed these failures through persistent state checkpointing, immutable action logging, and compensatory replanning, demonstrating consistent constraint enforcement and temporal consistency where the standalone models could not.

Plain-language abstract AiScientist is an AI system designed to autonomously carry out end-to-end machine learning research engineering — from reading a paper specification to implementing, running, and iteratively improving experiments — over hours or days without human intervention. It combines a hierarchical team of specialized agents with a shared file-based workspace that preserves project state across all stages, so later decisions stay coherent with earlier ones.

Motivation Existing AI research agents can handle narrow subtasks like idea generation or code synthesis, but consistently fail when tasks span many coupled stages over long time horizons. On PaperBench, the best prior agent achieved only 21% of the replication rubric, compared to 41% by expert PhD students, exposing a gap between local reasoning ability and the sustained, stateful coordination that real ML engineering demands.

Methodology AiScientist uses a top-level Orchestrator that maintains stage-level control through concise summaries and a workspace map, delegating to specialized agents for paper comprehension, task prioritization, implementation, and experimentation. Shared project state is stored as durable file artifacts — analyses, plans, code, and experimental logs — in a permission-scoped 'File-as-Bus' workspace, so agents re-ground on files rather than relying on conversational context handoffs. The system was evaluated on PaperBench, a benchmark for reproducing ML research papers from scratch, and MLE-Bench Lite, a competition-style ML optimization benchmark.

Results AiScientist improved PaperBench score by 10.54 points on average over the best matched baseline and achieved 81.82% Any Medal on MLE-Bench Lite. In one illustrative run on the Detecting Insults task, the system ran 74 experiment cycles autonomously over 23 hours, raising validation AUC from 0.903 to 0.982. Ablation studies showed that removing the File-as-Bus protocol reduced PaperBench by 6.41 points and MLE-Bench Lite by 31.82 points, identifying durable shared state as the key performance driver.

Plain-language abstract AgentForge is a multi-agent software engineering framework that uses large language models (LLMs) to automatically fix bugs and implement code changes in real software repositories. Unlike systems that guess whether code works, AgentForge requires every proposed change to pass actual execution inside a sandboxed Docker container before it is accepted. Five specialized agents — Planner, Coder, Tester, Debugger, and Critic — collaborate through shared memory to resolve software issues end-to-end.

Motivation LLMs can generate plausible-looking code but cannot verify whether it actually runs correctly. Existing multi-agent systems either simulate execution or treat verification as optional, which means errors can propagate unchecked. This paper addresses that gap by establishing execution-grounded verification as a mandatory first-class principle rather than an afterthought.

Methodology The framework instantiates five specialized LLM agents that coordinate through a dual-memory system combining episodic memory and a live repository index. Every code change must survive sandboxed Docker-based execution before being propagated to the next stage. The system is formalized as an iterative decision process over repository states, where execution feedback serves as the primary supervision signal. Performance was evaluated on SWE-bench Lite, a benchmark of real GitHub issues drawn from open-source Python repositories.

Results AgentForge achieved 40.0% resolution on SWE-bench Lite, outperforming single-agent baselines by 26–28 percentage points. Ablation studies confirmed that both execution feedback and role decomposition independently contribute to performance gains. The framework is released as open-source software.

Plain-language abstract MemReader is a family of two small language models designed to handle long-term memory for AI agents more reliably. Instead of simply transcribing conversation into stored records, MemReader actively decides whether incoming information is worth saving, whether it needs clarification from past context, or whether it should be ignored entirely — producing cleaner, more useful memory over time.

Motivation Existing agent memory systems such as Mem0, Zep, and MemOS treat memory extraction as a one-shot, passive task: a language model reads the current dialogue and writes structured entries. This approach fails in practice because it stores low-value chatter, handles ambiguous pronouns or incomplete information poorly, and cannot easily update stale memory when user state changes — leading to polluted, inconsistent long-term memory.

Methodology The authors introduce two complementary models. MemReader-0.6B is a compact (0.6-billion-parameter) model distilled from bilingual conversation and document data for accurate, schema-consistent structured extraction at low cost. MemReader-4B is a larger model trained with Group Relative Policy Optimization (GRPO) under a ReAct-style paradigm, in which the model first reasons about information value, reference ambiguity, and completeness, then selects one of four actions: write to memory, search historical context for disambiguation, buffer incomplete content, or ignore irrelevant input. Training trajectory data was constructed to cover all four decision paths. The models are evaluated on three public benchmarks: LOCOMO, LongMemEval, and HaluMem-Medium.

Results MemReader-0.6B outperforms a GPT-4o-mini-based passive extraction baseline in several settings, showing that a carefully supervised compact model can exceed general-purpose large-model baselines. MemReader-4B achieves state-of-the-art results on tasks involving knowledge updating, temporal reasoning, and hallucination reduction, with explicit decision-making reducing noise accumulation, state conflicts, and unusable memory entries. MemReader has been integrated into the MemOS system and deployed in real-world applications, with models and a public API released.

Plain-language abstract DeLM (Decentralized Language Models) is a multi-agent framework that drops the central controller. Instead of a main agent assigning subtasks, waiting, and merging results, parallel agents asynchronously claim tasks from a shared queue and read/write a shared 'verified context' of accumulated progress. It targets two settings — parallel exploration in software engineering and concurrent evidence processing in long-context QA — and improves accuracy while roughly halving cost.

Motivation Most multi-agent systems use centralized scatter-gather orchestration, which parallelizes sub-agent execution but not the coordination around it. Every finding must return to the main agent to be merged and rebroadcast, so progress-sharing becomes a serialized bottleneck as agents grow, and the controller can dilute, omit, or distort details. In long-context reasoning the main agent must pre-assign evidence clusters before knowing what is relevant, triggering extra delegation rounds. DeLM removes the controller as the coordination chokepoint.

Methodology Coordination is state-based, not prompt-routed. Two global structures: a shared context C of compact verified gists and a task queue T of pending subtasks. The pipeline initializes the queue from the input, executes ready subtasks in parallel, then compresses-verifies-admits each result into the shared context, generates more subtasks when the context is insufficient, and finalizes once none remain. The shared context is compact, global, and 'unfoldable' — agents read coarse gists by default and expand to detailed summaries or raw evidence only when needed. Admission-time verification checks each update against its underlying evidence and reasoning trajectory before it enters shared state, rejecting or regenerating unsupported updates so errors cannot propagate as reusable problem state.

Results On SWE-bench Verified, DeLM is strongest across test-time-scaling metrics, reaching 77.4% pass@4 at ~$0.12/task — roughly half the baselines' cost — with trace-level examples showing agents reuse each other's discoveries through the compact shared context. On LongBench-v2 Multi-Doc QA it leads four frontier model families by up to 5.7 points, with both admission-time verification and hierarchical summarization contributing. On OOLONG, vanilla DeLM underperforms RLM (which needs exact row-level aggregation via code execution), but RLM combined with DeLM yields the best accuracy and lowest cost, showing DeLM works as a coordination layer for programmatic reasoning too.

Plain-language abstract XFlow is a system for building multi-agent LLM workflows that are more reliable, and XPF is its protocol language. The core idea: today, constraints, rules, and process obligations are buried inside prompts that agents must remember and re-apply, with no way for the surrounding system to enforce them. XFlow draws an explicit prompt-harness boundary: informal semantic reasoning stays inside actors, but selected commitments are moved into harness structure that is compiled, checked, and enforced. A protocol is written as a readable literate document but compiled into a typed intermediate representation and executed as a program. At runtime, agent outputs are staged as lifecycle-governed symbols and only become shared state after passing checks.

Motivation In a multi-agent pipeline, one agent's hallucination, malformed output, or misread instruction becomes shared state and corrupts later decisions. Current frameworks sit at two extremes of the Chomsky hierarchy: markup/config tools describe workflow shape but say nothing about where knowledge leaves the prompt, and prompt-based tools keep formalizable constraints stuck in instructions. Developers have no language for drawing, testing, and adjusting which commitments are governable versus left to the actor.

Methodology Three-phase architecture. Specification: authors write XPF (YAML frontmatter for symbols/policies, Markdown stages, fenced semantic blocks for actor interfaces, guarded control flow, and responsibility handoffs). Compilation: mechanical passes parse the literate surface, resolve names, statically check that actor reads/writes are legal, judge outputs bind to declared symbols, flow targets exist, and call returns match child-protocol outputs, then lower to a typed IR. Execution: a runtime wraps each actor call in a typed interface, commits symbol writes only after schema validation/provenance/commit-policy checks inside atomic transactions with rollback, propagates staleness to derived values reactively, and persists scoped session frames for resume and audit. Evaluation uses Qwen3.5-9B and DeepSeek-V4-Flash across tau3-bench, CorpusQA (layered on the XpandA baseline), and SWE-bench Verified (around mini-SWE-agent).

Results On tau3-bench, wrapping a bare ReAct agent with XFlow raised constraint-compliance to 100% in Retail and Airline for Qwen3.5-9B (from 96.5% and 91.8%) and to 100% across all three domains for DeepSeek-V4-Flash (Telecom compliance +36.3pp), while task pass1 stayed comparable, separating 'reached the answer' from 'reached it via a valid path.' CorpusQA accuracy improved with XpandA+XFlow to 61.7 (+2.4) for Qwen and 75.7 (+0.9) for DeepSeek by encoding domain interpretation rules as deterministic derivations over extracted symbols. On SWE-bench Verified, XFlow around mini-SWE-agent raised the pass rate from 77.4 to 79.8 (+2.4) by gating patch submission on a passing local validation check; a case study shows the protocol rejecting a submission after a failed test run and issuing a targeted retry. The paper also presents cloud-edge use cases where edge worker outputs must pass schema and coverage checks before entering global state.

Plain-language abstract ESAA-Conversational is a shared memory layer that lets several LLM coding agents — Codex, Claude Code, Grok — hand work off to one another without copying the conversation by hand. It watches each agent's visible turns through hooks or watchers, writes them into one append-only log (activity.jsonl), and deterministically projects compact files — a handoff contract, current state, recorded decisions, and an open-task list — that the next agent reads to pick up where the last left off.

Motivation Developers increasingly switch among multiple coding agents as context windows fill or a different tool suits a subtask, but each agent keeps its conversation in a private, vendor-specific log. The result is 'conversational state drift': goals, rejected alternatives, decisions already made, and open tasks established with one agent are not reliably available to the next. The usual fix — copy-pasting context — is manual, lossy, expensive in tokens, and conflates capturing evidence with interpreting it.

Methodology The system applies event sourcing and CQRS: visible turns are captured mechanically, with no LLM inference, into an append-only activity.jsonl that is the single source of truth, while state.md, handoff.md, decisions.md, and tasks.json are reconstructible read models that are never hand-edited. 'Inverted ingestion' means the runtime reads native agent logs, hooks, or watchers and normalizes them into conversation_turn events rather than requiring agents to share a protocol. A strict boundary separates mechanical capture (turns are evidence) from curation (durable decisions and tasks entered through explicit decide/task commands). A paginated context command serves filtered windows (--last, --around, --before, --topic) so a cold agent reads a slice, not the whole log; workspace_root isolates projects and a lockfile serializes writes. The v1.1.0 release is a local PowerShell CLI.

Results A self-referential case study recorded 570 events (562 conversation turns) in a single workspace on 21 June 2026, distributed across Codex (304), Claude (79), and Grok (67). The three heterogeneous agents co-designed and reviewed the tool through the shared log alone, with no direct agent-to-agent channel — e.g., Codex was given a focused view of recent Grok iterations via `context --agent grok --last 20`, and one concrete defect (incomplete filtering of legacy events under context --topic) was found, fixed, and closed as a task. The public release ships 51 tests in its main battery. Reported limitations: the implementation is Windows/PowerShell-only, sync depends on third-party hook surfaces outside the authors' control, retrieval is purely textual with no embeddings, and the system offers operational but not forensic auditability (no hash chains or signatures); validation covers one workspace and three agents.

Plain-language abstract The paper presents ActiveGraph, an open-source (Apache-2.0) agent runtime that inverts the usual framework design: instead of a conversation loop with logging bolted on, the append-only event log is the source of truth, the working graph is a deterministic projection of that log, and agent behavior is a population of reactions that fire on graph changes and emit new events. No component instructs another; coordination happens through the shared graph. This buys deterministic replay of any run, cheap forking at any event without re-executing the shared prefix, and total lineage from goal to each model call.

Motivation Conventional LLM agent stacks grow by accretion: chat loop, then tools, then rules, then logging, with memory as a lossy similarity-queried store, so the log is a byproduct and questions like 'why is this fact in context', 'what did the agent believe before rule R changed', or 'what if step 42 had gone differently' are awkward or impossible. For long-running agentic work such as diligence, compliance, and research, where the reasoning matters as much as the answer, the authors argue the recoverable causal chain is the actual product.

Methodology A systems and architecture contribution, explicitly not a task-performance benchmark. The runtime defines events (id, type, payload, actor, caused_by, timestamp), behaviors as subscriptions over event types plus Cypher-subset graph-shape patterns, and a determinism contract (no random, wall-clock, fresh UUIDs, or outside I/O in behavior bodies) policed dynamically by strict replay. Nondeterministic model calls are handled by recording responses in a content-addressed cache keyed on a hash of the full request, so replay and forks make no new model calls. A bundled investment-diligence pack runs fully offline against recorded fixtures on three companies, with no API key, completing in under thirty seconds.

Results The reproducible quickstart run produced 671 events yielding 93 objects (3 companies, 24 questions, 9 documents, 25 claims, 25 evidence items, 1 contradiction, 3 risks, 3 memos) and 76 relations via 103 model calls and 48 tool calls, with zero orchestration code; re-running produced byte-identical logs. Forking a 200-step run at step 150 pays only for steps from 150 onward. Named costs and limits: replay time grows with log length (no checkpointing or compaction yet), schema evolution is a real operational burden, side-effecting tools still mutate the world on first execution, multi-writer ordering is unresolved, and self-improving agents are discussed only as an affordance, not demonstrated.

Nature paper: a host decomposes the case, invokes specialized servers, self-reflects, and emits ranked outputs with evidence-linked rationale. Validated on 6,401 cases with blinded expert review of the reasoning chain, not just final answers.

Why it matters In regulated/high-stakes settings, capture evidence references and a validation summary so a human can review the reasoning chain — auditability is a first-class output, not an afterthought.

Plain-language abstract Shepherd is a Python runtime framework that treats an AI agent's execution as a first-class object that higher-order "meta-agents" can inspect, fork, replay, and modify in real time. It introduces a Git-like execution trace where every model call, tool call, and environment change becomes a structured, replayable event, letting one agent supervise, optimize, or train another without bespoke plumbing.

Motivation As LLM-based agent systems tackle more complex tasks, they increasingly rely on meta-agents that act on other agents at runtime — for example, to prevent conflicts, fix failed runs, or improve training. Existing agentic substrates expose only plain transcripts and environment snapshots, forcing each meta-agent implementation to reinvent custom tooling to reconstruct and orchestrate execution state. Shepherd was built to close this gap by giving meta-agents a principled, unified interface over agentic execution.

Methodology Shepherd is grounded in functional programming principles: agents are typed tasks, and their execution is recorded in a Git-like trace where every action becomes a commit, every fork is a branch, and any past agent-environment state can be checked out and replayed. The framework is instantiated as a Python substrate; its core operations are formalized through an algebraic-effects calculus mechanized in Lean to provide precise semantic guarantees. The paper demonstrates the substrate through three concrete meta-agent use cases spanning live supervision, post-hoc counterfactual optimization, and tree-search reinforcement learning.

Results A live supervisor meta-agent using Shepherd raised CooperBench joint pass rate from 28.8% to 54.7% by intervening before parallel coding agents conflicted. A counterfactual replay meta-optimizer outperformed MetaHarness by up to 11 points on LiveCodeBench and TerminalBench-2 while cutting wall-clock time by up to 58%. A tree-search RL trainer using Shepherd-chosen fork points improved Qwen3.5-35B-A3B's avg@5 score on TerminalBench-2 by 5.2 points over GRPO. The substrate forks a 5.8 GB agent-environment state 5 times faster than a Docker commit and reuses over 95% of the LLM provider's KV cache.

Defines useful agent observability as correlation between high-level intent (LLM traffic) and low-level action (syscalls, file/process events), joined by time + process lineage. <3% overhead; exposes multi-agent coordination bottlenecks invisible to either stream alone.

Why it matters Correlate intent with action, not just log prompts. The join catches reasoning loops, prompt-injection exfiltration, and hidden coordination bottlenecks. Prefer standards-based telemetry over a bespoke tracer.

An open, vendor-neutral standard for telemetry from LLM and agent systems — spans/metrics/events for model calls, tool calls, token usage, and agent operations.

Why it matters Adopt this as your trace schema so observability is portable across vendors and tools. It is the closest thing to a common substrate for the execution traces the research above argues for.

Plain-language abstract Multi-agent LLM systems leave long execution traces in which a few events actually decide the outcome, buried in logs of messages, routes, memory writes, and tool calls. The standard way to find those events is counterfactual replay: rewind, edit, and re-run the trajectory to measure each event's effect, but its cost grows linearly with candidate events and is infeasible at scale. The authors compile each trace into a typed event knowledge graph and train a calibrated predictor that estimates which events the replay oracle would mark high-effect, without running the oracle. The named system, BranchPoint-Latent, uses a gradient-boosted learning-to-rank model over 13 graph features and raises per-trace localization (Branch Recall@5) from 0.73 to 0.93 on held-out families at zero replay cost.

Motivation Reliable operation of multi-agent LLM systems depends on debugging long traces, but the standard counterfactual-replay oracle costs O(T*|F|*N) model calls (seconds-to-minutes of GPU per replay), prohibitive at production scale. The authors take a direction orthogonal to oracle design: predict the oracle's per-event verdict cheaply instead of paying for it. Prediction is non-trivial because no single cheap signal works everywhere: graph centrality is strong on graph-friendly traces but near-constant on chains, Last-K fails when consequential events are early, and novelty/disagreement/uncertainty anti-correlate with the oracle in several regimes.

Methodology Each trace is compiled into an event knowledge graph with typed node attributes (route position, memory/retrieval persistence, tool metadata, uncertainty proxies, optional latent payloads). A deterministic remove-event replay oracle scores each event offline and supplies training labels once, then is removed from the deployment loop. From the graph, 13 CPU-cheap features are computed. A single gradient-boosted tree (depth 3, 400 trees, lr 0.08) with a within-trace learning-to-rank objective and family-balanced weights emits a redundancy-aware budget-bounded top-K replay agenda. Evaluation uses five-fold cross-validation grouped by trace and by trace-disjoint held-out family, over 37 trace families (163,815 events) drawn from HotpotQA, StrategyQA, GSM8K, ARC, XSum, and MBPP, plus six model-authored live-agent families (Qwen3-1.7B) and the Who&When leaderboard.

Results The interpretable linear scorer attains Spearman rho 0.58-0.59, AUPRC 0.61 (19 points over the 0.418 base rate), ECE 0.13, dominating centrality (0.52) and routing bottleneck (0.45), while single-feature heuristics anti-correlate. The learning-to-rank GBM improves every metric: Branch Recall@5 0.95 in-distribution and 0.93 on held-out families, NDCG@5 0.94/0.92, per-trace rho 0.80/0.77, beating the linear scorer on 28 of 37 families. Gains concentrate where linear scoring is blind: held-out MBPP tool traces go 0.00 to 1.00, GSM8K 0.23 to 0.98. The predictor recovers ~80% of the active-replay oracle's recall at zero replay cost. On live model-authored traces it beats centrality on all six families. On Who&When, the LLM-free CPU model reaches Acc@1 0.37 (algo) and 0.24 (hand-crafted), matching an RL-fine-tuned 8B LLM while the benchmark's own zero-shot LLM judges reach only <=0.14. A structural regime router does not beat the always-learned default (0.719 vs 0.731).

Plain-language abstract The paper presents ActiveGraph, an open-source (Apache-2.0) agent runtime that inverts the usual framework design: instead of a conversation loop with logging bolted on, the append-only event log is the source of truth, the working graph is a deterministic projection of that log, and agent behavior is a population of reactions that fire on graph changes and emit new events. No component instructs another; coordination happens through the shared graph. This buys deterministic replay of any run, cheap forking at any event without re-executing the shared prefix, and total lineage from goal to each model call.

Motivation Conventional LLM agent stacks grow by accretion: chat loop, then tools, then rules, then logging, with memory as a lossy similarity-queried store, so the log is a byproduct and questions like 'why is this fact in context', 'what did the agent believe before rule R changed', or 'what if step 42 had gone differently' are awkward or impossible. For long-running agentic work such as diligence, compliance, and research, where the reasoning matters as much as the answer, the authors argue the recoverable causal chain is the actual product.

Methodology A systems and architecture contribution, explicitly not a task-performance benchmark. The runtime defines events (id, type, payload, actor, caused_by, timestamp), behaviors as subscriptions over event types plus Cypher-subset graph-shape patterns, and a determinism contract (no random, wall-clock, fresh UUIDs, or outside I/O in behavior bodies) policed dynamically by strict replay. Nondeterministic model calls are handled by recording responses in a content-addressed cache keyed on a hash of the full request, so replay and forks make no new model calls. A bundled investment-diligence pack runs fully offline against recorded fixtures on three companies, with no API key, completing in under thirty seconds.

Results The reproducible quickstart run produced 671 events yielding 93 objects (3 companies, 24 questions, 9 documents, 25 claims, 25 evidence items, 1 contradiction, 3 risks, 3 memos) and 76 relations via 103 model calls and 48 tool calls, with zero orchestration code; re-running produced byte-identical logs. Forking a 200-step run at step 150 pays only for steps from 150 onward. Named costs and limits: replay time grows with log length (no checkpointing or compaction yet), schema evolution is a real operational burden, side-effecting tools still mutate the world on first execution, multi-writer ordering is unresolved, and self-improving agents are discussed only as an affordance, not demonstrated.

Plain-language abstract Glite ARF is an open-source framework for running many LLM coding agents in parallel on a shared research codebase without the whole thing quietly corrupting itself. It wraps each agent's work in an isolated task folder, makes finished work immutable (fixes happen in new tasks, not edits to old ones), and auto-generates a dashboard so a human can see the true state of a multi-week campaign instead of relying on an agent-written summary.

Motivation Delegating research experiments directly to coding agents doesn't scale: agents follow most instructions but the few they skip compound into corrupted data, fabricated citations, contaminated splits, and stale summaries. The authors' own prior audit found 13 such incidents in one campaign, including one where a single agent step recomputed and corrupted 20,304 historical training rows across 38 feature sets.

Methodology The framework defines a three-role stack (human chooses hypotheses, coding agents execute isolated tasks, deterministic Python 'verificator' scripts enforce structure) built on seven structural principles: task isolation via git worktrees, immutability with a corrections overlay, aggregators-only cross-task reading, and a materialized human-facing overview regenerated from committed artifacts. It's evaluated via a real external shared task (BEA 2026 vocabulary-difficulty) plus measured overhead across three author-run campaigns.

Results The BEA 2026 submission built with Glite ARF placed first (closed track) and second (open track) across all three target languages, cutting the baseline RMSE by 29.9% (closed) and 35.9% (open), across 273 tracked tasks run by up to twelve parallel agents from a single laptop at roughly $450 in LLM spend. Structured per-fold provenance caught and let them strip four target-leaking feature sets that had inflated one result to an implausible 0.609 RMSE (corrected to 0.802). The framework's structural machinery adds only about 1% wall-clock overhead across three campaigns in three domains.

Plain-language abstract TrajAudit automatically diagnoses why an AI coding agent failed a repository-level task by reading its execution trajectory - the recorded sequence of the agent's reasoning, tool calls, and observations - and pinpointing the earliest step at which the agent took an action that introduced an error, together with a justification.

Motivation As coding agents take on complex multi-file repository tasks, they fail in opaque ways, often as the cumulative consequence of a single early mistake such as a misunderstood requirement or a flawed plan. Existing trajectory-based diagnosis methods degrade badly on these traces, dropping below 40% on repository-level trajectories that often exceed 40 steps, because the trajectories are dominated by observational noise (tool outputs, redundant program structure, verbose code - over 70% of the content) and are simply too long for LLM long-context reasoning, while the methods passively consume the whole trace as if every step were equally relevant.

Methodology TrajAudit uses an investigator agent backed by two modules. Prior failure reasoning prompts an LLM to derive a preliminary diagnosis from the failed test code and its error report, directing the agent toward the most suspicious region. Semantic saliency folding compresses observations, retaining only failure-relevant context such as code patch structures and entries containing failure indicators like 'fail' or 'exception'. The investigator agent then retrieves folded content on demand through predefined interactive APIs, performing top-down diagnosis: begin with a high-level overview and drill into detail only where needed. The authors also introduce RootSE, a benchmark of 93 real-world agentic failure instances spanning over 4,500 execution steps, for locating the earliest decisive error step.

Results On RootSE, TrajAudit outperforms all existing baselines by over 24.4 percentage points in localization accuracy while reducing token consumption by at least 18%.

Plain-language abstract The paper turns a prompt-driven enterprise LLM prototype, an investment-briefing agent, into an auditable application by moving deterministic behavior out of prompts and into code: source manifests, source-backed claims, routing metadata, answer contracts, trace generation, and validators, arranged around a replaceable composition boundary where only phrasing is left to the model. It is instantiated on public data for five Korean corporate groups (25 listed companies, 113 source-backed runtime claims) and evaluated on whether the code-owned contracts hold, survive model substitution, and are load-bearing.

Motivation Enterprise LLM applications often start prompt-dominant, with product behavior carried by natural-language instructions and retrieval context rather than code, data contracts, or validation. Prompts can demonstrate behavior but not guarantee it: productization needs each visible claim traceable to bounded sources, routed to the correct entity, constrained in what it may assert, reproducible, and audited through versioned artifacts, none of which prompts alone enforce.

Methodology The harness relocates control into code: manifests define which sources may be used, source-backed claims define which statements may enter runtime context, routing metadata binds questions to entities, answer contracts define the visible answer, and traces record how each answer was assembled, with the LLM confined to a replaceable composition boundary. Evaluation covers three questions: contract preservation across a fixed validation set with a fault-injection negative control, behavior under three substituted hosted models across 270 composition-boundary runs, and an enforcement-layer ablation that disables the code-owned gate and compares it against a bolt-on external guardrail over 30 adversarial runs (15 recommendation-bait, 15 leak-bait).

Results The contracts held across the fixed validation set, and the fault-injection runs confirmed the validators flag deliberately broken source, routing, trace, answer, and leakage contracts. Under model substitution the enforced checks passed on all 270 composition-boundary runs, with failures confined to the model-composed side and recorded. In the ablation, prompt instructions alone let recommendation-language and trace-leakage violations reach the reader on all 30 adversarial runs, each blocked by the harness; the external guardrail also blocked them but over-refused, with 4 false refusals and 28 of 30 adversarial runs blocked, dropping utility to 88/120 where the harness preserved 120/120 by falling back to a deterministic composer.

Plain-language abstract This paper asks whether having multiple AI language model agents debate each other — a setup called Multi-Agent Debate (MAD) — actually helps with software engineering tasks like generating code summaries and translating code between programming languages. The researchers adapted MAD systems originally designed for general language tasks, ran them on two standard software engineering benchmarks, analyzed the debate logs when things went wrong, and proposed two targeted fixes.

Motivation Single AI language model agents struggle with tasks that require diverse expertise or multiple reasoning steps. MAD systems, where agents iteratively critique and refine each other's answers, had shown promise in general natural language tasks, but whether this structured debate could improve software engineering tasks — which mix natural language and source code — had not been studied.

Methodology The authors implemented a MAD framework adapted from prior NLP research, applying it to code summarization and code translation tasks. They evaluated the default MAD against state-of-the-art single-model baselines using metrics including BLEU, METEOR, ROUGE-L, BERTScore, CodeBLEU, and execution accuracy. To understand failures, they manually analyzed debate logs using an open-coding approach (88% inter-rater reliability), categorizing underperforming debate patterns. Based on these patterns, they proposed two enhancements: an Early Termination strategy that stops debate once a judge identifies an acceptable answer, and an Extended Reflection strategy that restarts debate with judge-provided feedback when no winner is found.

Results Default MAD performed well on code summarization but showed limited improvement on code translation compared to the state-of-the-art baseline. Manual analysis identified three failure patterns in debates: Forceful Agreement (agents converge on an incorrect answer), Ending Divergence (agents start agreeing but drift toward worse responses), and Prolonged Disagreement (agents remain stuck without reaching consensus). Ending Divergence dominated code summarization failures (76% of cases) while Forceful Agreement dominated code translation failures (62%). Both proposed enhancements improved code summarization quality with statistical significance; only the Extended Reflection strategy improved code translation. The enhanced MAD variants also reduced the number of API calls compared to the default configuration, though they still require substantially more LLM inferences than single-model approaches.

Plain-language abstract This paper investigates why multi-agent systems built from large language models so often fall short of expectations. The authors created MAST, a structured catalog of failure types, by carefully examining hundreds of conversation traces from seven widely-used multi-agent frameworks. The result is a practical taxonomy that names and organizes 14 distinct ways these systems break down, along with an automated tool to apply that taxonomy at scale.

Motivation Multi-agent LLM systems have attracted considerable interest because they can, in principle, divide complex tasks among specialized agents and coordinate their work. Yet empirical results consistently show their performance gains over single-agent setups are small or absent — for instance, one studied framework (ChatDev) solved only 33% of programming tasks. No systematic account of why these systems fail existed, leaving developers without a principled framework for diagnosis or improvement.

Methodology The authors analyzed more than 200 conversation traces drawn from seven open-source multi-agent frameworks (including MetaGPT, ChatDev, HyperAgent, AppWorld, AG2, and Magentic-One) running on diverse benchmarks. Six expert human annotators applied Grounded Theory to label failures in the traces, with inter-annotator agreement measured by Cohen's Kappa (reaching 0.88). An LLM-as-a-Judge pipeline using OpenAI's o1 was then developed and validated against expert labels (Kappa 0.77) to enable scalable automated annotation. Two case studies tested whether targeted interventions could reduce the identified failures.

Results The analysis identified 14 distinct failure modes organized into three categories: specification issues (system design problems, 41.77% of failures), inter-agent misalignment (coordination failures, 36.94%), and task verification failures (quality control, 21.30%). Prominent individual failure modes included step repetition (17.14%), reasoning-action mismatch (13.98%), and disobey task specification (10.98%). Targeted interventions such as improved role specification yielded only modest gains (e.g., +15.6% for ChatDev), indicating that the identified failures stem from fundamental system design challenges rather than easily patched prompt issues.

Plain-language abstract AgentForge is a multi-agent software engineering framework that uses large language models (LLMs) to automatically fix bugs and implement code changes in real software repositories. Unlike systems that guess whether code works, AgentForge requires every proposed change to pass actual execution inside a sandboxed Docker container before it is accepted. Five specialized agents — Planner, Coder, Tester, Debugger, and Critic — collaborate through shared memory to resolve software issues end-to-end.

Motivation LLMs can generate plausible-looking code but cannot verify whether it actually runs correctly. Existing multi-agent systems either simulate execution or treat verification as optional, which means errors can propagate unchecked. This paper addresses that gap by establishing execution-grounded verification as a mandatory first-class principle rather than an afterthought.

Methodology The framework instantiates five specialized LLM agents that coordinate through a dual-memory system combining episodic memory and a live repository index. Every code change must survive sandboxed Docker-based execution before being propagated to the next stage. The system is formalized as an iterative decision process over repository states, where execution feedback serves as the primary supervision signal. Performance was evaluated on SWE-bench Lite, a benchmark of real GitHub issues drawn from open-source Python repositories.

Results AgentForge achieved 40.0% resolution on SWE-bench Lite, outperforming single-agent baselines by 26–28 percentage points. Ablation studies confirmed that both execution feedback and role decomposition independently contribute to performance gains. The framework is released as open-source software.

Reports compile-success / test-pass / deploy-success / API-consistency, stratified by task complexity, with token + cost + time accounting. (Preprint; single underlying model is a validity threat.)

Why it matters For software workflows, measure the things that actually fail in production — compile, test, deploy — with cost per task, stratified by difficulty. One headline accuracy number hides the reliability story.

Plain-language abstract This paper lays out a comprehensive quality-assurance strategy for enterprise AI systems — products built on large language models, retrieval pipelines, and autonomous agents. It argues that traditional software testing is structurally mismatched with these systems and proposes a new framework centered on continuous risk reduction, a five-layer AI Assurance Pyramid, and treating evaluation as a core engineering discipline.

Motivation Enterprise AI systems fail in ways that conventional testing cannot detect: confident hallucinations, silent behavioral drift after a cloud provider updates a model, and coordination failures in multi-agent workflows that produce wrong answers visually indistinguishable from correct ones. Teams that test AI the same way they test deterministic software — with pass/fail test suites evaluated at release time — are systematically under-protected because AI outputs are probabilistic and cannot be verified for correctness in the classical sense.

Methodology The paper is a conceptual and prescriptive engineering strategy, not an empirical study. It introduces a structured AI Failure Taxonomy covering five categories of AI-native failure modes (including hallucination, instruction drift, trajectory collapse, and emergent coordination failure across fifteen specific modes), then maps these to a revised five-layer AI Assurance Pyramid ranging from Layer 0 (deterministic infrastructure validation) through Layer 4 (business outcome evaluations). It provides operational guidance on evaluation-driven development, RAG system testing using metrics such as those from the RAGAS framework, model lifecycle management including prompt regression testing, and governance including human-in-the-loop oversight and auditability.

Results The paper concludes that evaluation infrastructure must be treated as a shared platform capability — centralized datasets, judge pipelines, rubrics, and scoring pipelines — rather than rebuilt per project, drawing an analogy to the shift from per-project CI scripts to shared CI/CD platforms. It argues that the cost of insufficient evaluation (hallucination incidents, model drift detected weeks after onset, adversarial failures reaching production) consistently exceeds the investment in evaluation infrastructure, and that a pyramid weighted toward lower layers catches failures more cheaply and with better diagnostic precision than top-heavy end-to-end evaluation alone.

Plain-language abstract This paper introduces tau-bench, a benchmark for testing AI language agents on realistic customer-service tasks. The agent must hold a back-and-forth conversation with a simulated human user, call real database APIs, and follow written domain policies — all at once — to complete tasks like changing a flight reservation or processing a retail return.

Motivation Existing agent benchmarks give the agent all information upfront and have it interact only with software environments, not people. Real deployment requires agents that can gather information incrementally from human users, consult domain-specific rules, and behave consistently across many interactions — capabilities that no prior benchmark measured together.

Methodology The benchmark was built in three stages: manual design of realistic database schemas and APIs, language-model-assisted generation of synthetic data entries, and human-annotated scenario creation for a simulated user. Two customer-service domains were created — tau-retail and tau-airline. Evaluation compares the actual database state at the end of each conversation against an annotated ground-truth state. A new metric, pass^k, measures whether an agent succeeds consistently across k independent trials of the same task.

Results State-of-the-art models using function calling perform poorly: GPT-4o achieves roughly 61% task success on tau-retail and about 35% on tau-airline (pass^1). Consistency degrades sharply with more trials — pass^8 on tau-retail falls below 25% for the same model. Failure analysis shows agents struggle most with complex database reasoning, correctly applying domain policies, and handling requests that involve more than one action.

Plain-language abstract SWE-bench is a benchmark for testing whether language models can solve real software engineering problems. It presents models with 2,294 actual GitHub issues from 12 popular Python repositories and asks them to generate code patches that fix those issues, verified by running the repository's own test suite.

Motivation Existing coding benchmarks like HumanEval consist of self-contained problems solvable in a few lines, which no longer capture what frontier language models can and cannot do. Real software engineering requires navigating large codebases, understanding interactions across many files, and reasoning about complex bugs — a much harder and more realistic challenge that prior benchmarks did not test.

Methodology The authors scraped pull requests from 12 popular Python repositories, filtered for PRs that resolved a linked GitHub issue, included changes verified by tests that shifted from failing to passing, and excluded instances with installation or runtime errors. This pipeline reduced roughly 90,000 PRs to 2,294 curated task instances. Models are given an issue description and a codebase snapshot, and must produce a patch; evaluation uses BM25-based retrieval to provide relevant context and runs the repository's test suite to check correctness. The authors also released a training set of 19,000 instances from 37 repositories and two fine-tuned models, SWE-Llama 7b and 13b, built on CodeLlama.

Results State-of-the-art models performed very poorly on SWE-bench. The best-performing model, Claude 2, resolved only 1.96% of the issues when using a BM25 retriever. Fine-tuned SWE-Llama 13b was competitive with Claude 2 in some settings and could handle contexts exceeding 100,000 tokens, but overall results confirm that current language models struggle with realistic, multi-file software engineering tasks.

Plain-language abstract This paper asks whether a powerful AI language model can reliably act as a judge to evaluate other AI chatbots — replacing expensive, slow human raters. The authors build two benchmarks, MT-Bench and Chatbot Arena, to test this idea and measure how closely AI judges match real human opinions.

Motivation Standard language-model benchmarks such as MMLU and HELM score models on multiple-choice or short-answer tasks, but they fail to distinguish well-aligned chat assistants from unaligned base models. A scalable, automated way to measure how well a chatbot follows instructions and satisfies users in open-ended, multi-turn conversations was missing.

Methodology The authors introduced MT-Bench — 80 carefully designed multi-turn questions spanning eight categories (writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, and humanities) — and Chatbot Arena, a crowdsourced platform where users chat with two anonymous models simultaneously and vote for the better one, collecting about 30,000 votes over one month. They then tested the 'LLM-as-a-judge' approach (pairwise comparison, single-answer grading, and reference-guided grading) using GPT-4, GPT-3.5, and Claude-v1 as judges, and systematically examined biases including position bias, verbosity bias, and self-enhancement bias.

Results GPT-4 as a judge agreed with human preferences at a rate exceeding 80%, matching the level of agreement observed between two human raters. Position bias was a significant problem for Claude-v1 and GPT-3.5 (consistency rates of roughly 24% and 46% respectively), while GPT-4 was consistent in over 65% of cases. A 'repetitive list' adversarial attack caused Claude-v1 and GPT-3.5 to fail 91% of the time, versus only 8.7% for GPT-4. Providing chain-of-thought or reference solutions reduced GPT-4's math-grading failure rate from 14/20 to 3/20. The study concludes that LLM-as-a-judge is a scalable and explainable proxy for human preference evaluation.

Learns a policy over agent-call sequences with a length-regularization penalty and a STOP node; a regularized 3-agent sequence uses ~1/3 of the fixed-baseline calls.

Why it matters Adaptive selection works but is domain-specific. Record route/outcome/cost facts first; optimize selection from data, don't hand-author a routing table.

Plain-language abstract AgentDojo is a benchmarking framework for testing how well AI agents can resist prompt injection attacks — attempts by malicious content in tool outputs to hijack what the agent does. It provides a set of realistic tasks, security test cases, and an extensible environment where researchers can design new attacks and defenses against such vulnerabilities.

Motivation AI agents that combine large language models with external tools (email, banking, travel booking) cannot formally distinguish instructions from data, making them vulnerable to prompt injection: an attacker embeds malicious instructions in content the agent reads, causing it to execute unauthorized actions such as leaking user data or sending unauthorized messages. No rigorous, extensible benchmark existed to systematically measure agent robustness against these attacks.

Methodology The authors built AgentDojo as a dynamic, extensible framework rather than a static test suite. They populated it with 97 realistic agent tasks across domains such as email management, e-banking, and travel bookings, paired with 629 security test cases. Each security test specifies an attacker goal and an injection endpoint. The framework evaluates both utility (whether the agent completes its user task) and security (whether the attacker goal is achieved) using formal checks over environment state, not LLM-simulated judgments.

Results State-of-the-art LLMs solve fewer than 66% of AgentDojo tasks even without any attack present. Existing prompt injection attacks succeed against the best-performing agents in fewer than 25% of cases. Deploying a secondary attack-detector defense reduces the attack success rate further to 8%. Attacks benefit only marginally from side information about the system or victim, and rarely succeed when the attacker goal involves security-sensitive actions such as exfiltrating an authentication code.

Plain-language abstract MultiAgentBench is a benchmark for testing how well groups of AI language model agents work together or compete against each other across a range of realistic scenarios. It introduces MARBLE, an evaluation framework that scores both whether agents complete tasks and how well they collaborate, using milestone-based performance indicators. The benchmark covers scenarios from co-authoring research proposals to building structures in Minecraft to social deduction games.

Motivation Existing benchmarks for AI agents either focus on single agents working alone or test only narrow, specialized domains, which means they cannot capture what happens when multiple agents must coordinate, negotiate, or compete. There was no comprehensive way to measure the quality of multi-agent collaboration and competition across diverse, interactive settings.

Methodology The authors built MARBLE, a multi-agent coordination framework organized around a Coordination Engine that links an Agent Graph, Cognitive Module, and Coordinate Engine to support adaptive collaboration and communication. Scenarios include both established tasks (such as research collaboration following the ResearchTown setup and Minecraft-based building tasks) and LLM-generated tasks with human verification (such as Werewolf and bargaining games). The benchmark evaluates multiple coordination topologies—star, chain, tree, and graph—as well as strategies such as group discussion and cognitive planning.

Results Among the models tested, gpt-4o-mini achieved the highest average task score. Graph-structured coordination performed best among the topology options in the research scenario. Cognitive planning improved milestone achievement rates by 3%. Code and datasets are publicly available at the project's GitHub repository.

Plain-language abstract This paper identifies and studies a hidden bias problem called "preference leakage" that occurs when the same large language model (or a closely related one) is used both to generate synthetic training data and to evaluate model outputs. When a judge model and the model being evaluated share a close relationship — being the same model, one inheriting from the other, or both belonging to the same model family — the judge systematically favors the related model's outputs in ways that are difficult to detect.

Motivation As AI development increasingly relies on LLMs both to synthesize training data and to serve as automated evaluators (LLM-as-a-judge), a contamination risk emerges: popular benchmarks and research pipelines commonly use powerful models like GPT-4 for both data generation and evaluation, creating an undisclosed overlap. Existing work on LLM-as-a-judge biases (such as position or length bias) did not account for this structural conflict of interest, which the authors argue is subtler and harder to detect than previously identified biases.

Methodology The authors defined three types of relatedness between a data-generator LLM and a judge LLM: being the same model, an inheritance relationship (one trained on the other's outputs), and membership in the same model family. They selected three generator/judge models — GPT-4o, Gemini-1.5-Flash, and LLaMA-3.3-70B — and two student base models — Mistral-7B and Qwen-2.5-14B. Each student model was fine-tuned (SFT) on 30,000 synthetic responses from each generator, sampled from the Ultrafeedback dataset. Resulting student models were then evaluated on Arena-Hard (500 questions) and AlpacaEval 2.0 (805 questions) using each judge. The authors introduced a preference leakage score (PLS) to quantify judge bias toward related student models, and supplemented automated scoring with human annotation from three annotators on 100 AlpacaEval questions.

Results Experiments confirmed systematic bias: judge models consistently favored student models trained on their own synthetic data. For example, the GPT-4o and Gemini-1.5 judge pair showed preference leakage scores of 28.7% on Arena-Hard and 18.4% on AlpacaEval 2.0. Further analysis showed that the severity of preference leakage correlates with the degree of relatedness between generator and judge, and with the proportion of synthetic data used in training. The bias was found to be pervasive across relatedness types and harder to detect than previously identified LLM-as-a-judge biases, particularly affecting subjective questions and certain judgment dimensions.

Plain-language abstract This paper surveys how researchers evaluate AI chatbots and assistants built on large language models when those systems must hold multi-turn conversations — exchanges that unfold over many back-and-forth messages rather than a single prompt. The authors reviewed nearly 250 published studies to map out what is being evaluated and how, producing two classification frameworks that together cover the full range of current evaluation practice.

Motivation AI conversational agents powered by large language models are increasingly deployed in customer service, personal assistants, and other settings that require sustained, context-aware dialogue. Despite this growth, the field lacked a systematic overview of how such multi-turn systems are evaluated — what dimensions matter and what measurement methods are available — leaving practitioners without a consolidated reference.

Methodology Using a PRISMA-inspired systematic review process, the authors examined nearly 250 scholarly sources from a range of publication venues. From this corpus they constructed two interrelated taxonomy systems: one defining what to evaluate (task completion, response quality, user experience, memory and context retention, planning and tool integration) and one categorizing how to evaluate (annotation-based evaluations, automated metrics such as BLEU and ROUGE, hybrid human-plus-quantitative strategies, and self-judging methods that use LLMs as evaluators).

Results The survey produced a structured, dual-taxonomy framework covering both evaluation dimensions and evaluation methodologies for LLM-based multi-turn conversational agents. The framework captures traditional language-understanding metrics alongside newer techniques suited to the dynamic, interactive nature of multi-turn dialogue, offering a consolidated foundation for researchers and practitioners assessing conversational AI systems.

A voluntary, widely-referenced framework organizing AI risk management into Govern / Map / Measure / Manage functions.

Why it matters Map your agent controls to a recognized framework so 'reliability' becomes auditable. Pairs naturally with the assurance + governance research above.

Plain-language abstract SWE-Marathon is a benchmark for software work that takes hours and millions of tokens — porting whole libraries between languages, cloning full products, building ML systems — rather than the minute-scale single-patch tasks most benchmarks measure. Each of its 20 tasks ships an executable environment, a human reference solution, and a multi-layer hidden verifier, and even the strongest current coding agents finish fewer than 30%.

Motivation Capability claims now reach workflows that take human engineers days, but dominant benchmarks fall short on horizon (most tasks resolve within an hour) and on verifier strength (single committed-patch or single-test grading that agents can game). At hour-scale budgets, agents with filesystem and network access probe weak checks, so long, realistic, ungameable tasks need richer verifier surfaces than a single test suite.

Methodology Twenty tasks across library reproductions, product clones, ML engineering, and algorithmic optimization were authored by engineers familiar with each system and accepted only on specificity, solvability (reference oracle passes, no-op agent fails), and integrity (no readable answers or forbidden reference implementations), enforced by CI, rubric checks, piloted agent trials, and an adversarial cheating agent. Tasks run in Harbor/Modal sandboxes under 2–10h wall-clock limits; hidden verifiers span dense assertion suites, behavioral parity, performance gates, deterministic replay, audit checks, and computer-use UI judges. The authors evaluate 13 agent–model configurations under both native CLIs and a shared Terminus-2 scaffold, and audit every rollout for reward hacking with an LLM suspicion score.

Results Across 1,300 rollouts no configuration exceeds 30% pass@1. Of 526 agent-attributable failures, implementation failure (41.6%) and timeout (31.4%) dominate, followed by reward hacking (15.4%), premature termination (7.6%), and poor self-verification (4.0%). 13.8% of rollouts take an exploit-shaped action and 10.2% ship a bypass, but the three-layer defense catches all 132 shipped bypasses so none earns reward. Long context is not passive: pass rate falls monotonically with runs of identical consecutive tool calls (claude-code 41.9%→3.2%), 32% of one scaffold's tool calls are silent duplicates, and compaction tracks failure — 0 of 71 summarizer trials pass versus 8.9% without. Token spend is not a skill proxy; the lowest-token quintile passes 11.3% versus 8.3% for the highest, and per-(model,scaffold) token use varies up to 12×.

Plain-language abstract Multi-agent LLM systems leave long execution traces in which a few events actually decide the outcome, buried in logs of messages, routes, memory writes, and tool calls. The standard way to find those events is counterfactual replay: rewind, edit, and re-run the trajectory to measure each event's effect, but its cost grows linearly with candidate events and is infeasible at scale. The authors compile each trace into a typed event knowledge graph and train a calibrated predictor that estimates which events the replay oracle would mark high-effect, without running the oracle. The named system, BranchPoint-Latent, uses a gradient-boosted learning-to-rank model over 13 graph features and raises per-trace localization (Branch Recall@5) from 0.73 to 0.93 on held-out families at zero replay cost.

Motivation Reliable operation of multi-agent LLM systems depends on debugging long traces, but the standard counterfactual-replay oracle costs O(T*|F|*N) model calls (seconds-to-minutes of GPU per replay), prohibitive at production scale. The authors take a direction orthogonal to oracle design: predict the oracle's per-event verdict cheaply instead of paying for it. Prediction is non-trivial because no single cheap signal works everywhere: graph centrality is strong on graph-friendly traces but near-constant on chains, Last-K fails when consequential events are early, and novelty/disagreement/uncertainty anti-correlate with the oracle in several regimes.

Methodology Each trace is compiled into an event knowledge graph with typed node attributes (route position, memory/retrieval persistence, tool metadata, uncertainty proxies, optional latent payloads). A deterministic remove-event replay oracle scores each event offline and supplies training labels once, then is removed from the deployment loop. From the graph, 13 CPU-cheap features are computed. A single gradient-boosted tree (depth 3, 400 trees, lr 0.08) with a within-trace learning-to-rank objective and family-balanced weights emits a redundancy-aware budget-bounded top-K replay agenda. Evaluation uses five-fold cross-validation grouped by trace and by trace-disjoint held-out family, over 37 trace families (163,815 events) drawn from HotpotQA, StrategyQA, GSM8K, ARC, XSum, and MBPP, plus six model-authored live-agent families (Qwen3-1.7B) and the Who&When leaderboard.

Results The interpretable linear scorer attains Spearman rho 0.58-0.59, AUPRC 0.61 (19 points over the 0.418 base rate), ECE 0.13, dominating centrality (0.52) and routing bottleneck (0.45), while single-feature heuristics anti-correlate. The learning-to-rank GBM improves every metric: Branch Recall@5 0.95 in-distribution and 0.93 on held-out families, NDCG@5 0.94/0.92, per-trace rho 0.80/0.77, beating the linear scorer on 28 of 37 families. Gains concentrate where linear scoring is blind: held-out MBPP tool traces go 0.00 to 1.00, GSM8K 0.23 to 0.98. The predictor recovers ~80% of the active-replay oracle's recall at zero replay cost. On live model-authored traces it beats centrality on all six families. On Who&When, the LLM-free CPU model reaches Acc@1 0.37 (algo) and 0.24 (hand-crafted), matching an RL-fine-tuned 8B LLM while the benchmark's own zero-shot LLM judges reach only <=0.14. A structural regime router does not beat the always-learned default (0.719 vs 0.731).

Plain-language abstract SEAGym is a testing environment for AI agents that improve themselves over time by editing their own setup—their prompts, memory, skills, tools, and configuration—rather than by retraining the underlying model. Instead of just checking whether a self-improved agent scores higher on a final task, SEAGym measures the improvement process itself: whether each self-edit actually helps on new tasks, whether gains stick or later collapse, and what they cost.

Motivation Self-evolving agents improve mainly by changing their agent harness—the structured execution layer around a base model. Existing evaluations reduce this to isolated task scores or a single sequential learning curve, which obscures whether a given update produces reusable improvement, overfits the recent tasks, increases cost, or harms older behavior. Most agent benchmarks are built for static evaluation, resetting agent state between isolated episodes and removing exactly the state persistence that self-evolution depends on.

Methodology SEAGym uses an RL-style environment formulation in which the self-evolving agent supplies both the task policy and the harness-update rule, while the environment defines task sampling, feedback, schedules, and snapshot assessment. It converts Harbor-compatible static benchmarks into reusable task sources organized into train batches and frozen evaluation views—update-validation, held-out in-domain transfer, out-of-domain transfer, replay diagnostics, and cost records—and saves agent snapshots and metric artifacts. Explicit schedule parameters (state reset, task reuse, batch size, update timing) let single-task adaptation, online transfer, and epoch-based batch learning be studied under one protocol. Task rollout is separated from method update, so methods such as ACE, TF-GRPO, and AHE connect through thin wrappers while keeping their native update rules; experiments instantiate the environment on Terminal-Bench 2.0 and HLE.

Results The evaluation views provide complementary signals about the evolution process rather than one summary number. Frequent updates may fail to improve held-out performance, validation gains do not always transfer to in-domain or out-of-domain test views, and useful intermediate snapshots can collapse later or recover. Batch size, source diversity, and the rollout model backend all affect harness reliability, showing that self-evolution dynamics depend on the evaluation schedule and setup, not just the update method.

Plain-language abstract RigorBench is a benchmark that scores AI coding agents on how they work, not just whether their final code passes tests. It defines five process pillars — Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity — and combines them into a weighted RigorScore. The authors built 30 tasks in five categories (Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, Don't Break the Build) and ran four harnesses (Agent-Rigor, Agent-Skills, Superpowers, and a Baseline ReAct control) on the same underlying model, scoring the full execution trajectory rather than the final artifact. Structured process discipline raised process-quality scores by an average of 41% and downstream outcome correctness by 17%, and RigorScore correlated with outcome quality at r=0.87. They release the tasks, rubrics, and trajectory-analysis tools as open source.

Motivation Existing AI coding benchmarks (SWE-bench, HumanEval, MBPP, BigCodeBench, Terminal-Bench, AgentBench, and others) measure outcomes only — did the code pass the tests or resolve the issue. The authors survey major benchmarks and find none evaluate the engineering process. Their argument: an agent that stumbles onto a correct fix through reckless trial-and-error, without planning, verification, or graceful recovery, is less reliable than one that follows engineering discipline — yet every existing benchmark gives them the same score. They call the resulting risk the lab-to-production gap: outcome-only optimization breeds strategies (fragile fixes, token waste, false confidence, broken intermediates) that look fine on a benchmark but are hazardous in production. They ground this in software-engineering precedent (Humphrey's Personal Software Process, CMMI) that how software is built predicts its quality.

Methodology RigorBench has three design pieces. (1) A five-pillar scoring framework, each pillar built from weighted sub-metrics: Planning Fidelity from plan-artifact creation, decomposition quality on a 4-point rubric, and plan-execution alignment via Kendall tau; Verification Coverage from test-creation rate, coverage delta via instrumentation, and requirements traceability as recall; Recovery Efficiency from recovery-attempt count, strategy diversity, and token-waste ratio; Abstention Quality scored only on impossible/ambiguous tasks (correct abstention, false confidence, clarification seeking); Atomic Transition Integrity from build health, test-suite stability, and commit hygiene. The composite RigorScore = 0.20·PF + 0.25·VC + 0.25·RE + 0.15·AQ + 0.15·ATI, each pillar normalized to [0,1]. (2) 30 curated tasks, 6 per category, each designed to be discriminative, measurable, and realistic. (3) Trajectory-based evaluation: each agent runs in an isolated Docker container with a fresh task-repo clone, instrumented shell/filesystem, a 60-minute timeout, and a 200K-token budget; the pipeline parses raw logs into a trajectory, extracts signals (planning artifacts, test events, error-recovery cycles, abstention signals, codebase-health checkpoints), and scores each pillar with deterministic heuristics plus LLM-as-judge for qualitative sub-metrics, using a 3-judge panel with majority voting. Setup evaluates four harnesses — Agent-Rigor (a 6-phase discipline lifecycle), Agent-Skills, Superpowers, and a Baseline ReAct control — all on the same model, giving 120 executions, with process and outcome quality measured independently.

Results Process discipline improved process-quality scores by an average of 41% and downstream outcome correctness by 17%. Overall RigorScore: Agent-Rigor 0.61, Superpowers 0.48, Baseline ReAct 0.48, Agent-Skills 0.47; outcome scores rose from 0.64 (Baseline) to 0.83 (Agent-Rigor). The largest disciplined-vs-baseline gain was Planning Fidelity (+0.47) — baseline agents rarely produce explicit plans despite chain-of-thought ability. Abstention Quality showed the second-largest gain (+0.34): no baseline agent abstained on any of the 6 impossible tasks, and disciplined agents still abstained correctly on only 62%. Recovery Efficiency improved least (+0.25); token-waste ratio fell only 34% and doom loops persisted on hard tasks. RigorScore correlated with outcome quality at r=0.87 (p<0.001) across all 120 executions, and the with/without design supports attribution, not just correlation. Disciplined agents also used 12% fewer total tokens despite producing more artifacts, because saved recovery tokens outweigh planning/verification overhead. Inter-judge agreement was Fleiss' kappa 0.74. Limitations: only 30 tasks, a single discipline framework, LLM-judge bias risk, June-2025 temporal validity, and benchmark-contamination risk.

Plain-language abstract This study asks whether the test-running that code-repair agents do — the 'generate, run tests, revise' loop — is actually worth its cost. The authors analyze 7,745 public SWE-bench agent traces and then run 3,000 controlled repair attempts with three agents (Claude Code, Codex, and open-source OpenCode), turning execution on and off. Forbidding the agent from running tests barely changes how many bugs get fixed while cutting token and time costs by roughly half, so execution should be treated as a resource spent deliberately, not a default.

Motivation LLM repair agents have standardized on a 'generate-run-revise' loop that executes tests to validate and refine patches, but running code is expensive: it consumes tokens (generating commands, parsing verbose output), adds latency (a full test suite can take minutes to hours), and forces teams to maintain a working test environment — a Docker image with the right dependencies — for every target repository. Prior work studied model architectures, prompts, or search but treated execution as a necessary implicit component; even Agentless removed the loop and execution together, so execution's marginal contribution was never isolated.

Methodology A two-stage empirical study. Stage one characterizes execution behavior by analyzing 7,745 SWE-bench leaderboard traces spanning four execution-based agents, twelve LLMs, and the Lite and Verified datasets. Stage two isolates execution as a single controlled variable: the agent scaffold is held fixed (Claude Code with Claude-Sonnet-4.5, Codex with GPT-5.2-xhigh, OpenCode with Qwen2.5-Coder-32B) while execution access is varied across four paradigms over 3,000 end-to-end attempts on 200 SWE-bench instances, enforced at both the prompt and tool level, yielding a controlled measurement of execution's marginal value and cost.

Results Agents execute constantly — an average of 8.8 test runs per task, ranging from 2 to 19, with late-stage executions (66–100% of the conversation) more successful than early ones. Yet restricting execution barely hurts: the resolve-rate gap between Prohibited and Unrestricted is 1.25 percentage points on commercial agents (not significant, p>0.05) and roughly zero for OpenCode, while Prohibited saves 56–62% of tokens and 48–54% of wall-clock on Claude Code and removes per-repository test-environment maintenance. The benefit is concentrated rather than uniform: 54–66% of commercial-agent cases resolve in a single edit, localization accuracy without execution stays above 95%, and 81–100% of failed cases pass the agent's own executed validation but fail the official evaluation; OpenCode instead over-retries with only 11% of failed cases passing self-validation. The authors conclude execution should be treated as a resource with an explicit cost-benefit tradeoff, not a default capability.

Plain-language abstract Users can reach many LLMs, and the strongest model is not the best one for every coding task, so routing each task to the right model matters for both quality and cost. Existing routers treat this as a one-off classification and fall well short of a per-task oracle that always picks the best model. Agent-as-a-Router instead formalizes routing as a Context→Action→Feedback loop that verifies each decision by running the chosen model's output in a sandbox and accumulates that experience over the task stream. The framework is instantiated as ACRouter and evaluated on CodeRouterBench, an execution-verified environment of about 10K coding tasks scored across 8 frontier LLMs.

Motivation A zero-shot LLM router, even on a strong model like Claude Sonnet 4.6, lags the per-task oracle by a wide margin. An ablation shows the gap is information deficit rather than weak reasoning: adding per-dimension performance statistics to the router yields a 15.3% relative AvgPerf gain (41.41→47.74) and beats a heuristic encoding the same priors. Static routers cannot close this gap because their information state is frozen, which motivates a router that gathers execution-grounded signal during deployment and conditions future decisions on it.

Methodology Routing is cast as a contextual-bandit C-A-F loop: at each task the router observes context (prompt, optional metadata, and memory accumulated from prior loops), selects a model, and receives verifier feedback — a sandbox-observed score in [0,1] plus monetary cost computed from token consumption — which is memorized for the next decision; cumulative regret against a per-task oracle is the streaming metric. ACRouter realizes this with an Orchestrator (a fine-tuned Qwen3.5-0.8B policy combined with heuristic rules and top-10 kNN historical neighbors via weighted voting), a Verifier (AST parsing plus sandbox execution aggregated into a unified per-task score), and a Memory module. CodeRouterBench supplies ~10K tasks across 9 in-distribution coding dimensions plus an out-of-distribution agentic-programming testbed, with execution-verified scores from 8 frontier LLMs.

Results ACRouter attains the lowest cumulative regret (205.5) and highest AvgPerf (49.98%) among routers on the in-distribution stream, beating DimensionBest — which has a full dimension-level prior — by 2.48% AvgPerf, and reaches 62.50% AvgPerf on the out-of-distribution agentic-programming split. It is also more cost-efficient than always invoking the strongest model: Perf/$ of 3.79 (ID) and 1.18 (OOD) versus Always-Opus at 1.29 and 0.64. Lightweight static learners post fair in-distribution scores but fail to generalize out of distribution.

Plain-language abstract Verifying a coding agent's output was supposed to be easier than producing it; this position-plus-experiments paper from the Qwen team argues the asymmetry has inverted. Every verifier (tests, rubrics, judges) is a proxy for human intent, and optimization pressure widens the proxy-intent gap into reward hacking. The paper frames verification quality along scalability, faithfulness, and robustness, shows existing approaches achieve only two of the three, and walks through four reward constructions, concluding that verification must co-evolve with the generator.

Motivation Unit tests, LLM judges, and human review each fail one of scalability, faithfulness, or robustness. As foundation models and harnesses improve, generating candidate solutions has become easier than reliably checking them, and reward hacking is an inevitable consequence of sustained optimization against an imperfect proxy, not a patchable bug.

Methodology Four reward constructions studied on Qwen foundation models, each analyzed through the same lens (task characteristics, verification constraints, reward implementation, empirical observations): execution-based test verifiers built with the SWE-Universe pipeline plus an agentic quality judge that scores instruction clarity and instruction-test alignment against a human-annotated benchmark; rubric-based and interactive browser judges for frontend tasks; user interaction feedback mined as training signal; and an autonomous agentic evaluator for long-horizon tasks.

Results With the quality judge and trajectory-level behavior monitoring in place, the hacked resolved rate across three SWE-Bench variants drops from 28.57% to 0.56% and the clean resolved rate rises from 40.22% to 60.53%. Interactive judges grounded in observed runtime behavior resist the length-exploitation hacks static judges suffer. Training on user-feedback signal yields a 13.3-point gain on a private coding-agent benchmark, and long-horizon training data filtered by the agentic evaluator stably beats random sampling under a controlled budget.

Plain-language abstract Self-improving agent systems in the Darwin and Huxley Gödel Machine line search over self-modifying agents but assume a fixed benchmark stays valid as agents improve. The Red Queen Gödel Machine makes the evaluator part of the loop: learned evaluators co-evolve with the task agents they score, and the search objective may change only at epoch boundaries so per-epoch self-improvement guarantees still hold.

Motivation Static evaluation blocks self-improvement in three settings: tasks with no direct benchmark (paper writing, proof writing), evaluation that is slow or weakly informative, and benchmarks that saturate or get reward-hacked as agents strengthen. The Red Queen dynamic from evolution, adapting against co-evolving competitors, is the missing piece.

Methodology Tree search over an archive of multi-agent workspaces, with nodes selected by Thompson sampling over clade metaproductivity. Controlled utility evolution: within an epoch one evaluator is frozen and grades every task agent; challenger evaluators evolve in the shared codebase and are scored on a held-out ground-truth anchor; at checkpoints a challenger replaces the incumbent only if it wins on an epsilon-best-belief criterion, after which selective erasure discards utility records that depended on the displaced evaluator. Evaluated on coding (Polyglot), scientific paper writing and reviewing, and Olympiad proof writing and grading.

Results On verifiable coding, adding a cheap co-evolved code-review judge (queried once, versus multi-turn agent execution) reaches 71.7% held-out pass rate against the prior SOTA HGM-H's 69.9% with 1.35–1.72× fewer search tokens. Co-evolved paper writers raise acceptance from 21.8% to 40.5% under a diverse judge panel; a co-evolved grader beats static baselines at 3× lower search cost with 9% higher ground-truth accuracy. An adversarial-objective epoch corrects a reviewer that over-accepted AI-generated papers at up to 1.91× the human rate, yielding one equally stringent on both.

Plain-language abstract Training coding agents needs a verifier to filter SFT trajectories and provide RL rewards, and the gold standard, running unit tests in per-repository Docker environments, is expensive to build and infeasible for many real codebases. Dockerless verifies patches without executing them: an agent explores the repository to gather evidence and judges correctness, closing most of the gap to execution-based training.

Motivation Automated environment-building pipelines succeed on only a limited share of repositories, and private, enterprise, and legacy codebases often lack reproducible environments or test suites entirely. Existing environment-free verifiers score patches from surface-level information without inspecting the repository, which fails on tasks where correctness depends on deep context, such as whether a modified function is actually called or whether an alternative implementation integrates with surrounding modules.

Methodology Two stages: given the issue, reference patch, and candidate patch, the verifier generates verification questions (where the fix should take effect, what would confirm correctness, what else could break), then parallel sub-agents answer each with read-only shell tools; a judge conditions on the collected evidence and emits a binary verdict token whose logits give a continuous score. One shared backbone serves question generation, exploration, and judging, trained via rejection sampling on 3.7K issues from SWE-Gym and Multi-SWE-RL, keeping only trajectories whose verdict matches the ground-truth test outcome, with a capped negative-to-positive ratio.

Results Beats the strongest open-source verifier by 14.3 AUC points. As an SFT filter, training on the Dockerless-ranked top 25% of trajectories (4K of 16K) beats training on the full pool by 1.8, 6.4, and 3.4 points on SWE-bench Verified, Multilingual, and Pro; as an RL reward it beats the DeepSWE verifier by 1.4, 2.7, and 1.1 points. The end-to-end environment-free pipeline reaches 62.0%, 50.0%, and 35.2% resolve rates, up 2.4, 8.7, and 2.9 points over the Qwen3.5-9B baseline, matching environment-based post-training.

Plain-language abstract Performance-optimization benchmarks (GSO, SWE-Perf, SWE-fficiency) score coding agents by comparing patched runtime against unoptimized baselines and reference patches, but runtime is noisy and scoring rules are load-bearing. This audit replays every official reference patch across machines, re-derives rankings under the official rules, and checks task-level coverage across public submissions, finding fragile reference signals, rule-dependent rankings, and largely saturated tasks.

Motivation Unlike pass/fail repair benchmarks, performance benchmarks compare noisy runtime measurements that fluctuate with CPU scheduling, cache state, memory bandwidth contention, and microarchitecture. Benchmarks counter with repeated trials, outlier filtering, and statistical tests, but whether the resulting leaderboard scores can be trusted remained untested.

Methodology Replay of 740 official reference patches across four Google Cloud machine types (Cascade Lake, Milan, Emerald Rapids, Turin) over three rounds each, preserving each benchmark's official workloads and validity rules; a scoring-rule audit over eight public submissions shared by GSO and SWE-fficiency, including a bounded-penalty diagnostic; and task-level inspection of 10 public submissions per replay-valid task.

Results Reference patches satisfy their original validity rules in every cross-machine replay for only 39/102 GSO, 11/140 SWE-Perf, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime change. Official rankings disagree on 9 of 28 pairwise submission comparisons; SWE-fficiency's rule gives its ten worst tasks 58.5–82.8% of a submission's score weight, and a bounded-penalty variant changes 6 of 8 submission ranks. Across 450 replay-valid tasks, at least one public submission beats the base program on 449 and matches or beats the reference patch on 384, so the remaining gap is about reaching reference-level speed, not finding any working optimization.

Plain-language abstract NOVA is a system Tencent built to automate architecture changes to its production ad-recommendation models, the kind of structural redesign (new attention modules, feature interactions) that usually needs an expert engineer. It uses an agent to propose changes, but layers verification on top so it can catch a runnable-but-wrong candidate, code that passes tests but breaks a recommender-specific invariant, before wasting a training run on it, and routes the riskiest changes to a human-in-the-loop mode.

Motivation Generic coding agents optimize for code that runs and passes unit tests, but a recommender architecture can be syntactically valid and still be a bad or actively harmful architecture, for example silently dropping sequence masking or degenerating self-attention into a plain MLP. AutoML only tunes hyperparameters, not cross-module structural changes, leaving a gap between 'runs' and 'is architecturally sound.'

Methodology NOVA computes an architecture gradient, an SGD-inspired but non-differentiable update signal aggregating prior modifications, verification diagnostics, metric changes, and trajectory memory, to pick the next modification. A verification cascade checks structure semantics, local executability, offline effectiveness, and online impact before expensive training; failed candidates become reusable forbidden directions. An L1-L4 task-level control scheme routes high-risk changes to a human-supervised Copilot mode. It's deployed in a production advertising system serving over a billion users.

Results On the hardest task tier (L3, literature-to-production), NOVA reaches 86.7% valid-pass rate and 60.0% effective-pass rate, more than double the human expert loop's effective-pass rate, and shortens one literature-to-production cycle by over 13x in human-attended time. In live online A/B testing, the selected candidate improved GMV on three pCVR objectives by +1.25%, +1.70%, and +2.02% while reducing prediction bias by 37.3-66.7%.

Plain-language abstract TTHE (Test-Time Harness Evolution) lets an LLM agent improve itself during evaluation by rewriting its own harness - the executable program around the frozen model that constructs context, invokes tools, verifies intermediate results, and recovers from failures - using only the unlabeled execution traces it produces on the test inputs, with no weight updates and no gold labels.

Motivation An agent's behavior is set as much by its harness as by the underlying model; the same model placed in different harnesses produces substantially different outcomes on the same workload. Yet in most deployments the harness is frozen at test time, tuned once on development data, even though evaluation itself produces rich operational evidence - model calls, tool actions, test results, runtime errors, recovery decisions - that a human harness engineer would inspect to revise the scaffold. Prior adaptation either changes weights (fine-tuning, test-time training), revises a single response (reflection, self-debugging), or searches a workflow before evaluation and then freezes it for testing.

Methodology TTHE treats the executable harness as the state of test-time adaptation. On each unlabeled test batch it maintains a population of candidate harnesses, executes them under instrumentation, and refines them through agentic proposers that reason over the unlabeled execution traces and proxy-indicated weaknesses; a separate judge commits one candidate according to execution-derived signals, and the selected program persists to govern subsequent inputs. The solver, proposers, and judge are instantiated as different roles and harnesses around the same frozen LLM, so all adaptation occurs through changes to the surrounding program; gold labels remain outside the loop and are consulted only after selection, for measurement.

Results Across five execution-grounded domains - text-to-SQL, competitive programming, software engineering, data-science coding, and agentic tool use - TTHE improves fixed ReAct-style baseline harnesses and produces inspectable policies for grounding, verification, and repair. Gains are non-monotonic in search budget, and trace audits attribute the remaining errors to an agentic judge that can commit a plausible but incorrect program under imperfect proxy signals, identifying proxy quality and judge reliability as the central technical bottlenecks.

Plain-language abstract The paper turns a prompt-driven enterprise LLM prototype, an investment-briefing agent, into an auditable application by moving deterministic behavior out of prompts and into code: source manifests, source-backed claims, routing metadata, answer contracts, trace generation, and validators, arranged around a replaceable composition boundary where only phrasing is left to the model. It is instantiated on public data for five Korean corporate groups (25 listed companies, 113 source-backed runtime claims) and evaluated on whether the code-owned contracts hold, survive model substitution, and are load-bearing.

Motivation Enterprise LLM applications often start prompt-dominant, with product behavior carried by natural-language instructions and retrieval context rather than code, data contracts, or validation. Prompts can demonstrate behavior but not guarantee it: productization needs each visible claim traceable to bounded sources, routed to the correct entity, constrained in what it may assert, reproducible, and audited through versioned artifacts, none of which prompts alone enforce.

Methodology The harness relocates control into code: manifests define which sources may be used, source-backed claims define which statements may enter runtime context, routing metadata binds questions to entities, answer contracts define the visible answer, and traces record how each answer was assembled, with the LLM confined to a replaceable composition boundary. Evaluation covers three questions: contract preservation across a fixed validation set with a fault-injection negative control, behavior under three substituted hosted models across 270 composition-boundary runs, and an enforcement-layer ablation that disables the code-owned gate and compares it against a bolt-on external guardrail over 30 adversarial runs (15 recommendation-bait, 15 leak-bait).

Results The contracts held across the fixed validation set, and the fault-injection runs confirmed the validators flag deliberately broken source, routing, trace, answer, and leakage contracts. Under model substitution the enforced checks passed on all 270 composition-boundary runs, with failures confined to the model-composed side and recorded. In the ablation, prompt instructions alone let recommendation-language and trace-leakage violations reach the reader on all 30 adversarial runs, each blocked by the harness; the external guardrail also blocked them but over-refused, with 4 false refusals and 28 of 30 adversarial runs blocked, dropping utility to 88/120 where the harness preserved 120/120 by falling back to a deterministic composer.

Plain-language abstract AgentAbstain is an evaluation framework for agentic abstention: whether a tool-using LLM agent recognizes when not to act. It is a paired-task benchmark of 263 task pairs across 42 executable MCP sandbox environments, where each pair shares a sandbox but differs by a single controlled perturbation that turns a should-act task into a should-abstain variant. An automated pipeline, AbstainGen, synthesizes the environments and paired tasks end to end so fresh instances can be regenerated on demand.

Motivation LLM agents increasingly commit irreversible actions on a user's behalf, booking travel, managing files, running code, and calling APIs, yet evaluations score task success rather than whether an agent refrains under ambiguity, conflicting constraints, or tool failure. Prior abstention work is single-response question answering, where the worst case is a wrong answer and there is no tool-call trace against which to verify restraint. A tool-using agent instead faces a sequential decision grounded in observable environment state, and safety or tool-reliability benchmarks test refusal of malicious goals or correct tool calls, not when a well-intentioned task should be abandoned mid-execution.

Methodology Every instance is a pair sharing one sandbox and differing by exactly one trigger, so no always-act or always-refuse policy can exceed 50% Paired Accuracy. Eight abstention categories are organized by when the trigger becomes observable (pre-execution vs. runtime) and where it resides (query, environment state, or tools). Sandbox tools are typed as lookup (read-only), verify (validation gate), or commit (state-mutating); scoring pairs a deterministic commit-check on the tool-call trace with an LLM judge on the terminal response, and a Conditioned Abstention Rate restricts the abstain score to pairs whose act variant already succeeded. AbstainGen validates generated tasks through deterministic DAG replay and cross-family LLM critics, and three human annotators rated 94 to 98% of a stratified sample as well-designed.

Results Across 17 frontier LLMs in 4 agent harnesses, agentic abstention is unsolved: the best agent, Gemini 3.1 Pro, reaches 59.5% Paired Accuracy and 13 of 17 models stay below 50%, meaning agents systematically prioritize acting over abstaining. Abstention capability is largely independent of general task-solving capability, so scaling task-solving alone will not close the gap. A distinctive agentic failure mode, post-hoc abstention, has agents commit irreversible actions before recognizing the abstention trigger and then claim refusal.

Plain-language abstract Automatic harness evolution improves an LLM agent by iteratively searching over its external scaffold (prompts, tools, memory, verification routines, control logic) using task feedback, then reporting gains on the same benchmark. This paper argues that protocol conflates better harness design with more search, and that sharing the search and evaluation set invites overfitting. Under matched feedback and inference budgets on Terminal-Bench 2.1 with three frontier models, harness evolution does not consistently beat parallel sampling or sequential refinement, and evolved harnesses barely transfer to held-out tasks.

Motivation Prior work shows harness engineering substantially affects agent performance even at a fixed model, which motivated automatic harness evolution. But these methods are themselves an iterative search that repeatedly evaluates and revises candidate harnesses, so they should be compared against test-time scaling baselines that spend the same compute on the evaluation tasks (parallel sampling, sequential refinement, task-level revision) to see whether the gains come from harness design or from search alone. And because the search and the final evaluation share the same public benchmark, reported gains risk reflecting adaptation to specific instances rather than transferable harness design.

Methodology The authors formalize four methods under a unified budget of K=5: parallel sampling, sequential refinement, harness evolution (instantiated as AHE with its explore agent disabled so gains come from evolving the harness rather than retrieving benchmark-specific ones), and a new harness-scaling baseline that adapts the harness to a single evaluation instance. They evaluate on Terminal-Bench 2.1 (89 verified terminal tasks) with Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini, a 128k-token generation budget, high reasoning effort, averaged over two runs, across three settings: without unit tests (model self-selection), with unit tests (oracle selection, reporting pass@1 and pass@5), and a generalization setting with disjoint search and evaluation splits (45 training, 10 validation, 34 held-out test tasks).

Results Without unit tests, harness evolution averages 67.4 pass@1, below the 68.2 direct-sampling baseline and 72.3 for parallel sampling, and drops GPT-5.4 from 75.3 to 69.7, indicating self-generated feedback is too noisy to reliably revise a harness. With unit tests every method improves over direct sampling, but harness evolution still trails: parallel sampling leads pass@1 (86.0 average) and sequential refinement leads pass@5 (91.8 average), so the benefit of harness methods only appears when multiple attempts can be selected among. In the disjoint generalization setting an evolved harness yields +1.2 for Claude Opus 4.6, +0.0 for GPT-5.4, and +0.6 on average on held-out tasks, versus large gains on the tasks it was optimized on, evidence that current harness-evolution methods encode task-specific shortcuts and overfit the search set.

Plain-language abstract RouteLLM is a framework for training router models that decide, on a per-query basis, whether to send a user's question to a large, expensive language model or a smaller, cheaper one. By learning from human preference data and using data augmentation, the routers aim to preserve response quality while cutting the number of times the costly model is actually called.

Motivation Large language models vary enormously in cost: a state-of-the-art model like GPT-4 can cost 60 times more per token than a smaller model like Mixtral-8x7B. Sending every query to the best model is prohibitively expensive, but sending everything to a cheap model degrades quality. Prior routing approaches either relied on synthetic or biased labels, queried multiple models in sequence (increasing latency), or were tied to a fixed set of models, leaving the cost-quality trade-off unsolved in a general, practical way.

Methodology The framework trains a binary router that predicts the probability that a strong model will outperform a weak model for a given query, learned by maximizing the likelihood of human preference labels drawn from the Chatbot Arena dataset. A cost threshold parameter translates this probability into a routing decision. The paper implements and compares several router architectures—including BERT-style classifiers and matrix-factorization approaches—and applies data augmentation to the preference data to boost performance. Routers are evaluated on public benchmarks including MMLU and MT Bench, with both in-distribution and out-of-distribution splits to test generalization.

Results Across benchmarks, the trained routers reduced the number of calls to the strong model (and thus cost) by over 2 times without substantially lowering response quality. The routers also generalized to strong/weak model pairs not seen during training, demonstrating transfer capability. Data augmentation consistently improved performance across all router architectures tested.

Plain-language abstract CascadeDebate is a system that chains together AI language models of different sizes, inserting small teams of debating AI agents at the decision points where a cheaper model would normally hand off to a more expensive one. Instead of blindly escalating uncertain questions to a larger model or a human expert, the system first tries to resolve ambiguity through internal group discussion. A confidence-based router activates this debate step only when a single model's answer is uncertain, and an online optimizer continuously tunes when escalations occur.

Motivation Deployed language model systems must balance accuracy against the high cost of running large models. Existing cascade pipelines route uncertain queries from small models to large ones based on a single model's confidence score, but that score is often poorly calibrated, causing premature and wasteful escalations. Multi-agent deliberation is known to improve reasoning, but existing multi-agent systems operate as standalone setups rather than as embedded components within cost-controlled pipelines, leaving a gap between the two approaches.

Methodology The authors build a four-stage cascade — single-model inference with a small base model, multi-agent deliberation with that same base model, single-model inference with a larger model, and multi-agent deliberation with the larger model — with human experts as a final fallback. Confidence-based routers using Bayesian-calibrated token probabilities and agent-agreement scores control progression between stages. Four specialized role-prompted agents per stage (e.g., Experimental Scientist, Misconception Detector for science tasks) deliberate via majority vote. An online Adam-based threshold optimizer updates escalation thresholds from streaming human feedback. Experiments used Llama-3.2 (1B/3B) and Qwen2.5 (1.5B/3B) instruction-tuned models on 1,000-instance samples from five multiple-choice benchmarks: ARC-Easy, ARC-Challenge, MMLU, MedQA, and MedMCQA, run on a single NVIDIA A100 GPU.

Results CascadeDebate achieved the best accuracy on all five benchmarks for both model families, improving over the strongest single baseline by 1.43 to 18.24 percentage points for Llama-3.2 and yielding especially large gains on medical tasks (MedQA: 86.44% vs. 64.00% for the best single-scale multi-agent baseline). The online threshold optimizer delivered 20.98 to 52.33% relative accuracy improvement over fixed threshold policies. On ARC-Challenge with Llama-3.2, the full cascade raised accuracy from 50.67% to 92.89%, a gain 1.62 times larger than a standard cascade at 15.62 times the single-base-model compute cost.

Plain-language abstract This paper introduces Minions, a system that lets a small language model running on your personal device team up with a powerful cloud-based language model to answer complex questions over large documents — like financial reports, medical records, or scientific papers — while dramatically cutting the cloud computing cost. The key idea is that the local model handles the heavy lifting of reading the full document, while the cloud model directs the work and synthesizes the final answer.

Motivation Querying powerful cloud-hosted language models over large documents is expensive — processing a million-token document with a leading cloud API can cost over $15 per query. At the same time, small models (1–8 billion parameters) now run on ordinary laptops and phones but are mostly used for simple tasks. There was no established method for these two tiers to collaborate effectively on complex, data-intensive reasoning tasks, balancing cost reduction against answer quality.

Methodology The researchers designed and evaluated two communication protocols between a local small model and a remote frontier model (GPT-4o). The first, called Minion, is a simple back-and-forth chat where only the local model reads the full document and relays a compressed summary to the cloud model. The second, called MinionS, has the remote model write code that decomposes the task into many small single-step subtasks (jobs), which the local model executes in parallel over chunks of the document; the remote model then aggregates the results. Experiments were run on three benchmarks covering financial (FinanceBench), medical (LongHealth), and scientific (QASPER) question answering.

Results The naive Minion protocol reduced remote cloud costs by 30.4 times but recovered only 87% of the performance of using the cloud model alone. The improved MinionS protocol, using an 8-billion-parameter local model, recovered 97.9% of frontier model performance while reducing cloud costs by 5.7 times (to about 18% of the original cost). With a smaller 3-billion-parameter local model, MinionS achieved 93.4% of frontier performance at 16.6% of cloud cost. Ablations showed that splitting complex instructions and limiting context length per subtask were critical to bridging the performance gap of smaller local models.

Plain-language abstract Autellix is a new serving engine for running AI agents that make many sequential calls to large language models (LLMs). It treats each multi-step agent program as a first-class unit to be scheduled, rather than managing individual LLM calls in isolation, and uses information about how much work a program has already done to decide which calls to run next.

Motivation Existing LLM serving systems like vLLM schedule individual model calls without any awareness of the broader agent programs those calls belong to. As AI agents grow more complex — involving chains of reasoning steps, tool use, and parallel search — programs end up spending most of their time waiting in queues due to head-of-line blocking, where long calls delay short ones and earlier programs block later arrivals.

Methodology Autellix intercepts LLM calls issued by agent programs and attaches program-level context to each request before it reaches the scheduler. Two scheduling algorithms are proposed: one for single-threaded programs and one for distributed multi-threaded programs. Both are non-clairvoyant — they require only the cumulative service time of a program's previously completed calls, with no prior knowledge of the full execution graph. The system was evaluated across diverse agentic workloads (Chatbot, ReAct, Monte Carlo Tree Search) using models such as LLaMA-3.1-8B on A100 GPUs, compared against vLLM's first-come first-served and multilevel feedback queue baselines.

Results Autellix improves program throughput by 4–15x at the same latency compared to state-of-the-art systems such as vLLM. The scheduling approach reduces cumulative wait times by prioritizing programs that have used less service so far, which also increases LLM engine utilization because faster call completions cause programs to issue subsequent calls more quickly.

Learns a policy over agent-call sequences with a length-regularization penalty and a STOP node; a regularized 3-agent sequence uses ~1/3 of the fixed-baseline calls.

Why it matters Adaptive selection works but is domain-specific. Record route/outcome/cost facts first; optimize selection from data, don't hand-author a routing table.

Formalizes workflow management over a task-dependency graph with workers (capabilities, availability, cost rates), hard/soft constraints, and graph-modifying actions. Reactive managers that over-assign and under-inspect fail.

Why it matters Expose the task graph + resource facts to operators. Good orchestration inspects and decomposes; it doesn't just dump work onto workers.

Low-confidence preprint: decomposes tasks into skill graphs executed by heterogeneous models with rule-augmented orchestration; same-model parallelism alone underperforms heterogeneous execution.

Why it matters Provider heterogeneity is worth evaluating as config/eval metadata. Resist turning 'skills' into a rigid registry — keep model choice in configuration.

Plain-language abstract CARROT is a system for deciding which AI language model to send a given query to, balancing answer quality against cost. Rather than always using the most powerful (and expensive) model, CARROT predicts how well each available model will perform on a query and how much it will cost, then picks the best value option. The authors also release SPROUT, a new dataset of roughly 45,000 prompts run through 14 state-of-the-art models, designed to train and test such routing systems.

Motivation Running every user query through the most capable language model is prohibitively expensive at scale, but existing routing methods either ignore per-query cost variation or restrict choice to just two models (cheap vs. expensive). Prior datasets used to benchmark routers turned out to be too easy, showing no advantage for intelligent routing over random assignment, which obscured whether predictive routing was actually useful.

Methodology The authors conduct a minimax statistical analysis of the routing problem, establishing a theoretical lower bound on how well any router can perform given a training sample, and proving that a plug-in router that estimates both cost and accuracy for each model achieves this optimal rate. CARROT implements this two-stage approach: first train predictors for each model's expected cost and accuracy on a query (using embeddings and k-nearest neighbors or a small transformer), then select the model minimizing a weighted combination of predicted cost and predicted error. The SPROUT dataset was constructed by collecting zero-shot responses and token counts from 14 models across six benchmarks covering reasoning, science, retrieval-augmented generation, and open-ended user queries.

Results On the SPROUT dataset, CARROT matches or exceeds GPT-4o's performance at roughly 30% of GPT-4o's cost across multiple benchmarks. On the Open-LLM-Leaderboard-v2 dataset, CARROT outperforms the single best model (Qwen2-72B) by a large margin by intelligently combining the complementary strengths of models in the pool. The SPROUT dataset itself proved essential: on the older RouterBench dataset, predictive routing offered no measurable benefit over random assignment, whereas on SPROUT both CARROT and simpler cost-aware routers substantially beat the naive baseline, confirming that dataset quality is a key bottleneck for evaluating routers.

Plain-language abstract This paper is a comprehensive survey of LLM Ensemble — the practice of combining multiple large language models to handle user queries, so that each model's individual strengths can be exploited rather than relying on a single model. It is the first systematic review of the field, covering taxonomy, methods, benchmarks, applications, and future directions.

Motivation No single large language model excels at every task: models differ in architecture, parameter size, training data, and cost, and all suffer from accuracy issues, hallucinations, and misalignment with human intent. With over 182,000 models available on Hugging Face, there was a clear opportunity — and no prior survey — to assess how combining multiple models could overcome the weaknesses of any individual one.

Methodology The authors conducted a literature survey, organizing existing LLM Ensemble methods into a three-part taxonomy based on when the ensemble occurs relative to model inference: ensemble-before-inference (routing a query to the best model before any response is generated), ensemble-during-inference (aggregating token-level outputs from multiple models during decoding), and ensemble-after-inference (combining complete responses after generation). They reviewed all relevant methods under each category alongside associated benchmarks and applications.

Results The survey provides the first formal taxonomy and comprehensive review of LLM Ensemble methods, identifying three broad paradigms and their sub-approaches. It documents related benchmarks, real-world applications, and open research challenges, and makes a curated reading list publicly available. The work establishes a structured foundation for future research in multi-model collaboration.

Plain-language abstract RouteNLP is a system that automatically decides which AI language model should handle each incoming query in an enterprise setting, using cheaper smaller models for routine tasks and escalating only harder queries to expensive frontier models. It combines a difficulty-aware query router, statistically calibrated confidence thresholds, and a feedback loop that analyzes failures to improve the cheaper models over time. In an 8-week real-world deployment processing roughly 5,000 queries per day, the system cut inference costs by 58% while keeping 91% of responses accepted and reducing worst-case response latency from nearly 1.9 seconds to under 0.4 seconds.

Motivation Enterprise teams serving NLP workloads with large language models face inference costs that can exceed $200K per month, yet the majority of queries—over 70% in the studied deployment—are routine tasks that do not require frontier model capabilities. Existing routing approaches are typically evaluated on single benchmarks, ignore production constraints such as latency SLAs, and treat the model portfolio as fixed rather than something that can be improved in response to routing failures. RouteNLP addresses this gap by closing the loop between routing decisions and portfolio quality.

Methodology The framework routes queries across a tiered model portfolio using three integrated components: a multi-task difficulty-aware router with shared task-conditioned representations trained on preference data and per-task quality signals; confidence-calibrated cascading that uses conformal risk control to initialize routing thresholds in a distribution-free manner; and a distillation-routing co-optimization loop that clusters escalation failures, applies targeted knowledge distillation to cheaper-tier models, and automatically retrains the router and recalibrates thresholds. The system was evaluated on a six-task benchmark spanning finance, customer service, and legal domains, and validated in an 8-week pilot deployment at an enterprise customer-service division.

Results On the six-task benchmark, RouteNLP achieves 40–85% cost reduction while retaining 96–100% quality on structured tasks and 96–98% on generation tasks; human evaluation confirmed that 74.5% of routed generation outputs match or exceed frontier-model quality. In the live pilot, inference costs fell by 58% and p99 latency dropped from 1,847 ms to 387 ms with 91% response acceptance. The targeted distillation-routing co-optimization loop produced over twice the cost improvement of untargeted distillation at equal data volume (21.7% vs. 9.4%), and benchmark predictions matched pilot outcomes to within a 4-point gap on cost reduction (62% benchmark vs. 58% pilot).

Plain-language abstract The paper treats a coding agent's tool-use scheduling as a Bayesian control problem. The controller keeps a belief b=P(Y=1) that the current code candidate will pass an expensive oracle verifier, treats cheap critics (syntax check, public tests, an LLM judge) as noisy observations that update that belief by Bayes' rule, treats generator calls as stochastic transitions that may fix or break the candidate, and treats the oracle as a costly terminal action. At each step it picks the action — gather more critic evidence, regenerate, verify, or stop — that maximizes expected utility, defined as reward for a correct solution minus accumulated action costs. Two controllers are derived from the resulting POMDP Bellman equation: a one-step greedy controller and a finite-horizon dynamic-programming controller. Across six generators and nine coding benchmarks, Bayesian control wins specifically when verification is costly and critics are informative but imperfect, and the same belief state doubles as an uncertainty score that beats token-probability and tool-success baselines.

Motivation Modern coding agents wrap an LLM generator with tools that differ widely in cost and reliability — free syntax checks, public tests, LLM reviewers, and a high-fidelity oracle verifier. The orchestrators that decide when to refine, discard, or verify a candidate typically use fixed rules: always verify, best-of-N, single-critic gates, or predefined generate-critique-regenerate-verify loops. These rules ignore uncertainty: they maintain no posterior over candidate correctness and never explicitly weigh the value of a critic call against its cost, so they cannot adapt stopping to task difficulty, critic reliability, generator repair probability, or verifier cost. The engineering problem is to produce correct code while efficiently allocating limited, unequal-cost resources, and a static confidence threshold collapses that decision into a single scalar boundary.

Methodology Code-generation control is formalized as cost-sensitive sequential hypothesis testing over the latent correctness Y, encoded as a POMDP whose state is a belief b=P(Y=1 | history). A generator call propagates the belief through a 2x2 transition kernel parameterized by an empirical fix probability and break probability. A critic call updates b by Bayes' rule using per-critic likelihoods, each at a small cost. The oracle verifier returns the true label at the highest cost Cver and ends the episode. The Bellman equation V(b)=max(Qgen, max_i Qcrit_i, Qver) defines action values that each subtract the action's cost. Two controllers are derived: bayesian_greedy does one-step lookahead and resets belief to the prior after a generator call, so it cannot value multi-step chains; bayesian_DP runs backward induction over (belief, remaining-depth) on a 51-point belief grid with horizon H=3, using the measured transition kernel so it can value multi-step critic-refine-verify chains. All controllers run on top of frozen LLMs — optimization is entirely at the control layer. Prior pass rate, critic likelihoods, and fix/break transitions are estimated from held-out calibration trajectories. Cost vectors come from deployment telemetry (latency, tokens, API cost), instantiated as a slow-oracle regime (Cver=90, syntax 1, public-test 2, LLM critic 5, Cgen=10) and a fast-oracle regime (Cver=5), with R=100. Baselines are eight policies — always_verify (reference), best_of_N, three single-critic gates, a fixed AND-gate pipeline, plus Self-Refine and Reflexion — compared by mean utility minus always_verify with paired bootstrap CIs.

Results Pooling 7,020 sweep points across all 54 (benchmark, generator) pairs and the cost grid into a decision map over the (P(Y=1), Cver/R) plane yields three regimes. Regime A (Bayesian wins): when Cver/R is at least ~1, paying Cver on every patch is unprofitable, so the Bayesian controllers verify only when expected value of information exceeds Cver; bayesian_DP pulls ahead of bayesian_greedy by deferring verification across refinement rounds. Regime B (public-test gate wins): at moderate cost on cells where the public-test critic is near-oracle, a single PASS moves the posterior past threshold, so the gate ties or marginally beats the Bayesian controllers by avoiding inference overhead. Regime C (always_verify wins): when Cver/R is small or the prior is already high, any critic's cost approaches Cver, so blind verification dominates. On SWE-Bench Lite / claude-haiku-4.5 (slow oracle, regime A), both Bayesian controllers are effectively tied for best and substantially beat always_verify, while best-of-3, Self-Refine, and Reflexion all underperform always_verify. Critic informativeness drives the gains: the syntax critic is near-useless, the LLM critic is dominant on some benchmarks and weak on others, and Bayesian control wins when no single critic is near-oracle and it can compose several moderately informative ones into a sharper posterior. As a post-hoc uncertainty score, the Bayes belief state achieves the best average prediction-rejection ratio of 0.866, beating tool-success-rate 0.795, sequence-probability 0.801, and perplexity 0.367.

Plain-language abstract This study asks whether the test-running that code-repair agents do — the 'generate, run tests, revise' loop — is actually worth its cost. The authors analyze 7,745 public SWE-bench agent traces and then run 3,000 controlled repair attempts with three agents (Claude Code, Codex, and open-source OpenCode), turning execution on and off. Forbidding the agent from running tests barely changes how many bugs get fixed while cutting token and time costs by roughly half, so execution should be treated as a resource spent deliberately, not a default.

Motivation LLM repair agents have standardized on a 'generate-run-revise' loop that executes tests to validate and refine patches, but running code is expensive: it consumes tokens (generating commands, parsing verbose output), adds latency (a full test suite can take minutes to hours), and forces teams to maintain a working test environment — a Docker image with the right dependencies — for every target repository. Prior work studied model architectures, prompts, or search but treated execution as a necessary implicit component; even Agentless removed the loop and execution together, so execution's marginal contribution was never isolated.

Methodology A two-stage empirical study. Stage one characterizes execution behavior by analyzing 7,745 SWE-bench leaderboard traces spanning four execution-based agents, twelve LLMs, and the Lite and Verified datasets. Stage two isolates execution as a single controlled variable: the agent scaffold is held fixed (Claude Code with Claude-Sonnet-4.5, Codex with GPT-5.2-xhigh, OpenCode with Qwen2.5-Coder-32B) while execution access is varied across four paradigms over 3,000 end-to-end attempts on 200 SWE-bench instances, enforced at both the prompt and tool level, yielding a controlled measurement of execution's marginal value and cost.

Results Agents execute constantly — an average of 8.8 test runs per task, ranging from 2 to 19, with late-stage executions (66–100% of the conversation) more successful than early ones. Yet restricting execution barely hurts: the resolve-rate gap between Prohibited and Unrestricted is 1.25 percentage points on commercial agents (not significant, p>0.05) and roughly zero for OpenCode, while Prohibited saves 56–62% of tokens and 48–54% of wall-clock on Claude Code and removes per-repository test-environment maintenance. The benefit is concentrated rather than uniform: 54–66% of commercial-agent cases resolve in a single edit, localization accuracy without execution stays above 95%, and 81–100% of failed cases pass the agent's own executed validation but fail the official evaluation; OpenCode instead over-retries with only 11% of failed cases passing self-validation. The authors conclude execution should be treated as a resource with an explicit cost-benefit tradeoff, not a default capability.

Plain-language abstract Users can reach many LLMs, and the strongest model is not the best one for every coding task, so routing each task to the right model matters for both quality and cost. Existing routers treat this as a one-off classification and fall well short of a per-task oracle that always picks the best model. Agent-as-a-Router instead formalizes routing as a Context→Action→Feedback loop that verifies each decision by running the chosen model's output in a sandbox and accumulates that experience over the task stream. The framework is instantiated as ACRouter and evaluated on CodeRouterBench, an execution-verified environment of about 10K coding tasks scored across 8 frontier LLMs.

Motivation A zero-shot LLM router, even on a strong model like Claude Sonnet 4.6, lags the per-task oracle by a wide margin. An ablation shows the gap is information deficit rather than weak reasoning: adding per-dimension performance statistics to the router yields a 15.3% relative AvgPerf gain (41.41→47.74) and beats a heuristic encoding the same priors. Static routers cannot close this gap because their information state is frozen, which motivates a router that gathers execution-grounded signal during deployment and conditions future decisions on it.

Methodology Routing is cast as a contextual-bandit C-A-F loop: at each task the router observes context (prompt, optional metadata, and memory accumulated from prior loops), selects a model, and receives verifier feedback — a sandbox-observed score in [0,1] plus monetary cost computed from token consumption — which is memorized for the next decision; cumulative regret against a per-task oracle is the streaming metric. ACRouter realizes this with an Orchestrator (a fine-tuned Qwen3.5-0.8B policy combined with heuristic rules and top-10 kNN historical neighbors via weighted voting), a Verifier (AST parsing plus sandbox execution aggregated into a unified per-task score), and a Memory module. CodeRouterBench supplies ~10K tasks across 9 in-distribution coding dimensions plus an out-of-distribution agentic-programming testbed, with execution-verified scores from 8 frontier LLMs.

Results ACRouter attains the lowest cumulative regret (205.5) and highest AvgPerf (49.98%) among routers on the in-distribution stream, beating DimensionBest — which has a full dimension-level prior — by 2.48% AvgPerf, and reaches 62.50% AvgPerf on the out-of-distribution agentic-programming split. It is also more cost-efficient than always invoking the strongest model: Perf/$ of 3.79 (ID) and 1.18 (OOD) versus Always-Opus at 1.29 and 0.64. Lightweight static learners post fair in-distribution scores but fail to generalize out of distribution.

Plain-language abstract This paper deploys a production LLM agent that, instead of writing fresh code for every request, compiles the repeated steps of a standard operating procedure (SOP) into validated, versioned tools ahead of time; at runtime the agent calls those tools directly and falls back to code generation only when a tool is unavailable or fails. It triages alarms in an Amazon fulfillment-center outbound dock against a 44-node SOP over heterogeneous metric backends.

Motivation In the prevailing CodeAct-style paradigm the agent generates and executes fresh code for each request, so when the same workflow repeats against a stable backend it re-interprets the same instruction, rediscovers the same schema, and regenerates similar code, raising latency, cost, and run-to-run variance. In this setting most latency and correctness errors come from translating underspecified SOP text into a concrete query against a production metric backend, which motivates self-evolving agents that build reusable tools before they are needed in production.

Methodology Offline, each SOP node is compiled into a tool: a data-collector sub-agent runs against the live environment via MCP, producing a trace of query code, backend responses, observed schema (field names, datatypes, value ranges), and a graded verdict; a tool-maker LLM writes a candidate tool conditioned on the SOP node text, the node's position in the decision tree, and that trace; and a reflector-tool-maker loop repairs the candidate against labeled cases. Online, the production agent invokes these compiled tools inline, one call per node, falling back to code generation only when a tool is unavailable or fails.

Results In production, tool calls reduce p50 latency by 42%; on 1,500 historical alarms they reduce end-to-end error rate by up to 53% by suppressing run-to-run variance in repeated steps. Because tools return compact structured verdicts, a simpler direct-call architecture reduces p50 latency by a further 62% in a controlled ablation. Both the data-collection trace and the test-repair loop are necessary, residual errors concentrate on underspecified SOP steps where targeted fixes raise pass@1 from 94.5% to 99.9%, and versioned tools improve auditability while exposing specification gaps and upstream data drift.

Plain-language abstract LAMaS is an orchestration framework for LLM multi-agent systems that treats end-to-end latency as a first-class objective rather than a side effect of cost. It learns execution graphs under a constrained objective that minimizes latency and cost subject to an accuracy floor, attributes the latency penalty to operators in proportion to how close they sit to the critical path, and adds a small learned controller that drops redundant remaining agent calls at run time once the answer has stabilized.

Motivation Automated multi-agent orchestration has optimized task performance and inference cost, but has largely left latency alone, and state-of-the-art agent systems already take 10-60 minutes per task in computer-use and autonomous-research settings. Latency will not follow from cost reduction: cost sums over every operator in the graph while end-to-end time is the longest source-to-sink path, so operators off the critical path do not affect wall-clock time at all. That makes naive latency optimization fail in two ways, mis-assigning operator-level credit under a uniform penalty, and degrading task accuracy unpredictably when nothing holds accuracy down. A graph committed at training time also cannot use evidence that only appears as execution unfolds.

Methodology The orchestrator is a policy network over an agentic supernet, a probabilistic DAG over candidate operators, sampling operators per step until cumulative probability exceeds a threshold tau so graph width and depth adapt per query. Training minimizes E[lambda_t*T + lambda_c*C] subject to E[S] >= S0, relaxed to reward R = lambda_S*S/S0 - lambda_t*T - lambda_c*C with the dual variable lambda_S updated by projected gradient ascent, rising when batch accuracy falls below the floor and decaying when it is met. S0 is set to the validation accuracy of the same orchestrator trained at lambda_t = 0, making the constraint a do-no-harm floor. Each operator's share of the latency penalty is scaled by path-criticality w(o) = l(o)/T, where l(o) is the longest path through o, obtained with the critical path in a single forward/backward pass. The orchestrator trains by policy gradient with EMA reward normalization. After it converges, a small MLP controller trains by MSE regression on frozen-orchestrator traces to predict the control advantage delta_k = R_k - R_N from completion fraction, output agreement, and a compressed query embedding; at inference the predicted advantage becomes a Bernoulli retain/eliminate decision at each operator completion. Evaluation uses gpt-4o-mini-0718 on GSM8K, HumanEval, MATH and MMLU-Pro against Generate, CoT, SC, GDesigner, AgentDropout and MaAS, with latency measured as wall-clock time under one API provider and runtime, no response caching, averaged over 3 runs.

Results LAMaS cuts end-to-end latency 55.6-75.6% against the strongest learning-based MAS baselines while matching or beating accuracy. On GSM8K it reaches 93.65% at 11.73s and 622 critical-path tokens against MaAS at 93.36%, 48.11s and 2501; on HumanEval 95.42% at 21.00s against MaAS 93.38% at 47.32s; on MMLU-Pro 66.27% at 14.81s against MaAS 65.40% at 49.12s. Non-MAS prompting (Generate, CoT, SC) stays faster but consistently loses accuracy, most visibly on harder benchmarks. Ablations isolate each component: removing the inference controller holds accuracy within 0.38 points but adds 16.5-21.1% latency and 16.0-25.2% cost; removing critical-path credit degrades all three metrics; setting lambda_t = 0 leaves accuracy within 0.25 points while inflating latency 24-70% and cost 19-35%. A bare uniform latency term with no accuracy floor, no path-criticality weighting and no controller loses 1.30-5.34 accuracy points for only 2-11% latency savings over lambda_t = 0 and no cost reduction. The pipeline transfers to other learning-based orchestrators: GDesigner+LAMaS cuts GSM8K latency from 32.31s to 20.55s and cost from 17.86e-4 to 12.08e-4 USD at 93.27% vs 93.30% accuracy, with a comparable effect on AgentDropout.

Plain-language abstract Production LLM deployments combine two cost-reduction primitives: prompt caching, which charges a discounted rate for reused token prefixes, and prompt compression, which sends fewer tokens. The dominant query-aware compression methods rewrite the compressed prefix on every query, which silently invalidates the cache. The paper measures Anthropic's Sonnet 4.6 cache, finds a two-tier architecture with a sharp threshold near 3,500 tokens, and proposes Cache-Aware Prompt Compression (CAPC): query-agnostic compression plus explicit cache control, with a bound on the compression ratio that keeps the cached prefix out of the unreliable tier. CAPC is the cheapest strategy in every configuration tested.

Motivation The prompt-compression literature implicitly assumes a perfect cache hit rate and treats caching and compression as separable, but in production they conflict: a query-aware compressor makes every call a cache miss, paying full input rates plus repeated cache writes, so its tuned savings can be negative. Nobody had measured real cache behavior or modeled the joint cost.

Methodology Empirical characterization of Sonnet 4.6's prompt cache over 30-call sessions (three independent trials) establishes the two-tier structure: a hot tier below roughly 3,500 tokens where the hit rate plateaus at 0.83, and a persistent replicated tier above it near 1.0. A cost model parameterized by the measured hit rate subsumes the literature's ideal model and predicts a crossover at compression ratio r >= 6, validated on real API spend. CAPC pairs query-agnostic compression with explicit cache_control and a tier-preserving ratio bound rmax = floor(P/3500) so over-compression cannot push the cached prefix into the hot tier. Validation covers 16 document-size by ratio configurations on LongBench-v2, a production enterprise tool-using assistant with a 94k-token tools schema prefix, knowledge-graph RAG over the FastAPI and httpx codebases, and tau-bench retail with a deterministic database-state reward. Total API spend across all experiments was $98.96.

Results CAPC is cheapest in 16 of 16 LongBench-v2 configurations, with mean savings of 49% over cache-only (range 24-67%), 64% over query-aware compression, and 90% over vanilla, at quality within 0.05 of the uncompressed baseline. The enterprise assistant sees a 51.7% cost reduction on its 94k-token prefix. On knowledge-graph RAG, CAPC delivers 9.3x cost reduction on FastAPI and 2.4x on httpx versus cache-all at a stable 85% hit rate, with quality lift proportional to what the model does not already know: +0.8 points over vanilla on the familiar FastAPI, +142% on the less-familiar httpx. On tau-bench retail CAPC is cheapest of four strategies with task reward exactly equal to vanilla (36/50), while query-aware compression is the most expensive at +40.1% over vanilla, confirming the cost model's negative-ROI prediction on a public benchmark.

Plain-language abstract ReAct is a prompting framework for large language models (LLMs) that interleaves verbal reasoning traces with concrete actions in a single generation loop. Instead of reasoning alone (chain-of-thought) or acting alone (action-plan generation), ReAct lets a model think through a step, take an action such as querying a knowledge base, observe the result, and then continue reasoning — all within one prompted sequence. The paper was published as a conference paper at ICLR 2023.

Motivation Prior work on LLMs treated reasoning (chain-of-thought prompting) and acting (action-plan generation for interactive environments) as separate capabilities. Chain-of-thought reasoning is static and grounded only in the model's internal representations, making it prone to hallucination and error propagation. Action-focused approaches lacked high-level verbal reasoning and working memory. No prior work had systematically studied how combining the two in a synergistic, interleaved manner could benefit general task solving.

Methodology ReAct prompts an LLM to generate both reasoning traces and task-specific actions in an interleaved fashion. It was evaluated on four benchmarks: two knowledge-intensive language tasks — multi-hop question answering (HotpotQA) and fact verification (Fever), where the model calls a Wikipedia API for external information — and two interactive decision-making benchmarks (ALFWorld and WebShop). The approach uses few-shot in-context prompting, with only one or two in-context examples provided, and is compared against chain-of-thought (reason-only), act-only, imitation learning, and reinforcement learning baselines.

Results On HotpotQA and Fever, ReAct reduced hallucination and error propagation compared to chain-of-thought baselines by grounding reasoning in retrieved Wikipedia facts, while also producing more human-interpretable task-solving trajectories. On the interactive decision-making benchmarks ALFWorld and WebShop, ReAct outperformed imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, using only one or two in-context examples.

Plain-language abstract AutoGen is an open-source framework that lets developers build applications powered by large language models (LLMs) by having multiple software agents converse with each other to complete tasks. Each agent can be backed by an LLM, a human, a tool, or some combination, and developers can define how agents interact using natural language or Python code. The framework was evaluated across applications spanning mathematics, coding, question answering, operations research, online decision-making, and entertainment.

Motivation Building capable LLM-based applications is difficult because it requires coordinating multiple capabilities — reasoning, tool use, human feedback, and code execution — in ways that existing single-agent or ad hoc multi-agent approaches do not support cleanly. Prior work showed that using multiple cooperating agents can improve factuality, reasoning, and validation, but there was no general framework that made it easy to define, customize, and coordinate such agents across diverse domains and complexity levels.

Methodology AutoGen introduces two core abstractions: conversable agents and conversation programming. Conversable agents are entities that send and receive messages and can be backed by LLMs, human input, or tool execution; developers configure them by composing built-in capabilities or extending base classes such as AssistantAgent and UserProxyAgent. Conversation programming lets developers specify agent interaction patterns — including static flows, dynamic group chats, and human-in-the-loop rounds — via natural language prompts or Python code. The framework was applied to six demonstration applications and evaluated on benchmarks including the MATH dataset, using 120 randomly selected level-5 problems and the full test set.

Results On the MATH benchmark, AutoGen's built-in two-agent setup outperformed alternative approaches including Multi-Agent Debate, LangChain ReAct, vanilla GPT-4, and commercial products such as ChatGPT with Code Interpreter and the Wolfram Alpha plugin. Across all six demonstration applications, the framework enabled high-performing and in some cases novel interaction patterns — such as multi-human-in-the-loop problem solving and conversational chess — while reducing the development effort required to build each application.

Consensus-topology ablations: visible authorship raises self-voting and ties; visible live vote tallies raise herding and premature consensus.

Why it matters If you run review/debate quorums, blind the lanes and hide interim tallies by default — visibility measurably biases the consensus toward the wrong answer.

Plain-language abstract TraceFix is a system that automatically designs, verifies, and repairs coordination protocols for groups of AI agents working together. When multiple AI agents must share resources, pass messages, and coordinate actions concurrently, subtle bugs like deadlocks can emerge. TraceFix uses formal verification—exhaustive model checking with TLA+—to find these bugs before deployment and iteratively fixes the protocol until no violations can be found, then enforces the verified protocol at runtime.

Motivation As large language model (LLM) agents increasingly run concurrently and share external state, coordination failures—races, deadlocks, missed handshakes, and premature termination—become the dominant source of system failures rather than individual agent capability. These bugs depend on rare execution schedules and can remain hidden through all observed runs yet appear under untested interleavings. Prior approaches address coordination through orchestration frameworks or runtime guards but do not verify that the concurrent protocol itself is free of such hazards.

Methodology TraceFix operates as a four-stage pipeline: an orchestration agent synthesizes a protocol topology (a structured intermediate representation defining agents, shared locks, and directed message channels) from a natural-language task description; the topology is compiled into PlusCal coordination logic; the TLA+ model checker (TLC) exhaustively searches for counterexample traces under bounded assumptions; and a repair agent revises the PlusCal source based on the counterexample until TLC finds no further violations. Verified process bodies are then compiled into per-agent system prompts, and a runtime monitor rejects any coordination operations outside the verified topology. The approach was evaluated on a benchmark of 48 tasks spanning 16 scenario families at three difficulty tiers, with a 3,456-run runtime comparison across four architectures and two model capability tiers.

Results All 48 benchmark tasks reached full TLC verification; 62.5% passed on the first attempt and none required more than four repair iterations. Bounded model checking remained tractable across all tasks (median under 1 second, maximum under 60 seconds) even for state spaces reaching millions of distinct states. Topology-monitored execution achieved the highest task completion rates (89.4% average, 81.5% full completion) and degraded at roughly half the rate of prompt-only and chat-only baselines when model capability was reduced. A paired ablation showed that TLC-verified protocols cut deadlock and livelock occurrences from 31.1% to 14.1%, with the largest separation under fault injection conditions.

Plain-language abstract AiScientist is an AI system designed to autonomously carry out end-to-end machine learning research engineering — from reading a paper specification to implementing, running, and iteratively improving experiments — over hours or days without human intervention. It combines a hierarchical team of specialized agents with a shared file-based workspace that preserves project state across all stages, so later decisions stay coherent with earlier ones.

Motivation Existing AI research agents can handle narrow subtasks like idea generation or code synthesis, but consistently fail when tasks span many coupled stages over long time horizons. On PaperBench, the best prior agent achieved only 21% of the replication rubric, compared to 41% by expert PhD students, exposing a gap between local reasoning ability and the sustained, stateful coordination that real ML engineering demands.

Methodology AiScientist uses a top-level Orchestrator that maintains stage-level control through concise summaries and a workspace map, delegating to specialized agents for paper comprehension, task prioritization, implementation, and experimentation. Shared project state is stored as durable file artifacts — analyses, plans, code, and experimental logs — in a permission-scoped 'File-as-Bus' workspace, so agents re-ground on files rather than relying on conversational context handoffs. The system was evaluated on PaperBench, a benchmark for reproducing ML research papers from scratch, and MLE-Bench Lite, a competition-style ML optimization benchmark.

Results AiScientist improved PaperBench score by 10.54 points on average over the best matched baseline and achieved 81.82% Any Medal on MLE-Bench Lite. In one illustrative run on the Detecting Insults task, the system ran 74 experiment cycles autonomously over 23 hours, raising validation AUC from 0.903 to 0.982. Ablation studies showed that removing the File-as-Bus protocol reduced PaperBench by 6.41 points and MLE-Bench Lite by 31.82 points, identifying durable shared state as the key performance driver.

Plain-language abstract Shepherd is a Python runtime framework that treats an AI agent's execution as a first-class object that higher-order "meta-agents" can inspect, fork, replay, and modify in real time. It introduces a Git-like execution trace where every model call, tool call, and environment change becomes a structured, replayable event, letting one agent supervise, optimize, or train another without bespoke plumbing.

Motivation As LLM-based agent systems tackle more complex tasks, they increasingly rely on meta-agents that act on other agents at runtime — for example, to prevent conflicts, fix failed runs, or improve training. Existing agentic substrates expose only plain transcripts and environment snapshots, forcing each meta-agent implementation to reinvent custom tooling to reconstruct and orchestrate execution state. Shepherd was built to close this gap by giving meta-agents a principled, unified interface over agentic execution.

Methodology Shepherd is grounded in functional programming principles: agents are typed tasks, and their execution is recorded in a Git-like trace where every action becomes a commit, every fork is a branch, and any past agent-environment state can be checked out and replayed. The framework is instantiated as a Python substrate; its core operations are formalized through an algebraic-effects calculus mechanized in Lean to provide precise semantic guarantees. The paper demonstrates the substrate through three concrete meta-agent use cases spanning live supervision, post-hoc counterfactual optimization, and tree-search reinforcement learning.

Results A live supervisor meta-agent using Shepherd raised CooperBench joint pass rate from 28.8% to 54.7% by intervening before parallel coding agents conflicted. A counterfactual replay meta-optimizer outperformed MetaHarness by up to 11 points on LiveCodeBench and TerminalBench-2 while cutting wall-clock time by up to 58%. A tree-search RL trainer using Shepherd-chosen fork points improved Qwen3.5-35B-A3B's avg@5 score on TerminalBench-2 by 5.2 points over GRPO. The substrate forks a 5.8 GB agent-environment state 5 times faster than a Docker commit and reuses over 95% of the LLM provider's KV cache.

Formalizes workflow management over a task-dependency graph with workers (capabilities, availability, cost rates), hard/soft constraints, and graph-modifying actions. Reactive managers that over-assign and under-inspect fail.

Why it matters Expose the task graph + resource facts to operators. Good orchestration inspects and decomposes; it doesn't just dump work onto workers.

Plain-language abstract DynTaskMAS is a software framework for running teams of AI agents—each powered by a large language model—on complex tasks in parallel and without waiting for one another. It breaks a hard problem into subtasks automatically, tracks which subtasks depend on which others, and runs as many as possible at the same time, so the overall job finishes faster and the AI models stay busier.

Motivation Existing multi-agent systems built on large language models typically use simple, fixed workflows where agents work one at a time or in rigid sequences. This leaves computing resources underused and makes it hard to tackle tasks whose structure changes as work progresses, creating a need for frameworks that can decompose and schedule work dynamically.

Methodology The framework is built around four integrated components: a Dynamic Task Graph Generator that decomposes an input task into subtasks while preserving their logical dependencies; an Asynchronous Parallel Execution Engine that schedules those subtasks for concurrent execution; a Semantic-Aware Context Management System that shares relevant information among agents efficiently; and an Adaptive Workflow Manager that adjusts execution in real time. The work was presented as a conference paper at the 35th International Conference on Automated Planning and Scheduling, and performance was evaluated experimentally across varying task complexities and agent counts.

Results DynTaskMAS achieved a 21–33% reduction in execution time compared to traditional approaches, with larger gains for more complex tasks. Resource utilization improved from 65% to 88%, a 35.4% increase. Throughput scaled near-linearly up to 16 concurrent agents, reaching a 3.47× improvement when using 4× the agents.

Plain-language abstract This paper introduces APWA (Agent-Parallel Workload Architecture), a distributed system for running large groups of AI agents in parallel on complex tasks. Rather than having agents coordinate through a central bottleneck, APWA breaks a task into non-overlapping subtasks that independent agents solve simultaneously, drawing on distributed computing infrastructure. The system is general-purpose and was tested on tasks including redacting sensitive personal information from large document sets, structured data extraction, and hierarchically summarizing long literary texts.

Motivation Existing multi-agent AI systems struggle when tasks involve large volumes of data or many independent subtasks: they rely on a central orchestrator that can only handle messages one at a time, which prevents true parallelism and causes complete failure as task size grows. No prior system provided automated workload partitioning over distributed infrastructure that could scale to thousands of parallel agents without coordination bottlenecks.

Methodology APWA uses a Task Manager agent that dynamically decomposes a user query into non-interfering subtasks using novel programming abstractions for distributed data tables, parallel planning, and subtask delegation. Worker agents execute subtasks in parallel via a Ray-based distributed backend with no cross-communication required during execution. The system was evaluated on three benchmarks: a PII redaction task (AI4Privacy PII-300k dataset at 64, 512, and 4096 records), a structured JSON extraction task across heterogeneous document formats (SchemaBench), and a hierarchical summarization task on three literary corpora ranging from 166 kB to 10.5 MB, compared against direct LLM calls, Magentic-One, and MegaAgent baselines.

Results APWA achieved a 0% failure rate across all benchmark configurations, while Magentic-One failed 100% of the time on the larger summarization and PII tasks and MegaAgent failed 60–100% of the time on most settings. On the hierarchical summarization benchmark, APWA completed tasks in 157 s, 210 s, and 329 s for the three corpora while baselines either failed entirely or took comparable time only on the smallest task. On PII redaction, APWA achieved structural scores of 1.000 and semantic F1 scores up to 0.772 at intermediate scale (512 records), whereas baselines either failed or produced near-zero semantic scores. The topic-research task completed 10, 20, and 100 topics in 143 s, 157 s, and 595 s respectively.

Plain-language abstract This paper presents OneManCompany (OMC), an open-source framework that organizes teams of AI agents like a real company — with hiring, role assignment, task management, performance reviews, and continuous improvement. Rather than building yet another chatbot or single-agent tool, it creates an organizational layer so that many different AI agents can work together on complex, open-ended projects without being pre-configured for each one.

Motivation Existing multi-agent AI systems are brittle: they hardcode team structures before a project begins, lock all agents into the same software runtime, and lose any lessons learned when a session ends. There is no principled way to recruit new specialists mid-project, coordinate agents from different software families, or have the organization improve itself over time — the same gaps that would make a real company dysfunctional.

Methodology OMC introduces three core mechanisms. First, a Talent-Container architecture packages each agent's persona, skills, and tools into a portable 'Talent' that can run on any supported backend (LangGraph, Claude CLI, or script processes), with a community-driven Talent Market for on-demand recruitment. Second, an Explore-Execute-Review (E2R) tree search decomposes tasks top-down into a dependency-tracked DAG, enforces bottom-up completion propagation with formal termination and deadlock-freedom guarantees, and gates each subtask result behind supervisor review before downstream tasks proceed. Third, a self-evolution layer lets individual agents update their working principles after each task and manager feedback session, while organization-level standard operating procedures and culture rules accumulate across projects. The system was evaluated on PRDBench, a benchmark of 50 project-level software development tasks across 20+ domains, using a founding LangGraph agent plus three Claude Code-based specialists recruited from the Talent Market.

Results OMC achieved an 84.67% success rate on PRDBench, surpassing the previous state of the art by 15.48 percentage points, at an average cost of approximately $6.91 per task ($345.59 total across 50 tasks). Cross-domain case studies — including autonomous content generation (completed in under 10 minutes for ~$4.48), iterative game development with human-in-the-loop feedback, cross-modal audiobook production, and a research survey that generated 17 structured documents and three novel research proposals in under one hour for $16.26 — demonstrated that the same framework generalizes across heterogeneous agent backends and task types without any domain-specific reconfiguration.

Plain-language abstract This paper proposes a two-dimensional classification system for AI agent design patterns, combining what an agent does (cognitive function) with how it is structurally organized (execution topology). The resulting 7-by-6 matrix identifies 27 named patterns and provides a shared vocabulary for describing, comparing, and designing large-language-model-based agent systems.

Motivation Existing frameworks from major AI organizations each describe agent architectures from only one perspective: industry guides focus on how data flows (execution topology) while cognitive science surveys focus on what agents do (cognitive function). This single-axis view fails to distinguish architecturally distinct systems — for example, the same Orchestrator-Workers topology can implement task planning, hierarchical delegation, or observability monitoring, three patterns with fundamentally different failure modes — leaving architects without a principled, framework-neutral vocabulary.

Methodology The authors define two independent axes: a Cognitive Function axis with seven categories (Context Engineering, Memory, Reasoning, Action, Reflection, Collaboration, Governance) and an Execution Topology axis with six structural archetypes (Chain, Route, Parallel, Orchestrate, Loop, Hierarchy). They take the Cartesian product to produce a 42-cell matrix, populate 27 cells with named patterns (13 with original names), demonstrate that the two axes are orthogonal through systematic cross-axis analysis, and validate the framework's coverage by mapping four real-world deployment domains — financial lending, legal due diligence, network operations, and healthcare triage — onto the pattern catalog.

Results The framework identifies 27 named patterns across the 7-by-6 matrix, with 15 cells either structurally redundant or not yet observed in practice. Cross-domain analysis yields five empirical laws of pattern selection: time pressure favors Chain-of-Thought and Loop patterns; action authority favors Blast Radius Control; asymmetric failure costs shape how Reflection is parameterized; volume determines which collaboration patterns are needed; and the same pattern (such as Generator-Critic) appears across all four domains but with domain-specific parameterization. Four patterns — Context Triage, RAG Pipeline, Complexity-Based Routing, and Generator-Critic — appear in three or more domains, suggesting they are foundational to most production agent systems.

Plain-language abstract This paper introduces MPAC (Multi-Principal Agent Coordination Protocol), a new communication standard that lets AI agents owned by different people or organizations work together on shared tasks without a central authority overseeing them. Today's agent protocols assume one person or organization controls all the agents involved, which breaks down when independent parties — like two engineers whose coding agents both edit the same file — need to coordinate. MPAC defines how agents declare intentions before acting, handle conflicts explicitly, and escalate disagreements to their human owners when needed.

Motivation Existing AI agent protocols (MCP and A2A) both assume a single controlling principal who owns and trusts every participating agent, but many real coordination problems involve independent principals whose agents must work on shared state — two engineers' coding agents editing the same repository, family members' agents planning a trip together, or agents from different organizations drafting a joint contract. When this single-principal assumption breaks down, coordination collapses to ad-hoc chat, manual merging, or silent overwrites. No existing protocol addressed this multi-principal coordination gap.

Methodology MPAC is specified as a five-layer application-layer protocol (Session, Intent, Operation, Conflict, and Governance layers) with 21 message types, three state machines with normative transition tables, Lamport-clock watermarking for causal ordering, two execution models (pre-commit and post-commit), three security profiles, and an optimistic-concurrency-control mechanism for shared state. The authors released two interoperable reference implementations (Python with 122 tests; TypeScript with 101 tests), 66 adversarial enforcement tests, a machine-readable JSON Schema suite covering all 21 message types, and seven live multi-agent demos spanning scenarios such as code editing, trip planning, pre-commit authorization with fault recovery, and multi-level conflict escalation.

Results A controlled three-agent cross-module code review benchmark showed a 95% reduction in coordination overhead (from 68.65 seconds to 3.02 seconds) and a 4.8x wall-clock speedup (from 131.76 seconds to 27.38 seconds) under MPAC compared to a serialized human-mediated baseline. Per-agent decision time was preserved (63.11 seconds versus 57.13 seconds), confirming that the speedup comes from eliminating coordination waits rather than compressing model inference. The full specification, reference implementations, test suites, and demo transcripts were released as open source.

Plain-language abstract This paper introduces DiffMAS, a training framework that lets multiple AI language model agents learn how to communicate with each other through internal representations rather than plain text. Instead of passing messages as words, agents share compressed internal states called key-value caches, which can be jointly optimized during training. The result is a multi-agent system where the communication channel itself improves alongside the agents' reasoning abilities.

Motivation Most multi-agent AI systems built on large language models treat communication as a fixed interface — agents decode their internal reasoning into text and send that text to the next agent. This forces continuous internal representations to be compressed into discrete tokens, creating optimization boundaries that prevent the system from being improved end-to-end. The gap being addressed is the lack of a learnable communication mechanism that allows information to flow across agent boundaries without being lossy or disconnected from training.

Methodology DiffMAS operates in two stages using a sequential pipeline of four agents: Planner, Critic, Refiner, and Solver. In Stage 1, upstream agents sequentially build a shared key-value (KV) cache trace by prefilling and appending KV segments, forming a continuous latent communication channel without gradient updates. In Stage 2, the final agent decodes conditioned on the accumulated KV trace, and supervised fine-tuning via cross-entropy loss updates only low-rank adapter (LoRA) parameters while keeping the backbone model frozen. Experiments were conducted on Qwen3-4B, Qwen3-8B, Qwen3-14B, and DeepSeek-R1-Distill-Qwen-32B across mathematical reasoning (AIME24, AIME25), scientific QA (GPQA-Diamond), code generation (HumanEval+, MBPP+), and commonsense reasoning (OpenBookQA) benchmarks.

Results DiffMAS consistently outperformed single-agent inference, text-based multi-agent systems, and prior latent communication methods across all benchmarks. Qwen3-8B achieved 76.7% on AIME24 (+26.7% over the single-agent baseline) and 60.1% on GPQA-Diamond (+20.2%). At the 32B scale with DeepSeek-R1-Distill-Qwen-32B, AIME24 accuracy reached 70.0% (+3.3%). DiffMAS also showed lower token-level perplexity (mean 1.24 vs. 1.31 for the static latent baseline) and greater self-consistency across repeated samples, indicating that the gains stem from more stable multi-agent coordination rather than occasional correct guesses.

Low-confidence preprint: decomposes tasks into skill graphs executed by heterogeneous models with rule-augmented orchestration; same-model parallelism alone underperforms heterogeneous execution.

Why it matters Provider heterogeneity is worth evaluating as config/eval metadata. Resist turning 'skills' into a rigid registry — keep model choice in configuration.

Plain-language abstract This paper introduces Agent-in-the-Middle (AiTM), a new type of attack on AI systems built from multiple cooperating language models. Rather than taking over individual agents, AiTM intercepts and manipulates the messages that agents send to each other, steering the whole system toward harmful outputs without touching its underlying components.

Motivation Multi-agent systems built on large language models are increasingly used for complex tasks like software development and scientific research, relying on message-passing between specialized agents. Prior security research focused on compromising individual agents or feeding them adversarial inputs, leaving the communication channel itself — a critical and exploitable backbone — largely unexamined.

Methodology The authors designed an external LLM-powered adversarial agent that sits between agents, intercepts messages destined for a chosen victim agent, and injects contextually tailored malicious instructions. A reflection mechanism allows the adversarial agent to iteratively refine its injected content based on observed conversation dynamics. The attack was evaluated across multiple multi-agent frameworks (including AutoGen, MetaGPT, and ChatDev), different communication structures (debate, majority voting, task-specific dialogue), and varied attack goals.

Results AiTM achieved an attack success rate exceeding 40% in all tested scenarios and surpassing 70% in the majority of experiments. Applying the attack to real-world multi-agent applications MetaGPT and ChatDev demonstrated that it can meaningfully compromise their task performance, confirming that the inter-agent communication layer is a significant and practical security vulnerability in current LLM multi-agent systems.

Plain-language abstract This paper studies how the internal design of multi-agent AI systems — networks of two or more cooperating AI agents — affects how easily those systems can be manipulated into carrying out harmful tasks. The authors find that purely looking at individual agent safety is not enough: the way agents are connected and coordinated can open or close attack pathways independent of how safe any single agent is.

Motivation As AI systems shift from single models to multi-agent architectures that plan, use tools, and delegate subtasks to one another, new security risks emerge that single-agent evaluations miss entirely. A model that reliably refuses a harmful request when prompted directly may nonetheless enable that harm when the same request is decomposed into innocent-looking subtasks routed through multiple specialist agents — meaning architectural choices themselves constitute a security variable that had not been systematically studied.

Methodology The authors adapted three existing single-agent misuse benchmarks — BrowserART (web browser tasks), OS-Harm (desktop/OS tasks), and RedCode-Gen (code generation) — to the multi-agent setting, keeping task semantics identical while varying only architecture. They evaluated 13 architectural configurations across multiple base models (including GPT-4o, GPT-5.4, GPT-5-mini, Claude Sonnet 4, Qwen3-VL, and Llama 70B), systematically varying three design dimensions: agent role specialization, communication topology (standalone, star, chain, and mesh), and memory visibility (private, shared reasoning traces, full shared memory). Stage-wise metrics distinguished planning-stage refusal, execution-stage interception, partial harmful execution, and full attack completion.

Results Multi-agent architectures were more vulnerable than standalone agents in the majority of configurations tested. Attack success rates varied by up to 3.8 times across configurations at comparable or higher benign task accuracy, demonstrating that security and capability can diverge sharply. On BrowserART, a star topology with four specialists raised the harmful task completion rate from 10% (standalone) to 31%. Topology effects were environment-dependent: star was riskiest for browser tasks, while chain was riskiest for code generation (42.5% harmful task completion versus 9.4% for standalone). Memory visibility effects were similarly context-dependent, sometimes worsening and sometimes improving security depending on topology and environment, ruling out any single universal architectural recommendation.

Plain-language abstract MultiAgentBench is a benchmark for testing how well groups of AI language model agents work together or compete against each other across a range of realistic scenarios. It introduces MARBLE, an evaluation framework that scores both whether agents complete tasks and how well they collaborate, using milestone-based performance indicators. The benchmark covers scenarios from co-authoring research proposals to building structures in Minecraft to social deduction games.

Motivation Existing benchmarks for AI agents either focus on single agents working alone or test only narrow, specialized domains, which means they cannot capture what happens when multiple agents must coordinate, negotiate, or compete. There was no comprehensive way to measure the quality of multi-agent collaboration and competition across diverse, interactive settings.

Methodology The authors built MARBLE, a multi-agent coordination framework organized around a Coordination Engine that links an Agent Graph, Cognitive Module, and Coordinate Engine to support adaptive collaboration and communication. Scenarios include both established tasks (such as research collaboration following the ResearchTown setup and Minecraft-based building tasks) and LLM-generated tasks with human verification (such as Werewolf and bargaining games). The benchmark evaluates multiple coordination topologies—star, chain, tree, and graph—as well as strategies such as group discussion and cognitive planning.

Results Among the models tested, gpt-4o-mini achieved the highest average task score. Graph-structured coordination performed best among the topology options in the research scenario. Cognitive planning improved milestone achievement rates by 3%. Code and datasets are publicly available at the project's GitHub repository.

Plain-language abstract G-Safeguard is a security framework designed to protect multi-agent AI systems — networks of cooperating large language model (LLM) agents — from adversarial attacks. It treats the communications between agents as a graph, uses a graph neural network to detect compromised agents, and then surgically removes malicious connections to stop bad information from spreading.

Motivation Multi-agent LLM systems are increasingly used for complex tasks, but they inherit the security weaknesses of individual LLMs and add new ones through inter-agent communication. Existing defenses target single-agent threats and ignore the network topology of multi-agent systems, making them unable to detect or contain attacks that spread from one agent to another across the system.

Methodology G-Safeguard models a multi-agent system as a graph where nodes are agents and edges represent communication links. At each dialogue round it constructs a multi-agent utterance graph encoding what each agent said and to whom, then applies an edge-featured graph neural network to identify anomalous (high-risk) agents. Detected threats are neutralized through topological intervention — pruning edges that carry adversarial or misleading content — to prevent further propagation. Because the approach uses inductive graph learning, it transfers to multi-agent systems of any size without retraining.

Results On the MMLU and CSQA benchmarks, G-Safeguard blocked 10%–38.52% of agent infections in chain and star network topologies, reduced attack success rates for prompt injection by 21.38% and 22.01% on CSQA and MMLU respectively, cut tool-attack success by 12.67%, and cut memory-poisoning success by 16.27%. It also recovered over 40% of task performance degraded by prompt injection, and maintained stable defense at scale with 19.50%–39.23% attack-success-rate reductions in large multi-agent settings.

Plain-language abstract DeLM (Decentralized Language Models) is a multi-agent framework that drops the central controller. Instead of a main agent assigning subtasks, waiting, and merging results, parallel agents asynchronously claim tasks from a shared queue and read/write a shared 'verified context' of accumulated progress. It targets two settings — parallel exploration in software engineering and concurrent evidence processing in long-context QA — and improves accuracy while roughly halving cost.

Motivation Most multi-agent systems use centralized scatter-gather orchestration, which parallelizes sub-agent execution but not the coordination around it. Every finding must return to the main agent to be merged and rebroadcast, so progress-sharing becomes a serialized bottleneck as agents grow, and the controller can dilute, omit, or distort details. In long-context reasoning the main agent must pre-assign evidence clusters before knowing what is relevant, triggering extra delegation rounds. DeLM removes the controller as the coordination chokepoint.

Methodology Coordination is state-based, not prompt-routed. Two global structures: a shared context C of compact verified gists and a task queue T of pending subtasks. The pipeline initializes the queue from the input, executes ready subtasks in parallel, then compresses-verifies-admits each result into the shared context, generates more subtasks when the context is insufficient, and finalizes once none remain. The shared context is compact, global, and 'unfoldable' — agents read coarse gists by default and expand to detailed summaries or raw evidence only when needed. Admission-time verification checks each update against its underlying evidence and reasoning trajectory before it enters shared state, rejecting or regenerating unsupported updates so errors cannot propagate as reusable problem state.

Results On SWE-bench Verified, DeLM is strongest across test-time-scaling metrics, reaching 77.4% pass@4 at ~$0.12/task — roughly half the baselines' cost — with trace-level examples showing agents reuse each other's discoveries through the compact shared context. On LongBench-v2 Multi-Doc QA it leads four frontier model families by up to 5.7 points, with both admission-time verification and hierarchical summarization contributing. On OOLONG, vanilla DeLM underperforms RLM (which needs exact row-level aggregation via code execution), but RLM combined with DeLM yields the best accuracy and lowest cost, showing DeLM works as a coordination layer for programmatic reasoning too.

Checkpoint/restore duplicates irreversible effects (double commits, double payments, token reuse) unless the tool boundary enforces replay-or-fork: replay a recorded response when equivalent, require an explicit fork for a new irreversible op, block consumed-credential reuse. LLMs regenerate subtly different requests even at temperature 0.

Why it matters The single most under-appreciated agent reliability bug: a retry after a side effect lands repeats the side effect. Idempotency keys + replay-or-fork on every external write are table stakes.

Defines useful agent observability as correlation between high-level intent (LLM traffic) and low-level action (syscalls, file/process events), joined by time + process lineage. <3% overhead; exposes multi-agent coordination bottlenecks invisible to either stream alone.

Why it matters Correlate intent with action, not just log prompts. The join catches reasoning loops, prompt-injection exfiltration, and hidden coordination bottlenecks. Prefer standards-based telemetry over a bespoke tracer.

Plain-language abstract This paper lays out a comprehensive quality-assurance strategy for enterprise AI systems — products built on large language models, retrieval pipelines, and autonomous agents. It argues that traditional software testing is structurally mismatched with these systems and proposes a new framework centered on continuous risk reduction, a five-layer AI Assurance Pyramid, and treating evaluation as a core engineering discipline.

Motivation Enterprise AI systems fail in ways that conventional testing cannot detect: confident hallucinations, silent behavioral drift after a cloud provider updates a model, and coordination failures in multi-agent workflows that produce wrong answers visually indistinguishable from correct ones. Teams that test AI the same way they test deterministic software — with pass/fail test suites evaluated at release time — are systematically under-protected because AI outputs are probabilistic and cannot be verified for correctness in the classical sense.

Methodology The paper is a conceptual and prescriptive engineering strategy, not an empirical study. It introduces a structured AI Failure Taxonomy covering five categories of AI-native failure modes (including hallucination, instruction drift, trajectory collapse, and emergent coordination failure across fifteen specific modes), then maps these to a revised five-layer AI Assurance Pyramid ranging from Layer 0 (deterministic infrastructure validation) through Layer 4 (business outcome evaluations). It provides operational guidance on evaluation-driven development, RAG system testing using metrics such as those from the RAGAS framework, model lifecycle management including prompt regression testing, and governance including human-in-the-loop oversight and auditability.

Results The paper concludes that evaluation infrastructure must be treated as a shared platform capability — centralized datasets, judge pipelines, rubrics, and scoring pipelines — rather than rebuilt per project, drawing an analogy to the shift from per-project CI scripts to shared CI/CD platforms. It argues that the cost of insufficient evaluation (hallucination incidents, model drift detected weeks after onset, adversarial failures reaching production) consistently exceeds the investment in evaluation infrastructure, and that a pyramid weighted toward lower layers catches failures more cheaply and with better diagnostic precision than top-heavy end-to-end evaluation alone.

Plain-language abstract This paper introduces MPAC (Multi-Principal Agent Coordination Protocol), a new communication standard that lets AI agents owned by different people or organizations work together on shared tasks without a central authority overseeing them. Today's agent protocols assume one person or organization controls all the agents involved, which breaks down when independent parties — like two engineers whose coding agents both edit the same file — need to coordinate. MPAC defines how agents declare intentions before acting, handle conflicts explicitly, and escalate disagreements to their human owners when needed.

Motivation Existing AI agent protocols (MCP and A2A) both assume a single controlling principal who owns and trusts every participating agent, but many real coordination problems involve independent principals whose agents must work on shared state — two engineers' coding agents editing the same repository, family members' agents planning a trip together, or agents from different organizations drafting a joint contract. When this single-principal assumption breaks down, coordination collapses to ad-hoc chat, manual merging, or silent overwrites. No existing protocol addressed this multi-principal coordination gap.

Methodology MPAC is specified as a five-layer application-layer protocol (Session, Intent, Operation, Conflict, and Governance layers) with 21 message types, three state machines with normative transition tables, Lamport-clock watermarking for causal ordering, two execution models (pre-commit and post-commit), three security profiles, and an optimistic-concurrency-control mechanism for shared state. The authors released two interoperable reference implementations (Python with 122 tests; TypeScript with 101 tests), 66 adversarial enforcement tests, a machine-readable JSON Schema suite covering all 21 message types, and seven live multi-agent demos spanning scenarios such as code editing, trip planning, pre-commit authorization with fault recovery, and multi-level conflict escalation.

Results A controlled three-agent cross-module code review benchmark showed a 95% reduction in coordination overhead (from 68.65 seconds to 3.02 seconds) and a 4.8x wall-clock speedup (from 131.76 seconds to 27.38 seconds) under MPAC compared to a serialized human-mediated baseline. Per-agent decision time was preserved (63.11 seconds versus 57.13 seconds), confirming that the speedup comes from eliminating coordination waits rather than compressing model inference. The full specification, reference implementations, test suites, and demo transcripts were released as open source.

Plain-language abstract This paper identifies and demonstrates a new class of security attack called Indirect Prompt Injection, where malicious instructions are hidden inside content that an AI assistant retrieves from the web or other sources. When the AI reads that content, the hidden instructions hijack its behavior—without the user or attacker ever typing anything directly into the chat. The authors show these attacks work against real deployed systems and can cause the AI to steal user data, spread misinformation, or act as an automated social engineer.

Motivation Prior work on prompt injection assumed an attacker had direct access to the AI's input. As AI assistants began integrating with search engines, email clients, and code tools—routinely ingesting untrusted external data—a new and largely unexamined attack surface opened up. The paper argues that the rapid deployment of LLM-integrated applications outpaced safety evaluations, leaving millions of users potentially exposed to adversaries who never interact with the system directly.

Methodology The authors developed a comprehensive threat taxonomy mapping classic computer-security concepts (intrusion, persistence, malware, denial of service, data exfiltration) to the novel LLM-integrated application setting. They then constructed and tested concrete attack prompts against both synthetic GPT-4-based applications with controlled functionality and real-world systems including Bing Chat (GPT-4 powered) and GitHub Copilot. For Bing Chat, injections were delivered by embedding instructions in HTML comments on pages read via the Edge sidebar feature, allowing local testing without public poisoning. Attack scenarios covered information gathering, fraud, malware spreading, prompt-worm propagation, and code-completion manipulation.

Results The attacks proved practically viable across all tested systems. Indirectly injected prompts successfully steered model behavior in ways that direct-interface jailbreak filters blocked: Bing Chat halted sessions for directly entered jailbreaks but obeyed the same instructions when they arrived via retrieved content. The compromised model retained injected instructions across multiple conversation turns, used conversation context to augment persuasion, and could exfiltrate user-disclosed information (such as a journalist's identity) through markdown hyperlinks or search queries to attacker-controlled URLs. GitHub Copilot was also shown to be susceptible to injections placed in retrieved code. The authors conclude that effective mitigations for these attacks are currently lacking.

Plain-language abstract AgentDojo is a benchmarking framework for testing how well AI agents can resist prompt injection attacks — attempts by malicious content in tool outputs to hijack what the agent does. It provides a set of realistic tasks, security test cases, and an extensible environment where researchers can design new attacks and defenses against such vulnerabilities.

Motivation AI agents that combine large language models with external tools (email, banking, travel booking) cannot formally distinguish instructions from data, making them vulnerable to prompt injection: an attacker embeds malicious instructions in content the agent reads, causing it to execute unauthorized actions such as leaking user data or sending unauthorized messages. No rigorous, extensible benchmark existed to systematically measure agent robustness against these attacks.

Methodology The authors built AgentDojo as a dynamic, extensible framework rather than a static test suite. They populated it with 97 realistic agent tasks across domains such as email management, e-banking, and travel bookings, paired with 629 security test cases. Each security test specifies an attacker goal and an injection endpoint. The framework evaluates both utility (whether the agent completes its user task) and security (whether the attacker goal is achieved) using formal checks over environment state, not LLM-simulated judgments.

Results State-of-the-art LLMs solve fewer than 66% of AgentDojo tasks even without any attack present. Existing prompt injection attacks succeed against the best-performing agents in fewer than 25% of cases. Deploying a secondary attack-detector defense reduces the attack success rate further to 8%. Attacks benefit only marginally from side information about the system or victim, and rarely succeed when the attacker goal involves security-sensitive actions such as exfiltrating an authentication code.

Plain-language abstract This paper introduces Agent-in-the-Middle (AiTM), a new type of attack on AI systems built from multiple cooperating language models. Rather than taking over individual agents, AiTM intercepts and manipulates the messages that agents send to each other, steering the whole system toward harmful outputs without touching its underlying components.

Motivation Multi-agent systems built on large language models are increasingly used for complex tasks like software development and scientific research, relying on message-passing between specialized agents. Prior security research focused on compromising individual agents or feeding them adversarial inputs, leaving the communication channel itself — a critical and exploitable backbone — largely unexamined.

Methodology The authors designed an external LLM-powered adversarial agent that sits between agents, intercepts messages destined for a chosen victim agent, and injects contextually tailored malicious instructions. A reflection mechanism allows the adversarial agent to iteratively refine its injected content based on observed conversation dynamics. The attack was evaluated across multiple multi-agent frameworks (including AutoGen, MetaGPT, and ChatDev), different communication structures (debate, majority voting, task-specific dialogue), and varied attack goals.

Results AiTM achieved an attack success rate exceeding 40% in all tested scenarios and surpassing 70% in the majority of experiments. Applying the attack to real-world multi-agent applications MetaGPT and ChatDev demonstrated that it can meaningfully compromise their task performance, confirming that the inter-agent communication layer is a significant and practical security vulnerability in current LLM multi-agent systems.

Plain-language abstract This paper studies how the internal design of multi-agent AI systems — networks of two or more cooperating AI agents — affects how easily those systems can be manipulated into carrying out harmful tasks. The authors find that purely looking at individual agent safety is not enough: the way agents are connected and coordinated can open or close attack pathways independent of how safe any single agent is.

Motivation As AI systems shift from single models to multi-agent architectures that plan, use tools, and delegate subtasks to one another, new security risks emerge that single-agent evaluations miss entirely. A model that reliably refuses a harmful request when prompted directly may nonetheless enable that harm when the same request is decomposed into innocent-looking subtasks routed through multiple specialist agents — meaning architectural choices themselves constitute a security variable that had not been systematically studied.

Methodology The authors adapted three existing single-agent misuse benchmarks — BrowserART (web browser tasks), OS-Harm (desktop/OS tasks), and RedCode-Gen (code generation) — to the multi-agent setting, keeping task semantics identical while varying only architecture. They evaluated 13 architectural configurations across multiple base models (including GPT-4o, GPT-5.4, GPT-5-mini, Claude Sonnet 4, Qwen3-VL, and Llama 70B), systematically varying three design dimensions: agent role specialization, communication topology (standalone, star, chain, and mesh), and memory visibility (private, shared reasoning traces, full shared memory). Stage-wise metrics distinguished planning-stage refusal, execution-stage interception, partial harmful execution, and full attack completion.

Results Multi-agent architectures were more vulnerable than standalone agents in the majority of configurations tested. Attack success rates varied by up to 3.8 times across configurations at comparable or higher benign task accuracy, demonstrating that security and capability can diverge sharply. On BrowserART, a star topology with four specialists raised the harmful task completion rate from 10% (standalone) to 31%. Topology effects were environment-dependent: star was riskiest for browser tasks, while chain was riskiest for code generation (42.5% harmful task completion versus 9.4% for standalone). Memory visibility effects were similarly context-dependent, sometimes worsening and sometimes improving security depending on topology and environment, ruling out any single universal architectural recommendation.

Plain-language abstract This paper is a comprehensive survey and security analysis of the Model Context Protocol (MCP), an open standard introduced by Anthropic in late 2024 that gives AI models a unified way to discover and interact with external tools, APIs, databases, and files. The authors map the current MCP ecosystem, catalogue security risks across the full server lifecycle, and lay out research directions for making the protocol secure and sustainable.

Motivation Before MCP, connecting an AI application to external tools required writing custom API integrations for every service, leading to fragmented, platform-specific ecosystems and heavy maintenance burdens. MCP was designed to eliminate this fragmentation with a single standardized interface, but its rapid adoption outpaced any systematic analysis of its architecture or security posture, leaving a gap that this paper addresses.

Methodology The study performs an ecosystem-wide analysis of MCP as of March 2025, examining the protocol's architecture (host, client, and server roles), the three-phase lifecycle of MCP servers (creation, operation, and update), and the security risks at each phase. The authors survey publicly available MCP server collections—cataloguing registries such as MCP.so (4,774 servers), Glama (3,356), and Smithery (2,942)—review unofficial auto-installers, and analyse real-world adoption by industry platforms including OpenAI, Cursor, and Cloudflare.

Results The analysis identifies a range of concrete security threats: name collision attacks (malicious servers impersonating legitimate ones), installer spoofing through unofficial auto-installers, code injection and backdoors during server creation, and configuration drift in remote multi-tenant deployments. The paper finds that MCP currently lacks a centralized security oversight model, standardized authentication and authorization, and an official package management system. It concludes with recommendations for cryptographic server verification, namespace policies, and reputation-based trust mechanisms to support the protocol's sustainable growth.

Plain-language abstract AgentShield is a security framework that detects when an AI agent has been hijacked by a hidden malicious instruction embedded in the data it processes. Instead of trying to block every attack upfront, it plants traps inside the agent's tool interface and watches for any agent behavior that touches those traps — a reliable sign the agent is following an attacker's commands rather than the user's.

Motivation AI agents that can call external tools are vulnerable to "indirect prompt injection": attackers hide instructions in web pages, documents, or API responses that the agent reads, causing it to act against the user's interests. Existing defenses all try to prevent such attacks from succeeding, but some attacks slip through regardless. Additionally, all prior work was evaluated only in English, leaving speakers of languages like Kurdish and Arabic with no tested protection.

Methodology AgentShield embeds three layers of deception traps directly in the agent's tool interface: fake tools, fake credentials, and allowlisted parameters. When a compromised agent follows a hidden attacker instruction, it nearly always interacts with one of these traps, producing an immediate compromise signal. The same trap-trigger events also serve as automatically labeled training data for a self-supervised downstream classifier. The system was evaluated on 176 cross-lingual attack prompts spanning multiple languages, against four large language models from three different providers, and subjected to a systematic adaptive-attack evaluation where the attacker knows about the trap design.

Results On commercial models, AgentShield detected 90.7% to 100% of attacks that successfully bypassed the models' own built-in refusals, with zero false alarms across 485 normal-use tests. It was not evaded by any adaptive attack on commercial models. The self-supervised classifier trained on trap-trigger labels transferred across models and languages without retraining, demonstrating practical deployability beyond English.

Plain-language abstract G-Safeguard is a security framework designed to protect multi-agent AI systems — networks of cooperating large language model (LLM) agents — from adversarial attacks. It treats the communications between agents as a graph, uses a graph neural network to detect compromised agents, and then surgically removes malicious connections to stop bad information from spreading.

Motivation Multi-agent LLM systems are increasingly used for complex tasks, but they inherit the security weaknesses of individual LLMs and add new ones through inter-agent communication. Existing defenses target single-agent threats and ignore the network topology of multi-agent systems, making them unable to detect or contain attacks that spread from one agent to another across the system.

Methodology G-Safeguard models a multi-agent system as a graph where nodes are agents and edges represent communication links. At each dialogue round it constructs a multi-agent utterance graph encoding what each agent said and to whom, then applies an edge-featured graph neural network to identify anomalous (high-risk) agents. Detected threats are neutralized through topological intervention — pruning edges that carry adversarial or misleading content — to prevent further propagation. Because the approach uses inductive graph learning, it transfers to multi-agent systems of any size without retraining.

Results On the MMLU and CSQA benchmarks, G-Safeguard blocked 10%–38.52% of agent infections in chain and star network topologies, reduced attack success rates for prompt injection by 21.38% and 22.01% on CSQA and MMLU respectively, cut tool-attack success by 12.67%, and cut memory-poisoning success by 16.27%. It also recovered over 40% of task performance degraded by prompt injection, and maintained stable defense at scale with 19.50%–39.23% attack-success-rate reductions in large multi-agent settings.

A voluntary, widely-referenced framework organizing AI risk management into Govern / Map / Measure / Manage functions.

Why it matters Map your agent controls to a recognized framework so 'reliability' becomes auditable. Pairs naturally with the assurance + governance research above.

A community-maintained catalogue of the top LLM-application security risks — prompt injection, insecure output handling, excessive agency, supply chain, and more.

Why it matters The practical security checklist for agent systems. 'Excessive agency' and 'prompt injection' map directly to the topology and injection findings above.

Plain-language abstract AgentArmor studies how AI coding agents fail in everyday, non-adversarial use — not jailbreaks or prompt injection, but the 'hot mess' cases where an agent deletes the wrong thing or skips a safety step. It frames unsafe behavior as three sequential failure points and proposes a set of agent-harness modifications that make current coding agents measurably safer.

Motivation As coding agents such as Cursor, Claude Code, Codex, and OpenCode take over the full software lifecycle — not just code generation but deployment and monitoring — rare but highly destructive failures surface, yet few works rigorously evaluate the safety gaps that trustworthy deployment requires. The authors deliberately set aside adversarial threats like refusals, prompt injection, and jailbreaking to focus on the non-adversarial case.

Methodology They model misalignment as three failure points that must all hold for safe behavior: forming the correct target (fails under underspecification when default behavior is unsafe), actively pursuing it (fails on capability errors from bias, refusal, or limits), and executing it through the harness (fails on stochastic sampling and context decay), combined via a chain rule P(unsafe)=1-(1-f1)(1-f2)(1-f3) with scenarios that isolate each stage. They taxonomize agent behavior into four active modes — greenfield, editing, deployment, monitoring — and curate 8 scenarios across 20 coding environments and 59 synthetic transcript templates, run at n>=500 samples over Claude Opus 4.6, GPT 5.4, and Gemini 3.1 Pro. The proposed mitigation, AgentArmor, adds an extended system prompt, a LoRA-trained command classifier for risk and user-intent alignment with a '3 strikes' policy and persuasion-blindness against goal drift, deterministic guardrails (run ls -la before deleting, read scripts before executing), and tools letting the agent make files immutable or prune its own transcript context.

Results Across the evaluations — escalation, disregarding CLAUDE.md, skipping security practices, dangerous command templates, stochastic generation, and long-context degradation — AgentArmor is safer by a statistically significant margin relative to the unmodified base models. The authors frame the result as concrete, adoptable mitigations for today's coding agents and a design philosophy for future agent-harness features, not as a population-level safety guarantee.

Plain-language abstract PORTICO is a reference monitor that controls which tool capabilities a coding agent's planner can even see at any moment. It targets 'lingering authority' — the gap where a permission granted for one subgoal stays available after that subgoal is done — by granting capabilities as short-lived, epoch-bound handles that are revoked the moment their justifying episode closes.

Motivation Coding agents turn natural-language tasks into tool calls over repositories, tests, shells, package managers, network clients, and version control, and they usually start with more authority than any single subgoal needs. The problem is temporal: authority justified for one subgoal becomes stale once that subgoal closes, yet a static allowlist or sandbox keeps it reachable for the whole run. Because a broad option left in the planner interface lets the model keep planning around it even when a later monitor would reject the call, the exposed interface is itself part of the security state.

Methodology Given an explicit task contract and a typed tool catalog, PORTICO compiles an initial capability envelope, grant rules, trusted closure predicates, and global deny rules. A request-grant-invoke lifecycle materializes expansions as opaque, epoch-bound handles; closure removes those handles from the next planner interface and rejects stale replay before any side effect. It distinguishes capability exposure from runtime availability, sandbox reachability, and post-selection execution checks, positioning itself as a complement to sandboxing and content defenses. Evaluation uses three controlled fixture suites plus one pinned real-repository suite, with a non-revoking comparator (same envelope and grants at the same turns) to isolate the effect of closure and an all-visible same-policy comparator to isolate interface breadth.

Results On the closure slice, PORTICO and the non-revoking comparator match on task success, scope compliance, and all pre-closure decisions; PORTICO then rejects 10/10 post-closure capability reuses while the comparator permits 10/10, and a deterministic stale-write audit records 0/6 executed forbidden effects versus 6/6. The same split holds across file writes, git mutation, and network egress in scripted and six live-model traces. A four-episode same-policy diagnostic shows that narrowing what the planner sees, rather than only filtering execution, reduces wasted planning: broad exposure preserves zero executed forbidden effects but raises blocked proposals from 67 to 84.

Plain-language abstract AgentLens is a white-box safety system for LLM coding agents that run shell commands over many turns. Instead of an external guardrail that watches from outside, it reads the agent's own internal hidden states at each execution step, runs a lightweight linear probe to decide whether the current state is harmful, and when it is, nudges the agent's internal representation inside a small 10-dimensional subspace to push it toward refusing. It needs no weight changes — only one layer and one probe per model, with the steering strength tuned at inference time. The authors also build the Mechanistic Agent Safety (MAS) benchmark of step-annotated agent trajectories. Across three open-weight models it detects harmful steps accurately, gives early signals of harm one step ahead, and sharply reduces the rate of successful attacks.

Motivation Coding agents (Claude Code, Codex, Cursor CLI, Gemini CLI) operate a computer directly through the shell and can be weaponized for end-to-end malicious operations mapped to MITRE ATT&CK techniques. Because these agents loop, feeding environment feedback back into context, risk is rarely confined to one action: vulnerabilities accumulate across steps (create an empty script, chmod it, only later write a harmful payload and set it to auto-run). Existing defenses are mostly external guardrails that rely on predefined or auto-generated rules; they can monitor but offer limited fine-grained behavioral control per step. Mechanistic-interpretability safety methods exist but are confined to single-turn or jailbreak-style QA on static prompt-response pairs, so they cannot capture the evolving risk dynamics of multi-turn execution. The central question is whether step-level internal representations can detect harmful execution states and steer behavior during multi-turn interaction.

Methodology A frozen LLM is treated as the agent. At each step AgentLens extracts the last-token residual-stream hidden state at one selected intermediate layer. Multi-turn linear probing tests whether harmfulness is linearly decodable: a logistic probe is trained with binary cross-entropy on step examples — harmful execution steps labeled 1, refusal/benign steps labeled 0 — and the same current-step probe is reused for lookahead detection (target = next step's label). Detection: apply the probe to the step hidden state; if it flags harmful, trigger mitigation before the next action. Mitigation: from the two-class probe weights, score each hidden dimension by the absolute weight difference and keep the top-10 to form a sparse steering vector, broadcast across all token positions at the same layer, then regenerate the action. Only the steering strength alpha is adapted at inference: an LLM judge scores each candidate action's safety and utility (weights 0.6 and 0.4) and picks the alpha that maximizes the combination. Evaluation uses the MAS benchmark: 194 tasks (98 malicious, 96 benign) drawn from AdvCUA, Safe-OS, and AgentBench across 10 MITRE ATT&CK categories, run by three white-box ReAct agents (LLaMA-3.1-8B, Qwen-2.5-7B, Gemma-2-9B) in Docker. Trajectories are step-labeled via keyword matching plus LLM annotation and human review, with counterfactual benign steps that mimic malicious syntax so the probe captures intent, not surface form. Tasks are split 6:4 at the task level. Metrics are accuracy/F1 for detection and attack success rate plus collapse for mitigation; baselines include LLM-as-judge, guardrails (LLaMA-Guard, AGrail) for detection and RepE and self-reminder for mitigation.

Results Safety signals are linearly decodable from early-to-middle layers: probe accuracy peaks at 99.15% (LLaMA-3.1-8B, layer 12), 97.46% (Qwen-2.5-7B), and 95.35% (Gemma-2-9B). Current-step detection averages 97.32% accuracy, improving over the strongest baseline by 2.5–5.8 points per model. Lookahead reaches up to 96.77% accuracy, though the authors caution lookahead examples are sparse and frame this as feasibility evidence. Mitigation gives the lowest attack success on all three agents — LLaMA 73.80%->16.70%, Qwen 91.30%->4.35%, Gemma 92.86%->14.29% — averaging 85.99%->13.36%, a 72.63 pp absolute drop, versus RepE and self-reminder which remain far higher. The trade-off is generation collapse rising to 18.54% average (vs 2.24% vanilla), i.e. steering can over-correct into degenerate output. Causal validation via negative steering: reversing the direction on naturally-occurring refusal steps converts refusals into malicious commands — 100% ASR on 16 LLaMA refusal steps and 63% on 27 Gemma steps — evidence the sparse direction exerts bidirectional causal control rather than encoding lexical artifacts. On an out-of-distribution prompt-injection stress test (injection via environment observation), the probe does not reliably trigger, but applying the steering direction at the injection step drops ASR 86.7%->6.7% at 0% collapse — a detection-control gap. Stated limitations: requires white-box access, generation-quality trade-offs under strong steering, sparse lookahead data, and label ambiguity in malicious trajectories.

Plain-language abstract The paper introduces Semantic Compliance Hijacking (SCH), a supply-chain attack on LLM coding agents that ships no explicit code. A poisoned third-party 'Agent Skill' file carries only natural-language directives disguised as compliance or telemetry rules; the agent reads them as authoritative instructions and writes and runs the malicious code itself. This moves the attack from a detectable code payload onto the agent's own generative step, so credential exfiltration and remote code execution are synthesized at runtime from prose.

Motivation Modern coding agents (OpenClaw, Claude Code, Codex) load community skills like software libraries and run with file, shell, and network privileges, and they treat skill documentation as authoritative operational directives rather than passive data. Existing auditors (SkillScan's AST/regex sweeps, LLM Guard's classifiers) reliably catch explicit payloads, so the paper attacks the open question of whether a skill containing no code at all can still cause systemic compromise, a blind spot underscored by the real ClawHavoc marketplace-poisoning incident.

Methodology They formalize SCH with a payload-less constraint (keep a syntactic detector's toxicity score below the gateway threshold) and build 12 baseline adversarial skill seeds (2 attack dimensions x 6 skills) using three linguistic principles: control-flow eradication, topological/path steganography (spelling IP octets and ports out in words), and emotional/authoritative masking. A Multi-Skill Automated Optimization (MS-AO) loop (Algorithm 1, up to 5 rounds, quality score Q = ASR - 25*refusal - 10*syntax - 5*runtime, with best-state rollback) refines failed skills from execution tracebacks. Evaluation ran 600 contextualized cases (12 skills x 50 benign tasks drawn from DS-1000, BigCodeBench, SecurityEval, AgentBench os_interaction, InterCode nl2bash) across 3 frameworks and 3 models (GPT-5.4 mini, GLM-5, MiniMax-M2.7) in Docker-isolated sandboxes with a mock exfiltration server capturing network telemetry.

Results SCH executed on every framework-model pair, with peak 77.67% complete confidentiality leakage (GPT-5.4 mini on Codex) and RCE up to 67.33% (MiniMax-M2.7 on OpenClaw); average complete leakage ran 36.00-62.11% and average RCE 30.56-64.44%. Confidentiality beat integrity because frameworks lack data-flow awareness to block benign-looking telemetry POSTs (Claude Code cut GLM-5 RCE to 2.67% but did not stop exfiltration). The Alignment-Security Paradox held: the most-aligned GPT-5.4 mini was most vulnerable (62.11% avg complete leakage). Both defenses detected SCH at 0.00% versus 62.00-96.00% (SkillJect) and 91.61-99.81% (DDIPE); MS-AO raised RCE volatilely (GLM-5 peaked 31.67% at round 1, collapsed to 5.00% at round 3, recovered to 28.33% at round 5) and only ever hit a 33.33% max detection on discarded intermediate payloads, so final optimized skills preserved full evasion.

Plain-language abstract This paper proposes Knowledge-Based Pull Requests (KPR), a workflow for when an external collaborator with their own coding agent wants to contribute to a project. Instead of sending a code diff to be reviewed and merged directly, KPR has the collaborator's agent package up what it learned, code, tests, and a cleaned exploration trace, as a knowledge package. A human on the receiving side reviews that package, and only then does a project-owned trusted agent regenerate the actual mergeable code inside the project's own environment.

Motivation AI coding agents let external collaborators generate plausible pull requests very quickly, which shifts the bottleneck from writing code to reviewing it: deciding whether a change is warranted, respects project boundaries, and can be trusted. Empirical studies the authors cite show agent-authored PRs integrate faster but merge less often than human PRs, and that only about 44% of agent-produced code in real sessions survives into user commits, suggesting the diff itself isn't the most useful artifact to review.

Methodology KPR defines an artifact schema (knowledge package: rationale, evidence, rejected alternatives, human corrections), a cost-accounting view, and a collaboration gateway architecture. External code, tests, and agent traces are knowledge sources, never the direct merge candidate. A minimal controlled simulation pilot instantiates KPR packages from seven real merged public pull requests and stress-tests them under description-ablation, diff-ablation, and synthetic poisoned-patch conditions.

Results This is a conceptual framework and evaluation-agenda paper, not a large empirical study. The pilot shows KPR packages can be built from real PR material and survive the three stress-test ablations; the authors are explicit that broader claims about enterprise, vendor, and contractor deployments are proposed extensions of the pattern, not validated in production.

Plain-language abstract The paper turns a prompt-driven enterprise LLM prototype, an investment-briefing agent, into an auditable application by moving deterministic behavior out of prompts and into code: source manifests, source-backed claims, routing metadata, answer contracts, trace generation, and validators, arranged around a replaceable composition boundary where only phrasing is left to the model. It is instantiated on public data for five Korean corporate groups (25 listed companies, 113 source-backed runtime claims) and evaluated on whether the code-owned contracts hold, survive model substitution, and are load-bearing.

Motivation Enterprise LLM applications often start prompt-dominant, with product behavior carried by natural-language instructions and retrieval context rather than code, data contracts, or validation. Prompts can demonstrate behavior but not guarantee it: productization needs each visible claim traceable to bounded sources, routed to the correct entity, constrained in what it may assert, reproducible, and audited through versioned artifacts, none of which prompts alone enforce.

Methodology The harness relocates control into code: manifests define which sources may be used, source-backed claims define which statements may enter runtime context, routing metadata binds questions to entities, answer contracts define the visible answer, and traces record how each answer was assembled, with the LLM confined to a replaceable composition boundary. Evaluation covers three questions: contract preservation across a fixed validation set with a fault-injection negative control, behavior under three substituted hosted models across 270 composition-boundary runs, and an enforcement-layer ablation that disables the code-owned gate and compares it against a bolt-on external guardrail over 30 adversarial runs (15 recommendation-bait, 15 leak-bait).

Results The contracts held across the fixed validation set, and the fault-injection runs confirmed the validators flag deliberately broken source, routing, trace, answer, and leakage contracts. Under model substitution the enforced checks passed on all 270 composition-boundary runs, with failures confined to the model-composed side and recorded. In the ablation, prompt instructions alone let recommendation-language and trace-leakage violations reach the reader on all 30 adversarial runs, each blocked by the harness; the external guardrail also blocked them but over-refused, with 4 false refusals and 28 of 30 adversarial runs blocked, dropping utility to 88/120 where the harness preserved 120/120 by falling back to a deterministic composer.

Nature paper: a host decomposes the case, invokes specialized servers, self-reflects, and emits ranked outputs with evidence-linked rationale. Validated on 6,401 cases with blinded expert review of the reasoning chain, not just final answers.

Why it matters In regulated/high-stakes settings, capture evidence references and a validation summary so a human can review the reasoning chain — auditability is a first-class output, not an afterthought.

Formalizes workflow management over a task-dependency graph with workers (capabilities, availability, cost rates), hard/soft constraints, and graph-modifying actions. Reactive managers that over-assign and under-inspect fail.

Why it matters Expose the task graph + resource facts to operators. Good orchestration inspects and decomposes; it doesn't just dump work onto workers.

A voluntary, widely-referenced framework organizing AI risk management into Govern / Map / Measure / Manage functions.

Why it matters Map your agent controls to a recognized framework so 'reliability' becomes auditable. Pairs naturally with the assurance + governance research above.

Plain-language abstract When coding agents make implementation cheap, the engineering problem shifts to keeping high-velocity, AI-mediated development inspectable, correctable, and maintainable. This 12-week first-person case study, one expert engineer building a document accessibility remediation system with frontier coding agents, yields a process theory of governance conversion: failures surfaced by agentic speed are converted, by human judgment, into durable governance mechanisms.

Motivation Prior process models trade velocity against quality: velocity-centric multi-agent workflows underspecify quality control, oversight-centric models make human attention the throughput bottleneck, and governance-centric approaches derive controls only from obligations known before agents act. None explains how engineers revise the control environment when agent-produced failures reveal obligations not specified in advance.

Methodology First-person case study of a bounded 12-week greenfield development effort. The empirical record comprises 88 contemporaneous field notes, design records, deployment data, and repository history covering 420 KLOC of production code and 1.16 MLOC of tests, lints, supporting documentation, and agent tooling, analyzed to build a candidate middle-range theory with testable propositions.

Results A process model of governance conversion (failure, then judgment, then governance): agentic implementation velocity exposes recurring structural failure classes; engineering judgment interprets which failures reveal missing governance; new mechanisms, either controls that detect and contain failures or architecture that eliminates them by construction, encode that judgment into the engineering environment to constrain subsequent agent work. The paper contributes a catalog of governance mechanisms and argues the scarce human capacity is direction, interpretation, and abstraction rather than implementation-level review.

Plain-language abstract This paper proposes Knowledge-Based Pull Requests (KPR), a workflow for when an external collaborator with their own coding agent wants to contribute to a project. Instead of sending a code diff to be reviewed and merged directly, KPR has the collaborator's agent package up what it learned, code, tests, and a cleaned exploration trace, as a knowledge package. A human on the receiving side reviews that package, and only then does a project-owned trusted agent regenerate the actual mergeable code inside the project's own environment.

Motivation AI coding agents let external collaborators generate plausible pull requests very quickly, which shifts the bottleneck from writing code to reviewing it: deciding whether a change is warranted, respects project boundaries, and can be trusted. Empirical studies the authors cite show agent-authored PRs integrate faster but merge less often than human PRs, and that only about 44% of agent-produced code in real sessions survives into user commits, suggesting the diff itself isn't the most useful artifact to review.

Methodology KPR defines an artifact schema (knowledge package: rationale, evidence, rejected alternatives, human corrections), a cost-accounting view, and a collaboration gateway architecture. External code, tests, and agent traces are knowledge sources, never the direct merge candidate. A minimal controlled simulation pilot instantiates KPR packages from seven real merged public pull requests and stress-tests them under description-ablation, diff-ablation, and synthetic poisoned-patch conditions.

Results This is a conceptual framework and evaluation-agenda paper, not a large empirical study. The pilot shows KPR packages can be built from real PR material and survive the three stress-test ablations; the authors are explicit that broader claims about enterprise, vendor, and contractor deployments are proposed extensions of the pattern, not validated in production.

Plain-language abstract NOVA is a system Tencent built to automate architecture changes to its production ad-recommendation models, the kind of structural redesign (new attention modules, feature interactions) that usually needs an expert engineer. It uses an agent to propose changes, but layers verification on top so it can catch a runnable-but-wrong candidate, code that passes tests but breaks a recommender-specific invariant, before wasting a training run on it, and routes the riskiest changes to a human-in-the-loop mode.

Motivation Generic coding agents optimize for code that runs and passes unit tests, but a recommender architecture can be syntactically valid and still be a bad or actively harmful architecture, for example silently dropping sequence masking or degenerating self-attention into a plain MLP. AutoML only tunes hyperparameters, not cross-module structural changes, leaving a gap between 'runs' and 'is architecturally sound.'

Methodology NOVA computes an architecture gradient, an SGD-inspired but non-differentiable update signal aggregating prior modifications, verification diagnostics, metric changes, and trajectory memory, to pick the next modification. A verification cascade checks structure semantics, local executability, offline effectiveness, and online impact before expensive training; failed candidates become reusable forbidden directions. An L1-L4 task-level control scheme routes high-risk changes to a human-supervised Copilot mode. It's deployed in a production advertising system serving over a billion users.

Results On the hardest task tier (L3, literature-to-production), NOVA reaches 86.7% valid-pass rate and 60.0% effective-pass rate, more than double the human expert loop's effective-pass rate, and shortens one literature-to-production cycle by over 13x in human-attended time. In live online A/B testing, the selected candidate improved GMV on three pCVR objectives by +1.25%, +1.70%, and +2.02% while reducing prediction bias by 37.3-66.7%.

Plain-language abstract This paper builds an explanatory theory of how coding agents that author entire pull requests are reshaping code review, synthesized from practitioner writing at scale. It collects 38,709 grey-literature documents (engineering blogs and Reddit threads), codes a stratified random sample of 3,100 with an LLM-assisted pipeline, and derives a causal model of 26 constructs and 67 relationships whose organizing claim is that review is the control point through which a coding agent's effect on software is decided.

Motivation Practitioners sharply disagree about AI's effect on review - whether it becomes the delivery bottleneck, whether human review is still necessary, and whether it quietly erodes the comprehension that review once built. Repository-mining studies measure surface trends but seldom explain the mechanisms, and the trends are unstable: the authors' own analysis of public GitHub activity finds agent-authored PRs reviewed less often, merged several times faster, and discussed less than human ones, yet the direction of these trends flips under different but equally defensible analysis choices, so understanding requires a theory of the underlying causal mechanisms.

Methodology The authors collected 38,709 public documents (7,630 web articles and 31,079 Reddit threads), filtered to those substantively about code review, and coded a stratified random sample of 3,100 (about 12,000 pages) with an LLM-assisted thematic-analysis pipeline, then constructed the causal model in a largely manual but LLM-assisted process, using LLMs to organize codes and to search among codes and their quotes. A motivating observational study re-scraped full pull-request histories from the public Agents-in-the-Wild corpus for a longitudinal view.

Results The resulting theory has 26 constructs and 67 relationships (64 directed, 3 contested), centered on review load, review thoroughness (efficiency, depth, effectiveness), code quality, comprehension debt, and reviewer skill, and their feedback loops; its claim is that AI does not fix the sign of its effect on software - the team sets it through the expertise its humans bring and how it structures review. The observational study finds 40.1% of agent-authored PRs examined only by the developer who invoked the agent (versus 21.5% of human PRs), and the share of merged agent-authored PRs receiving no human review falling from over 50% in mid-2025 toward the roughly 14% human baseline by early 2026.