Closing: the evidence chain behind reliable agents

The most useful result in my retrieval work was a disagreement between two instruments. Three retrieval measures agreed that the tool had become much better at placing relevant repository evidence in front of the agent. The paired end-to-end reward across 370 tasks moved by +0.0349.

The first interval I reported around that difference likely understates the uncertainty and should not be read as a valid bound. It resampled tasks as though they were independent even though they came from 73 repositories and 20 suites.

Interpreting that experiment required two parts of this monograph. The stage decomposition from Part IV located where the improvement occurred. The resampling rule from Part I determined whether the end-to-end difference had been measured adequately at all.

The dependency chain introduced at the beginning therefore returns as the conclusion. A defect in an earlier layer rarely announces itself downstream. It arrives as a clean score, a confident verdict, or a plausible artifact.

Each layer determines what the next may trust

Part I sits beneath the rest of the monograph because every later practice is adopted or rejected through a comparison. Before a difference can describe the system rather than one draw from it, the evaluation must measure local run-to-run variation and be capable of resolving an effect large enough to matter.

Nineteen practices developed in later parts execute a method introduced in Part I. Removing that dependency does not remove the requirement. It leaves the requirement unstated.

Part II turns measurements into operating verdicts. A model grader, consensus vote, and proxy score are all instruments, and Part I supplies the methods for estimating their errors. In Skalse et al.’s (2022) linear expected-return formalization over all stochastic policies, two rewards can be mutually unhackable only when at least one is constant. This bounded theorem supplies a structural warning against treating any nonconstant proxy as final authority.

A grading system therefore produces a verdict at an operating point under a stated distribution. Layering is necessary because any nonconstant verdict can become an optimization target.

Part III asks whether the record underlying those verdicts is evidence at all. Four ordinary failures can corrupt that record in ways no later statistic repairs:

• One identity can reach both primary data and its recovery material.
• A run can die without a durable account of completed work.
• A trace can fail to distinguish tool dispatch from external commitment.
• A quarantine policy can remove failed runs before anyone examines them.

Preserving the record is necessary and insufficient. Zhang et al. (2025) gave expert-annotated failure logs from 127 multi-agent systems to the strongest automated attribution methods they evaluated. The best method identified the decisive step in 14.2 percent of cases.

A complete trace supports a causal explanation. It does not generate one automatically.

Part IV determines what evidence reached the model. A failure should not be assigned to model reasoning until the evaluation records whether the required evidence:

• existed in the searchable corpus;
• was present in the current index;
• was returned by retrieval;
• survived ranking and context selection; and
• described the repository state being modified.

Stale context is the sharpest case. It converts an ordinary absence of evidence into a confident implementation against an interface that no longer exists.

Part V treats human review capacity as a finite system resource. Its allocation depends on monitors calibrated in Part II. Its interfaces determine whether challenging a claim costs less than accepting it. Its gates depend on the authority and evidence preserved in Part III.

A gate that cannot change the execution path records assent and nothing more.

Part VI depends on artifacts produced by every earlier part. Routing and scheduling policies are evaluated by replaying recorded arrivals while holding demand fixed and forcing sophisticated policies to compete against cheap alternatives.

That discipline appears at both ends of the monograph. Kapoor et al. (2024) found that retrying a model matched more elaborate architectures on a function-level coding benchmark at a fraction of their inference cost. Chen et al. (2016) proposed inexpensive random sampling as the baseline a search-based method should beat before earning adoption.

The dependency chain also limits what a repair can accomplish. Compensating for an earlier defect with machinery from a later part is among the most expensive mistakes available here, and it is easy to make because the later component is often easier to deploy.

More samples do not repair a task distribution that excludes production work. More judges do not repair a rubric that experts apply inconsistently. More agents do not repair a retrieval boundary that treats an empty result as authoritative. More context does not repair a run that has lost its governing specification.

In each case, the added machinery is evaluated using the instrument the earlier layer was supposed to repair.

Engineering controls can precede trial evidence

Evidence strength varies across the dependency chain and does not track operational consequence. Strong-graded items account for 59 percent of the evidence in Part I but 14 percent in Part III. Chapter 7 has no strong-graded or direct scholarly evidence item, even though its containment practices may be the first controls needed by a system with production write access. Among the 52 practices ranked by consequence, Spearman’s rho between urgency rank and the presence of at least one strong item was -0.004.

This is not evidence-derived consensus. It is an authorial selection among controls supported in different ways. Some recommendations follow measured effects. Others follow a structural failure argument: the failure is possible under the current authority or state model, its consequence is material, the proposed control is reversible, and the boundary can be tested directly. A denied write, a durable completion record, and a replay in which the same arrivals face two schedulers are observations rather than prevalence claims.

The imperative headings in thin-evidence chapters should be read in that sense. They name an engineering action and the check that can falsify its local justification; they do not assert a universal effect size. A mechanism that cannot be reduced to an executable observation has not yet been specified well enough to recommend.

Two studies would materially strengthen the record: a controlled comparison of isolation designs scored against observed incidents, and a recovery benchmark with a published fault menu, several kill placements, recurring failures, and upgrades across runtime versions. Part VI remains more transfer-heavy: its topology and scheduling proposals are executable research questions for coding-agent fleets, not established production effects.

Personal starting points and a constrained-budget order

The literature does not supply universal values for repeat count, inter-rater agreement, context limits, refresh cadence, routing cost, or re-solve frequency. The following are my starting points for a new system. They are deliberately labeled as personal defaults: each is a trigger for measurement or review, not evidence that the value transfers.

Decision	Personal starting point	What changes it
Repeated evaluation	Use (k=3) paired runs per item for inexpensive screening; do not promote from that screen. Size the release comparison from a pilot and the smallest decision-relevant effect.	Higher variance, clustered tasks, or a smaller meaningful effect requires more runs.
Reliability reporting	Report both (\mathrm{pass}@k), where any attempt succeeds, and (\mathrm{pass}^{k}), where every attempt succeeds, with (k) printed beside the result. I begin with (k=3).	The deployed retry policy and cost of intermittent failure determine the useful (k).
Human-label agreement	Treat Cohen’s or Fleiss’s kappa below 0.60 as a rubric-debugging trigger. A value above 0.60 is not a correctness gate; promotion still depends on class-specific error rates against expert adjudication.	Class imbalance, ambiguous labels, and high-consequence errors make the confusion matrix more important than kappa.
Promotion threshold	Before execution, require either a three-percentage-point absolute success gain at no material cost increase or a ten-percent cost reduction at no success loss. Treat smaller effects as uncredited until a decision justifies them.	Task value, baseline rate, and operating cost should replace these round numbers.
Failure review	Read twenty stratified failures after a material model, harness, or policy change and keep an abstain category when the trace cannot support attribution.	Rare high-consequence classes are oversampled regardless of frequency.
Context restart	Restart or consolidate at 60 percent of the shortest context length at which the weakest required task shows material degradation in a local sweep.	New models, tools, instruction files, or task mix require a new sweep.
Freshness	Require exact repository-revision identity for code-bearing retrieval and reject an index with an unknown source revision.	A coarser content-addressed equivalence is acceptable only when it is proved by the indexer.

I account for the apparatus as part of system cost. Each comparison records inference and tool cost, elapsed time, labeling and adjudication hours, reviewer queue time, storage, and maintenance work. The denominator is accepted work, not model calls. A control whose ongoing measurement cost is invisible will eventually be bypassed or defended with stale evidence.

When capacity is constrained, I remove apparatus in this order:

1. dynamic scheduling and learned routing;
2. multi-agent debate, specialist roles, and elaborate aggregation;
3. extra model judges and retrieval lanes that have not shown unique contribution;
4. benchmark breadth, while retaining a small stratified set of consequential tasks; and
5. rich live interfaces and secondary taxonomies whose questions can still be answered from the durable record.

I keep four elements until last: an executable acceptance check, separation between ordinary and recovery authority, versioned task and configuration identity, and a paired comparison against the cheapest credible baseline. A small team can operate those controls without reproducing the full apparatus in this monograph.

Several questions remain research problems rather than missing defaults. No controlled result in the reviewed set shows that a dynamic dependency graph beats a well-designed fixed schedule for repository work. Propagation of deletion through summaries, embeddings, caches, and graph edges is largely unmeasured. Current context, attribution, grader, and routing results are dated snapshots. The structural claims are more durable: one run is one draw; a component that did not execute did not cause the result; a stale artifact can remain fluent after losing authority; and one identity that reaches production data and recovery material defines one failure domain.

A minimum pass through the dependency chain

For an existing system, one compact pass produces the minimum record on which later decisions can build:

1. Reopen one decision based on an aggregate score. Run the cheapest credible baseline and the candidate on identical task versions, initially three times per item, and preserve per-item outcomes.
2. Record success, reliability, cost, latency, model, harness, prompt, permissions, and pricing snapshot separately.
3. Exercise one permitted and one prohibited action with the ordinary identity, including the boundary between primary and recovery resources.
4. Verify one recent completion claim from repository or system state and rerun the executable check that makes it true.
5. Read twenty failed or unverifiable runs, label the first upstream failure where the trace permits it, and repair the first ordinary causal question the schema cannot answer.
6. Before the next promotion run, record the success floor, cost ceiling, task and baseline versions, mechanism condition, and fault-containment guard.

The result is not a reliability certificate. It is a set of challengeable artifacts: a paired distribution, a cost-quality record, an observed authority boundary, an independently verified state transition, a seed failure corpus, and a decision rule fixed before the result was known.

Begin with the comparison you currently trust most. Keep the per-item record even when the rerun agrees with you.

Sources and evidence

No evidence is introduced here. Each identifier below is carried by the chapter named, and the evidence grouping comes from that chapter’s record.

• Strong evidence: Skalse, Howe, Krasheninnikov & Krueger (2022). Defining and Characterizing Reward Hacking. NeurIPS 2022. arXiv:2209.13085. Chapter 6, layer-signals-beyond-single-proxy.
• Strong evidence: Zhang, S., et al. (2025). Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. ICML 2025. arXiv:2505.00212. Chapter 10, keep-humans-in-failure-attribution.
• Directional evidence: Kapoor, Stroebl, Siegel, Nadgir & Narayanan (2024). AI Agents That Matter. arXiv:2407.01502. Chapter 2, report-cost-accuracy-pareto.
• Strong evidence: Chen, J., et al. (2016). Sampling as a Baseline Optimizer for Search-Based Software Engineering. arXiv:1608.07617. Chapter 18, replay-traces-before-policy-changes.

The part-level and chapter-level evidence shares restated above were recomputed from the companion catalog: Part I has 19 of 32 items grouped as strong, Part III has 4 of 29, Part IV has 20 of 38, and Part VI has 6 of 27. Chapter 7 has no strong or direct scholarly evidence item.