Stephanie Jarmak
Writing Book

Book

Engineering Reliable Coding Agents

Evaluating and Operating the System Around the Model

By Stephanie Jarmak

The goal of this book is to enable others to build and maintain agentic systems whose reliability can be observed, tested, and defended.

19 chapters · 6 parts · approximately 314 pages

Contents

Start here Engineering Reliable Coding Agents

Part 1

Evaluation measurement and experiment design

  1. Run-to-run variance, statistical power, and paired comparisons
  2. Baselines, ablations, and cost-accuracy tradeoffs
  3. Benchmark contamination, oracle strength, and workload validity

Part 2

Evaluation and grading systems

  1. Execution-based evaluation, correction gates, and release tests
  2. Calibrating model graders and separating agreement from correctness
  3. Proxy metric gaming and layered evaluation signals

Part 3

Containment, durable execution, and recovery engineering

  1. The software factory as a distributed system
  2. Agent isolation, injection defenses, and independent verification
  3. Persistent agent state, durable workflows, and idempotent retries
  4. Replayable traces and fault-injection recovery testing
  5. Human-auditable failure analysis and taxonomy development

Part 4

Context engineering: retrieval, budgets, and memory

  1. Measuring and designing repository retrieval
  2. Localization funnels, repository indexes, and freshness checks
  3. Usable context budgets, consolidated-spec restarts, and file-based tool output
  4. Cross-session memory, raw traces, and compaction policies

Part 5

Human review and accountability engineering

  1. Efficient verification interfaces and risk-based human escalation
  2. Autonomy calibration, provenance, effective gates, and accountability

Part 6

Research agenda: work allocation and cost engineering

  1. Agent topology selection and dynamic task allocation
  2. Cost-aware fleet scheduling and model routing
Closing the evidence chain behind reliable agents Reference Glossary