Stephanie Jarmak

concepts

The concept map

The canonical concept vocabulary drawn as a graph: concepts, the topics they anchor to, and co-occurrence edges derived from digest issues and explorer papers. Click a concept for its evidence — digest issues, explorer sections, papers — or read the structured list below.

Agent memory

How agents store, retrieve, and forget context across turns and sessions; memory architectures and their design tradeoffs.

tagged in 51 of 185 digest issues, most recently 2026-08-24

Digest issues

Agent tooling

Harnesses, skills, and tool interfaces that let agents act: the plumbing between a model and the systems it operates.

tagged in 82 of 185 digest issues, most recently 2026-09-05

Digest issues

Agentic coding

Agents that write, refactor, and maintain software: coding assistants, autonomous dev loops, and the workflows around them.

tagged in 112 of 185 digest issues, most recently 2026-09-06

Digest issues

AI agents

Systems that plan, call tools, and act over multiple steps to accomplish a goal.

tagged in 0 of 185 digest issues

AI economics

Cost structure of AI: token pricing, inference margins, usage limits, and unit economics.

tagged in 30 of 185 digest issues, most recently 2026-09-05

Digest issues

AI for science

AI applied to scientific discovery and research workflows, from literature synthesis to hypothesis generation.

tagged in 8 of 185 digest issues, most recently 2026-08-30

Digest issues

AI governance

Rules for AI systems: regulation, policy, export controls, and organizational governance of agents.

tagged in 18 of 185 digest issues, most recently 2026-08-02

Digest issues

AI industry

The business landscape of AI: labs, funding, acquisitions, and competitive strategy.

tagged in 6 of 185 digest issues, most recently 2026-09-01

Digest issues

AI infrastructure

Compute, serving, and platform layers under AI systems: GPUs, inference stacks, and datacenter buildout.

tagged in 19 of 185 digest issues, most recently 2026-09-04

Digest issues

AI safety

Preventing harmful model and agent behavior: alignment, oversight, and safety evaluation.

tagged in 15 of 185 digest issues, most recently 2026-09-05

Digest issues

AI security

Securing AI systems and using AI in security: prompt injection, sandboxing, supply-chain risk, and agent attack surfaces.

tagged in 50 of 185 digest issues, most recently 2026-09-03 · 20 papers · 1 explorer section

Digest issues

Explorer sections

Papers

Code review

Automated and agent-assisted review of code changes: quality gates, review bots, and human-agent review workflows.

tagged in 16 of 185 digest issues, most recently 2026-09-03

Digest issues

Context engineering

Deciding what goes into a model's context window and when: packing, pruning, and structuring working context for long-running tasks.

tagged in 11 of 185 digest issues, most recently 2026-08-31

Digest issues

Developer productivity

How AI tooling changes software work: measured impact, adoption patterns, and workflow shifts.

tagged in 13 of 185 digest issues, most recently 2026-08-27

Digest issues

Evaluation

Measuring whether AI systems work: benchmark design, eval harnesses, and comparison of models, agents, and search systems.

tagged in 128 of 185 digest issues, most recently 2026-09-06 · 22 papers · 4 explorer sections

Digest issues

Explorer sections

Papers

Human in the loop

Where people sit in agent workflows: approval gates, escalation, and oversight interfaces.

tagged in 0 of 185 digest issues

Information retrieval

Finding the right information at the right time: ranking, hybrid search, and retrieval over scientific and code corpora.

tagged in 80 of 185 digest issues, most recently 2026-09-06 · 3 papers · 1 explorer section

Digest issues

Explorer sections

Papers

  • Engram: A Bi-Temporal Memory Engine Where a Lean Retrieved Context Beats the Full History — The read path retrieves through four channels in parallel — dense semantic, BM25 lexical, graph n-hop from query entities, and recency/salience — fuses them with Reciprocal Rank Fusion, then applies an 'as-of' filter and an abstention gate; the assembled context is hybrid (conflict-resolved facts plus raw session chunks) because facts alone lose recall.
  • Infini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory — At read time an agentic procedure lets the LLM iteratively call memory tools — inspect intermediate results, expand local context around matches, assemble evidence — rather than take a single top-k step; the agentic variant beats a hybrid summary+BM25 reader (79.3% vs 76.0% on LongMemEval_S).
  • T-Mem: Memory That Anticipates, Not Archives — T-Mem retrieves through a top-down topic -> scene -> item cascade, scoring each layer with reciprocal rank fusion over both a shared BM25 lexical index and a per-type dense index (bge-m3). On top of these node indices, four write-time 'trigger' families surface host nodes on the query's behalf: items expose three independently-encoded views (concept-only, bridge-only, joint) and the trigger score is the nan-aware max across views, attributed back to the host item. Scenes/items reached through associative triggers bypass the topic prefilter, because gating them by surviving topics would re-impose the similarity-only neighbourhood the system is built to escape.

Memory consolidation

Distilling episodic traces into durable knowledge and skills, and deciding what an agent should forget.

19 papers · 2 explorer sections · tagged in 0 of 185 digest issues

Explorer sections

Papers

Model Context Protocol

The open protocol connecting models to tools and data sources, and the server/client ecosystem built on it.

tagged in 11 of 185 digest issues, most recently 2026-08-30

Digest issues

Model releases

New model launches and availability changes across frontier and open providers.

tagged in 60 of 185 digest issues, most recently 2026-09-05

Digest issues

Multi-agent orchestration

Coordinating multiple agents on shared work: topologies, delegation patterns, shared memory, and production reliability.

tagged in 89 of 185 digest issues, most recently 2026-09-06

Digest issues

Open models

Open-weight model releases and the ecosystem around running, fine-tuning, and evaluating them.

tagged in 32 of 185 digest issues, most recently 2026-09-04

Digest issues

Reliability

Failure modes, recovery, and durable state for agent systems running unattended or at production scale.

tagged in 51 of 185 digest issues, most recently 2026-09-06 · 18 papers · 1 explorer section

Digest issues

Explorer sections

Papers

  • ReAct: Synergizing Reasoning and Acting in Language Models — Interleaves chain-of-thought reasoning with tool actions so a model can plan, query external sources, and self-correct — reducing hallucination on decision tasks.
  • Is Multi-Agent Debate (MAD) the Silver Bullet? Empirical Analysis in Code Summarization & Translation — Structured multi-agent debate yields minimal-to-inconsistent gains over a strong single-agent baseline on software-engineering tasks.
  • Why Do Multi-Agent LLM Systems Fail? (MAST failure taxonomy) — 14 failure modes in 3 categories — specification issues, inter-agent misalignment, task verification — built with an LLM-as-judge pipeline at Cohen's κ=0.88.
  • λ_A: A Typed Lambda Calculus for LLM Agent Composition — Well-formedness / termination guarantees for agent composition via a typed calculus.
  • TraceFix: Repairing Agent Coordination Protocols with TLA+ Counterexamples — Uses TLA+ counterexamples to repair coordination protocols.
  • TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems — RootSE distills 93 real repository-level agent failures (over 4,500 execution steps) into a diagnosis benchmark whose target is the earliest decisive error step - the single early mistake, such as a misread requirement or flawed plan, whose cumulative consequence is the eventual system failure.
  • SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? — A 5-bucket failure taxonomy over 526 agent-attributable failures: implementation failure (41.6%) and timeout (31.4%) dominate, then reward hacking (15.4%), premature termination (7.6%), and poor self-verification (4.0%); long context degrades behavior actively — pass rate falls monotonically with consecutive-duplicate run length (claude-code 41.9%→3.2%) and compaction/summarizer trials pass at 0% vs 8.9% without.
  • XFlow: An Executable Protocol Programming System for Reliable Multi-Agent Workflows — The paper makes the failure taxonomy explicit before scaling fan-out: a single agent's hallucinated/malformed/misinterpreted output becomes shared state and corrupts downstream decisions. Concrete failure modes observed: tau3-bench baseline reaches the right end state via an invalid path (treats a parameter clarification as authorization to mutate); CorpusQA fails on the interpretation rule not the retrieved value; SWE-bench baseline submits a patch after local validation already failed. The cloud-edge fan-out is gated: edge workers see only assigned chunks, write only declared outputs, and must pass schema + coverage checks before entering global state.
  • AgentArmor: A Framework, Evaluation, & Mitigation of Coding Agent Failures — Decomposes non-adversarial coding-agent failure into three sequential points — forming the correct target (underspecification), pursuing it (capability error), and executing it through the harness (stochastic sampling, context decay) — with a chain-rule risk P(unsafe)=1-(1-f1)(1-f2)(1-f3) and scenarios that isolate each stage, cross-cut by four active modes (greenfield, editing, deployment, monitoring) over 8 scenarios, 20 environments, and 59 transcript templates at n>=500 across three frontier models.
  • RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents — Names a failure taxonomy for the lab-to-production gap — fragile fixes (patches pass tests but leave latent bugs), token waste (trial-and-error instead of a planned approach), false confidence (never abstaining on impossible/ambiguous tasks), and broken intermediates (codebase left broken between steps). Baseline ReAct agents fail badly against it: no baseline abstained on any of the 6 impossible tasks, and even disciplined agents abstained correctly only 62% of the time. Recovery is the hardest mode and the one scaffolding does NOT fix — smallest gain of all five pillars, token-waste cut only 34%, doom loops persist when root cause isn't in the error message; the authors conclude recovery may need architectural changes beyond configuration-level frameworks.
  • NOVA: A Verification-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems — NOVA's verification cascade checks architecture semantics before expensive training or deployment, catching runnable-but-structurally-invalid candidates that pass unit tests but violate recommender-specific invariants like sequence masking; failed-verification diagnostics become reusable 'forbidden directions' that shape later search.
  • Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents — Glite ARF enforces task-level worktree isolation and immutability with a corrections overlay after an incident where one agent step recomputed and corrupted 20,304 historical training rows across 38 feature sets; completed task folders can't be modified, so fixes are separate, auditable downstream tasks.
  • Govern the Repository, Not the Agent: Measuring Ecosystem-Level Risk in AI-Native Software — Across 930,000+ agent-authored pull requests, multilevel models show about half the variance in integration friction survives after accounting for the contribution, its author, size, and agent, and it is a repository-level property. Agent-authored contributions concentrate this repository-level friction roughly twice as much as human ones (ICC 0.30 vs 0.16).
  • AgentAbstain: Do LLM Agents Know When Not to Act? — The paper gives an agent-native failure taxonomy of 8 abstention scenarios, organized by when a trigger becomes observable (pre-execution reasoning vs. runtime discovery) and where it resides (query, environment state, or tools). Across 17 frontier LLMs in 4 harnesses the best agent, Gemini 3.1 Pro, reaches only 59.5% Paired Accuracy and 13 of 17 stay below 50%, and abstention capability tracks largely independently of general task-solving skill. The signature failure is post-hoc abstention: an agent commits an irreversible action, such as cancelling a flight before checking rebooking availability, then verbally acknowledges the problem, leaving unrecoverable side effects.
  • Proof-or-Stop: Don't Trust the Agent, Trust the Evidence -- Loop Engineering for Verifiable Evidence-Gated Lifecycle Control — Treats every agent-emitted lifecycle claim (reviewed, tested, done, ready-to-merge) as inadmissible until fresh, tracked-source-state-bound evidence satisfies a gate. In a 9,240-cell powered ablation, the pre-registered contrast between the gated loop and a compute-budgeted naive loop cut visible-pass/hidden-fail amplification from 31/1,800 injected cells to 2/1,800 (+1.6pp not-amplified, 95% CI [0.8, 2.5]); a near-compute contrast (14/1,800 vs 2/1,800) shows the gain tracks enforcing the review signal as a hard gate, not just adding a reviewer.
  • Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent — Decomposes a production enterprise agent's uplift over its bare base model (+11.0 pp on SpreadsheetBench Verified, +7-10 pp on BullshitBench v2, ~+15 pp on GAIA validation) by architectural layer: scaffolding and prompting account for +9.5 of the +11.0 pp on SpreadsheetBench, while the verification loop's isolated contribution is +1.5 pp (6 rescued tasks) - small but concentrated at the top of the score distribution, where every competitive system has already exhausted scaffolding gains. A specialist-swap ablation shows the loop's value rides on observer independence: moving observe/compare from the small trained verifier back onto the frontier model that generated the artifact cuts SpreadsheetBench rescues from 6 tasks to 2 and drops BullshitBench correct rejection 4-5 pp.
  • When Do Agent Loops Mistake Stagnation for Progress? Self-Evaluation Bias and Externally Grounded Verification in Long-Running Autonomous LLM Agent Loops — The paper names the failure mode, the progress mirage, and defines it as a measurable rate rather than an anecdote: the fraction of evaluator-accepted cycles whose oracle-measured delta is zero or below. The definition was deliberately widened from the preregistered wording of exactly zero, because a delta of zero excludes the worse error of accepting an outright regression. Alongside it the paper reports a wasted-cycle ratio, acceptance rates conditioned separately on non-positive and positive deltas, and time-to-first-positive with right-censoring recorded, all computed mechanically from raw cycle logs with no human discretion. The oracle is a synthetic user simulator kept unreachable by container and network isolation, with a boundary proof run at every run and a canary scan at every cycle, so the ground truth cannot be quietly contaminated by the agent under test. The proposed architecture puts the evaluator in a separate process reading databases, version-control history, web analytics and append-only ledgers, and forces a rejected cycle to leave an artifact the next iteration must read first at boot. Four supporting mechanisms (append-only state, file-based handoff between sessions, periodic frame-adversarial self-review, a self-healing scheduler) are presented as field-derived design notes and are not individually validated.
  • Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs Single LLMs

Retrieval-augmented generation

Grounding model output in retrieved evidence: dense retrieval foundations, agentic search loops, and RAG system design.

tagged in 0 of 185 digest issues

Synthetic data

Model-generated training and evaluation data: generation pipelines, quality control, and contamination risk.

8 papers · 1 explorer section · tagged in 0 of 185 digest issues

Explorer sections

Papers

Test-time compute

Spending inference-time computation to improve results: reasoning-intensive retrieval, reranking, and search at query time.

6 papers · 1 explorer section · tagged in 0 of 185 digest issues

Explorer sections

Papers

  • Rank1: Test-Time Compute for Reranking in Information Retrieval — The first reranker trained to spend test-time compute: distills R1/o1 reasoning traces so a small model reasons before scoring relevance.
  • Distillation and Refinement of Reasoning in Small Language Models for Document Re-ranking — Distills then RL-refines reasoning into a 3B reranker that generates relevance explanations at inference; third on BRIGHT, beating 20x-larger models.
  • Verbal-R3: Verbal Reranker as the Missing Bridge between Retrieval and Reasoning — A verbal reranker that reasons in natural language to bridge retrieval and the generator, instead of injecting raw passages.
  • GRC: Unifying Reasoning-Driven Generation, Retrieval and Compression — Unifies reasoning-driven generation, retrieval, and compression in one LLM, sharing training across embedding and generative tasks.
  • RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents — Critic-free policy optimization that turns retrieval interactions into localized credit signals for latent reasoning steps.
  • EMBL AI Librarian: Life-Sciences Knowledge Layer for AI Agents — The paper is a direct argument against paying for a static embedding index when inference-time reasoning can do the ranking. Librarian maintains no index of its own; it queries Europe PMC's live search and spends LLM calls at request time on query planning and paragraph-level relevance scoring against the question. The authors give four reasons for refusing the dense datastore. It is expensive to build and serve, with OpenScholar's alone at roughly 744 GB. Its margin is narrowing, since a well-tuned BM25 backbone driven by a capable LLM issuing multiple keyword queries now matches or exceeds dense retrieval on scientific benchmarks. Embeddings flatten structured metadata, blocking direct queries over explicit fields such as gene, protein, organism and chemical annotations, which is exactly the structure Europe PMC exposes. And a nearest-neighbour lookup is not inspectable, where a fielded keyword query is. The trade lands: more than 16 points of Citation F1 over strong recent baselines on ScholarQA-Bench, higher expert-consensus agreement as a drop-in retrieval layer, and about 8 points on LitQA2 over web search.

Verification

Checking that an agent's output actually satisfies the task: test oracles, property checks, and validation of generated artifacts.

tagged in 3 of 185 digest issues, most recently 2026-09-05

Digest issues