ERPBench: Agents Save the ERP Form in 85% of Runs and Store the Right Value in 3%

Six computer-use agents were pointed at a live ERP system and asked to do what a procurement clerk does: open the right form, fill it in, save it. Some of them saved in up to 85% of runs. The stored record carried the correct value in as few as 3%. That gap, between an action that looks finished on screen and a database row that is right, runs through most of today’s items.

ERPBench (Bhagtani et al.) is the benchmark behind that number. The authors argue that enterprise software is a different evaluation problem from the desktop and web tasks most computer-use benchmarks are built on: interfaces are dense, tasks span coordinated multi-step interactions, and the errors that matter alter persistent business records rather than surfacing anywhere visible. So they score each task against ground-truth values in the ERP database, not against the screenshot trajectory, and they run the agents on a live, reproducible instance rather than a simulated approximation. Strong general GUI scores did not transfer. The paper also ships a production-grade harness that gates agent actions behind human approval, which the benchmark itself runs autonomously, plus a taxonomy of enterprise-specific failure modes. For anyone deploying agents against systems of record, the scoring choice is the transferable lesson: if the thing you care about is state, grade state.

The same principle, applied to observability, produces EviRCA (Wang et al.). Agentic root-cause analysis for microservices usually hands a model raw metrics, traces and logs and lets it write and execute code to poke around, which is expensive, unstable, and not very accurate. EviRCA splits the job: a deterministic, system-agnostic extraction stage turns raw telemetry into a compact set of multimodal “evidence cards”, and the LLM reasons only over those cards through a handful of read-only tools, never touching raw telemetry or running code. On OpenRCA, built from real telemetry across three enterprise systems, that design reaches a correct rate of 40.6% to 43.9% across two LLMs against prior baselines topping out at 15.2%, with 15 to 26 times fewer tokens and 3 to 20 times less wall-clock time. It also solves the hard cases that need simultaneous reasoning over time, component and root cause, where earlier approaches reported near-zero. The process-level failure analysis carries the sharper finding: remaining errors come from misjudging evidence the extractor had already surfaced, not from failing to find it. The extraction layer, not the reasoning layer, sets the ceiling.

Then there is the cost of running many agents at once, measured by someone who watched a 5-hour usage window vanish. A Claude Code user’s write-up (r/ClaudeCode) traces the burn to one mechanism: sub-agents default to a 5-minute prompt cache while the main session gets an hour, so any sub-agent that blocks inside a tool call longer than five minutes (a test sweep, a build, a git hook) comes back to a cold cache and rewrites its whole accumulated context, easily 300K to 600K tokens, on the next request. In their test-heavy multi-agent setup, one sub-agent rewrote a roughly 590K-token context eight times in a day, 5.4M tokens of cache writes on its own; across five agents it was 26 full rewrites and 12.2M tokens. Setting subagentPromptCacheTtl to 1h (Claude Code v2.1.242 or later) brought a comparable day down to 3.0M tokens of writes, about 75% fewer, and the 5-hour window moved from 0% to 22% instead of 2% to 100%. The half the setting does not cover: even an hour expires under a 40-minute sweep, so run the long thing in the background and have the agent poll. The orchestration lesson generalizes past one vendor’s cache: a blocking call inside a context-heavy worker is the most expensive idle time in the system.

On the agentic coding side, IBM Research’s Spotlights (Barzelay et al.) works on the step before the coding agent gets a task. Systems like AlphaEvolve and OpenEvolve can improve an implementation once someone names the target and the metric; Spotlights takes a repository, a high-level engineering objective such as “reduce serving latency”, and optional runtime evidence like profiles, and returns candidate source regions with a rationale, a proposed change, and an optional link to a relevant technique from the literature. No defect, bottleneck or file location is specified up front. It gets there through a logical repository map, successive agent reviews, and a research step. Evaluated on vLLM and llm-d, ColPali, Fabric-X, and an OCR pipeline, it recovered seven of nine expert-selected targets across three cases, 70% of its top ten candidates met the stated correctness and severity thresholds, and 73.6% of candidate occurrences pointed at the same source region across five repeated runs. One discovered change cut end-to-end page-processing runtime by 10.6% with measured output quality preserved. The 73.6% stability figure is the one to track; a discovery tool that finds different things each run is hard to build a workflow around.

Two retrieval papers round out the day. SELF-INDEX (Lee et al.) treats the index itself as the thing to optimize. Retrieval quality depends on how well index keys expose what each document contains, the right representation varies by corpus and retriever, and today the loop of diagnosing retrieval failures, revising the keys and reprocessing is done by hand. SELF-INDEX’s Optimizer diagnoses shortfalls, selectively rewrites the responsible keys, and validates each revision before committing it; a Query Simulator generates demands beyond the queries already observed, so the index evolves ahead of traffic. The authors report consistent gains across corpora and retrievers over existing index-optimization methods, with the benefit carrying into search agents and into agent memory systems retrieving past interactions. F²DR (Xiong et al., Baidu and Tianjin University) works the evaluation side of the same agentic search loop. Reward models built for single-turn tasks do not see the plan, retrieve, reflect, answer pipeline, so F²DR scores a DeepSearch workflow on three axes, Content, Trajectory and Answer, and the paper introduces DeepSearch RM-Bench to test reward models against it. They report higher evaluation consistency than self-evaluation baselines and enough spread across open-source reward models for the benchmark to discriminate between them. The dataset release is promised, not yet posted.

Nothing new landed on semantic governance in this window; the only hits were papers already featured earlier this week.

What to watch: ERPBench’s approval-gated harness and EviRCA’s read-only tool surface are two designs for the same instinct, keep the model away from the raw substrate and grade what it changed. Whether that instinct holds when the substrate is a warehouse and the record is a metric is the question the governed-analytics papers have not yet answered with numbers of this kind.

Feed note: the item mirror reports its last recorded sync as 2026-09-01, while the candidate pool carries items dated through 2026-09-20. Selections were made from the pool’s own dates.