Pinterest puts a graded metrics layer behind MCP; coding agents over-defend in up to 58.7% of runs

Nearly 150 new metrics a month now land in Pinterest’s Metrics Board, and over 98% of the metrics used for experimentation go through it. Pinterest Engineering described the platform on October 6 as a thin semantic layer: each metric is a YAML spec in a Git repo, reviewed like code, compiled into its own Airflow DAG, validated by rule-based checks and the Warden anomaly detector, and published from one definition to the DataHub-based catalog, the Helium experimentation platform, Superset, and StarRocks. The design call worth reading closely is the one they declined to make. Rather than enforce a modeled semantic layer over every source table, Metrics Board accepts arbitrary SQL and asks the author to supply the modelling metadata around it: temporal columns, grain, dimensions. The team says the variety of source tables made a full model unrealistic. Governance then rides on metadata instead of on the model. Every metric carries a certification grade from Unknown through C, B and A to A+, the top rung manually verified by a data quality steward; dashboards sit in three tiers that drive retention, Presto cluster routing and documentation requirements; and all of it is exposed through an MCP server so Pinterest’s Analytics Agent answers natural-language questions from golden queries and certified definitions rather than raw tables. The post reports one engagement-metric revamp that saved roughly two engineering-months, and metric creation dropping from days to one or two hours. What it does not report is agent answer accuracy before and after grounding on certified definitions, which is the number this charter cares about. The grade and tier system is the mechanism; someone should measure its effect.

Text2Dashboard (arXiv 2610.06914, October 7) is the research-prototype counterpart, and its numbers are small enough to be useful. Wu et al. built a Codex plugin plus a standalone agent runtime over an enterprise DataBrain that turns analytic requests into dashboards. The model proposes actions; deterministic hooks own approval, audit, checkpointing, recovery and failure handling; SQL is forced read-only; output passes static checks, a dynamic preflight and browser inspection. On frozen real tasks, metadata selection passed 4 of 4, all four SQL tasks met semantic criteria but only 2 of 4 met the exact output-column contract, four single-panel dashboards passed, and a parameterised task blew through its step limit. All ten injected hook faults resolved as specified with no unapproved external side effects. Model inference was over 97% of runtime in every group. The authors state outright that this does not establish production readiness or general text-to-SQL accuracy. Read it for the architecture pattern, where the model proposes and software disposes with every state transition recorded, and for the fault-injection protocol, not for the scores.

ParanoiaEval (arXiv 2610.08662, October 6) measures something nobody had isolated: whether a coding agent’s defensive work is warranted. Luo et al. ground the benchmark in the Avoidance-Transfer-Mitigation-Acceptance framework from software risk management and build 200 repository-level task pairs that differ only in the evidence determining which treatment is correct. Across 8 models, unnecessary risk treatment shows up in 11.2% to 58.7% of runs despite explicit evidence that it is not needed, and stronger task capability does not predict better risk judgement. A post-hoc human study finds that treatment violations measurably degrade developer experience. The paper argues risk treatment is an independent capability axis, and the spread between agent configurations suggests harness and prompt choices move it as much as the base model does. If you have watched an agent wrap a two-line change in a try/except, a feature flag and a fallback path, this is the benchmark for that behaviour.

StateWise (arXiv 2610.05241, October 6) attacks a failure mode that compounds across sessions: agents reusing stored operational records, a path, a config value, a build command, that have since gone stale. Peng et al. audited coding-agent trajectories and found failure chains where invalid records are reused as premises, producing task failures and unsafe modifications. Their framework runs record-level counterfactual replanning to find which records are decision-critical, then checks each one’s current validity through reliability checks, read-only verification of machine-observable facts, and targeted clarification for developer-owned intent. Corrections are typed, scoped and carry repair lineage, the agent replans from the repaired state, and an independent state-action check gates execution. On 150 executable cases under corrupted persistent state, StateWise reaches 93.3% correctness against 38.7% for the baseline agent, with zero unsafe actions. The design maps onto any memory layer: verify the premise before acting on it, and persist the fix so the next session inherits the correction rather than the corruption.

Two retrieval papers today pull in opposite directions. NVIDIA’s Beyond Semantic Similarity (arXiv 2610.05750, October 6) wraps a retriever in a ReAct loop and gains 8.7 nDCG@10 points with the same embedding model, and the same pipeline lands competitive results on both ViDoRe v3 and BRIGHT, where specialized retrievers tend to fail out of domain. The bill is 107.4 seconds per query against 0.67 for standard retrieval, plus 764.1K input tokens and 5.8K output tokens per query. That is a 160x latency multiplier and a token budget that makes agentic retrieval a tool for a handful of high-value queries, not a default path. RAGFlip (arXiv 2610.07266, October 7) looks at the opposite risk. Swapping BM25 for BGE-large, E5-large-v2 or SPLADE improves aggregate coverage on Natural Questions, HotpotQA and FiQA in every setting, yet at k=1 between 8.6% and 37.5% of the queries BM25 already served correctly are lost. On HotpotQA at k=10, requiring every positive passage pushes the negative-flip rate to 12.3% to 17.3%, and a reader experiment shows some of those flips become answer regressions. Fixed-budget hybrids with BM25 cut the damage. Aggregate nDCG hides regressions on the queries your users already trusted, so ship retriever upgrades with a query-level compatibility check, the same way you would ship a model upgrade with a per-slice eval.

Attention Tax, Handoff Tax (arXiv 2610.06069, October 6) offers a reliability model for the single-agent-versus-decomposition argument that has produced contradictory results all year. Anchan and Sen frame it as two taxes: long context degrades a single agent’s attention, while decomposition pays a handoff cost each time information is compressed and transferred between agents. Redundancy via parallel sampling helps only to the extent that the samples’ failures are not shared. The model yields crossover conditions, and the test is the useful part. On a ledger-reconciliation task they measured the context-degradation curve and the handoff tax from single-agent and handoff runs alone, predicted a crossover at depth 10, and then found decomposition winning at depths 20, 50 and 100, with observed success within 9 percentage points of the prediction at every depth. Measure your own two taxes before choosing an orchestration topology.

What to watch: whether Pinterest or anyone else publishes agent accuracy deltas attributable to certified metric definitions, and whether the ParanoiaEval spread across configurations turns into harness-level mitigations rather than another model leaderboard.

Feed note: the code-intel mirror reported its last sync as 2026-09-01, yet the candidate pool contained items published through 2026-10-07. The status row looks stale; the data does not.