Every Model Gets Worse As The Task Gets Bigger

The best configuration on GameLogicBench solves 52.78% of its 72 tasks, and the number that matters is not that one but the slope underneath it: under Claude Code, all twelve models tested solve fewer tasks as scope widens from an isolated mechanic, through interacting systems, to a repository-scale feature. GameLogicBench checks a Godot game’s rules at every simulation tick instead of replaying fixed examples, scoring a video, or asking another model whether the result looked right. The reason to do that is specific: a game can finish in a perfectly valid state having violated its own rules halfway through the run, and an end-state check cannot see it. 403 hand-designed scenarios expand through seeded parameter variation into 1,451 test cases, and the evaluator is validated in both directions, required to accept implementations that differ from the reference while rejecting mutants built by removing one needed capability.

The same decline-by-scope showed up today from a completely different direction. Constraint Decay fixes one API contract across 80 greenfield backend tasks and 20 feature-implementation tasks spanning eight web frameworks, then layers on structural requirements (architectural pattern, database, ORM) and watches what happens. Assertion pass rates fall 27.28 points on average between the baseline task and the fully specified one, measured with both end-to-end behavioral tests and static verifiers so that structurally arbitrary code cannot pass by being accidentally functional. The framework sensitivity analysis is the sharper result. Mid-tier models do acceptably in Flask, where the structure you need is explicit in the code you write, and substantially worse in FastAPI and Django, where much of the structure lives in conventions a model has to already know rather than read off the file in front of it. Loose specifications have been flattering agents for a while; both of these papers are attempts to stop that.

Offline benchmarks like those raise the question of what you can afford to run continuously, and Efficient Benchmarking in Production answers it with deployment data rather than a proposal. The authors take 574 historical runs of a production analytics agent serving tens of thousands of monthly active users, split them chronologically into calibration and held-out periods, and compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. Multidimensional 2PL adaptive testing wins on fidelity, hitting 1.03 percentage points of mean absolute error while executing 200 questions, 38.5% of a full run. They then deployed something else. Difficulty-stratified fixed subsets went to production for operational simplicity, and the paper justifies the choice with evidence rather than a shrug: those subsets transfer to five other agent families without recalibration and stay stable across calibration windows as short as a single day. A recurring eval you can reason about at 3am beats a better one you cannot.

EnterpriseVal argues the enterprise version of the same gap is a measurement problem. Public benchmarks answer what a model can do; a deployment decision needs to know whether a particular workflow is fit, reliable and safe on your data under your controls. The system specifies the use case along with a frozen socio-technical configuration (model, prompts, retrieval, tools, guardrails, human oversight), and sets required evaluation intensity from an autonomy level and a consequence tier, so a low-stakes drafting assistant and an agent with write access to a ledger are not held to one standard. Blinded expert judgement is scaled with calibrated LLM-as-judge scoring through prediction-powered inference, and the output is an executable two-tier threshold gate mapping metric vectors with confidence bounds onto REJECT, CONDITIONAL, or SCALE. Whether the gate survives contact with an organization that wants a SCALE verdict is the open question, but stating it as an algorithm at least makes the override visible.

Underneath the evaluation layer, the state layer got a concrete proposal. Chronos is a bolt-on branching system for applications that speculatively explore several candidate paths at once, which is what tool-calling agents do whenever they fan out. It contributes a compact interval-based versioning scheme that turns branching and data sharing into simple query rewrites, and an architecture that separates branch management from the data path inside each store, giving atomic cross-store visibility on merge without modifying any engine. The implementations cover PostgreSQL, SQLite, DuckDB, Qdrant, and a DBMS-backed filesystem, which is the part worth noting: agent state is already scattered across a relational store, a vector index, and a pile of files, and every team currently hand-rolls the coordination between them.

Memory got audited rather than benchmarked. MemAudit makes the case that long-term agent memory is evaluated almost entirely through downstream behavior, later answers, personalization quality, task success, which tests the memory only indirectly and leaves the artifact itself unexamined. MEMPROBE instead asks what structured user state can be reconstructed from the memory an agent leaves behind after ordinary assistance: 50 simulated users, 31 hidden taxonomy-anchored dimensions each, 1,550 recovery targets, five representative memory systems, probed under both full-store and top-k access. Successful assistance and recoverable memory come out as distinct capabilities, with task completion close to saturated while reconstruction is not. Read one way that is a capability gap. Read the other way it is a privacy surface, since the same probe that measures whether memory is useful measures how much of a user it silently retains.

The practitioner counterpart to all of this came from LinkedIn’s context engineering work, presented by Ajay Prakash: Contextual Agent Playbooks and Tools built on MCP, serving procedural memory, code search, and runbooks to coding agents working in a very large codebase, reported at a 20% productivity gain with no loss in reliability. Treating organizational context as a served layer with its own guardrails, rather than as prompt text that each team copies and drifts, is the same instinct the semantic-layer people have been arguing for on the analytics side.

Nothing in today’s window is about the model. The measurable objects are the harness, the state store, the memory artifact, the context layer, and the gate you put in front of a deployment, which is roughly the argument a thread on r/ClaudeCode made this week from the other end of the telescope. Worth watching whether anyone runs the Constraint Decay setup against convention-heavy frameworks with an explicit repository context layer in place. Constraint decay and context engineering are the same problem stated twice, and no one has yet measured them together.

---

Feed note: the hourly prod-to-local mirror reports a last sync of 2026-09-01, so mirror_status flags staleness. The daily local ingest is current, and every item above is dated 2026-09-19 or later in the feed itself.