A tied resolve rate hid a 2.25x cost gap between Opus 4.7 and Gemini 3.5 Flash

Two frontier models plugged into the same coding agent solved exactly the same number of tasks on one of JetBrains’ private benchmarks. Claude Opus 4.7 got there in an average of 184 steps at $2.79 per run. Gemini 3.5 Flash took 271 steps and cost $1.24. Identical score, 2.25x the price, and nothing else about the two runs was alike.

That tie is the starting point for JetBrains’ trajectory-level evaluation of Junie, published August 31, and it is the most useful piece of agent-eval work to land in the last day. Across 523 tasks in four benchmark datasets, Opus resolved 267 (51.1%) and Gemini resolved 254 (48.6%). The two models produced the same outcome on 430 of those tasks, solving 214 together and failing 216 together, so only 93 tasks separate the leaderboard positions at all. JetBrains scored each run on four axes instead of one: outcome, execution efficiency, patch quality, and process quality, with a deterministic layer pulled from logs and repository data (tokens, tool calls, modified symbols, repeated file reads, unchanged command retries, tool failure loops) and LLM judges assessing milestones like whether the agent reproduced the defect, found the root cause, and validated the fix.

The per-model profiles that fall out are specific enough to act on. Opus skipped executable validation entirely in 123 runs, including 68 runs that still resolved the task, which means a fifth of its wins were never confirmed by running anything. Gemini went the other way: it almost always ran a check, but 195 of its runs (37.3%) were judged to contain moderate or severe hallucination against 130 for Opus, and 80 runs (15.3%) showed significant or severe patch redundancy against Opus at 6.5%. On the wider four-model sweep over 522 shared tasks, GPT-5.5 took the top resolve rate at 51.5% and was the only model that always ran an executable check, Opus led every patch-quality metric, and Qwen 3.6 27B FP8 resolved 38.9% at three percent of GPT-5.5’s cost per run. GPT-5.5 and Opus finish four tenths of a point apart and within a cent per run, which a leaderboard reports as interchangeable; their patches are not, at 24.7% moderate-or-severe hallucination for Opus versus 33.7% for GPT-5.5 and 6.3% versus 13.2% on patch redundancy. JetBrains states the limits plainly: one Junie scaffold, and judges anchored to a single golden patch that is rarely the only valid solution.

The same problem shows up as a measurement artifact in Resource Constraints and Performance in Agentic AI Systems (arXiv 2608.27886, August 31), where Amaz Salman, Malka Halgamuge, and Teo Susnjak run OpenClaw against NanoBot on a paired benchmark plus an instrumented subset. On the primary benchmark, full task completion came in at 31% for OpenClaw and 25% for NanoBot, a six-point gap whose 95% task-bootstrap interval runs from -3 to +15 points, establishing nothing. On the instrumented layer the two systems tied at 26% full completion, but NanoBot reached at least partial completion on 43% of prompts against OpenClaw’s 26%. Resource use was not close: OpenClaw was slower on 83% of prompts and had a higher peak-memory value on every single one, with geometric mean ratios of 2.98 for wall time and 19.44 for peak memory. The authors then undercut their own dominance count, noting that of NanoBot’s eighteen dominance cases across all 23 prompts, ten were cheaper joint failures. Two evidence layers, the same two systems, different outcome labels. If you are picking an agent runtime on published completion rates, that pair of layers is the reason to ask which harness produced the number before you trust it.

Ranking instability shows up along a third axis in Terminal-Bench-LILT (arXiv 2608.28641, September 1), a suite of 300 coding tasks in Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese, each authored by native-speaker programmers and targeting problems with no direct English equivalent: internationalization, encoding, text normalization, cultural conventions. The strongest of six frontier models reached 63.1% pass rate, many tasks went unsolved by every model, and per-language performance does not track general coding benchmark rankings. Sample tasks are at github.com/lilt/terminal-bench-lilt.

Two papers out the same morning attack the localization stage that JetBrains’ judges keep flagging as the place runs go wrong. SpIDER builds a graph from per-repository syntax trees at the start of a developer session rather than precomputing an index, then expands dense-retrieval candidates along that graph, attaching to each surfaced function the seed and edge type that pulled it in so the candidate set stays auditable under a fixed retrieval budget. Recall@20 improves across every language and benchmark in the accompanying SpIDER-Bench (curated from SWEPolyBench, SWEBench-Verified, and Multi-SWE-bench over Python, Java, JavaScript, and TypeScript) by at least 13% relative along containment edges and at least 27% relative once call edges are explored. A^2Agent diagnoses a different failure in the same stage: agents frequently find the right code region while exploring and then never commit it, because trajectory-level reward gives no per-turn signal about which action mattered. Their fix pairs a per-turn reward sequence that pays for both discovery and commitment with an action-level advantage estimate that groups turns sharing an exploration context, gaining 1.58% average F1 over the prior state of the art on SWE-Bench Verified and 8.55% on SWE-Bench Pro, with a 4B model beating baselines up to 8x its size.

Where the gains come from is itself contested. openJiuwen (arXiv 2608.27969, August 31) holds the model policy fixed and moves the work into the harness, composing capabilities along “Rails” across single agents, delegated sub-agents, and a Swarm Flow mode on one shared execution substrate, then letting runtime evidence (semantic diagnostics, execution outcomes, task progress, shifting context relevance) steer context and task control mid-run. It reports 82.6% on SWE-bench Verified and 87.19% on Terminal-Bench 2.1, 3.4 and 3.39 points above the strongest selected official-leaderboard point estimates. A different bet on the same margin: Decispher, a Show HN from August 31, argues the recoverable waste is organizational, since agents re-derive context that already exists in PRs, tickets, Slack threads, and ownership boundaries. It reports 89% on the LongMemEval oracle split with GPT-4.1-mini as both extractor and reader, 81% on LongMemEval-S (89% with frontier models), and a 38x median token reduction, with a worker agent that asks a human when context is insufficient instead of guessing.

What to watch: whether trajectory-level metrics get published by anyone who is not also the vendor running the scaffold, and whether the two-evidence-layer disagreement in the OpenClaw comparison replicates. A resolve rate you cannot decompose is a number you cannot act on.