A Kotlin Benchmark, a Closed-Loop Reviewer, and Memory on a Budget

Claude Code running Opus 4.7 at xhigh effort resolved 90 of 105 tasks on JetBrains’ new Kotlin Benchmark, an 85.71% clear rate that edged out JetBrains’ own Junie on Opus 4.7 max (81.9%) and Codex on GPT-5.5 xhigh (81.9%). The benchmark, published July 8, ports the SWE-bench methodology to real Kotlin repositories: 105 engineering tasks pulled from active open-source projects, each requiring an agent to read an issue, navigate the codebase, and produce a patch that passes containerized test verification. JetBrains built it on the open Multi-SWE-bench infrastructure and says this first run predates the newest model releases, so read the numbers as a floor, not a ceiling.

The benchmark’s more interesting admission is buried in its “what’s next” section: pass/fail resolution rate is only one axis, and JetBrains plans to add cost, performance, maintainability, and code quality to future iterations. That’s the same complaint driving AgentLens, a new evaluation framework that treats a coding agent’s entire trajectory, not just the final patch, as the unit of measurement. AgentLens pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side run comparisons, producing a readable explanation for every score instead of a single pass/fail bit. The authors run it as a nightly regression pipeline against their own production agent, which is the real test of a trajectory-level eval: does it catch regressions that a resolve-rate benchmark would miss.

Closing the loop from the other direction, SWE-Review puts a second agent in the review seat. Given an issue and an AI-generated pull request, a reviewer agent explores the repository, decides whether to accept, and writes structured revision feedback, then the cycle repeats. Across the authors’ SWE-Review-Bench, the generate-review-revise loop beat single-turn fixed-context review on both decision accuracy and post-revision resolve rate, and the reviewer’s skills transferred to improve issue-resolution models directly. The implicit claim is that one-shot PR generation was never the end state, just the version that shipped first because it was easier to build.

Memory work this window clustered around bounding context rather than expanding it. SelfMem drops the fixed retrieval-and-summarization pipeline most memory frameworks hardcode and instead gives the agent memory tools plus feedback signals, letting it learn its own storage and retrieval strategy. On the BEAM benchmark, across conversations from 100K to 1M tokens, SelfMem beat the strongest baseline by 48.7%, 40.8%, and 41.9% at each scale respectively, with a model-guided refinement pass adding further gains on top. Ensemble QSP takes a more structural approach: a three-layer hierarchical memory that caps each state category and evicts finished work, holding injected context to a median of 301 tokens and a max of 4,050 across 104 autonomous pharmacokinetic-modeling runs. Five specialist worker agents report to domain-expert principal agents that enforce physics-based checklists, and the architecture is domain-agnostic by construction: swap in a new principal-agent configuration and it runs a different science.

Both memory papers are solving the problem coding agents hit constantly: context that grows without bound degrades everything downstream. That’s the exact complaint behind Magic Compact, a Claude Code plugin one developer open-sourced this week after watching the built-in /compact flatten hours of reverse-engineering work into a generic summary that captured, in their words, maybe 10% of what mattered. Instead of collapsing an entire conversation into one blob, Magic Compact keeps every user message intact, replaces old assistant turns with per-turn summaries, and caches large tool output for later retrieval, so the agent wakes up from compaction still knowing what it already tried. It’s a plugin, not a paper, but it’s the same architectural bet as SelfMem and Ensemble QSP: bound what gets replayed, don’t try to remember everything.

On the infrastructure side, former GitHub CEO Thomas Dohmke’s new startup Entire came out of preview with a distributed Git network aimed at agent-driven read load: regional mirrors of GitHub repos that let coding agents clone from nearby infrastructure instead of hammering one central host. Entire’s internal benchmarks claim roughly 570,000 clones per hour from a single repository and 586 pushes per second. The more interesting bet is Entire Blame, which extends git blame to show which AI conversation and prompt produced a given line, alongside semantic search over why code was written rather than just what changed.

Watch whether AgentLens’ and SWE-Review’s trajectory-level and closed-loop evaluation ideas make it into the next round of public benchmarks; resolve rate alone is starting to look like a floor, not a scoreboard.