Agentic fix-PRs get rejected 46% of the time, and instruction files are a coin flip

Here’s a number to start with. 4% of BugFix pull requests opened by Copilot Devin, Cursor, and Claude end up rejected. Nearly half. That’s from a new study coming to the Mining Software Repositories Conference next year, out of a cold to technology superior in Montreal.

And the reason I want to spend time on it is not the headline percentage. It’s the autopsy underneath it, because the autopsy is more surprising than the number. The authors took a representative sample of 306 pull requests that were closed without being merged, and they read every one of them by hand. The discussion threads, the CI results, the comments.

Then they sorted the rejections into 14 reasons grouped under four headings. Now, if you’d asked me beforehand why agent fixes get rejected, I would have guessed. The code was wrong, the tests failed, the approach was bad, and those are in there. But the single most common reason at 17% is just inactivity.

The pull requests sat there untouched until a bot closed it for being stale. Nobody rejected it on the merits, it just rotted. Another 6% were superseded, meaning a human got impatient and wrote their own competing fix that landed first. So almost a quarter of these rejections aren’t about code quality at all.

They’re about the agent producing something that nobody ever engaged with, or engaged with too slowly. After that you get into the failures you’d expect. 5%. 5%.

CI and test failures, 7%. And then a category I found genuinely striking, the agent just failing, becoming unreachable, the session dying mid-task, producing no commits at all. One example in the paper is a Devin run where the message literally reads, Devin is currently unreachable, the session may have died, let’s just close this one. 5% of all the rejections.

The agent didn’t write bad code, it evaporated. Here’s the cost angle that makes this matter. These rejected fixes aren’t tiny. The median code churn runs between 81 and 293 lines, so every one of them required a human to read a real diff before deciding to throw it away.

The authors point out that almost half of all the review comments in the entire dataset land on pull requests that never merge. That’s the hidden tax on agentic contribution. It’s not just wasted tokens and compute on the agent’s side. It’s wasted human attention on the review side, which is the scarcer resource.

So the obvious question is, can you fix this with guidance? Can you write the agent a better instruction file and get a better merge rate? And the same research group published a companion paper the very same day asking exactly that. It’s called Toward Instructions as Code.

They looked at over 15,000 agentic pull requests across 148 projects, and for each project they compared the merge rate, the code churn, and the time to merge before and after that project added an instruction file. md, that kind of thing. And the result is humbling. Adding instructions is basically a coin flip.

35% dropped by at least that much. It almost perfectly cancels, writing an instruction file does not, on average, make your agent better at getting code merged. But there’s a signal in the noise, and it’s the part worth remembering. The projects that improved had substantially longer instruction files, broken into more sections and subsections.

md does nothing. The file only helps when it actually encodes the stuff the agent is missing. How to navigate the project, how to run the tests, what approaches are off-limits. Which, if you go back to the rejection taxonomy, from the first paper, is exactly the gap.

The agents fail on approach, on validation, on knowing what not to do. A real instruction file can close that. A token one can’t. So the two papers fit together.

Here’s why they fail, and here’s the partial, conditional fix. Now let me turn to the code that actually does land, because there’s a third paper this week that asks a different question. Not whether the PR merges, but whether the architecture rots underneath you when you adopt these tools. This one’s a little bit more complicated, but it’s a causal study from the University of Southern Denmark, and it’s the most careful version of this analysis I’ve seen.

They mined 151 Java repositories. md files, cursor rules, co-authored by trailers in the commit history. And they matched those against 77 control repositories that looked similar but had no AI markers. 1800 monthly snapshots, a proper staggered difference-in-differences design.

And the first result looks like great news for the AI tools. Architectural smell density, which is their measure of structural rot per thousand lines of code, drops almost 7% after a project adopts agentic tooling. You could write the headline, AI tools improve your architecture, and it would be wrong, because when they decompose it, the raw count of architectural smells is completely flat, plus 1%, not significant. What actually changed is the denominator.

Lines of code grew almost 13%, so the density went down purely because the code base got bigger at the same structural quality, not because anything got cleaner. The agents poured in more code at roughly the same architecture per feature. The improvement is a mirage created by dividing by a bigger number. I love this finding because it’s a methodological lesson that generalizes way past this one paper.

Anytime you measure something per thousand lines of code, defects per kLOC, or a number of lines of code, you can measure something per thousand lines of code. And you can measure something per thousand lines of code, defects per kLOC, or a number of lines of code, and you can measure something per thousand lines of code, and you can measure something per thousand lines of code, vulnerabilities per kLOC, churn per kLOC, and your treatment changes how much code people write, the ratio will lie to you. You have to report the raw counts. Otherwise you’ll announce an improvement that’s just a volume effect.

And it’s worth noting this actually contradicts an earlier code-level study on Cursor that found 30% more warnings and 41% more complexity. The Denmark authors reconcile it nicely. The AI may write messier code inside a module while still respecting the complexity of the code, while still respecting the boundaries between modules. Code-level mess and architecture-level mess are not the same thing.

Let me shift to memory and retrieval, because that side had a good week too. There’s a paper called Geelong that goes after long-term dialogue memory, and the angle is cost. Instead of paying a frontier model to manage memory on every single turn, they fine-tune a small language model to extract triples for a memory graph, and they add a clever importance score that reuses the cross-attention signals from a T5 summer riser to decide which memories actually matter. The payoff is up to nearly 10% better response quality on one benchmark and over 40% better retrieval recall on another, all while being dramatically cheaper to run.

And it lands in the same place a lot of recent memory work has landed. A lean, structured store beats both extremes. Beats dumping your whole raw history into context and beats running an expensive summer riser over everything every time you need to recall something. But the result I’d actually steal from my own work comes from an unlikely place, an e-commerce search paper called Iterating Toward Better Search.

They built a two-agent evaluation harness. One agent plays the buyer, configured with different personas and different patience levels, and it talks to a swappable responder architecture that hits a real-product search API. They held the buyer constant so they could compare responder designs on identical scenarios. 2,000 conversations, 14 persona buckets, and 3,000 questions.

4,000 questions, 14 personas, and 3,000 results. buckets, and three findings come out that apply far beyond shopping. First, rolling window memory beat intent extraction memory on every quality metric, and ran 35% faster per query. That’s now the third or fourth result this month pointing the same direction, keeping recent raw context often beats eagerly compressing it into extracted intent.

370b, the quality only moved by a fraction of a point on identical architecture. The scaffold mattered more than the model, and third the one that should make every eval engineer nervous, given the exact same scoring rubric Gemini and Claude systematically disagreed as judges. Gemini rewarded process correctness did the assistant follow good steps, Claude demanded concrete outcomes did the user actually get what they wanted, same prompt different values. Your LLM judge has a personality and it leaks straight into your numbers.

If you’re running one judge model and treating its scores as ground truth, you’re measuring that model’s temperament as much as your system’s quality. And to ground all of this, there was a practitioner write up making the rounds comparing the production memory systems people actually ship with, Letta, Mem0, Graffiti, and Cogni, mapping where each one sits from flat vector stores up to full knowledge graphs. It’s a good thing to read right next to the academic papers because the research keeps converging on structure and small items. So where does this leave us?

The thing I’m watching is whether anyone closes the loop these first two papers leave wide open. One paper hands you a taxonomy of exactly why agent fixes get rejected, the other tells you instruction files only help when they’re long and well structured. The experiment that writes itself is to take that rejection taxonomy, compile it directly into an instruction system. It’s the same template, the approach hints, the validation steps, the priority rules, and then measure merge rate against a control that didn’t get it.

Nobody has run that yet, when somebody does, that’s the number I want to see.