Code Retrieval & Enterprise Codebases

Plain-language abstract This paper introduces the CodeSearchNet Corpus and Challenge, a large benchmark for evaluating systems that retrieve relevant code given a plain-language description. The corpus contains about 6 million functions from open-source code across six programming languages, along with roughly 2 million automatically generated natural-language descriptions scraped from function documentation. The challenge itself is built around 99 natural-language queries with approximately 4,000 expert-annotated relevance judgments.

Motivation Finding code using natural language queries requires bridging the gap between the abbreviated, technical vocabulary used in code and the more informal language people use to describe what they want. Without a shared, large-scale benchmark with human relevance judgments, it was difficult to measure progress or compare approaches on this task.

Methodology The authors assembled the CodeSearchNet Corpus by scraping open-source repositories for functions in Go, Java, JavaScript, PHP, Python, and Ruby, then mechanically extracted and preprocessed associated documentation to produce query-like natural-language strings for roughly 2 million functions. A set of 99 natural-language queries was constructed and expert annotators labeled the relevance of candidate code results, yielding about 4,000 annotations. Several simple baseline retrieval models were implemented and evaluated against these labels.

Results The paper releases the corpus, the expert-annotated challenge set, and baseline model results, establishing a leaderboard and competition to track future progress. The baseline solutions provide initial reference scores against the 99 query challenge set, and the authors report that the corpus spans approximately 6 million functions across six languages with 2 million documentation-derived natural-language pairs available for training.

Plain-language abstract CodeBERT is a pre-trained model that learns to understand both natural language (like English) and programming languages (like Python or Java) at the same time. By training on millions of paired code-and-documentation examples from GitHub, it builds representations that can be fine-tuned to help with tasks such as searching for code using plain-language queries or automatically generating documentation for a function.

Motivation Large pre-trained language models had dramatically improved natural language processing, and multimodal models had extended this to language-image and language-video pairs. However, no large pre-trained model existed that jointly captured the semantic connection between natural language and programming languages across multiple languages, leaving tasks like natural language code search and code documentation generation without a strong shared foundation.

Methodology CodeBERT uses a multilayer Transformer architecture and is trained on GitHub repositories in six programming languages (Python, Java, JavaScript, and others). Training uses a hybrid objective combining standard masked language modeling and replaced token detection: bimodal NL-PL pairs provide paired supervision, while unimodal code (functions without documentation) helps train better token generators for the replaced token detection objective. The model is then fine-tuned for downstream tasks; evaluation covers natural language code search on the CodeSearchNet corpus (measured by Mean Reciprocal Rank) and code-to-documentation generation (measured by smoothed BLEU-4 score). A NL-PL probing dataset is also constructed to assess what knowledge is encoded in the pre-trained model in a zero-shot setting.

Results Fine-tuned CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation across six programming languages. On code documentation generation, CodeBERT pre-trained with both replaced token detection and masked language modeling objectives gains 1.3 BLEU score over the RoBERTa baseline overall. In zero-shot probing experiments, CodeBERT consistently outperforms RoBERTa, indicating that it encodes knowledge about the relationship between natural language and code beyond what a purely text-based model captures.

Plain-language abstract GraphCodeBERT is a pre-trained neural model for programming languages that learns to understand code by taking into account how data flows between variables, not just the sequence of tokens. It was published as a conference paper at ICLR 2021 and evaluated on four practical coding tasks: searching for code using natural-language queries, detecting duplicate code, translating code between languages, and automatically fixing bugs.

Motivation Prior pre-trained models for code treat source code as a flat sequence of tokens, ignoring the structural relationships that give code its meaning. Because variable names can be ambiguous or non-descriptive, understanding what a variable represents requires knowing where its value comes from — information encoded in data flow but absent from token sequences alone. The paper addresses this gap by incorporating semantic-level code structure into pre-training.

Methodology The authors built GraphCodeBERT on top of the Transformer architecture, extending it with a graph-guided masked attention function that incorporates a data-flow graph, where nodes are variables and edges represent where each variable's value originates. They introduced two new structure-aware pre-training tasks alongside masked language modeling: predicting data-flow edges, and aligning variable representations between source code and the data-flow graph. The model was pre-trained on the CodeSearchNet dataset of 2.3 million functions across six programming languages, using two DGX-2 machines with 16 NVIDIA Tesla V100 GPUs each, and trained for 200,000 batches (roughly 83 hours).

Results GraphCodeBERT achieved state-of-the-art performance on all four downstream tasks evaluated. On natural language code search, it reached an overall mean reciprocal rank of 0.774 across six languages, outperforming the prior best model CodeBERT (0.760). Ablation experiments confirmed that both the data-flow code structure and the newly introduced pre-training tasks contributed to the improvements, and analysis showed the model consistently preferred attending to data-flow structure over token-level connections in the code search task.

Plain-language abstract CoIR (Code Information Retrieval Benchmark) is a standardized evaluation framework for testing how well AI models can search and retrieve code. It bundles ten curated datasets spanning eight retrieval tasks across seven coding domains, and packages them in a Python library compatible with the widely used BEIR and MTEB evaluation ecosystems so researchers can benchmark models with a single pip install.

Motivation Existing code retrieval benchmarks such as CodeSearchNet, CosQA, and XcodeEval cover only a narrow slice of real-world retrieval needs, focus almost exclusively on text-to-code search, and have been used so heavily that many models have overfit their leaderboards. There was also no shared evaluation framework, making it impossible to compare results across benchmarks consistently.

Methodology The authors assembled ten datasets — eight adapted from existing sources and two newly created — covering four primary retrieval task types (Text-to-Code, Code-to-Code, Code-to-Text, and Hybrid Code Retrieval) across fourteen programming languages. Each dataset was manually inspected and cleaned. Ten retrieval models were then evaluated on CoIR: one sparse baseline (BM25), several open-source dense models (E5-Base, GTE-Base, BGE-Base, Contriever, E5-Mistral, BGE-M3, UniXcoder), and two proprietary models (OpenAI-Ada-002, Voyage-Code-002). Models were scored using NDCG@10 following the BEIR protocol.

Results No single model dominated all tasks. Voyage-Code-002 achieved the highest mean score of 56.26 but with high variance, indicating weak generalization. BGE-M3 showed the lowest variance despite moderate average performance, suggesting stronger robustness. Top-performing text-retrieval models such as E5-Mistral did not consistently excel across CoIR sub-tasks, underscoring that code retrieval requires capabilities beyond general text retrieval. The evaluation confirmed that even state-of-the-art systems struggle on code retrieval and that many models have overfit existing benchmarks.

Plain-language abstract This paper introduces CoSQA+, a new benchmark for evaluating code search systems — tools that retrieve code snippets matching natural-language queries from programmers. Unlike earlier benchmarks that pair each query with just one correct code example, CoSQA+ pairs queries with multiple valid code snippets, reflecting how developers actually work. The authors also build an automated pipeline using AI agents that verify code correctness through test execution rather than relying on human judgment alone.

Motivation Existing code search benchmarks such as CoSQA and CodeSearchNet assume each query has a single correct answer and rely on human annotators who judge matches by reading code rather than running it. A survey of 102 Python programmers found that developers typically consult 2–3 code examples per query, and perform about 8.31 code searches per programming day, revealing a mismatch between how benchmarks are designed and how code search is actually used. Human and LLM-based annotation also suffer from accuracy limitations because they assess semantic similarity rather than functional correctness verified by test execution.

Methodology The authors constructed CoSQA+ with 412,080 query-code pairs by combining web queries from CoSQA with repository code from CodeSearchNet. Multiple models were used to select the top 20 candidate code snippets by similarity for each query. A novel test-driven agent pipeline then annotated these pairs through five stages: a preliminary screener, a test program generator, a test executor, a bug fixer, and a final arbiter, using test execution to determine whether each code snippet functionally satisfies the query. Annotation quality was evaluated against ground truth labels established by three Python experts on 400 randomly selected pairs. The paper also introduced NDCG@10 as an evaluation metric and benchmarked 8 code search methods, 7 of which are embedding-based.

Results The test-driven agents achieved an annotation accuracy of 92.0% on the 400-pair evaluation set, outperforming both a single LLM annotator and human Python experts annotating without test-based verification. Notably, 83.67% of the test programs automatically generated by the agents were executable. Models fine-tuned on CoSQA+ — specifically CodeBERT, UniXcoder, and CodeT5+ embeddings — consistently outperformed models trained on the original CoSQA on the CSN99 Python benchmark as measured by both NDCG@10 and MRR metrics.

Plain-language abstract This paper presents DeepAPI, a system that takes a plain-English question like "how do I parse XML files?" and automatically generates the sequence of API calls a developer should use to accomplish that task. It treats the problem as machine translation, converting a natural language query into an ordered list of API method calls from software libraries like Java's JDK.

Motivation Developers frequently struggle to discover which APIs to use and in what order when working with large, unfamiliar libraries — the .NET framework and JDK each contain hundreds to thousands of APIs, and documentation is often inadequate. Existing search approaches treat queries and APIs as unordered bags of words, so they fail to capture word order or semantics and cannot, for example, distinguish "convert int to string" from "convert string to int".

Methodology DeepAPI adapts an RNN Encoder-Decoder neural language model that encodes a sequence of query words into a fixed-length context vector and then decodes an API call sequence from that vector. The model was trained on more than 7 million annotated Java code snippets mined from GitHub, with 10,000 instances held out for testing and the remainder used for training over 240 hours (1 million iterations). The authors also augmented the base model to weight individual APIs by their importance.

Results DeepAPI achieved an average BLEU score of 54.42 on the test set, substantially outperforming a code-search-plus-pattern-mining baseline (BLEU 11.97) and the prior SWIM word-alignment approach (BLEU 19.90). On 30 real API queries drawn from query logs and related work, the average rank of the first relevant result was 1.6; 80% of top-5 results and 78% of top-10 results were judged relevant.

Plain-language abstract This paper studies how to build better search systems that let developers find code snippets using plain English questions. The key insight is that code snippets often come with short human-written descriptions of what they do, and the authors show that exploiting those descriptions dramatically improves search quality compared to systems that look only at raw code.

Motivation Searching large code collections using natural language is hard because the meaning of a code fragment is rarely obvious from its tokens alone — even experienced developers struggle to judge relevance without extra context. Prior neural code-search systems all tried to infer intent from unannotated source code, leaving a gap: no system had systematically studied what happens when brief human-written descriptions accompany the snippets and are used at retrieval time.

Methodology The authors defined an 'annotated code search' task where every snippet is paired with a short natural language description, then built three benchmark datasets (PACS) drawn from existing resources including the CoNaLa corpus. They fine-tuned a pre-trained NLP transfer-learning model (the Universal Sentence Encoder, chosen because its training data included Stack Overflow content) to predict semantic similarity between a user query and a snippet's description, and compared this approach against classic information-retrieval baselines and neural code-search models such as NCS and UNIF that operate on unannotated code.

Results The fine-tuned description-based retrieval model substantially outperformed all code-only baselines on all three benchmarks, achieving absolute gains of up to 20.6 percentage points in mean reciprocal rank. The experiments also showed that including the snippet source code alongside the description provides additional benefit beyond descriptions alone, but descriptions are the dominant signal for obtaining relevant results.

Plain-language abstract CodeMatcher is a code search tool that helps software developers find relevant code snippets using plain-language queries. It combines the speed of traditional keyword-based search with the semantic understanding of deep learning models, allowing developers to describe what they need in natural language and retrieve matching code methods from large open-source codebases.

Motivation Developers spend the majority of their coding time searching for reusable code, yet existing approaches faced a fundamental trade-off: simple keyword-matching tools were fast but semantically shallow, while deep learning models like DeepCS could understand query meaning and sequential word relationships but were impractically slow for large codebases. There was no method that achieved both semantic understanding and search efficiency at scale.

Methodology CodeMatcher first collects metadata about query words to identify and filter out irrelevant or noisy terms, retaining only the important ones. It then performs iterative fuzzy search over a codebase indexed with Elasticsearch, and finally reranks the returned candidate code snippets by how well their tokens sequentially match the important query words. The system was evaluated on a large-scale codebase comprising approximately 41,000 GitHub repositories.

Results CodeMatcher achieved a Mean Reciprocal Rank (MRR) of 0.60, outperforming DeepCS by 82%, CodeHow by 62%, and UNIF by 46%. It was also more than 1,200 times faster than DeepCS. Compared to existing online search engines, CodeMatcher surpassed GitHub search by 46% and Google search by 33% in MRR. The authors additionally observed that improving method naming quality aids code search, as method names play a key role in bridging natural language queries and code.

Plain-language abstract This paper is a comprehensive survey of code search — the tools and techniques that help software developers find relevant source code examples within a codebase or across open-source repositories. It covers 30 years of research, synthesizing 109 papers, and organizes the field around the full pipeline of a code search engine: how queries are expressed, how code is indexed, and how results are ranked.

Motivation Massive amounts of source code exist across modern software projects and platforms like GitHub, where over 60 million new projects were created in 2020 alone. Most code a developer needs to write has likely already been written somewhere, making effective code search a high-value activity — yet no unified survey had mapped the landscape of techniques for building code search engines across the full research history.

Methodology The authors surveyed 109 research papers on code search published over approximately 30 years (roughly 1990 to 2022). They organized the literature around the key components of a code search engine: the kinds of queries supported (natural language, code snippets, formal specifications, test cases, and dedicated query languages), query preprocessing and expansion techniques, methods for indexing and retrieving code, and strategies for ranking and pruning search results. The survey also covers empirical studies of how developers use code search in practice.

Results The survey produces a structured taxonomy of the code search problem and existing solutions, identifying the major technical challenges and how prior work has addressed them. Based on this synthesis of prior work, the paper concludes with an outline of open challenges and research opportunities for future code search systems.

Plain-language abstract This paper is a large-scale survey of how language models — the AI systems behind tools like GitHub Copilot — are used in software engineering. It covers more than 70 models, 40 evaluation tasks, 180 datasets, and around 900 related papers, tracing how AI assistance has grown from simple code completion to supporting the entire software development lifecycle.

Motivation The NLP research community and the software engineering community have studied language models for code largely in isolation, with existing reviews focusing on one side or the other but not both. The authors identified a gap: no prior work had jointly examined how SE uses language models for development automation and how NLP uses SE tasks as benchmarks, nor had any review covered LLM applications across the full software development lifecycle beyond programming.

Methodology The authors conducted a systematic literature survey, organizing their coverage around a taxonomy of software engineering stages. They reviewed code-processing models by distinguishing general language models (such as the GPT family) from specialized code models pretrained with tailored objectives, and traced the historical evolution from statistical models and RNNs through pretrained Transformers. Coverage extends beyond code generation to requirement engineering, testing, deployment, and operations. The survey is maintained as a living document on GitHub.

Results The survey documents that advanced NLP techniques — including instruction tuning, infilling objectives, revised scaling laws, architectural improvements, and autonomous agents — have been progressively introduced into code modeling, while SE tasks in turn provide real-world benchmarks that drive LLM development. The authors identify the bidirectional relationship between the two communities as a key structural finding, and flag key open challenges and future directions across the full software engineering lifecycle.

Plain-language abstract This paper introduces CodeBLEU, an automatic metric for evaluating how good a piece of synthesized code is. Unlike the standard BLEU score borrowed from machine translation, CodeBLEU is designed specifically for programming languages: it checks not just whether tokens match a reference, but also whether the code has the right syntactic structure and correct logical behavior.

Motivation Researchers building systems that automatically generate code had no reliable way to measure output quality. BLEU ignores the special importance of programming keywords and tree-structured syntax, while exact-match accuracy is so strict that it penalizes correct programs written in a different but equivalent style. A metric that better tracks how human programmers actually judge code quality was needed.

Methodology CodeBLEU combines four components into a weighted sum: the standard BLEU n-gram match, a weighted n-gram match that gives programming keywords five times higher weight than other tokens, a syntactic match based on sub-trees extracted from abstract syntax trees (AST) parsed with tree-sitter, and a semantic match based on data-flow graphs. The authors evaluated the metric by computing Pearson correlation coefficients between CodeBLEU scores and quality ratings (1–5 scale) assigned by ten human programmers, across three code synthesis tasks: text-to-code (C#/Java, 100k training samples), code translation between Java and C# (11.8k method pairs), and code refinement for bug fixing (Java, ~46–52k samples per subset). Four systems of varying capability were scored per task.

Results CodeBLEU achieved higher Pearson correlation with human programmer scores than standard BLEU on all three tasks, with improvements of 1.0%, 3.0%, and 5.6% for text-to-code, code translation, and code refinement respectively. It also outperformed exact-match accuracy on text-to-code and code translation. An ablation study showed that the syntactic AST match and semantic data-flow match components individually contributed the strongest correlation with human judgments on the text-to-code and code translation tasks, while the weighted n-gram and data-flow components mattered most for code refinement. Statistical testing confirmed the score differences between systems were significant.

Plain-language abstract This paper tackles a fundamental obstacle in applying machine learning to source code: the vocabulary problem. When neural language models are trained on code, the sheer number of unique identifier names makes the vocabulary unmanageably large and causes frequent failures on names never seen during training. The authors study how vocabulary design choices affect this problem, propose an open-vocabulary neural language model using Byte-Pair Encoding (BPE), and demonstrate that it scales to a corpus of over 13,000 software projects — far larger than prior work.

Motivation Neural language models have shown strong results for natural language but struggle with source code because programmers coin new identifiers freely, producing vocabularies that grow without bound. Prior work showed that neural models could not scale past roughly a hundred projects before performance degraded due to rare and out-of-vocabulary tokens. There was no principled, large-scale study of how vocabulary choices interact with model scalability, nor a proven technique for making neural code models open-vocabulary.

Methodology The authors first conducted an empirical study on a corpus of 13,362 open-source projects across Java, C, and Python, systematically varying vocabulary design choices such as handling of comments, string literals, whitespace, identifier splitting by camelCase and underscores, and frequency filtering, measuring the impact on vocabulary size and out-of-vocabulary rate. They then built a neural language model that applies Byte-Pair Encoding to tokenize source code into sub-word units, combined with beam search and a caching mechanism, enabling it to predict tokens not seen during training. The model was trained at scales up to 13,362 projects.

Results The study found that vocabulary design choices cause variations in vocabulary size of up to three orders of magnitude, and that common splitting strategies such as camelCase and underscore splitting are insufficient to bring vocabularies to a manageable size — BPE is required. The open-vocabulary BPE neural language model outperformed both n-gram language models and closed-vocabulary neural models on code completion across all three languages (Java, C, Python), and was more effective than prior language models at the downstream task of highlighting buggy code.

Plain-language abstract MISIM is a neural system for determining whether two pieces of code do the same thing, even when they look very different on the surface. It introduces a new way of representing code called the Context-Aware Semantics Structure (CASS) that strips away syntactic clutter to expose underlying meaning, and pairs it with a learned similarity scoring algorithm tested across three neural network architectures. The system was evaluated against four state-of-the-art code similarity methods over roughly 328,000 programs comprising more than 18 million lines of code.

Motivation Existing code similarity systems rely on structural representations like abstract syntax trees or simplified parse trees that are either too syntactically dense, require compilation to an intermediate representation, or introduce ambiguities—causing models to memorize syntax rather than learn semantic meaning. There was no representation purpose-built to lift semantics from code syntax while remaining language-extensible and usable in interactive developer environments.

Methodology The authors built MISIM around a configurable code representation called CASS, which uses a global attributes table and a structured tree that resolves syntactic ambiguities present in ASTs and SPTs. They trained three neural network variants on top of CASS—a bag-of-features model (MISIM-BoF), a bidirectional GRU-based recurrent model (MISIM-RNN), and a relational graph convolutional network (MISIM-GNN)—using metric learning with cosine similarity and Circle loss. Experiments used the GCJ and POJ-104 benchmark datasets and compared against code2vec, code2seq, Neural Code Comprehension, and Aroma, with MAP@R and Average Precision as evaluation metrics.

Results MISIM-GNN achieved at least 8.08% better MAP@R accuracy than the next best performing system across all experiments. Ablation studies confirmed that CASS outperformed both AST- and SPT-based representations under the same neural architectures, demonstrating that the structural representation itself—not just the neural model—was responsible for the accuracy gains.

Plain-language abstract This paper is a systematic literature review of research on automatically improving the search queries that software developers use when looking for code. Developers routinely search both their local codebase and the Internet for code relevant to fixing bugs or adding features, but picking the right search keywords is hard — 76% of developer queries need at least one reformulation to succeed. The authors surveyed 70 carefully selected studies on automated tools that construct or rewrite those queries, answering seven research questions about how those tools work, what they accomplish, and where they fall short.

Motivation Software maintenance consumes up to 80% of a project's total budget, and a large fraction of that time is spent searching for the right code to change. Developers routinely fail to choose effective search queries from bug reports or feature requests, partly because there is only a 10–15% chance of guessing the exact vocabulary used in the codebase. No prior work had comprehensively catalogued the landscape of automated query reformulation techniques, their methodologies, limitations, and open problems, leaving researchers without a clear picture of what had been tried and what remained unsolved.

Methodology The authors conducted a systematic literature review following established guidelines, screening 2,970 candidate studies and selecting 70 primary studies for in-depth analysis. They applied Grounded Theory coding — spending approximately 100 man-hours on open coding (producing 209 open codes for methodologies alone) and axial coding to establish relationships among codes — then organized findings around seven research questions covering methodologies, evaluation practices, limitations, and the distinction between local and Internet-scale code search.

Results Eight major methodology families were identified for query reformulation, including term weighting, relevance feedback, semantic relations, word co-occurrence analysis, and thesaurus lookup; about 40% of studies use term weighting or relevance feedback and 46% use semantic or co-occurrence approaches. Of the 70 studies, 58% target local code search (bug localization, feature location) and 42% target Internet-scale code search. The review identifies eight recurring limitations across primary studies — including noisy keywords and vocabulary mismatch — and concludes that current keyword-selection algorithms are insufficient, pointing to integrating richer context and semantic understanding as the most promising direction for future work.

Plain-language abstract This paper introduces Contriever, a system for searching large document collections without any labeled training data. Instead of relying on keyword matching (as traditional search engines do) or needing human-annotated query-document pairs (as most modern neural retrievers require), Contriever learns to retrieve relevant documents purely from raw text using a technique called contrastive learning. The approach works across languages and can even retrieve documents across different writing scripts.

Motivation Neural network-based document retrievers achieve strong performance on well-resourced benchmarks but fail to generalize when no in-domain training data is available, consistently losing to classical keyword-matching methods like BM25 in zero-shot settings. This gap is especially acute for non-English languages where large annotated retrieval datasets simply do not exist. The paper asks whether contrastive self-supervised learning can close this gap without any manual supervision.

Methodology The model uses a bi-encoder architecture built on BERT base, where queries and documents are encoded independently and relevance is scored by the dot product of their representations. Training is fully unsupervised: positive pairs are constructed from a single document by independently cropping two random spans of text (treated as a query and a matching key), and the model is trained with the InfoNCE contrastive loss to bring representations of spans from the same document closer together while pushing apart spans from different documents. The authors compare this cropping strategy against the Inverse Cloze Task and other augmentations, and also train a multilingual variant. Evaluation uses the BEIR benchmark (15 diverse retrieval datasets) plus multilingual benchmarks.

Results The unsupervised Contriever model outperforms BM25 on 11 out of 15 BEIR datasets at Recall@100, the first dense retriever to broadly surpass BM25 without any supervision. When used as pre-training before fine-tuning on a few thousand in-domain examples or on the full MS MARCO dataset, it further improves over strong supervised baselines on BEIR. The multilingual model achieves strong unsupervised cross-lingual retrieval, including retrieving English documents from Arabic queries — a task impossible for term-matching methods.

Plain-language abstract CORE-Bench is a benchmark for code retrieval as coding agents actually do it: instead of matching a natural-language query to one isolated snippet, it tests whether a retriever can, given a real development request and a snapshot of a repository, find the files and functions that need editing and the surrounding context required to make the change. It is organized in three levels — basic code understanding, issue-to-edit localization, and broader context retrieval — built largely from SWE-bench-series issues and their repositories.

Motivation In agentic and 'vibe' coding, retrieval is a step inside the agent's loop: given a bug report or feature request the agent must decide which files and functions to inspect before editing. Existing benchmarks such as CodeSearchNet and the code tasks in MTEB/CoIR only test decontextualized snippet matching under fixed corpora, missing five properties that matter in practice — a large intent-to-implementation gap, evidence scattered across code, configuration, and dependencies, long function-level chunks that dilute a single embedding, requests that need multiple edit locations, and dense in-repository distractors. Strong scores on those benchmarks can therefore overstate how useful a model is to a coding agent.

Methodology The benchmark has three levels. Level-1 reuses curated traditional retrieval tasks (text-to-code, code-to-code, hybrid). Level-2 (issue-to-edit localization) is built from SWE-bench-series instances: each repository is checked out to the commit immediately before the PR so the corpus matches the pre-edit state, source and documentation files are AST-chunked, and the PR patch is aligned back to those chunks to produce relevance labels, with an LLM filter (Qwen3.5-397B) removing answer-leaking or underspecified descriptions. Level-3 (broader context) reruns resolution with a code agent (Mini-SWE-Agent), extracts the files it browses from execution traces, judges each with a three-vote LLM ensemble (two Qwen votes plus one Claude Sonnet 4.6 vote), then verifies usefulness by re-running the agent under a closed allowlist of only the annotated files. Models are scored by NDCG@10 and Recall@100, and rewrite variants convert raw PR/issue text into concise assistant-style queries.

Results Retrieval models that look strong on traditional code search drop sharply on the agentic levels: Qwen3-Embedding-8B goes from 71.7 NDCG@10 / 96.9 Recall@100 on Level-1 to 20.3 / 48.0 on Level-2 and 34.4 / 41.5 on Level-3, and different models collapse into a similar low band, indicating Level-2/3 are not just scaled-up versions of code search. In-domain supervised fine-tuning on GitHub pull-request supervision improves performance across all difficulty regimes, but a clear gap remains and Recall@100 still falls as repositories grow denser and larger, leaving substantial headroom for requirement-driven repository retrieval.

Plain-language abstract This paper introduces the CodeSearchNet Corpus and Challenge, a large benchmark for evaluating systems that retrieve relevant code given a plain-language description. The corpus contains about 6 million functions from open-source code across six programming languages, along with roughly 2 million automatically generated natural-language descriptions scraped from function documentation. The challenge itself is built around 99 natural-language queries with approximately 4,000 expert-annotated relevance judgments.

Motivation Finding code using natural language queries requires bridging the gap between the abbreviated, technical vocabulary used in code and the more informal language people use to describe what they want. Without a shared, large-scale benchmark with human relevance judgments, it was difficult to measure progress or compare approaches on this task.

Methodology The authors assembled the CodeSearchNet Corpus by scraping open-source repositories for functions in Go, Java, JavaScript, PHP, Python, and Ruby, then mechanically extracted and preprocessed associated documentation to produce query-like natural-language strings for roughly 2 million functions. A set of 99 natural-language queries was constructed and expert annotators labeled the relevance of candidate code results, yielding about 4,000 annotations. Several simple baseline retrieval models were implemented and evaluated against these labels.

Results The paper releases the corpus, the expert-annotated challenge set, and baseline model results, establishing a leaderboard and competition to track future progress. The baseline solutions provide initial reference scores against the 99 query challenge set, and the authors report that the corpus spans approximately 6 million functions across six languages with 2 million documentation-derived natural-language pairs available for training.

Plain-language abstract CodeBERT is a pre-trained model that learns to understand both natural language (like English) and programming languages (like Python or Java) at the same time. By training on millions of paired code-and-documentation examples from GitHub, it builds representations that can be fine-tuned to help with tasks such as searching for code using plain-language queries or automatically generating documentation for a function.

Motivation Large pre-trained language models had dramatically improved natural language processing, and multimodal models had extended this to language-image and language-video pairs. However, no large pre-trained model existed that jointly captured the semantic connection between natural language and programming languages across multiple languages, leaving tasks like natural language code search and code documentation generation without a strong shared foundation.

Methodology CodeBERT uses a multilayer Transformer architecture and is trained on GitHub repositories in six programming languages (Python, Java, JavaScript, and others). Training uses a hybrid objective combining standard masked language modeling and replaced token detection: bimodal NL-PL pairs provide paired supervision, while unimodal code (functions without documentation) helps train better token generators for the replaced token detection objective. The model is then fine-tuned for downstream tasks; evaluation covers natural language code search on the CodeSearchNet corpus (measured by Mean Reciprocal Rank) and code-to-documentation generation (measured by smoothed BLEU-4 score). A NL-PL probing dataset is also constructed to assess what knowledge is encoded in the pre-trained model in a zero-shot setting.

Results Fine-tuned CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation across six programming languages. On code documentation generation, CodeBERT pre-trained with both replaced token detection and masked language modeling objectives gains 1.3 BLEU score over the RoBERTa baseline overall. In zero-shot probing experiments, CodeBERT consistently outperforms RoBERTa, indicating that it encodes knowledge about the relationship between natural language and code beyond what a purely text-based model captures.

Plain-language abstract GraphCodeBERT is a pre-trained neural model for programming languages that learns to understand code by taking into account how data flows between variables, not just the sequence of tokens. It was published as a conference paper at ICLR 2021 and evaluated on four practical coding tasks: searching for code using natural-language queries, detecting duplicate code, translating code between languages, and automatically fixing bugs.

Motivation Prior pre-trained models for code treat source code as a flat sequence of tokens, ignoring the structural relationships that give code its meaning. Because variable names can be ambiguous or non-descriptive, understanding what a variable represents requires knowing where its value comes from — information encoded in data flow but absent from token sequences alone. The paper addresses this gap by incorporating semantic-level code structure into pre-training.

Methodology The authors built GraphCodeBERT on top of the Transformer architecture, extending it with a graph-guided masked attention function that incorporates a data-flow graph, where nodes are variables and edges represent where each variable's value originates. They introduced two new structure-aware pre-training tasks alongside masked language modeling: predicting data-flow edges, and aligning variable representations between source code and the data-flow graph. The model was pre-trained on the CodeSearchNet dataset of 2.3 million functions across six programming languages, using two DGX-2 machines with 16 NVIDIA Tesla V100 GPUs each, and trained for 200,000 batches (roughly 83 hours).

Results GraphCodeBERT achieved state-of-the-art performance on all four downstream tasks evaluated. On natural language code search, it reached an overall mean reciprocal rank of 0.774 across six languages, outperforming the prior best model CodeBERT (0.760). Ablation experiments confirmed that both the data-flow code structure and the newly introduced pre-training tasks contributed to the improvements, and analysis showed the model consistently preferred attending to data-flow structure over token-level connections in the code search task.

Plain-language abstract CodeT5 is a pre-trained AI model designed to help computers both understand and generate source code. Unlike earlier models that were good at only one of those jobs, CodeT5 handles both in a single unified system. It pays special attention to the meaningful names developers give to variables and functions, using those names as extra signals about what the code actually does.

Motivation Most pre-trained models for code were either encoder-only (good at understanding, weak at generation) or decoder-only (the reverse), forcing practitioners to bolt on extra components and miss out on pre-training benefits for whichever task the architecture didn't natively support. Existing models also treated source code like plain text, ignoring the fact that developer-chosen identifiers carry rich semantic meaning about program behavior.

Methodology CodeT5 is built on the T5 encoder-decoder Transformer architecture and pre-trained on the CodeSearchNet dataset covering six programming languages, augmented with C and C# data from open-source GitHub repositories. A custom Byte-level BPE tokenizer was trained on all pre-training data, reducing tokenized code sequence lengths by 30–45% compared to T5's default tokenizer. The core novelty is an identifier-aware pre-training objective: the model is trained to identify which tokens are developer-assigned identifiers and to recover them when masked, using abstract syntax trees (via tree-sitter) to extract token-type information. A bimodal dual generation task simultaneously optimizes NL-to-PL and PL-to-NL generation to improve natural language and programming language alignment. The model is fine-tuned on CodeXGLUE benchmark tasks spanning code summarization, generation, translation, refinement, defect detection, and clone detection, with multi-task learning explored via task-control prefix tokens.

Results CodeT5 achieved state-of-the-art results across all CodeXGLUE tasks evaluated. On code summarization, CodeT5-base improved overall BLEU-4 by more than 1.2 absolute points over the prior best model PLBART. On code generation, CodeT5-base achieved roughly 4.7 points improvement on CodeBLEU over PLBART. On code refinement, CodeT5-base surpassed GraphCodeBERT by over 4.8 exact-match points on the harder medium-length task (13.96 vs. 9.10). On defect detection, CodeT5-base achieved a 2.6 accuracy-point gain over PLBART, and both model sizes outperformed all baselines on that task. Ablation studies confirmed that the identifier-aware pre-training objective contributed meaningfully across code summarization, generation, and refinement tasks.

Plain-language abstract UniXcoder is a pre-trained neural model for programming languages that can handle both understanding tasks (such as finding similar code or searching code by description) and generation tasks (such as summarizing code or completing it). It is trained on source code together with two additional information sources—code comments and abstract syntax trees—so that it can capture both the meaning and the structure of programs.

Motivation Existing pre-trained code models were either encoder-only (good at understanding but needing a randomly initialized decoder for generation) or decoder-only (good at generation but weak at understanding). No unified model handled both families of tasks well, and none efficiently supported auto-regressive tasks like code completion that require a decoder-only inference mode.

Methodology UniXcoder is built on a multi-layer Transformer that uses mask attention matrices with prefix adapters to switch between encoder, decoder, and encoder-decoder modes. Abstract syntax trees, which are inherently tree-structured, are converted to sequences via a proposed one-to-one mapping that preserves all structural information. The model is pre-trained with three language modeling objectives (masked, unidirectional, and denoising) plus two new tasks: multi-modal contrastive learning (using AST to strengthen code-fragment embeddings) and cross-modal generation (using code comments to align embeddings across programming languages). It is evaluated on five tasks across nine public datasets, and a new zero-shot code-to-code search dataset constructed from the CodeNet corpus.

Results UniXcoder achieves state-of-the-art performance on most of the five evaluated tasks—clone detection, code search, code summarization, code generation, and code completion—across nine datasets. Ablation analysis confirms that both AST and code comments independently improve the model's ability to capture code semantics, and the model outperforms prior unified encoder-decoder approaches on auto-regressive tasks such as code completion.

Plain-language abstract CoIR (Code Information Retrieval Benchmark) is a standardized evaluation framework for testing how well AI models can search and retrieve code. It bundles ten curated datasets spanning eight retrieval tasks across seven coding domains, and packages them in a Python library compatible with the widely used BEIR and MTEB evaluation ecosystems so researchers can benchmark models with a single pip install.

Motivation Existing code retrieval benchmarks such as CodeSearchNet, CosQA, and XcodeEval cover only a narrow slice of real-world retrieval needs, focus almost exclusively on text-to-code search, and have been used so heavily that many models have overfit their leaderboards. There was also no shared evaluation framework, making it impossible to compare results across benchmarks consistently.

Methodology The authors assembled ten datasets — eight adapted from existing sources and two newly created — covering four primary retrieval task types (Text-to-Code, Code-to-Code, Code-to-Text, and Hybrid Code Retrieval) across fourteen programming languages. Each dataset was manually inspected and cleaned. Ten retrieval models were then evaluated on CoIR: one sparse baseline (BM25), several open-source dense models (E5-Base, GTE-Base, BGE-Base, Contriever, E5-Mistral, BGE-M3, UniXcoder), and two proprietary models (OpenAI-Ada-002, Voyage-Code-002). Models were scored using NDCG@10 following the BEIR protocol.

Results No single model dominated all tasks. Voyage-Code-002 achieved the highest mean score of 56.26 but with high variance, indicating weak generalization. BGE-M3 showed the lowest variance despite moderate average performance, suggesting stronger robustness. Top-performing text-retrieval models such as E5-Mistral did not consistently excel across CoIR sub-tasks, underscoring that code retrieval requires capabilities beyond general text retrieval. The evaluation confirmed that even state-of-the-art systems struggle on code retrieval and that many models have overfit existing benchmarks.

Plain-language abstract This paper introduces CoSQA+, a new benchmark for evaluating code search systems — tools that retrieve code snippets matching natural-language queries from programmers. Unlike earlier benchmarks that pair each query with just one correct code example, CoSQA+ pairs queries with multiple valid code snippets, reflecting how developers actually work. The authors also build an automated pipeline using AI agents that verify code correctness through test execution rather than relying on human judgment alone.

Motivation Existing code search benchmarks such as CoSQA and CodeSearchNet assume each query has a single correct answer and rely on human annotators who judge matches by reading code rather than running it. A survey of 102 Python programmers found that developers typically consult 2–3 code examples per query, and perform about 8.31 code searches per programming day, revealing a mismatch between how benchmarks are designed and how code search is actually used. Human and LLM-based annotation also suffer from accuracy limitations because they assess semantic similarity rather than functional correctness verified by test execution.

Methodology The authors constructed CoSQA+ with 412,080 query-code pairs by combining web queries from CoSQA with repository code from CodeSearchNet. Multiple models were used to select the top 20 candidate code snippets by similarity for each query. A novel test-driven agent pipeline then annotated these pairs through five stages: a preliminary screener, a test program generator, a test executor, a bug fixer, and a final arbiter, using test execution to determine whether each code snippet functionally satisfies the query. Annotation quality was evaluated against ground truth labels established by three Python experts on 400 randomly selected pairs. The paper also introduced NDCG@10 as an evaluation metric and benchmarked 8 code search methods, 7 of which are embedding-based.

Results The test-driven agents achieved an annotation accuracy of 92.0% on the 400-pair evaluation set, outperforming both a single LLM annotator and human Python experts annotating without test-based verification. Notably, 83.67% of the test programs automatically generated by the agents were executable. Models fine-tuned on CoSQA+ — specifically CodeBERT, UniXcoder, and CodeT5+ embeddings — consistently outperformed models trained on the original CoSQA on the CSN99 Python benchmark as measured by both NDCG@10 and MRR metrics.

Plain-language abstract This paper presents DRACO, a system that helps AI code-completion tools work better inside private software repositories. Instead of searching for similar-looking code snippets to use as context, DRACO analyzes how data flows through a codebase, building a graph of code relationships that lets it fetch the specific background knowledge a language model needs to complete code correctly.

Motivation Pre-trained code language models struggle to complete code in private repositories because they have no knowledge of the repository's unique naming conventions, custom types, or module structure. Prior retrieval approaches that find context by text similarity often fail when the relevant code does not look textually similar to the target, causing models to hallucinate plausible-sounding but incorrect completions.

Methodology DRACO applies an extended dataflow analysis to a private repository, building a repo-specific context graph that captures type-sensitive data dependency relations among code entities. At completion time it uses dataflow information from the unfinished code to retrieve precisely relevant entities from the graph and constructs well-formed prompts for any code language model. Experiments used the existing CrossCodeEval benchmark and a newly constructed Python dataset, ReccEval, collected from the Python Package Index, testing across adapted code models, specialized code models, and GPT models.

Results DRACO improved code exact match by 3.43% and identifier F1-score by 3.27% on average compared to the previous best approach (RepoCoder). It also reduced prompt generation time by 100x on average, making it practical for real-time code completion. The approach is plug-and-play across different code language models without requiring model retraining.

Plain-language abstract CodeRAG-Bench is a benchmark for evaluating whether retrieving external documents can help language models write better code. It covers nine thousand coding tasks spanning basic programming challenges, open-domain problems, and repository-level tasks, paired with a retrieval corpus of twenty-five million documents drawn from competition solutions, tutorials, library documentation, StackOverflow posts, and GitHub repositories. The benchmark provides standardized evaluation pipelines for both retrieval quality and end-to-end code generation correctness.

Motivation Most code-generation models rely entirely on knowledge baked into their parameters during training, with no mechanism to look up relevant libraries, documentation, or examples at inference time. This limits their ability to handle unfamiliar APIs, evolving libraries, or private codebases. Retrieval-augmented generation had demonstrated strong gains on text tasks, but its usefulness for diverse code generation scenarios had not been systematically studied.

Methodology The authors assembled CodeRAG-Bench by unifying existing Python coding datasets into four categories (basic programming, open-domain, repository-level, and code retrieval) and annotating canonical reference documents for each task. They built a 25-million-document open retrieval corpus aggregating five source types. They then evaluated 10 retrieval models and 10 language models in three settings: generation with canonical (gold) documents, generation with retrieved documents, and open retrieval from the full corpus, using execution-based metrics for functional correctness.

Results Providing canonical documents substantially improves code generation: GPT-4o gains 27.4% on SWE-Bench and 6.9% on the harder ODEX subset when gold documents are supplied. Some models using retrieved documents even matched or exceeded performance with gold documents, showing strong potential for retrieval-augmented approaches. However, current retrieval models frequently fail to surface useful documents—especially for open-domain and repository-level tasks—and generation models with limited context windows show smaller gains, pointing to concrete bottlenecks that remain unsolved.

Curated contrastive data and hard negatives for both the retriever and the reranker

Why it matters The data that makes code retrievers/rerankers competitive; metrics unverified via live lookup

Plain-language abstract This paper presents DeepAPI, a system that takes a plain-English question like "how do I parse XML files?" and automatically generates the sequence of API calls a developer should use to accomplish that task. It treats the problem as machine translation, converting a natural language query into an ordered list of API method calls from software libraries like Java's JDK.

Motivation Developers frequently struggle to discover which APIs to use and in what order when working with large, unfamiliar libraries — the .NET framework and JDK each contain hundreds to thousands of APIs, and documentation is often inadequate. Existing search approaches treat queries and APIs as unordered bags of words, so they fail to capture word order or semantics and cannot, for example, distinguish "convert int to string" from "convert string to int".

Methodology DeepAPI adapts an RNN Encoder-Decoder neural language model that encodes a sequence of query words into a fixed-length context vector and then decodes an API call sequence from that vector. The model was trained on more than 7 million annotated Java code snippets mined from GitHub, with 10,000 instances held out for testing and the remainder used for training over 240 hours (1 million iterations). The authors also augmented the base model to weight individual APIs by their importance.

Results DeepAPI achieved an average BLEU score of 54.42 on the test set, substantially outperforming a code-search-plus-pattern-mining baseline (BLEU 11.97) and the prior SWIM word-alignment approach (BLEU 19.90). On 30 real API queries drawn from query logs and related work, the average rank of the first relevant result was 1.6; 80% of top-5 results and 78% of top-10 results were judged relevant.

Plain-language abstract This paper studies how to build better search systems that let developers find code snippets using plain English questions. The key insight is that code snippets often come with short human-written descriptions of what they do, and the authors show that exploiting those descriptions dramatically improves search quality compared to systems that look only at raw code.

Motivation Searching large code collections using natural language is hard because the meaning of a code fragment is rarely obvious from its tokens alone — even experienced developers struggle to judge relevance without extra context. Prior neural code-search systems all tried to infer intent from unannotated source code, leaving a gap: no system had systematically studied what happens when brief human-written descriptions accompany the snippets and are used at retrieval time.

Methodology The authors defined an 'annotated code search' task where every snippet is paired with a short natural language description, then built three benchmark datasets (PACS) drawn from existing resources including the CoNaLa corpus. They fine-tuned a pre-trained NLP transfer-learning model (the Universal Sentence Encoder, chosen because its training data included Stack Overflow content) to predict semantic similarity between a user query and a snippet's description, and compared this approach against classic information-retrieval baselines and neural code-search models such as NCS and UNIF that operate on unannotated code.

Results The fine-tuned description-based retrieval model substantially outperformed all code-only baselines on all three benchmarks, achieving absolute gains of up to 20.6 percentage points in mean reciprocal rank. The experiments also showed that including the snippet source code alongside the description provides additional benefit beyond descriptions alone, but descriptions are the dominant signal for obtaining relevant results.

Plain-language abstract deGraphCS is a deep learning system for code search: given a plain-English description, it retrieves matching code snippets from a large repository. Instead of representing code as raw text or a syntax tree, it converts source code into a variable-based flow graph derived from compiler intermediate representation (LLVM IR), then uses a graph neural network to match that graph against the natural-language query.

Motivation Developers routinely search large code repositories like GitHub using natural language, but existing deep learning code-search methods rely on token sequences or abstract syntax trees (ASTs) that fail to capture true code semantics. Code written in different styles can perform the same function yet have very different ASTs, and code with similar structure can mean entirely different things, so structural representations produce imprecise matches.

Methodology The authors compile C source code through LLVM to obtain intermediate representation (IR), then construct a variable-based flow graph where nodes are IR tokens and edges encode data or control dependencies between variables. A graph optimization step removes redundant nodes (reducing graph size by 51.88%) without changing semantics. An attentional gated graph neural network embeds these graphs into vector representations that are matched against natural-language query embeddings. The system was trained and evaluated on a dataset of 41,152 C code snippets collected from GitHub, with comparisons against DeepCS, UNIF, and MMAN baselines, plus a qualitative user study in which five experienced developers rated results for 50 held-out queries.

Results deGraphCS outperforms the prior state-of-the-art method (MMAN) by 19.14% in mean reciprocal rank (MRR) and 26.43% in SuccessRate@1. Its top-1 hit rate rises from 34.05% (best baseline) to 43.05%. In the user study it achieves an average SuccessRate@10 of 0.65 and an average MRR of 0.48, with improvements over MMAN, UNIF, and DeepCS of 14%, 20%, and 55% respectively.

Plain-language abstract GraphSearchNet is a neural network framework that helps developers find relevant code snippets by typing a natural language query. It builds graph representations of both source code and queries, then uses a combination of graph neural networks and attention mechanisms to match them by meaning rather than keywords, enabling more accurate code retrieval from large repositories like GitHub or Stack Overflow.

Motivation Existing deep learning approaches to code search largely treat programs as flat sequences of tokens, ignoring the structural information encoded in syntax trees and data-flow graphs. Graph Neural Networks can capture some of that structure, but they struggle to model long-range dependencies within a graph, limiting how well they learn program semantics and how accurately they map code to natural language queries.

Methodology The authors build a graph for each source code snippet using syntactic edges (AST edges, NextToken, SubToken) and data-flow edges (ComputedFrom, LastUse, LastWrite), and a separate graph for each natural language query using dependency parsing. Both graphs are encoded by a Bidirectional Gated Graph Neural Network (BiGGNN), which is then augmented with a multi-head attention module to capture global dependencies that BiGGNN alone misses. The combined encoder is trained and evaluated on Java and Python code from the public CodeSearchNet benchmark, which contains over two million functions, and assessed against 13 baseline systems using metrics including R@k, MRR, and NDCG, with an additional quantitative analysis on 99 real natural language queries.

Results GraphSearchNet outperforms all 13 state-of-the-art baselines by a significant margin on the CodeSearchNet benchmark across both Java and Python datasets and across all five evaluation metrics. The qualitative analysis on 99 real queries further confirms that the model returns high-quality code candidates in response to natural language input.

Plain-language abstract This paper presents DGMS (Deep Graph Matching and Searching), a system that helps developers find relevant code snippets by searching large repositories using plain-English descriptions. Instead of treating code as flat text, DGMS converts both the natural-language query and the source code into graph structures that capture how different parts of the code relate to each other, then uses neural networks to compare them semantically.

Motivation Existing code search tools either rely on keyword matching—which misses semantic meaning—or use neural networks that encode text as flat sequences, losing the structural relationships within code such as long-range dependencies between variables. No prior approach represented both the natural-language query and the code snippet using the same graph-based representation that preserves both structural and semantic information simultaneously.

Methodology DGMS converts natural-language queries and source code snippets into unified graph-structured data, preserving structural and semantic information for both. A graph encoding module (using Relational Graph Convolutional Networks, RGCN) learns node embeddings for each graph individually. A semantic matching module then applies cross-attention between the text graph and the code graph to capture fine-grained correspondence. Finally, a code searching module aggregates node embeddings into graph-level vectors and computes similarity scores using a ranking loss, trained end-to-end. The model was evaluated on two public datasets—FB-Java and CSN-Python—using mean reciprocal rank (MRR) and success-at-k metrics, with each query matched against 100 candidate code snippets.

Results DGMS outperformed all seven baseline methods on both datasets across all evaluation metrics. On CSN-Python, DGMS achieved an MRR of 92.2% and S@1 of 87.6%, exceeding the best baseline by margins of 22.1 MRR points and 27.9 S@1 points—a 31.5% and 46.7% relative improvement respectively. On FB-Java, DGMS reached 87.9% MRR and 81.7% S@1, with S@5 scores exceeding 95% on both datasets, meaning the correct code snippet was found in the top five results more than 95% of the time.

Plain-language abstract CodeMatcher is a code search tool that helps software developers find relevant code snippets using plain-language queries. It combines the speed of traditional keyword-based search with the semantic understanding of deep learning models, allowing developers to describe what they need in natural language and retrieve matching code methods from large open-source codebases.

Motivation Developers spend the majority of their coding time searching for reusable code, yet existing approaches faced a fundamental trade-off: simple keyword-matching tools were fast but semantically shallow, while deep learning models like DeepCS could understand query meaning and sequential word relationships but were impractically slow for large codebases. There was no method that achieved both semantic understanding and search efficiency at scale.

Methodology CodeMatcher first collects metadata about query words to identify and filter out irrelevant or noisy terms, retaining only the important ones. It then performs iterative fuzzy search over a codebase indexed with Elasticsearch, and finally reranks the returned candidate code snippets by how well their tokens sequentially match the important query words. The system was evaluated on a large-scale codebase comprising approximately 41,000 GitHub repositories.

Results CodeMatcher achieved a Mean Reciprocal Rank (MRR) of 0.60, outperforming DeepCS by 82%, CodeHow by 62%, and UNIF by 46%. It was also more than 1,200 times faster than DeepCS. Compared to existing online search engines, CodeMatcher surpassed GitHub search by 46% and Google search by 33% in MRR. The authors additionally observed that improving method naming quality aids code search, as method names play a key role in bridging natural language queries and code.

Plain-language abstract This paper presents CasCode, a two-stage system for searching codebases using plain English queries. When you type a description of what you want code to do, the system first quickly narrows down candidates from a large index, then applies a more thorough but slower matching step to the shortlist. The result is a system that is nearly as accurate as the slow-but-careful approach while being much faster to use in practice.

Motivation Existing semantic code search systems face a fundamental tradeoff: fast encoder-based retrieval independently encodes queries and code snippets into vector representations, enabling scalable search but sacrificing accuracy; slow classifier-based approaches jointly process query-code pairs for better matching but are computationally infeasible for large candidate sets. Neither approach alone was both effective and efficient enough for practical deployment, motivating a system that combines the strengths of both.

Methodology The authors built CasCode, a cascaded two-stage framework using transformer encoder models fine-tuned on the CodeSearchNet benchmark, which covers six programming languages (Ruby, JavaScript, Go, Python, Java, PHP). In the first stage, a fast CodeBERT encoder retrieves the top-K candidate code snippets via nearest-neighbor lookup on a preconstructed index. In the second stage, a slow binary classifier jointly processes the query paired with each of the K candidates and re-ranks them by confidence score. To reduce memory overhead, the authors also propose a shared-parameter variant where a single transformer serves both roles, trained jointly with a combined infoNCE and binary cross-entropy objective. Experiments were run on 8 NVIDIA A100 GPUs with K set to 10 or 100.

Results CasCode (separate, K=100) achieved an overall mean reciprocal ranking (MRR) score of 0.7795 across six programming languages on the CodeSearchNet test set, surpassing the previous state-of-the-art of 0.713 MRR. The shared-parameter variant reached 0.7700 MRR at roughly half the parameter count. On the Ruby subset, CasCode (separate, K=10) processed queries at 9.78 queries per second compared to 0.11 queries per second for the slow classifier alone, demonstrating that the cascaded approach achieves near-optimal accuracy with substantially lower inference cost.

Plain-language abstract This paper introduces ContraCode, a self-supervised pre-training method that teaches neural networks to understand source code by its functionality rather than its surface appearance. By training a model to recognize semantically equivalent programs that have been rewritten in different but functionally identical ways, ContraCode learns representations of code that are robust to stylistic variation and adversarial rewrites.

Motivation Existing code representation models like RoBERTa learn by reconstructing masked tokens, which causes them to be sensitive to superficial syntactic changes even when those changes preserve program semantics. This brittleness is exploited by adversarial attacks—just three minor label-preserving edits to code syntax are enough to push RoBERTa below random-classifier performance on clone detection, a vulnerability that could, for example, be used to circumvent malware detectors.

Methodology ContraCode frames pre-training as a contrastive discrimination task: given a program, the model must identify which of many candidate programs is functionally equivalent to it. Functionally equivalent variants are generated automatically using a source-to-source compiler that applies transformations such as dead code elimination, constant folding, and obfuscation, providing scalable data augmentation without human labeling. The encoder is trained to maximize similarity between representations of equivalent program variants while minimizing similarity with non-equivalent distractors.

Results Contrastive pre-training with ContraCode outperforms RoBERTa on an adversarial code clone detection benchmark by 39% AUROC. Improved adversarial robustness also generalizes to natural code understanding tasks: ContraCode improves TypeScript type inference top-1 accuracy by 9%, learned type inference by 2–13 percentage points, code summarization F1 score by up to 8%, and clone detection AUROC by 2–46% over competitive baselines.

Plain-language abstract This paper is a comprehensive survey of code search — the tools and techniques that help software developers find relevant source code examples within a codebase or across open-source repositories. It covers 30 years of research, synthesizing 109 papers, and organizes the field around the full pipeline of a code search engine: how queries are expressed, how code is indexed, and how results are ranked.

Motivation Massive amounts of source code exist across modern software projects and platforms like GitHub, where over 60 million new projects were created in 2020 alone. Most code a developer needs to write has likely already been written somewhere, making effective code search a high-value activity — yet no unified survey had mapped the landscape of techniques for building code search engines across the full research history.

Methodology The authors surveyed 109 research papers on code search published over approximately 30 years (roughly 1990 to 2022). They organized the literature around the key components of a code search engine: the kinds of queries supported (natural language, code snippets, formal specifications, test cases, and dedicated query languages), query preprocessing and expansion techniques, methods for indexing and retrieving code, and strategies for ranking and pruning search results. The survey also covers empirical studies of how developers use code search in practice.

Results The survey produces a structured taxonomy of the code search problem and existing solutions, identifying the major technical challenges and how prior work has addressed them. Based on this synthesis of prior work, the paper concludes with an outline of open challenges and research opportunities for future code search systems.

Plain-language abstract This paper presents a method for automatically detecting when two pieces of source code do the same thing, even when they look very different on the surface. The authors build a new type of program representation called a flow-augmented abstract syntax tree (FA-AST) and use graph neural networks to compare code pairs for similarity.

Motivation Existing code clone detectors work well for syntactically similar code but struggle with semantic clones — code fragments that implement the same functionality in structurally different ways (Type-4 clones). Prior deep learning approaches used abstract syntax trees as input but those trees omit control flow and data flow information that is essential for recognizing semantic equivalence.

Methodology The authors augment standard abstract syntax trees with explicit control flow and data flow edges to produce FA-ASTs. They then apply two graph neural network architectures — a gated graph neural network (GGNN) and a graph matching network (GMN) — to learn vector representations of code fragments and measure pairwise similarity. The approach is evaluated on two Java datasets: the Google Code Jam dataset and the widely used BigCloneBench benchmark.

Results FA-AST with GMN achieves an F1 of 0.95 and recall of 0.94 on BigCloneBench, outperforming prior AST-based deep learning approaches including ASTNN, CDLH, and RtvNN. On the harder weakly-typed semantic clone category (WT3/T4), FA-AST+GMN reaches an F1 of 94.6 versus ASTNN's 92.8. The ROC-AUC of FA-AST+GMN (0.996) also exceeds both ASTNN (0.988) and FA-AST+GGNN (0.986) on BigCloneBench.

Plain-language abstract TranS3 is a software engineering framework that combines two common developer tasks — automatically summarizing what a piece of code does and searching a large codebase for relevant code using plain-English queries — into a single unified system built on the Transformer neural network architecture. The framework generates natural-language comments for code snippets and uses those comments to improve search accuracy, while a reinforcement-learning loop continuously refines comment quality.

Motivation Code summarization (generating human-readable descriptions of code) and code search (finding relevant code given a natural-language query) had been developed largely in isolation, even though they share a common technical basis and are mutually reinforcing. State-of-the-art summarization accuracy was low — around 20% BLEU-1 on standard benchmarks — limiting the usefulness of injecting generated comments into search, and no prior work had tackled the challenge of optimizing both tasks jointly without trade-offs.

Methodology TranS3 was built around a two-component pipeline. The code summarization component used a Transformer encoder paired with a tree-Transformer encoder (which captures the abstract syntax tree structure of programs) inside an actor-critic reinforcement learning framework: the actor network generates comments and the critic network evaluates their accuracy against ground-truth comments and feeds back rewards. The resulting trained encoder was reused directly by the code search component, which encoded queries, generated comments, and code snippets with the same encoders, then computed and combined similarity scores between query-code and query-comment vector pairs to rank results. Experiments used a GitHub dataset of over 120,000 Python functions with their corresponding comments.

Results TranS3 outperformed multiple state-of-the-art baselines on both tasks. For code summarization, it improved BLEU-1 accuracy by between 47.2% and 141.6% relative to the best competing approaches. For code search, it improved Mean Reciprocal Rank (MRR) by between 5.1% and 28.8% relative to the selected baselines. Case studies with developers further supported the practical effectiveness of the generated comments and search results.

Plain-language abstract ColBERT is a passage-ranking model that makes powerful neural language models practical for large-scale document search. It independently encodes queries and documents using BERT and then computes a fine-grained similarity score between them in a lightweight final step, allowing document representations to be precomputed and cached rather than reprocessed for every query.

Motivation BERT-based ranking models achieve strong retrieval quality but are 100 to 1000 times more computationally expensive than earlier approaches because they must process every query-document pair together through a large neural network. This cost makes them impractical for real-time search over millions of documents, creating a need for architectures that preserve BERT's expressive power while dramatically reducing per-query computation.

Methodology ColBERT introduces a late interaction architecture: the query and the document are each encoded independently through BERT into sequences of contextualized token embeddings, and relevance is then estimated by a cheap MaxSim operator that sums the maximum cosine similarities between each query token embedding and all document token embeddings. Because document embeddings can be precomputed offline, ColBERT supports both re-ranking a candidate set retrieved by a traditional model and end-to-end retrieval from a full collection via vector-similarity indexes. The model was evaluated on two passage search benchmarks, MS MARCO Ranking (9 million passages, 1 million queries) and TREC CAR.

Results ColBERT's retrieval effectiveness is competitive with existing BERT-based rankers and outperforms all non-BERT baselines on MS MARCO, while running two orders of magnitude faster and requiring four orders of magnitude fewer floating-point operations per query. The late interaction design also enables end-to-end retrieval directly from a large document collection, a capability not available to prior BERT re-ranking approaches.

Plain-language abstract This paper from OpenAI introduces a family of embedding models — called cpt-text and cpt-code — that convert text or source code into compact numerical vectors suitable for tasks like semantic search, classification, and code retrieval. The models are trained without labeled data by using a contrastive learning approach on naturally occurring paired text found on the internet and in code repositories.

Motivation Previous embedding models for text and code were trained separately, with different datasets, objectives, and architectures for tasks like sentence similarity versus information retrieval. There was no unified, scalable approach that could produce a single set of high-quality embeddings capable of performing well across both classification and search tasks without task-specific fine-tuning.

Methodology The authors train Transformer encoder models using a contrastive learning objective with in-batch negatives on unlabeled data. Text models treat neighboring passages on the internet as positive pairs; code models pair a function's top-level docstring with its implementation. Models are initialized from existing pretrained language models (including GPT-3 scale), trained with large batch sizes, and scaled from 300 million to 175 billion parameters to study the effect of model size.

Results The largest unsupervised cpt-text model achieves a 4% relative improvement over the previous best unsupervised text embedding model and a 1.8% improvement over the best supervised model on linear-probe classification across 7 tasks. On large-scale semantic search, the same embeddings yield relative improvements of 23.4%, 14.7%, and 10.6% over prior best unsupervised methods on MSMARCO, Natural Questions, and TriviaQA respectively. Code embedding models achieve a 20.8% relative improvement over prior work on code search.

Plain-language abstract This paper is a large-scale survey of how language models — the AI systems behind tools like GitHub Copilot — are used in software engineering. It covers more than 70 models, 40 evaluation tasks, 180 datasets, and around 900 related papers, tracing how AI assistance has grown from simple code completion to supporting the entire software development lifecycle.

Motivation The NLP research community and the software engineering community have studied language models for code largely in isolation, with existing reviews focusing on one side or the other but not both. The authors identified a gap: no prior work had jointly examined how SE uses language models for development automation and how NLP uses SE tasks as benchmarks, nor had any review covered LLM applications across the full software development lifecycle beyond programming.

Methodology The authors conducted a systematic literature survey, organizing their coverage around a taxonomy of software engineering stages. They reviewed code-processing models by distinguishing general language models (such as the GPT family) from specialized code models pretrained with tailored objectives, and traced the historical evolution from statistical models and RNNs through pretrained Transformers. Coverage extends beyond code generation to requirement engineering, testing, deployment, and operations. The survey is maintained as a living document on GitHub.

Results The survey documents that advanced NLP techniques — including instruction tuning, infilling objectives, revised scaling laws, architectural improvements, and autonomous agents — have been progressively introduced into code modeling, while SE tasks in turn provide real-world benchmarks that drive LLM development. The authors identify the bidirectional relationship between the two communities as a key structural finding, and flag key open challenges and future directions across the full software engineering lifecycle.

Plain-language abstract MISIM is a neural system for determining whether two pieces of code do the same thing, even when they look very different on the surface. It introduces a new way of representing code called the Context-Aware Semantics Structure (CASS) that strips away syntactic clutter to expose underlying meaning, and pairs it with a learned similarity scoring algorithm tested across three neural network architectures. The system was evaluated against four state-of-the-art code similarity methods over roughly 328,000 programs comprising more than 18 million lines of code.

Motivation Existing code similarity systems rely on structural representations like abstract syntax trees or simplified parse trees that are either too syntactically dense, require compilation to an intermediate representation, or introduce ambiguities—causing models to memorize syntax rather than learn semantic meaning. There was no representation purpose-built to lift semantics from code syntax while remaining language-extensible and usable in interactive developer environments.

Methodology The authors built MISIM around a configurable code representation called CASS, which uses a global attributes table and a structured tree that resolves syntactic ambiguities present in ASTs and SPTs. They trained three neural network variants on top of CASS—a bag-of-features model (MISIM-BoF), a bidirectional GRU-based recurrent model (MISIM-RNN), and a relational graph convolutional network (MISIM-GNN)—using metric learning with cosine similarity and Circle loss. Experiments used the GCJ and POJ-104 benchmark datasets and compared against code2vec, code2seq, Neural Code Comprehension, and Aroma, with MAP@R and Average Precision as evaluation metrics.

Results MISIM-GNN achieved at least 8.08% better MAP@R accuracy than the next best performing system across all experiments. Ablation studies confirmed that CASS outperformed both AST- and SPT-based representations under the same neural architectures, demonstrating that the structural representation itself—not just the neural model—was responsible for the accuracy gains.

Plain-language abstract This paper presents CoCoSoDa, a system that helps developers find relevant code snippets by typing a plain-language description of what they need. It improves a machine learning technique called contrastive learning — which teaches a model to match queries to code by pulling related pairs together and pushing unrelated ones apart — through smarter data preparation and better handling of training examples.

Motivation Developers routinely search large codebases and open-source repositories for code that matches a natural-language description, but bridging the gap between human language and programming language is hard. Prior contrastive learning approaches for this task used relatively simple data augmentation strategies and limited sets of negative training examples, leaving substantial room for improvement in retrieval accuracy.

Methodology CoCoSoDa introduces two main components applied on top of pre-trained code models such as RoBERTa, CodeBERT, and GraphCodeBERT. First, soft data augmentation (SoDa) dynamically masks or replaces a small fraction (5–20%) of code tokens with their syntactic types to generate augmented positive samples without changing meaning. Second, a momentum mechanism maintains a large queue of consistently encoded negative samples across mini-batches using a slowly-updated momentum encoder. These are combined with a multimodal contrastive loss that aligns code-query pairs both within and across modalities. Experiments used the CodeSearchNet benchmark covering six programming languages.

Results CoCoSoDa outperformed 18 baseline systems on the CodeSearchNet benchmark. Compared to strong pre-trained baselines, it improved mean reciprocal rank (MRR) by 13.3% over CodeBERT, 10.5% over GraphCodeBERT, and 5.9% over UniXcoder on average across six languages. Ablation studies confirmed that each component — soft data augmentation, momentum negatives, and multimodal contrastive loss — contributed independently to the gains, and the model was robust across different hyperparameter settings.

Plain-language abstract RocketQA is a training method for dense passage retrieval systems that answer natural-language questions by searching large document collections. The paper identifies three specific weaknesses in how these retrieval models are trained and introduces targeted fixes for each, then tests the improved system on standard open-domain QA benchmarks.

Motivation Dense retrieval models encode questions and passages as vectors and find answers by comparing those vectors, but they are hard to train well for three reasons: the training setup (small batches on a single GPU) does not match the inference setup (searching millions of passages); training datasets contain many unlabeled relevant passages that get mistakenly treated as negatives; and large annotated datasets for open-domain QA are expensive to build, leaving models data-hungry.

Methodology The authors propose three complementary training strategies applied to a dual-encoder architecture built on pre-trained language models. Cross-batch negatives share passage encodings across GPUs to increase the number of negatives per question without proportionally increasing memory. Denoised hard negatives use a more accurate cross-encoder model to filter out false negatives from the top-retrieved candidates before using them as training signal. Data augmentation uses the cross-encoder to label large amounts of unannotated text, generating pseudo-labeled training pairs to supplement the limited human-annotated data. The system is evaluated on MSMARCO and Natural Questions datasets, with ablations isolating each strategy's contribution.

Results RocketQA outperforms all prior dense and sparse retrieval baselines on both the MSMARCO passage ranking and Natural Questions benchmarks. Ablation experiments confirm that each of the three strategies contributes independently: cross-batch negatives improve over in-batch negatives (MRR@10 rising from 32.39 to 33.32 on MSMARCO in the ablation table), hard negatives without denoising actually hurt performance relative to the baseline, while denoised hard negatives recover and exceed the baseline, and adding data augmentation improves results further. On end-to-end QA, pairing the RocketQA retriever with a re-trained DPR reader achieves 42.8 exact-match on Natural Questions, surpassing all compared extractive systems.

Plain-language abstract This paper introduces Contriever, a system for searching large document collections without any labeled training data. Instead of relying on keyword matching (as traditional search engines do) or needing human-annotated query-document pairs (as most modern neural retrievers require), Contriever learns to retrieve relevant documents purely from raw text using a technique called contrastive learning. The approach works across languages and can even retrieve documents across different writing scripts.

Motivation Neural network-based document retrievers achieve strong performance on well-resourced benchmarks but fail to generalize when no in-domain training data is available, consistently losing to classical keyword-matching methods like BM25 in zero-shot settings. This gap is especially acute for non-English languages where large annotated retrieval datasets simply do not exist. The paper asks whether contrastive self-supervised learning can close this gap without any manual supervision.

Methodology The model uses a bi-encoder architecture built on BERT base, where queries and documents are encoded independently and relevance is scored by the dot product of their representations. Training is fully unsupervised: positive pairs are constructed from a single document by independently cropping two random spans of text (treated as a query and a matching key), and the model is trained with the InfoNCE contrastive loss to bring representations of spans from the same document closer together while pushing apart spans from different documents. The authors compare this cropping strategy against the Inverse Cloze Task and other augmentations, and also train a multilingual variant. Evaluation uses the BEIR benchmark (15 diverse retrieval datasets) plus multilingual benchmarks.

Results The unsupervised Contriever model outperforms BM25 on 11 out of 15 BEIR datasets at Recall@100, the first dense retriever to broadly surpass BM25 without any supervision. When used as pre-training before fine-tuning on a few thousand in-domain examples or on the full MS MARCO dataset, it further improves over strong supervised baselines on BEIR. The multilingual model achieves strong unsupervised cross-lingual retrieval, including retrieving English documents from Arabic queries — a task impossible for term-matching methods.

Plain-language abstract This paper is a survey of how transformer-based language models — most notably BERT — have been applied to text ranking, meaning the task of returning an ordered list of relevant documents or passages in response to a query. It covers a wide range of techniques, from multi-stage reranking pipelines that first retrieve candidates with a fast method (like BM25) and then rescore them with a transformer, to newer dense retrieval methods that encode both queries and documents into vectors and use nearest-neighbor search directly.

Motivation Before transformers, text ranking systems relied on exact keyword matching (like BM25) or earlier neural approaches that struggled with vocabulary mismatch and could not fully leverage the meaning of queries and documents. The arrival of BERT produced a step-change in ranking quality, but also raised new challenges around handling documents longer than a transformer's input limit and around the high computational cost of running a large model at query time — problems that needed systematic treatment.

Methodology The survey organizes existing approaches into two main families. The first covers multi-stage reranking architectures, including models such as monoBERT (which frames relevance as a classification task), CEDR (which integrates contextualized embeddings into existing neural ranking models), and sequence-to-sequence rerankers like monoT5; it also covers document and query expansion techniques such as doc2query and DeepCT. The second family covers dense retrieval, where transformer encoders produce dense vector representations of queries and documents and ranking is performed via approximate nearest-neighbor search. Both families are analyzed for the tradeoff between retrieval effectiveness and query-time efficiency.

Results BERT fine-tuned for passage reranking substantially outperforms BM25 alone, with MRR@10 rising from around 0.197 to 0.289 on the MS MARCO passage ranking task in reported experiments. ColBERT, a late-interaction dense model, largely closes the latency gap between monoBERT reranking and pre-BERT neural models while retaining most of the effectiveness gain. The survey also finds that improvements in first-stage retrieval do not always translate into proportional gains when a strong reranker like monoBERT is applied on top, and it identifies open research questions including better handling of very long documents and further reducing inference cost.

Plain-language abstract This paper introduces a new way to represent source code as a graph so that machine learning models can better understand programs. Rather than treating all parts of a program's syntax tree as interchangeable, the authors build a "heterogeneous program graph" (HPG) that preserves the distinct types of nodes and edges — for example, distinguishing identifiers from operators and left-children from right-children. A matching graph neural network architecture (HGT) then uses those type labels when computing learned representations of the code.

Motivation Existing graph-based code representation methods convert programs into homogeneous graphs, discarding the type information present in a program's abstract syntax tree. This means a model cannot distinguish, say, an identifier node from an operator node, and two semantically different expressions like "a-b" and "b-a" can produce identical graph structures — an ambiguity that limits how accurately a model can understand code.

Methodology The authors define the Heterogeneous Program Graph (HPG) by reading a program's abstract syntax tree according to its ASDL grammar specification and annotating every node and every edge with its declared type. They then train a Heterogeneous Graph Transformer (HGT), which applies separate attention weights for each node-type and edge-type pair, to learn vector representations from HPG. The combined HPG+HGT framework is evaluated on four benchmark datasets covering two tasks: method name prediction and code classification, and compared against state-of-the-art baselines including Code Transformer and Cognac.

Results Experiments on all four datasets show that adding heterogeneous type information to the graph consistently improves the performance of GNN-based code representation models. The proposed HPG+HGT framework outperforms state-of-the-art baselines on most of the evaluated tasks and datasets. An ablation study further confirms that each component of the design contributes to the final performance.

Plain-language abstract This paper presents IDECoder, a framework that connects large language model (LLM) coding assistants directly to the static analysis capabilities built into Integrated Development Environments (IDEs). Rather than having a model guess at cross-file context, IDECoder feeds it precise, real-time information that IDEs already compute — such as class hierarchies, function signatures, and type details — then uses the IDE's linting feedback to automatically correct errors in the model's output.

Motivation Current LLM-based code assistants are trained on single-file contexts and struggle when completing code in large software repositories, where understanding a variable's type or a method's signature requires looking at other files. Existing approaches that try to retrieve cross-file context using identifier matching or semantic similarity produce inaccurate results and often exceed LLMs' context-length limits, leaving repository-level code completion a largely unsolved problem.

Methodology IDECoder operates in three phases. First, it uses the IDE's native static analysis (abstract syntax tree parsing, symbol tables, reference indexing) to accurately identify cross-file contexts such as docstrings, method signatures, and class attributes. Second, it organizes these contexts using a chain-of-thought prompting strategy that introduces information top-down — from functional role to specific type details — before sending the prompt to an LLM. Third, it passes the LLM's output back through the IDE's as-you-type linting service (e.g., Pylance), detects type errors and usage issues, and resends those diagnostics to the LLM for self-refinement. A proof-of-concept was implemented using GPT-3.5 hooked into Pylance in VS Code, and evaluated on 10 Python repositories sampled from the CROSSCODEEVAL benchmark using function-body completion as the task.

Results IDECoder outperformed all three baselines — in-file completion, all-import, and RAG — across every metric. On Exact Match it scored 10.46% versus 9.77% for the next-best (RAG); on CodeBLEU it reached 34.16% versus 31.65%; and on Syntax Match it achieved 50.73% versus 48.92%. The authors note that the current implementation is limited by restricted access to Pylance's internals and expect larger gains with fuller IDE integration.

Plain-language abstract This paper introduces RepoRift, a system that uses AI agents and Retrieval Augmented Generation (RAG) to improve semantic code search — the task of finding relevant code snippets from a codebase given a plain-English description. The agents enrich a user's query with context fetched from the relevant GitHub repository, then a multi-stream ensemble approach compares the augmented query against code embeddings to return the best-matching snippets.

Motivation Current semantic code search tools struggle when user queries are vague or use different terminology than the codebase — a problem called the Vocabulary Mismatch Problem. A user searching for 'model' might mean a machine-learning model, a database schema, or a design pattern, leading to poor results. Prior work improved the mapping between natural language and code but did not address the upstream problem of ambiguous or under-specified queries.

Methodology The system uses a CrewAI-based agent built on GPT-4 that performs an internet search scoped to a target GitHub repository and appends relevant technical context to the user's original query. The enriched query is then processed through a three-stream ensemble: one stream embeds the augmented query and ranks functions by cosine similarity; a second generates equivalent Python code via GPT-3.5-turbo and matches its embedding against repository functions and classes; a third decomposes that generated code into component functions and finds the closest match for each. A final GPT-4o re-ranking step selects the most relevant snippets from the combined candidate set. The system was evaluated on 101 manually processed queries drawn from the Python split of the CodeSearchNet dataset.

Results RepoRift achieved a 78.2% success rate at Success@10 and 34.6% at Success@1, outperforming the best prior baseline (PSCS) which scored 47.6% and 22.9% respectively on the Java split of the same dataset. The lower-bound accuracy at Success@10 was approximately 22.5 percentage points above the next-best method. The system also handled queries written in Russian, raw URLs, and vague conceptual descriptions without additional preprocessing.

Plain-language abstract SWE-Explore is a benchmark that isolates one capability of coding agents — how well they explore a repository to find the code relevant to an issue — instead of scoring only the final pass/fail repair. Given an issue and a repository, an explorer returns a ranked list of line-level code regions under a fixed budget, scored against the regions that agents which actually solved the issue relied on.

Motivation Repository-level benchmarks like SWE-bench reduce each attempt to a single resolved/unresolved bit, which conflates exploration, localization, and patch synthesis and hides which step failed. Two distinct failure modes exist: an agent never explores the relevant code, or it finds the evidence but fails to synthesize a correct patch. Existing executable benchmarks capture the latter; the former is largely invisible, and no common, precise target lets classical retrievers, search agents, and long-context selectors be compared on exploration quality. File- or function-level localization only says an agent reached the right neighborhood, not which lines it consulted.

Methodology Repository exploration is formalized as a standalone task f:(issue, repo) -> ranked list of regions (file path + line range), requiring no patch and no repository interaction. Line-level ground truth is trajectory-grounded: read actions are extracted from independent agent trajectories that successfully solved the same issue (kept only when >=2 trajectories resolve it under the executable harness), deduplicated per file and aggregated by consensus into high-confidence core regions and lower-confidence optional regions, with human QA. Explorers are scored on coverage, ranking, and budget-efficiency metrics. A separate restricted-context repair bridge — feeding only the explorer's output to a fixed coding agent and checking whether the patch passes the original tests — validates that the metrics track repair, as a one-time external-validity check rather than part of the standard loop.

Results SWE-Explore spans 848 instances across 10 programming languages and 203 repositories (64.5% Python, the rest Go/JavaScript/Rust/Java/PHP/TypeScript/Ruby/C/C++), built from SWE-bench Verified, SWE-bench-Pro, and SWE-bench Multilingual. Across retrieval methods, general coding agents, and specialized localizers, agentic explorers form a clear tier above classical retrieval. File-level localization is already strong for modern methods, but line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers, and the exploration metrics strongly track downstream repair behavior in the restricted-context validation.

Plain-language abstract LLM code agents mostly navigate repositories with keyword (grep) search, which returns lines by lexical match and hides the structural relationships - who calls whom, inheritance, configuration dependencies - that determine whether a line is actually relevant. This paper asks how much that missing structure is worth. It builds CodeAnchor, which runs lightweight static analysis offline and injects the resulting structural facts back into source files as plain-text comments ('anchor tags'), so an unchanged grep-first agent sees structural links inline as it searches. Tested on top of Codex on SWE-bench Lite and Verified, the tags help less by making the agent smarter and more by making its navigation disciplined and reproducible.

Motivation Grep-first agents are fast, language-agnostic, and competitive - often beating graph-based approaches - but they suffer a representation mismatch: lexical retrieval cannot expose structural topology, so agents rediscover links through ad-hoc multi-hop queries and produce brittle, stochastic trajectories where two runs on the same task visit different files and take different paths. For software engineering, that unpredictability is a quality concern: practitioners need agent behavior to be inspectable and reproducible, not just correct in aggregate.

Methodology CodeAnchor extracts call graphs (PyCG), inheritance, data flow, and configuration usage via offline static analysis and AST extractors, then injects them as compact plain-text 'CodeAnchor tags' colocated with functions, classes, and config entries. At runtime the agent issues ordinary text searches, but results now carry nearby tags, enabling 'retrieval short-circuiting' - following structural links directly instead of re-searching. The study instantiates this on Codex and evaluates on SWE-bench Lite and Verified across three axes (localization accuracy, trajectory behavior, run-to-run stability), varying annotation granularity (basic topology vs. dense data/config flow) and link directionality (bidirectional vs. inverse-only).

Results Lightweight call/inheritance topology improves function-level localization (+2.2pp Func@5), shortens trajectories (-1.6 rounds), raises link-following rate from 0.15-0.18 to 0.21-0.24, roughly halves run-to-run variance, and lifts single-run reliability (+3.4pp Pass@1) on medium-scale repositories, at roughly 10% more input tokens. Granularity saturates: dense data-/config-flow tags add overhead (+4.9 rounds on Lite) and only rescue a handful of implicit-dependency cases (3/274 Lite, 15/500 Verified). Directionality is scale-dependent: medium repos (~35k LOC) benefit from bidirectional links while large hub-heavy repos do better with inverse-only edges. Practical guidance: default to lightweight topology on medium projects, prune forward edges in large hub-heavy repos, and reserve dense tags for implicit-dependency cases.

Plain-language abstract ProjAgent is a repository-level code generation system that adds procedural similarity as an explicit retrieval signal. It decomposes a target function into reasoning steps and uses an agentic workflow to retrieve repository functions that implement similar procedures at each step, represented through LLM hidden-state projections, then combines that procedural context with conventional lexical and semantic retrieval and repairs the generated code with a static-analysis feedback loop.

Motivation Repository-level code generation depends on retrieving context across cross-file dependencies and project conventions, but lexical and semantic retrieval surface only textually or semantically similar code. Useful context often comes from functions that implement the same procedure, such as input validation, unit conversion, or state transformation, while sharing little naming, type, or domain overlap, so surface-similarity retrieval overlooks them; the paper's example pair of guard-clause validators scores just 0.38 BM25 and 0.59 embedding similarity.

Methodology ProjAgent represents procedural similarity with projections of LLM hidden states, which encode implementation behavior beyond surface text. An agentic retrieval workflow decomposes the target function into steps, identifies and validates a small set of procedurally related functions, then expands the set via projection-similarity retrieval and merges it with lexical and semantic retrieval to capture project APIs, symbols, and structure. A conservative static-analysis feedback loop iteratively repairs the generated code using compiler and static-analysis feedback. It is evaluated on REPOCOD, with an ablation on the 85 Astropy problems.

Results On REPOCOD, ProjAgent reaches 41.14% Pass@1, improving Pass@1 by 12.31% over sparse, dense, and same-file retrieval baselines and outperforming SpecAgent. Ablations show procedural and semantic retrieval are complementary and both load-bearing: removing procedural context drops Pass@1 from 41.14% to 25.76%, a larger fall than removing semantic context, while the static-analysis feedback loop adds a modest but measurable improvement.

Plain-language abstract This paper presents DRACO, a system that helps AI code-completion tools work better inside private software repositories. Instead of searching for similar-looking code snippets to use as context, DRACO analyzes how data flows through a codebase, building a graph of code relationships that lets it fetch the specific background knowledge a language model needs to complete code correctly.

Motivation Pre-trained code language models struggle to complete code in private repositories because they have no knowledge of the repository's unique naming conventions, custom types, or module structure. Prior retrieval approaches that find context by text similarity often fail when the relevant code does not look textually similar to the target, causing models to hallucinate plausible-sounding but incorrect completions.

Methodology DRACO applies an extended dataflow analysis to a private repository, building a repo-specific context graph that captures type-sensitive data dependency relations among code entities. At completion time it uses dataflow information from the unfinished code to retrieve precisely relevant entities from the graph and constructs well-formed prompts for any code language model. Experiments used the existing CrossCodeEval benchmark and a newly constructed Python dataset, ReccEval, collected from the Python Package Index, testing across adapted code models, specialized code models, and GPT models.

Results DRACO improved code exact match by 3.43% and identifier F1-score by 3.27% on average compared to the previous best approach (RepoCoder). It also reduced prompt generation time by 100x on average, making it practical for real-time code completion. The approach is plug-and-play across different code language models without requiring model retraining.

Plain-language abstract CodeRAG-Bench is a benchmark for evaluating whether retrieving external documents can help language models write better code. It covers nine thousand coding tasks spanning basic programming challenges, open-domain problems, and repository-level tasks, paired with a retrieval corpus of twenty-five million documents drawn from competition solutions, tutorials, library documentation, StackOverflow posts, and GitHub repositories. The benchmark provides standardized evaluation pipelines for both retrieval quality and end-to-end code generation correctness.

Motivation Most code-generation models rely entirely on knowledge baked into their parameters during training, with no mechanism to look up relevant libraries, documentation, or examples at inference time. This limits their ability to handle unfamiliar APIs, evolving libraries, or private codebases. Retrieval-augmented generation had demonstrated strong gains on text tasks, but its usefulness for diverse code generation scenarios had not been systematically studied.

Methodology The authors assembled CodeRAG-Bench by unifying existing Python coding datasets into four categories (basic programming, open-domain, repository-level, and code retrieval) and annotating canonical reference documents for each task. They built a 25-million-document open retrieval corpus aggregating five source types. They then evaluated 10 retrieval models and 10 language models in three settings: generation with canonical (gold) documents, generation with retrieved documents, and open retrieval from the full corpus, using execution-based metrics for functional correctness.

Results Providing canonical documents substantially improves code generation: GPT-4o gains 27.4% on SWE-Bench and 6.9% on the harder ODEX subset when gold documents are supplied. Some models using retrieved documents even matched or exceeded performance with gold documents, showing strong potential for retrieval-augmented approaches. However, current retrieval models frequently fail to surface useful documents—especially for open-domain and repository-level tasks—and generation models with limited context windows show smaller gains, pointing to concrete bottlenecks that remain unsolved.

Plain-language abstract RepoCoder is a code completion framework that helps AI tools finish partially written code by drawing on the full context of a software repository, not just the file being edited. It pairs a similarity-based code retriever with a pre-trained language model in a loop: the model's first-pass completion is fed back as a new search query to find better matching snippets, and then a second completion is generated with that richer context. The paper also introduces RepoEval, a benchmark of real GitHub repositories for evaluating completions at three levels of granularity.

Motivation Existing automated code completion tools work mostly within a single file, missing the broader context of a repository such as shared utilities, cross-file API calls, and project-specific naming conventions. Simply adding a one-shot retrieval step leaves a gap: the unfinished code alone is often a poor query for finding the snippets the model actually needs, leading to plausible but incorrect completions.

Methodology RepoCoder partitions repository files into overlapping code snippets using a sliding window, then retrieves the most similar snippets to the unfinished code using a similarity-based retriever. In the first iteration this produces a draft completion; in subsequent iterations the draft completion itself becomes the retrieval query, retrieving a new set of snippets and producing an improved completion. The framework is model-agnostic and requires no retraining, static analysis tools, or heuristic rules. Experiments were run with language models of varying sizes including GPT-3.5-Turbo and CodeGen, evaluated on the new RepoEval benchmark covering line, API invocation, and function body completion tasks.

Results RepoCoder improves over the in-file completion baseline by more than 10% across all experimental settings and consistently outperforms the vanilla retrieval-augmented generation approach. The iterative pipeline delivers gains over single-pass retrieval across different model sizes, confirming that using the model's own generated output to refine retrieval is effective.

Plain-language abstract RepoBench is a benchmark for evaluating AI code auto-completion systems at the level of entire software repositories, not just single files. It covers both Python and Java and breaks the evaluation into three linked tasks: retrieving relevant code snippets from other files, predicting the next line of code given cross-file and in-file context, and running an end-to-end pipeline that combines retrieval with completion.

Motivation Existing code completion benchmarks such as CodeXGLUE focus on single-file tasks, which do not reflect how developers actually work — navigating multi-file projects where the relevant context often lives in other files across a repository. RepoBench was created to fill this gap and provide a more realistic, comprehensive evaluation platform.

Methodology The benchmark draws on two data sources: the public github-code dataset (used for optional fine-tuning training data) and freshly crawled GitHub repositories created after February 2023 (used exclusively as the test set to avoid data leakage). Code dependencies are identified with tree-sitter by parsing import statements. Three task settings vary difficulty by whether the cross-file line appears for the first time or has prior in-file usage. Baselines include random retrieval, lexical retrieval (Jaccard and Edit Similarity), and semantic retrieval (CodeBERT and UniXcoder embeddings) for RepoBench-R, and Codex, CodeGen (350M–16B), and StarCoder (15.5B) for RepoBench-C.

Results UniXcoder consistently outperformed other retrieval methods, while Jaccard Similarity beat Edit Similarity for lexical retrieval, and Python tasks generally showed higher retrieval accuracy than Java. For code completion, Codex, CodeGen, and StarCoder performed comparably on Python in the 2k-token setting, but Codex was clearly superior on Java and on the longer 8k-token setting, where StarCoder showed pronounced out-of-distribution degradation. The results highlight that models capable of handling extended contexts and generalizing across languages are essential for real-world repository-level completion.

Plain-language abstract CrossCodeEval is a benchmark for testing how well AI coding assistants can complete code that depends on content from other files in a software project. It covers four programming languages — Python, Java, TypeScript, and C# — and tests whether models can find and use the right context from across a multi-file repository, not just the single file being edited.

Motivation Existing code completion benchmarks like HumanEval and MBPP evaluate models on self-contained, single-file tasks, which do not reflect how real software is written. Real repositories span many files with cross-file dependencies, so models must locate and incorporate external context to produce correct completions — a capability that prior benchmarks could not measure.

Methodology The benchmark was built from real-world, open-source, permissively licensed repositories in Python, Java, TypeScript, and C#. A static-analysis-based approach was used to identify locations in source files where cross-file context is actively required, ensuring that each benchmark example genuinely necessitates cross-file understanding. The authors also evaluated several methods for retrieving relevant cross-file context to include in the model prompt.

Results Experiments with state-of-the-art models including CodeGen and StarCoder showed that CrossCodeEval is extremely difficult without the relevant cross-file context. Providing that context in the prompt produced clear improvements in completion accuracy, but even the best-performing models fell well short of solving the benchmark, demonstrating that leveraging cross-file context at scale remains an open challenge. The benchmark also proved useful for comparing code retrieval methods.

Plain-language abstract RLCoder is a reinforcement learning framework for repository-level code completion — the task of suggesting the next lines of code while taking into account the broader codebase a developer is working in. Rather than relying on hand-labeled training data, it teaches a retrieval model to find useful code snippets from a repository by rewarding it when the snippets it selects make the target code easier to predict. The system also learns when retrieval is unnecessary, avoiding cases where extra context would hurt rather than help.

Motivation Large language models for code have a limited context window, so they cannot simply read an entire repository when completing a file. Existing retrieval approaches either use keyword matching (BM25), which misses code semantics, or learned dense retrieval, which requires labeled data that is expensive to produce. No prior system addressed all three gaps together: the labeled-data bottleneck, the risk of poorly segmented code candidates, and the failure to decide when retrieval is even worth doing.

Methodology RLCoder trains a retriever called RLRetriever using an iterative reinforcement learning loop without labeled data. At each step the retriever proposes candidate code snippets, and an evaluator measures whether providing those snippets as context reduces the perplexity of the target code. This perplexity signal is used as a reward to update the retriever. A weighted perplexity mechanism places extra emphasis on API and identifier tokens to reduce hallucinations. A stop signal mechanism lets the retriever autonomously decide whether to retrieve at all and which candidates to discard. Candidates are built with a Split-Aggregate strategy designed to preserve code continuity. Experiments were conducted on the CrossCodeEval and RepoEval benchmarks across multiple programming languages.

Results RLCoder consistently outperforms prior state-of-the-art methods on CrossCodeEval and RepoEval, achieving a 12.2% improvement in exact match (EM) over the best previous approach. The framework generalizes across different programming languages and can also be applied on top of existing methods such as RepoCoder to further improve their performance. Ablation experiments confirm that both the stop signal and the weighted perplexity mechanism contribute to correctness gains, even though they can reduce surface-level code similarity metrics.

Plain-language abstract EVOR is a new pipeline for helping large language models write code in unfamiliar programming libraries or languages. Rather than looking up information once from a fixed reference collection, EVOR repeatedly refines both its search queries and its knowledge base as it generates and tests code, using execution feedback to guide the process.

Motivation Existing retrieval-augmented code generation systems pull information from a single, static knowledge source using a fixed query, which limits how well language models can adapt to rapidly evolving libraries or niche programming languages they were not trained on. There was no mechanism to incorporate on-the-fly feedback — such as runtime error messages or newly generated code snippets — into the retrieval process.

Methodology EVOR runs a multi-round loop in which a retriever, a language model, and a code executor interact: after each generation attempt, execution feedback and LLM outputs are used to update both the search query and the knowledge base. The knowledge base ('knowledge soup') combines library documentation, web search results, execution feedback, and LLM-generated code snippets. The authors also compiled EVOR-BENCH, four new datasets covering frequently updated libraries and long-tail programming languages, to evaluate the approach under two realistic settings that require external knowledge.

Results EVOR achieves two to four times the execution accuracy of existing methods such as Reflexion and DocPrompting across the four EVOR-BENCH datasets. EVOR is also composable — combining it with prior retrieval-augmented generation baselines yields further gains — and ablation analysis confirms that both the synchronous evolution of queries and documents and the diversity of information sources each contribute to the performance improvement.

Plain-language abstract SWE-bench is a benchmark for testing whether language models can solve real software engineering problems. It presents models with 2,294 actual GitHub issues from 12 popular Python repositories and asks them to generate code patches that fix those issues, verified by running the repository's own test suite.

Motivation Existing coding benchmarks like HumanEval consist of self-contained problems solvable in a few lines, which no longer capture what frontier language models can and cannot do. Real software engineering requires navigating large codebases, understanding interactions across many files, and reasoning about complex bugs — a much harder and more realistic challenge that prior benchmarks did not test.

Methodology The authors scraped pull requests from 12 popular Python repositories, filtered for PRs that resolved a linked GitHub issue, included changes verified by tests that shifted from failing to passing, and excluded instances with installation or runtime errors. This pipeline reduced roughly 90,000 PRs to 2,294 curated task instances. Models are given an issue description and a codebase snapshot, and must produce a patch; evaluation uses BM25-based retrieval to provide relevant context and runs the repository's test suite to check correctness. The authors also released a training set of 19,000 instances from 37 repositories and two fine-tuned models, SWE-Llama 7b and 13b, built on CodeLlama.

Results State-of-the-art models performed very poorly on SWE-bench. The best-performing model, Claude 2, resolved only 1.96% of the issues when using a BM25 retriever. Fine-tuned SWE-Llama 13b was competitive with Claude 2 in some settings and could handle contexts exceeding 100,000 tokens, but overall results confirm that current language models struggle with realistic, multi-file software engineering tasks.

Plain-language abstract SWE-agent is a system that lets a language model (LM) autonomously fix real software bugs by interacting with a computer through a purpose-built interface. Just as developers use IDEs to navigate and edit code more effectively, SWE-agent gives the language model specialized tools — file browsing, search, and editing commands — designed around the model's own strengths and limitations rather than those of human programmers.

Motivation Prior work showed LM agents could handle simple code generation tasks, but applying them to realistic software engineering — finding bugs in large codebases and writing correct patches — remained largely unexplored. Existing approaches used raw Linux shells or Python interpreters, interfaces designed for humans, not models; this mismatch left substantial performance on the table. The paper investigates whether a carefully designed agent-computer interface (ACI) tailored to LM capabilities could close that gap.

Methodology The authors built SWE-agent, which wraps a language model in an agent-computer interface providing LM-friendly commands for navigating repositories, searching files, viewing and editing code, and executing tests. They evaluated the system on SWE-bench, a benchmark of 2,294 real GitHub issues from 12 popular Python repositories, and conducted ablation studies on SWE-bench Lite (300 instances) to isolate the contribution of individual ACI design choices such as window size, history processing, and decoding temperature. The primary model tested was GPT-4 Turbo, with portability also tested using Claude 3 Opus.

Results Using GPT-4 Turbo, SWE-agent resolved 12.47% of the 2,294 full SWE-bench test tasks (286 instances) and 18.00% of the 300-instance SWE-bench Lite subset — substantially outperforming the previous best resolve rate of 3.8% achieved by a non-interactive retrieval-augmented system. The ACI alone accounted for 10.7 percentage points of improvement over a baseline agent using only the default Linux shell. SWE-agent with Claude 3 Opus resolved 10.5% of the benchmark tasks, demonstrating that the ACI design transfers across different underlying language models.

Plain-language abstract OpenHands (formerly OpenDevin) is an open-source platform for building AI agents that can do software development tasks the way a human developer would: writing and editing code, running commands in a terminal, and browsing the web. The platform provides sandboxed execution environments, a flexible event-stream architecture for agent-environment interaction, support for multiple agents working together, and a suite of evaluation benchmarks.

Motivation AI agents powered by large language models are becoming capable of tackling complex, real-world software tasks, but building and evaluating such agents requires infrastructure for safe code execution, flexible environment interaction, and standardized benchmarking. Existing open-source frameworks lacked a unified, immediately usable platform that addressed all of these needs together, slowing research and development in the field.

Methodology OpenHands is built around an event-stream architecture through which user interfaces, agents, and environments communicate. Each task session runs inside a Docker-sandboxed operating system that exposes a bash shell, a Jupyter/IPython server, and a web browser to the agent. The platform includes an agent hub with over 10 implemented agents — including a generalist agent based on the CodeAct architecture — multi-agent delegation support, and an evaluation framework. Agents are assessed across 15 benchmark tasks spanning software engineering (including SWE-Bench) and web navigation (including WebArena).

Results OpenHands successfully demonstrated agents performing across 15 challenging task categories including software engineering and web browsing benchmarks. The project has attracted more than 2,100 contributions from over 188 contributors and is released under the permissive MIT license, reflecting broad adoption across academia and industry. The platform was accepted as a conference paper at ICLR 2025.

Plain-language abstract AutoCodeRover is an automated system that takes real-world GitHub issues written in plain English and produces code patches to fix bugs or add features, without human intervention. It combines a large language model with structured code search tools that understand a program's class and method layout, rather than treating the codebase as a flat collection of files. The system was evaluated on 300 real GitHub issues and outperformed comparable automated agents at lower cost and much faster speed.

Motivation Developers spend enormous time manually fixing bugs and implementing feature requests in existing software projects. While large language models have made automated code generation easier, they do not reliably understand the structure of a full codebase well enough to resolve issues independently. There was a gap between LLM-assisted coding and fully autonomous program improvement that could handle real maintenance tasks end-to-end.

Methodology AutoCodeRover works by parsing a GitHub issue's natural language description to extract keywords corresponding to files, classes, methods, or code snippets. It then uses AST-based (abstract syntax tree) code search APIs in an iterative, stratified strategy to retrieve relevant code context. When a test suite is available, spectrum-based fault localization is used to further sharpen the context before the LLM generates a patch. The approach was benchmarked on SWE-bench-lite, a set of 300 real GitHub issues drawn from open-source Python projects.

Results AutoCodeRover resolved 19% of the 300 issues on SWE-bench-lite (pass@1), exceeding the performance of SWE-agent at the time of publication. It fixed 57 GitHub issues in approximately 4 minutes each, compared to an average of more than 2.68 days per issue for human developers. The average cost per resolved issue was $0.43 USD, substantially lower than other baseline systems.

Plain-language abstract LocAgent is a framework that helps AI systems find the exact places in a software codebase that need to be changed to fix a bug or add a feature. It works by first converting the codebase into a graph structure, then letting a language model navigate that graph to pinpoint relevant files, classes, and functions. A new benchmark called Loc-Bench was also introduced to evaluate code localization across a wider range of software tasks than existing benchmarks cover.

Motivation Developers spend a large fraction of debugging time just understanding where in a codebase a change needs to be made, and automated tools struggle with the same challenge. Existing code localization approaches rely on lexical or semantic text matching and fail to reason across the hierarchical structure and cross-file dependencies that real codebases have. Benchmarks like SWE-Bench also skew heavily toward bug reports and do not represent the full range of software maintenance tasks such as feature requests, security fixes, and performance improvements.

Methodology LocAgent parses a codebase into a directed heterogeneous graph capturing nodes at file, class, and function granularity and edges representing containment, import, inheritance, and invocation relationships. LLM agents then search and navigate this graph using unified retrieval tools that support multi-hop reasoning across dependencies. To make the approach cost-effective, open-source Qwen-2.5-Coder-Instruct models (7B and 32B variants) were fine-tuned on 768 training samples from SWE-Bench using LoRA, with training trajectories generated by Claude-3.5. A new benchmark, Loc-Bench (560 examples), was constructed with a balanced distribution of bug reports, feature requests, security issues, and performance issues to complement SWE-Bench-Lite.

Results Experiments on real-world benchmarks show LocAgent significantly improves code localization accuracy. The fine-tuned Qwen-2.5-Coder-Instruct-32B model achieves results comparable to state-of-the-art proprietary models at approximately 86% lower cost, reaching up to 92.7% accuracy at the file level. The fine-tuned Qwen2.5-7B model costs only $0.05 per example while remaining competitive with Claude-3.5, demonstrating that fine-tuned open-source models can serve as efficient alternatives to proprietary APIs for this task.

Plain-language abstract ARISE is a software engineering tool that helps AI agents find bugs and fix code across large multi-file software repositories. It builds a detailed graph of a codebase—down to individual statements and how data flows between them—and gives agents a structured API to query that graph. On a standard benchmark of 300 real GitHub bug reports, ARISE outperforms the baseline agent at both locating the faulty lines and producing correct patches.

Motivation Automated bug-finding and repair tools struggle at the scale of real repositories, where a fault may touch many files and depend on how data flows across functions. Existing graph-based code representations capture structural relationships (files, classes, functions) but stop short of modeling how variable values flow within procedures, leaving AI agents without the fine-grained, semantic information needed to pinpoint the exact faulty statement.

Methodology The authors built ARISE (Agentic Repository-level Issue Solving Engine), which constructs a multi-granularity program graph that extends structural repository relationships down to statement-level nodes connected by intra-procedural definition-use (data-flow) edges. This graph is exposed to a large language model agent through a three-tier tool API that makes data-flow slicing a directly queryable primitive—allowing the model to trace, in a single call, which statements define or consume a given variable. ARISE was evaluated on SWE-bench Lite (300 real GitHub issues across 11 Python repositories) using Qwen2.5-Coder-32B-Instruct as the backbone LLM.

Results Compared to the unmodified SWE-agent baseline, ARISE improves Function Recall@1 by 17.0 percentage points and Line Recall@1 by 15.0 percentage points for fault localization. These localization gains translate into better patch generation: ARISE achieves 22.0% Pass@1 (66 out of 300 issues resolved), a 4.7 percentage-point improvement over SWE-agent. Controlled ablations confirm that the data-flow graph drives the improvement rather than the tool schema alone, and that large code models can consume structured slice output directly without needing a natural-language summarization layer.

Plain-language abstract LingmaAgent is an AI-powered software engineering system from Alibaba that can automatically fix real-world software bugs reported as GitHub issues. Deployed inside Alibaba's TONGYI Lingma IDE coding assistant, it reads and understands an entire software repository — not just the local file where the bug appears — and then generates a patch to resolve the issue.

Motivation Existing AI agents for automated software engineering focus mainly on local code context around a reported issue, ignoring the broader structure and relationships of the full repository. This narrow view limits their ability to understand root causes and make correct fixes, especially in large industrial codebases with complex interdependencies.

Methodology LingmaAgent condenses the full repository into a knowledge graph using a top-down traversal of the code structure. It then applies a Monte Carlo Tree Search (MCTS) strategy to explore that graph, using an explore-and-exploit approach to simulate multiple reasoning trajectories, score them by reward, and progressively focus on the most relevant parts of the codebase. Agents are guided to summarize and plan using this repository-level knowledge before dynamically querying local code details and generating patches. The system was evaluated on the SWE-bench benchmark and also deployed in production at Alibaba Cloud.

Results On the SWE-bench Lite benchmark, LingmaAgent achieved an 18.5% relative improvement over SWE-agent. In production at Alibaba Cloud, the system automatically resolved 16.9% of real in-house issues filed by engineers, and resolved 43.3% of problems when engineers provided manual guidance alongside the automated agent.

Plain-language abstract CodePlan is a framework that uses large language models (LLMs) to automatically edit entire code repositories, not just individual functions or files. It treats repository-wide coding tasks — such as migrating a codebase to a new library version or adding type annotations throughout — as a planning problem, generating a multi-step sequence of targeted edits that propagates changes across all affected files.

Motivation Existing LLM-powered coding tools like GitHub Copilot excel at localized edits within a single file or function, but they cannot handle tasks that require coordinated changes spanning an entire repository. Code within a repository is heavily interdependent, and the full codebase is too large to fit in a single LLM prompt, leaving a gap for automating the class of 'outer loop' software engineering tasks that touch many files at once.

Methodology CodePlan builds a dependency graph of the repository by parsing all source files using the tree-sitter library to produce abstract syntax tree (AST) structures, then connecting AST nodes with dependency edges. Given seed edit specifications (e.g., an API change description), it uses incremental dependency analysis and a change may-impact analysis to identify which code locations are affected, then runs an adaptive planning algorithm that iteratively calls an LLM on each affected location with context drawn from the repository, prior edits, and task instructions. After each round of edits, a correctness oracle (such as a build system or static analyzer) checks the repository and any reported errors become seed specifications for the next round. The approach was evaluated on two task types: C# package migration and Python temporal code edits, across multiple repositories requiring inter-dependent changes to between 2 and 97 files.

Results CodePlan substantially outperformed baselines that provided the same contextual information but without the planning component. On validity checks (such as building without errors and producing correct edits), CodePlan succeeded on 5 out of 6 repositories, while all baselines succeeded on 0 out of 6 repositories. The paper notes these are the most complex repository-level coding tasks automated with LLMs to date.

Plain-language abstract AOCI (AI-Oriented Code Indexing) is a system for helping large language models understand large software repositories by building a compact, structured index of the entire codebase before any task begins. Each entry in the index pairs a symbolic tag with a short semantic description covering a file's function, dependencies, and design constraints. The paper describes the indexing protocol, an automated platform that keeps the index in sync as code changes, and an evaluation showing AOCI outperforms retrieval, summarization, and agent-based alternatives on both academic benchmarks and real industrial tasks.

Motivation Large language models frequently fail on codebases with hundreds of thousands of lines because existing approaches — retrieval, natural-language summarization, and agent-based exploration — each construct a different, inconsistent view of the repository at query time. Larger context windows do not solve the problem: prior work has shown that performance degrades as irrelevant code fills the context (a phenomenon called Context Rot), and retrieval methods miss cross-file dependencies, leading to bugs such as using nonexistent fields or adding incorrect tables. The gap is a stable, LLM-readable file-level intent layer that captures each module's role, dependencies, and design constraints in one place.

Methodology The authors define an AOCI index as a structured file with encoding rules followed by per-file entries, each combining a multi-dimensional symbolic tag (carrying architectural coordinates) and a compact semantic description (covering function, relations, and interface). An AOCI Platform automates generation, incremental update, and consistency checking so the index stays aligned with the codebase over time. Evaluation was conducted on four open-source projects across three frontier LLMs and six context conditions (2,160 total evaluations), and on 19 end-to-end development tasks across five real industrial systems built in Go and Node.js. An ablation study isolated contributions of individual encoding components, and an LLM-as-judge scoring pipeline (aligned with human ratings at Spearman rho=0.90) was used for grading open-ended answers.

Results AOCI ranked first in overall accuracy among all deployable methods across the academic benchmark, achieving file-localization accuracy of 97.67%, only 0.66 percentage points below the Oracle upper bound. On the 19 industrial tasks, AOCI produced zero final-state defects, while three mainstream agent-based tools introduced defects in 12 of those tasks and consumed 4 to 130 times more tokens (p < 0.001). The advantage increased with task complexity, particularly on schema-sensitive and cross-module changes where token savings exceeded 100x.

Plain-language abstract This paper presents QVoG, a software analysis platform that automatically scans source code for bugs and security vulnerabilities without running the code. It represents programs as compressed graphs, runs structured queries over those graphs to find known vulnerability patterns, and optionally uses machine learning to catch more general cases. For very large codebases (over one million lines of code), QVoG completes its analysis in about 15 minutes.

Motivation Finding bugs and security vulnerabilities early in development is far cheaper than fixing them later, but existing graph-based static analysis tools struggle with large codebases because the graph structures they build grow enormous and consume too much memory. Additionally, the query languages these tools use are complex and hard to learn, and hand-crafted detection rules tend to be overly specific, causing missed detections or false alarms on similar but slightly different vulnerable patterns.

Methodology QVoG builds a compressed version of the Code Property Graph (CPG) by replacing each statement's full Abstract Syntax Tree with a single statement node that stores AST details as attributes, substantially shrinking graph size and reducing traversal steps. On top of this graph, the authors designed a declarative domain-specific query language to make writing detection rules simpler. They also integrated machine learning models trained on existing vulnerability datasets to generalize detection beyond hard-coded rules. Performance was evaluated by running QVoG, Joern, and CodeQL on the same datasets with semantically equivalent queries, measuring query accuracy (precision and recall on common CWE vulnerability types), execution time, and memory usage.

Results For projects with over one million lines of code, QVoG completed analysis in approximately 15 minutes compared to 19 minutes for CodeQL, demonstrating better scalability. The evaluation covered query accuracy on common CWE vulnerabilities and resource consumption metrics, with QVoG's compressed CPG representation reducing graph complexity and improving overall query efficiency relative to the comparison tools.

Plain-language abstract This paper presents a machine-learning approach to automatically finding security vulnerabilities in software source code. It converts code into a structured graph representation called a Code Property Graph, then trains a convolutional neural network to classify functions as secure or insecure. The authors also release a new dataset of labeled C/C++ functions collected from open-source repositories.

Motivation Traditional tools for finding software vulnerabilities — such as static analysis and fuzzing — struggle to scale to modern software complexity and often require significant manual effort. Existing machine-learning approaches rely on simplified code representations (like abstract syntax trees alone) that miss important control-flow and data-dependency information, and labeled datasets with fine-grained, function-level annotations are scarce.

Methodology Source code functions were transformed into Code Property Graphs (CPGs) using the Joern tool; CPGs unify abstract syntax trees, control flow graphs, and program dependency graphs into a single directed multigraph. A convolutional neural network adapted for arbitrary graphs via the PATCHY-SAN framework was trained on this representation to classify functions as secure or insecure. The dataset, collected from merged GitHub pull requests, was split 70/15/15 for training, validation, and testing, and the model was trained with stochastic gradient descent (learning rate 0.001) for up to 100 epochs with early stopping.

Results The model achieved 92% overall accuracy on the test set, with precision, recall, and F1-score of 91%, 95%, and 93% respectively for secure functions, and 89%, 85%, and 87% for insecure functions. It outperformed a Support Vector Machine baseline with a Weisfeiler-Lehman graph kernel by approximately 8% in F1-score. The model performed well on common vulnerability patterns such as buffer overflows and null pointer dereferences, but showed reduced recall for vulnerabilities arising from subtle logic flaws.

Plain-language abstract VulWeaver is a system for automatically finding security vulnerabilities in source code by combining program analysis with large language models. It builds a more accurate map of how code pieces depend on each other, gathers the relevant context around suspicious code, and uses structured reasoning with LLMs to decide whether a vulnerability is present. The system works on Java and C/C++ and was tested both on benchmark datasets and in real industrial use.

Motivation Conventional static analysis tools suffer from inaccurate program representations, producing many false alarms, while recent LLM-based detection approaches are typically limited to single functions and miss the inter-procedural context needed to catch complex vulnerabilities. There was no approach that combined accurate cross-function program semantics with grounded LLM reasoning at scale.

Methodology VulWeaver first builds an enhanced unified dependency graph (UDG) by combining deterministic static-analysis rules with LLM-based semantic inference to repair gaps left by tools such as Joern. It then extracts holistic vulnerability context through program slicing for explicit data/control dependencies, augmented with implicit usage, definition, and declaration information. Finally, it uses meta-prompting with vulnerability-type-specific expert guidelines to guide LLM reasoning, aggregating verdicts via majority voting for robustness. Experiments were conducted on the PrimeVul4J (Java) and PrimeVul (C/C++) benchmark datasets.

Results On PrimeVul4J, VulWeaver achieved precision 0.81, recall 0.70, and F1-score 0.75, outperforming learning-based, LLM-based, and agent-based baselines by 23%, 15%, and 60% in F1-score respectively. Its VP-S score of 0.58 was 164% higher than the best baseline, indicating strong ability to distinguish vulnerable from patched code. On the C/C++ PrimeVul dataset it reached F1 0.78 with minimal adaptation. In real-world evaluation it detected 26 true vulnerabilities across 9 open-source Java projects (15 confirmed by developers, 5 assigned CVE identifiers), and found 40 confirmed vulnerabilities in an industrial internal repository.

Plain-language abstract This paper introduces the Code Digital Twin, a framework that gives AI coding assistants access to the hidden, informal knowledge embedded in large software projects — things like why design decisions were made, how responsibilities are divided across modules, and what the system is actually meant to do. By building a structured representation of this knowledge alongside the code itself, the framework aims to make large language models (LLMs) significantly more useful for maintaining and evolving complex software systems.

Motivation LLMs perform well on simple, self-contained coding tasks but struggle with complex, long-lived software systems. The core problem is that much critical knowledge about such systems — design rationales, high-level functionality, responsibility boundaries — is never written down formally; it lives in developer minds, informal mailing-list discussions, or commit messages. Existing LLM-based tools lack a mechanism to capture and use this tacit knowledge, making them unreliable for real-world maintenance tasks such as issue localization and repository-level code generation.

Methodology The Code Digital Twin is built through a three-phase construction pipeline. First, an artifact-oriented backbone is extracted using top-down schema-guided and schema-free LLM extraction from documentation, combined with bottom-up functionality summarization from source code. Second, rationale-centric explanatory knowledge is extracted by having LLMs identify and structure design decisions from unstructured sources such as mailing-list archives and issue threads, using predefined knowledge frame schemas. Third, an artifact-knowledge reflection step links extracted concepts and functionalities back to concrete software artifacts to form a cohesive representation. The pipeline supports incremental updates so the twin stays synchronized as the codebase evolves, and it integrates knowledge graphs, frames, and natural-language descriptions for a hybrid structured/unstructured representation.

Results This is a conceptual framework and research agenda paper rather than an empirical study, so it does not report quantitative experimental results. The paper outlines three application modes enabled by the framework — software question answering, a context-aware development co-pilot, and an autonomous development auto-pilot — and identifies open research challenges around scalable knowledge representation, automated construction pipeline accuracy, human-in-the-loop knowledge accumulation, LLM integration, and continuous co-evolution of the twin with the software.

Plain-language abstract This paper presents IDECoder, a framework that connects large language model (LLM) coding assistants directly to the static analysis capabilities built into Integrated Development Environments (IDEs). Rather than having a model guess at cross-file context, IDECoder feeds it precise, real-time information that IDEs already compute — such as class hierarchies, function signatures, and type details — then uses the IDE's linting feedback to automatically correct errors in the model's output.

Motivation Current LLM-based code assistants are trained on single-file contexts and struggle when completing code in large software repositories, where understanding a variable's type or a method's signature requires looking at other files. Existing approaches that try to retrieve cross-file context using identifier matching or semantic similarity produce inaccurate results and often exceed LLMs' context-length limits, leaving repository-level code completion a largely unsolved problem.

Methodology IDECoder operates in three phases. First, it uses the IDE's native static analysis (abstract syntax tree parsing, symbol tables, reference indexing) to accurately identify cross-file contexts such as docstrings, method signatures, and class attributes. Second, it organizes these contexts using a chain-of-thought prompting strategy that introduces information top-down — from functional role to specific type details — before sending the prompt to an LLM. Third, it passes the LLM's output back through the IDE's as-you-type linting service (e.g., Pylance), detects type errors and usage issues, and resends those diagnostics to the LLM for self-refinement. A proof-of-concept was implemented using GPT-3.5 hooked into Pylance in VS Code, and evaluated on 10 Python repositories sampled from the CROSSCODEEVAL benchmark using function-body completion as the task.

Results IDECoder outperformed all three baselines — in-file completion, all-import, and RAG — across every metric. On Exact Match it scored 10.46% versus 9.77% for the next-best (RAG); on CodeBLEU it reached 34.16% versus 31.65%; and on Syntax Match it achieved 50.73% versus 48.92%. The authors note that the current implementation is limited by restricted access to Pylance's internals and expect larger gains with fuller IDE integration.

Plain-language abstract RealBench is a new benchmark for evaluating how well large language models (LLMs) can generate entire software repositories, designed to match how code is actually written in industry: from structured specifications that include both written requirements and UML design diagrams. It contains 61 real-world repositories spanning 20 programming domains, with four difficulty levels based on code size and an average of 50 human-verified test cases per repository.

Motivation Existing code generation benchmarks ask models to produce individual functions from plain natural language descriptions, but real software development starts from structured designs translated from stakeholder requirements — not raw prose. This gap means high benchmark scores do not reliably predict how useful LLMs would be in actual enterprise or team development workflows.

Methodology The authors collected 61 Python repositories from GitHub created after December 2024 (reducing data contamination risk) and annotated each with natural language requirements and UML diagrams in JSON format. They evaluated six LLMs — GPT-4o, Claude Sonnet 4, Gemini 2.5 Flash, DeepSeek-V3, Qwen3-235B-A22B, and Qwen2.5-Coder-7B-Instruct — using three generation strategies (holistic, incremental, and retrieval-augmented generation) and measured performance at two granularities with five metrics: Completion@k, Execution@k, Pass@k, Requirement@k, and Architecture@k.

Results All LLMs performed substantially worse on repository-level generation than on standard function-level benchmarks, with significant gaps among models. Models were generally able to identify and scaffold modules defined in UML diagrams but produced poor-quality implementations due to grammar and logic errors, circular imports, and attribute mismatches. Generating the entire repository at once worked best for smaller codebases, while a module-by-module strategy outperformed alternatives on larger, more complex repositories. Ablation studies confirmed that providing detailed UML system design significantly improves generation quality.

Plain-language abstract This paper surveys how AI agents built on large language models (LLMs) are being applied to software engineering tasks. Unlike standalone LLMs, these agents can perceive and use external tools and resources, making them more capable at complex tasks. The authors collected and categorized 106 research papers, examining both what software engineering problems are being tackled and how the agents themselves are designed.

Motivation LLM-based agents represent a new paradigm beyond standalone language models, and their application to software engineering had grown rapidly without a systematic, comprehensive overview. The paper addresses the gap of understanding how these agents are designed, what SE tasks they address, and where the field is heading—including the additional promise that comes from combining multiple agents with human interaction.

Methodology The authors conducted a systematic literature survey, collecting 106 papers on LLM-based agents applied to software engineering. They categorized the papers from two complementary perspectives: the software engineering perspective (covering tasks such as requirements, development, and maintenance) and the agent perspective (covering agent design and capabilities). The survey structure also identifies open challenges and future research directions.

Results The survey finds that LLM-based agents have shown effectiveness across a wide range of SE tasks. For example, in requirements engineering, systems like Elicitation can uncover and categorize hidden needs at lower cost than conventional user studies, and SpecGen generates and validates formal requirement specifications through iterative agent-verifier feedback. The paper identifies open challenges and future directions, and maintains a public repository of covered papers at https://github.com/FudanSELab/Agent4SE-Paper-List.

Plain-language abstract This paper is the first systematic survey of how AI agents built on large language models (LLMs) are being applied across software engineering tasks. It maps 115 papers into a unified framework, explains how these agents are structured, and identifies open problems and future directions for the field.

Motivation Many researchers were already building LLM-based agents for software engineering tasks such as code generation, bug detection, and code summarization, but there was no shared vocabulary or framework to describe what these agents actually do. The lack of a structured survey made it hard to compare approaches, identify gaps, or understand how the field was developing.

Methodology The authors collected and filtered the literature, arriving at 115 papers on LLM-based agents applied to software engineering. They then developed a three-module conceptual framework — perception, memory, and action — grounded in classical agent theory, and used it to classify and analyze each paper. The perception module covers how agents receive inputs (text, visual, auditory, tree/graph-based); the memory module covers semantic, episodic, and procedural memory; and the action module covers internal actions (reasoning, retrieval, learning) and external actions (interacting with environments or humans).

Results The survey produced a taxonomy of LLM-based agents in software engineering organized around the perception-memory-action framework. The authors identified current challenges in combining LLMs with SE and proposed future research opportunities in response to those challenges. A curated repository of the 115 surveyed papers was released publicly on GitHub.

Plain-language abstract This paper introduces A3-CodGen, a framework that helps large language models (LLMs) write better code for software projects by making them aware of the surrounding codebase. When a developer asks an LLM to add a new function, the framework extracts relevant context from the current file, other files in the project, and available third-party libraries, then feeds that context into the LLM's prompt so the generated code fits naturally into the existing project.

Motivation LLM-based code generation tools such as ChatGPT and GitHub Copilot often produce code that is disconnected from the actual codebase a developer is working in. Without knowledge of local helper functions, reusable utilities in other files, or which third-party libraries are already installed, LLMs generate code with logical errors, redundancy, and library compatibility problems — code that a human developer familiar with the project would not write.

Methodology The framework identifies three categories of repository context: local-aware information from the current code file, global-aware information from functions and modules in other files of the same repository, and third-party-library information about packages already available in the project environment. These three types of context are extracted, fused, and injected into the LLM prompt through prompt engineering, guiding the model to generate code that reuses existing components rather than reinventing them.

Results Experiments demonstrated that A3-CodGen successfully extracts, fuses, and feeds repository information into an LLM to produce more accurate, efficient, and reusable code. The framework achieved a higher code reuse rate compared to human developers, indicating that repository-aware prompting can guide LLMs to leverage existing code more effectively than baseline approaches.

Plain-language abstract CodexGraph is a system that connects large language models (LLMs) to entire code repositories by building a graph database from the repository's structure and letting the LLM query it. Instead of guessing which files to look at, the model writes graph queries to navigate the code's relationships—classes, functions, inheritance, imports—and retrieves exactly what it needs. The system is evaluated on three code benchmarks and deployed in five real-world software engineering applications.

Motivation LLMs perform well on self-contained coding problems but struggle when the task spans a full software repository with many interdependent files. Existing approaches either rely on similarity-based retrieval (which has low recall on reasoning-heavy tasks) or hand-crafted, task-specific APIs that require expert knowledge and do not generalize. A flexible, task-agnostic way for an LLM agent to navigate and retrieve code from large repositories was missing.

Methodology CodexGraph uses static analysis to extract a graph database from a code repository. Nodes represent source code symbols—modules, classes, and functions—enriched with metadata, and edges capture relationships such as CONTAINS, INHERITS, and USES. A unified, task-agnostic schema defines this graph. At query time, an LLM agent writes graph queries in a query language (analogous to Cypher or SQL) to locate relevant nodes and edges; a separate translation LLM agent handles the conversion from natural language intent to executable graph queries, reducing the reasoning burden on the primary agent. The system was evaluated on CrossCodeEval, SWE-bench, and EvoCodeBench benchmarks and integrated into the ModelScope-Agent framework for five real-world application scenarios.

Results With GPT-4o as the backbone, CodexGraph outperforms other retrieval-augmented code generation baselines on CrossCodeEval Lite (Python) and EvoCodeBench, and achieves results comparable to AutoCodeRover on SWE-bench Lite. The translation LLM agent is shown to be critical: removing it causes the Exact Match score on CrossCodeEval Lite (Python) to drop from 27.90% to 8.30% with GPT-4o. Performance scales clearly with LLM capability—Qwen2-72b, DeepSeek-Coder-v2, and GPT-4o show progressively better results—demonstrating that the graph interface benefits more from stronger underlying models.

Plain-language abstract CodeGRAG is a framework that helps large language models write code more accurately by giving them a structured, graph-based view of how code actually works — showing the flow of data and control — rather than treating code as a flat sequence of words. It works with general-purpose models either without any retraining (using a text-based prompt summarizing the graph) or with fine-tuning (injecting graph knowledge directly into model weights via a pretrained graph neural network).

Motivation Large language models struggle with code generation because programming languages and natural language have a fundamental mismatch: code uses special tokens, structural branching, and non-sequential execution patterns that sequential text models are not built to handle. Existing retrieval-augmented approaches retrieve raw code blocks as context, which carry this same vocabulary mismatch and syntactic noise, so the gap between natural and programming language remains unaddressed.

Methodology CodeGRAG builds a knowledge base of code blocks represented as composed graphs combining control flow graphs and data flow graphs extracted from abstract syntax trees. At inference time, the graph most relevant to the user's task is retrieved and presented to the language model in one of two ways: a hard meta-graph prompt that summarizes the graph structure as readable text for tuning-free models, or a soft prompting technique that encodes the graph with a GNN pretrained using alignment and structure-preserving contrastive objectives and injects the resulting embedding into the language model via supervised fine-tuning. Experiments were run on four datasets — HumanEval-X and a collected CodeForce dataset — covering C++ and Python, with Pass@1 as the evaluation metric.

Results Using the hard meta-graph prompt with GPT-3.5-Turbo, CodeGRAG raised Pass@1 on HumanEval-X from 57.93% to 64.02% for C++ and from 71.95% to 77.44% for Python compared to no retrieval; the multi-lingual meta-graph variant pushed scores further to 65.24% (C++) and 77.44% (Python). The soft prompting technique consistently outperformed plain supervised fine-tuning across all tested models (Gemma 7B, Llama2 13B, CodeLlama 7B) on both the CodeForce C++ and APPS Python benchmarks, with CodeLlama 7B reaching 30.26% Pass@1 on APPS Python versus 26.15% for SFT alone. Ablations confirmed that both the alignment and structure-preserving pretraining objectives for the GNN expert each contribute independently to performance.

Plain-language abstract MAGIS is a multi-agent software system that uses large language models (LLMs) to automatically resolve GitHub issues — the bug reports and feature requests that drive real-world software evolution. Rather than feeding an entire code repository directly to a single LLM (which exceeds context limits and performs poorly), MAGIS coordinates four specialized AI agents that divide up the work of understanding, locating, changing, and reviewing code.

Motivation Popular open-source projects accumulate thousands of GitHub issues requiring changes across large codebases, yet LLMs fail to resolve more than 95% of such issues even when given the relevant file paths. The core difficulties are that repositories are too large to fit in an LLM's context window, and that correctly locating the exact files and lines to modify is a prerequisite for making valid code changes — a skill LLMs lack when working alone.

Methodology The authors first conducted an empirical study identifying the main failure modes of LLMs on GitHub issue resolution, finding a correlation between accurate file/line localization and overall success. They then built MAGIS, a framework with four agent roles: a Manager that coordinates the overall process, a Repository Custodian that identifies which files need to change, a Developer that locates the specific lines and implements the code edits, and a Quality Assurance Engineer that reviews the resulting changes. The framework was evaluated on the SWE-bench benchmark, comparing against direct use of GPT-3.5, GPT-4, and Claude-2.

Results MAGIS resolved 13.94% of GitHub issues in the SWE-bench benchmark, significantly outperforming the baseline LLMs. Using GPT-4 as its underlying model, MAGIS achieved an eight-fold increase in the issue resolution rate compared to applying GPT-4 directly without the multi-agent framework. Further analysis showed that planning code changes, localizing the correct lines within files, and the code review step each contributed meaningfully to this improvement.

Plain-language abstract LibEvolutionEval is a benchmark and study that tests whether AI code-completion models can handle the fact that popular Python libraries change over time. The benchmark covers eight libraries—including PyTorch, Matplotlib, pandas, and scipy—across multiple versions, and evaluates whether models produce correct code for the specific library version a developer is using.

Motivation Code AI models are trained on large snapshots of open-source code, so they may have learned API usage patterns from one library version that are wrong or broken in another. Existing benchmarks focus on local file context and do not capture how rapidly-evolving public libraries affect model accuracy, leaving a gap in understanding how well these tools actually serve developers who work with different library versions simultaneously.

Methodology The authors built a dataset of version-specific code-completion tasks drawn from two sources: real GitHub repositories (where requirements files record the exact library version used) and controlled synthetic examples generated from official API documentation. Eight libraries were covered across their evolution over multiple years, with APIs annotated as introduced, deprecated, modified, or unchanged. Evaluations were run with StarCoder2, Mistral, and GPT-4o-mini using three context settings—in-file context only, version-aware prompting, and version-aware retrieval-augmented generation (RAG) with documentation retrieved using CodeSage embeddings.

Results Library evolution substantially affects model performance across all tested models and libraries, with accuracy fluctuating as APIs change across versions. Providing version-specific documentation via RAG consistently improved code completion accuracy, but did not fully eliminate version-based bias—both language models and embedding models showed persistent performance gaps tied to specific library versions, likely reflecting uneven representation of those versions in training data. Models performed better on indirect API completions (where they infer the API from an object reference) than on direct completions, and newly introduced or deprecated APIs posed the greatest difficulty.

Plain-language abstract AI coding agents spend a large fraction of their tool calls just finding their way around a codebase before they can do any real work. This paper asks whether giving an agent a compact, structured document that declares a project's module boundaries, function signatures, and design constraints can cut that exploration overhead — and if so, what format that document should take.

Motivation When an AI coding agent lacks a map of a codebase it must probe file after file to locate relevant code, wasting tool calls and increasing the chance of error. The paper addresses this navigational overhead and the open question of whether the format of an architecture descriptor (S-expression, JSON, YAML, or Markdown) meaningfully affects agent performance.

Methodology The authors ran three complementary studies using Claude Sonnet 4.6 at temperature 0. First, a controlled experiment with 24 code-localization tasks across 4 conditions tested whether architecture context reduces navigation steps. Second, a 20-question study across 4 descriptor formats measured accuracy, and a separate 15-task artifact-vs-process experiment isolated whether the descriptor file itself provides value independent of developer self-clarification. Third, an observational field study examined 7,012 Claude Code sessions, and a writer-side experiment ran 96 generation runs with 96 error injections across formats on 646,000 lines of production code.

Results Providing any architecture context reduced navigation steps by 33–44% regardless of format (Wilcoxon signed-rank p = 0.009, Cohen's d = 0.92). All four formats achieved identical 95% accuracy. An automatically generated descriptor with no human refinement reached 100% task accuracy versus 80% blind (p = 0.002, d = 1.04), proving the file itself has direct navigational value. Formal declaration correlated with a 52% reduction in agent behavioral variance across 7,012 sessions. The authors propose intent.lisp, an S-expression format, citing its non-atomic failure mode, compression density (22% shorter than JSON, with 5:1–64:1 compression ratios across the production code corpus), and syntactic enforcement of hierarchy as its key advantages over alternatives.

Plain-language abstract This paper introduces KGCompass, a system that helps large language models fix bugs in large software repositories more accurately and cheaply. It builds a knowledge graph linking issue reports, pull requests, and code entities, then uses that graph to guide the model to the right functions before generating a patch.

Motivation Fixing bugs automatically in real-world codebases is hard because bug reports are written in natural language while the code spans thousands of files. Most LLM-based repair systems either miss the buggy location entirely or flood the model with too much context, causing hallucinations and high cost. Only about 32.7% of bug reports in the SWE-bench Lite benchmark explicitly name the file or function containing the bug, so models need a smarter way to navigate the codebase.

Methodology KGCompass constructs a repository-aware knowledge graph that links repository artifacts—issues and pull requests—to code entities such as files, classes, and functions. Given a new bug report, it traverses this graph using multi-hop paths to narrow the search space down to the 20 most relevant candidate functions. A path-guided repair mechanism then feeds the traced entity paths and their context to an LLM, which generates a patch with an explanation. The system was evaluated on the SWE-bench Lite benchmark of 300 Python repository tasks.

Results KGCompass achieves a 45.67% repair resolution rate and 51.33% function-level fault localization accuracy on SWE-bench Lite, state-of-the-art among open-source single-LLM approaches, at a cost of only $0.20 per repair. Among successfully localized bugs, 69.7% required multi-hop traversal through the knowledge graph—paths that pure LLM approaches cannot reliably follow. When combined with stronger models, the graph-guided approach raised resolution rates by 50.8% over a Claude Sonnet baseline and by over 150% over weaker baselines.

Plain-language abstract R2C2-Coder is a system developed at Alibaba for improving and evaluating how well AI coding assistants can complete code at the level of an entire software repository, not just a single file. It introduces both a smarter prompt-construction method (R2C2-Enhance) and a new benchmark (R2C2-Bench) designed to reflect the complexity of real-world software projects.

Motivation Existing repository-level code completion methods rely on simple retrieval-augmented generation that breaks source files into small, disconnected code snippets. These fragments lose important project-wide context such as class hierarchies, cross-file dependencies, and shared utility definitions, and existing benchmarks cover only limited completion scenarios that do not faithfully represent how developers actually work.

Methodology R2C2-Enhance builds a two-tier retrieval pool from each repository: an abstraction context capturing coarse-grained, file-level structure using a parser-generator tool (Tree-sitter), and a snippet context providing fine-grained local code fragments. For each cursor position during completion, a retrieval query is formed and used to pull relevant entries from this pool; the retrieved context is combined with the current file to construct the prompt sent to a code LLM. Building on this, R2C2-Bench adds training, validation, and test splits along with a context perturbation strategy designed to test robustness to noisy or irrelevant retrieved context.

Results Extensive experiments across multiple benchmarks demonstrate the effectiveness of R2C2-Coder, showing that the richer retrieval pool and structured prompt construction improve repository-level code completion compared to baseline retrieval-augmented approaches.

Plain-language abstract CodeAgent is an AI-based framework that helps large language models write code for real software repositories, not just isolated functions. It gives the model access to five programming tools — for searching documentation, finding existing code, and running tests — and uses four different strategies to guide how the model uses those tools. The authors also built a new benchmark, CodeAgentBench, with 101 real-world repository-level coding tasks to measure performance.

Motivation Most code-generation research focuses on producing short, standalone functions that do not depend on a surrounding codebase. Yet more than 70% of functions in real open-source projects are non-standalone, relying on repository-specific classes, utilities, and documentation. Existing large language models struggle with this complexity, creating a gap between benchmark performance and practical software development.

Methodology The authors built CodeAgent, an LLM-based agent framework that equips language models with five external programming tools enabling information retrieval, code-symbol lookup, and automated testing within a repository context. Four agent strategies — ReAct, Tool-Planning, OpenAIFunc, and a Rule-based approach — guide the model in selecting and sequencing tool use. They evaluated CodeAgent on nine open-source and closed-source LLMs ranging from 13B to 175B parameters, using both their new CodeAgentBench (101 repository-level functions and classes from real projects) and the established HumanEval function-level benchmark.

Results On CodeAgentBench, CodeAgent improved pass rates by 2.0 to 15.8 percentage points across all tested LLMs compared to direct generation without tools. On the HumanEval benchmark, CodeAgent also demonstrated competitive performance, outperforming commercial tools including GitHub Copilot in accuracy. These results held across models of different sizes and licensing types, indicating broad applicability of the approach.

Plain-language abstract RAMBO is a system that helps AI coding tools write complete function bodies in large software repositories. Rather than just finding code that looks similar to what is needed, RAMBO identifies the specific classes, methods, and variables that a new function actually needs to use, then feeds those ingredients—along with examples of how they are used—to a large language model so it can generate correct, project-aware code.

Motivation Completing entire method bodies in large repositories is much harder than completing a single line of code, because the generated code must correctly use project-specific APIs, follow inter-module dependencies, and respect custom conventions. Existing retrieval-augmented approaches like RepoCoder retrieve similar code snippets, but studies show that up to 80% of such retrievals do not help—and can actively hurt—because method bodies are usually too unique for similarity-based search to surface useful examples.

Methodology RAMBO uses a retrieval-augmented generation pipeline in which the retrieval step is redesigned to extract essential repository-specific code elements—declared classes, methods, and variables/fields—and their concrete usages relevant to the method being completed. These elements and usage examples are assembled into the prompt context for a code large language model. The approach was evaluated on 40 Java projects using several leading code LLMs.

Results RAMBO significantly outperformed state-of-the-art repository-level method body completion approaches, achieving improvements of up to 46% in BLEU score, 57% in CodeBLEU, 36% in Compilation Rate, and up to 3x in Exact Match. It also surpassed the RepoCoder Oracle method—which uses an idealized retrieval upper bound—by up to 12% in Exact Match, setting a new benchmark for this task.

Plain-language abstract RepoGenReflex is a framework for AI-assisted code completion that works at the repository scale, meaning it can draw on code from many files across an entire codebase rather than just the file being edited. It combines a retrieval step — finding relevant code snippets from the repository — with a feedback loop that iteratively refines both the retrieved context and the generated code, using natural-language feedback rather than retraining the model.

Motivation Existing code completion tools either rely on rigid rules or on large language models that are constrained by their context window size and their localized training knowledge. When a developer works in a large codebase, relevant context may span dozens of files, and prior retrieval-augmented approaches like RepoCoder used fixed iteration limits rather than dynamically stopping when results were good enough. There was no mechanism to learn from the quality of each generated attempt and redirect subsequent retrieval accordingly.

Methodology The framework consists of five components: a Retriever, an Actor (a generative language model, specifically CodeGen-Mono-6B), an Evaluator, a Reflector, and an Experience cache. In each iteration the Retriever fetches relevant code snippets using Jaccard similarity, the Actor generates a candidate completion, and the Evaluator scores it with Exact Match (EM) and Edit Similarity (ES) metrics. The Reflector — using Meta-Llama-3-8B, selected experimentally over a code-specialized model — then produces natural-language feedback including evaluation analysis, contextual analysis, and specific suggestions. This feedback is stored in Experience and used to construct new retrieval targets for the next iteration. The loop stops when ES improvement falls below 1% for three consecutive iterations or a perfect EM score is reached. Experiments used both the existing RepoEval benchmark and a new benchmark, RepoGenEval, built from 8 high-quality Python repositories on GitHub with 700–40,000 stars, requiring over 90% Python content and explicit unit tests, yielding 1,600 test samples.

Results RepoGenReflex consistently achieved higher EM and ES scores than four state-of-the-art baselines — CodeT5+ 2B, CodeLlama-7b-hf, StarCoder, and CodeGemma — on both the RepoEval and RepoGenEval benchmarks. On RepoGenEval, RepoGenReflex reached EM scores of approximately 0.438–0.455 and ES scores of approximately 0.735–0.740 across repositories, compared to the next-best model (CodeGemma) at roughly 0.421–0.441 EM and 0.700–0.728 ES. An ablation study confirmed that removing the Reflector and Experience components caused a significant drop in both metrics, and that removing the Evaluator also hurt performance, validating the contribution of each component.

Plain-language abstract SWE-Explore is a benchmark that isolates one capability of coding agents — how well they explore a repository to find the code relevant to an issue — instead of scoring only the final pass/fail repair. Given an issue and a repository, an explorer returns a ranked list of line-level code regions under a fixed budget, scored against the regions that agents which actually solved the issue relied on.

Motivation Repository-level benchmarks like SWE-bench reduce each attempt to a single resolved/unresolved bit, which conflates exploration, localization, and patch synthesis and hides which step failed. Two distinct failure modes exist: an agent never explores the relevant code, or it finds the evidence but fails to synthesize a correct patch. Existing executable benchmarks capture the latter; the former is largely invisible, and no common, precise target lets classical retrievers, search agents, and long-context selectors be compared on exploration quality. File- or function-level localization only says an agent reached the right neighborhood, not which lines it consulted.

Methodology Repository exploration is formalized as a standalone task f:(issue, repo) -> ranked list of regions (file path + line range), requiring no patch and no repository interaction. Line-level ground truth is trajectory-grounded: read actions are extracted from independent agent trajectories that successfully solved the same issue (kept only when >=2 trajectories resolve it under the executable harness), deduplicated per file and aggregated by consensus into high-confidence core regions and lower-confidence optional regions, with human QA. Explorers are scored on coverage, ranking, and budget-efficiency metrics. A separate restricted-context repair bridge — feeding only the explorer's output to a fixed coding agent and checking whether the patch passes the original tests — validates that the metrics track repair, as a one-time external-validity check rather than part of the standard loop.

Results SWE-Explore spans 848 instances across 10 programming languages and 203 repositories (64.5% Python, the rest Go/JavaScript/Rust/Java/PHP/TypeScript/Ruby/C/C++), built from SWE-bench Verified, SWE-bench-Pro, and SWE-bench Multilingual. Across retrieval methods, general coding agents, and specialized localizers, agentic explorers form a clear tier above classical retrieval. File-level localization is already strong for modern methods, but line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers, and the exploration metrics strongly track downstream repair behavior in the restricted-context validation.

Plain-language abstract CORE-Bench is a benchmark for code retrieval as coding agents actually do it: instead of matching a natural-language query to one isolated snippet, it tests whether a retriever can, given a real development request and a snapshot of a repository, find the files and functions that need editing and the surrounding context required to make the change. It is organized in three levels — basic code understanding, issue-to-edit localization, and broader context retrieval — built largely from SWE-bench-series issues and their repositories.

Motivation In agentic and 'vibe' coding, retrieval is a step inside the agent's loop: given a bug report or feature request the agent must decide which files and functions to inspect before editing. Existing benchmarks such as CodeSearchNet and the code tasks in MTEB/CoIR only test decontextualized snippet matching under fixed corpora, missing five properties that matter in practice — a large intent-to-implementation gap, evidence scattered across code, configuration, and dependencies, long function-level chunks that dilute a single embedding, requests that need multiple edit locations, and dense in-repository distractors. Strong scores on those benchmarks can therefore overstate how useful a model is to a coding agent.

Methodology The benchmark has three levels. Level-1 reuses curated traditional retrieval tasks (text-to-code, code-to-code, hybrid). Level-2 (issue-to-edit localization) is built from SWE-bench-series instances: each repository is checked out to the commit immediately before the PR so the corpus matches the pre-edit state, source and documentation files are AST-chunked, and the PR patch is aligned back to those chunks to produce relevance labels, with an LLM filter (Qwen3.5-397B) removing answer-leaking or underspecified descriptions. Level-3 (broader context) reruns resolution with a code agent (Mini-SWE-Agent), extracts the files it browses from execution traces, judges each with a three-vote LLM ensemble (two Qwen votes plus one Claude Sonnet 4.6 vote), then verifies usefulness by re-running the agent under a closed allowlist of only the annotated files. Models are scored by NDCG@10 and Recall@100, and rewrite variants convert raw PR/issue text into concise assistant-style queries.

Results Retrieval models that look strong on traditional code search drop sharply on the agentic levels: Qwen3-Embedding-8B goes from 71.7 NDCG@10 / 96.9 Recall@100 on Level-1 to 20.3 / 48.0 on Level-2 and 34.4 / 41.5 on Level-3, and different models collapse into a similar low band, indicating Level-2/3 are not just scaled-up versions of code search. In-domain supervised fine-tuning on GitHub pull-request supervision improves performance across all difficulty regimes, but a clear gap remains and Recall@100 still falls as repositories grow denser and larger, leaving substantial headroom for requirement-driven repository retrieval.

Plain-language abstract LLM code agents mostly navigate repositories with keyword (grep) search, which returns lines by lexical match and hides the structural relationships - who calls whom, inheritance, configuration dependencies - that determine whether a line is actually relevant. This paper asks how much that missing structure is worth. It builds CodeAnchor, which runs lightweight static analysis offline and injects the resulting structural facts back into source files as plain-text comments ('anchor tags'), so an unchanged grep-first agent sees structural links inline as it searches. Tested on top of Codex on SWE-bench Lite and Verified, the tags help less by making the agent smarter and more by making its navigation disciplined and reproducible.

Motivation Grep-first agents are fast, language-agnostic, and competitive - often beating graph-based approaches - but they suffer a representation mismatch: lexical retrieval cannot expose structural topology, so agents rediscover links through ad-hoc multi-hop queries and produce brittle, stochastic trajectories where two runs on the same task visit different files and take different paths. For software engineering, that unpredictability is a quality concern: practitioners need agent behavior to be inspectable and reproducible, not just correct in aggregate.

Methodology CodeAnchor extracts call graphs (PyCG), inheritance, data flow, and configuration usage via offline static analysis and AST extractors, then injects them as compact plain-text 'CodeAnchor tags' colocated with functions, classes, and config entries. At runtime the agent issues ordinary text searches, but results now carry nearby tags, enabling 'retrieval short-circuiting' - following structural links directly instead of re-searching. The study instantiates this on Codex and evaluates on SWE-bench Lite and Verified across three axes (localization accuracy, trajectory behavior, run-to-run stability), varying annotation granularity (basic topology vs. dense data/config flow) and link directionality (bidirectional vs. inverse-only).

Results Lightweight call/inheritance topology improves function-level localization (+2.2pp Func@5), shortens trajectories (-1.6 rounds), raises link-following rate from 0.15-0.18 to 0.21-0.24, roughly halves run-to-run variance, and lifts single-run reliability (+3.4pp Pass@1) on medium-scale repositories, at roughly 10% more input tokens. Granularity saturates: dense data-/config-flow tags add overhead (+4.9 rounds on Lite) and only rescue a handful of implicit-dependency cases (3/274 Lite, 15/500 Verified). Directionality is scale-dependent: medium repos (~35k LOC) benefit from bidirectional links while large hub-heavy repos do better with inverse-only edges. Practical guidance: default to lightweight topology on medium projects, prune forward edges in large hub-heavy repos, and reserve dense tags for implicit-dependency cases.

Plain-language abstract ProjAgent is a repository-level code generation system that adds procedural similarity as an explicit retrieval signal. It decomposes a target function into reasoning steps and uses an agentic workflow to retrieve repository functions that implement similar procedures at each step, represented through LLM hidden-state projections, then combines that procedural context with conventional lexical and semantic retrieval and repairs the generated code with a static-analysis feedback loop.

Motivation Repository-level code generation depends on retrieving context across cross-file dependencies and project conventions, but lexical and semantic retrieval surface only textually or semantically similar code. Useful context often comes from functions that implement the same procedure, such as input validation, unit conversion, or state transformation, while sharing little naming, type, or domain overlap, so surface-similarity retrieval overlooks them; the paper's example pair of guard-clause validators scores just 0.38 BM25 and 0.59 embedding similarity.

Methodology ProjAgent represents procedural similarity with projections of LLM hidden states, which encode implementation behavior beyond surface text. An agentic retrieval workflow decomposes the target function into steps, identifies and validates a small set of procedurally related functions, then expands the set via projection-similarity retrieval and merges it with lexical and semantic retrieval to capture project APIs, symbols, and structure. A conservative static-analysis feedback loop iteratively repairs the generated code using compiler and static-analysis feedback. It is evaluated on REPOCOD, with an ablation on the 85 Astropy problems.

Results On REPOCOD, ProjAgent reaches 41.14% Pass@1, improving Pass@1 by 12.31% over sparse, dense, and same-file retrieval baselines and outperforming SpecAgent. Ablations show procedural and semantic retrieval are complementary and both load-bearing: removing procedural context drops Pass@1 from 41.14% to 25.76%, a larger fall than removing semantic context, while the static-analysis feedback loop adds a modest but measurable improvement.

Plain-language abstract SWE-bench is a benchmark for testing whether language models can solve real software engineering problems. It presents models with 2,294 actual GitHub issues from 12 popular Python repositories and asks them to generate code patches that fix those issues, verified by running the repository's own test suite.

Motivation Existing coding benchmarks like HumanEval consist of self-contained problems solvable in a few lines, which no longer capture what frontier language models can and cannot do. Real software engineering requires navigating large codebases, understanding interactions across many files, and reasoning about complex bugs — a much harder and more realistic challenge that prior benchmarks did not test.

Methodology The authors scraped pull requests from 12 popular Python repositories, filtered for PRs that resolved a linked GitHub issue, included changes verified by tests that shifted from failing to passing, and excluded instances with installation or runtime errors. This pipeline reduced roughly 90,000 PRs to 2,294 curated task instances. Models are given an issue description and a codebase snapshot, and must produce a patch; evaluation uses BM25-based retrieval to provide relevant context and runs the repository's test suite to check correctness. The authors also released a training set of 19,000 instances from 37 repositories and two fine-tuned models, SWE-Llama 7b and 13b, built on CodeLlama.

Results State-of-the-art models performed very poorly on SWE-bench. The best-performing model, Claude 2, resolved only 1.96% of the issues when using a BM25 retriever. Fine-tuned SWE-Llama 13b was competitive with Claude 2 in some settings and could handle contexts exceeding 100,000 tokens, but overall results confirm that current language models struggle with realistic, multi-file software engineering tasks.

Plain-language abstract This paper introduces the Code Digital Twin, a framework that gives AI coding assistants access to the hidden, informal knowledge embedded in large software projects — things like why design decisions were made, how responsibilities are divided across modules, and what the system is actually meant to do. By building a structured representation of this knowledge alongside the code itself, the framework aims to make large language models (LLMs) significantly more useful for maintaining and evolving complex software systems.

Motivation LLMs perform well on simple, self-contained coding tasks but struggle with complex, long-lived software systems. The core problem is that much critical knowledge about such systems — design rationales, high-level functionality, responsibility boundaries — is never written down formally; it lives in developer minds, informal mailing-list discussions, or commit messages. Existing LLM-based tools lack a mechanism to capture and use this tacit knowledge, making them unreliable for real-world maintenance tasks such as issue localization and repository-level code generation.

Methodology The Code Digital Twin is built through a three-phase construction pipeline. First, an artifact-oriented backbone is extracted using top-down schema-guided and schema-free LLM extraction from documentation, combined with bottom-up functionality summarization from source code. Second, rationale-centric explanatory knowledge is extracted by having LLMs identify and structure design decisions from unstructured sources such as mailing-list archives and issue threads, using predefined knowledge frame schemas. Third, an artifact-knowledge reflection step links extracted concepts and functionalities back to concrete software artifacts to form a cohesive representation. The pipeline supports incremental updates so the twin stays synchronized as the codebase evolves, and it integrates knowledge graphs, frames, and natural-language descriptions for a hybrid structured/unstructured representation.

Results This is a conceptual framework and research agenda paper rather than an empirical study, so it does not report quantitative experimental results. The paper outlines three application modes enabled by the framework — software question answering, a context-aware development co-pilot, and an autonomous development auto-pilot — and identifies open research challenges around scalable knowledge representation, automated construction pipeline accuracy, human-in-the-loop knowledge accumulation, LLM integration, and continuous co-evolution of the twin with the software.

Plain-language abstract RealBench is a new benchmark for evaluating how well large language models (LLMs) can generate entire software repositories, designed to match how code is actually written in industry: from structured specifications that include both written requirements and UML design diagrams. It contains 61 real-world repositories spanning 20 programming domains, with four difficulty levels based on code size and an average of 50 human-verified test cases per repository.

Motivation Existing code generation benchmarks ask models to produce individual functions from plain natural language descriptions, but real software development starts from structured designs translated from stakeholder requirements — not raw prose. This gap means high benchmark scores do not reliably predict how useful LLMs would be in actual enterprise or team development workflows.

Methodology The authors collected 61 Python repositories from GitHub created after December 2024 (reducing data contamination risk) and annotated each with natural language requirements and UML diagrams in JSON format. They evaluated six LLMs — GPT-4o, Claude Sonnet 4, Gemini 2.5 Flash, DeepSeek-V3, Qwen3-235B-A22B, and Qwen2.5-Coder-7B-Instruct — using three generation strategies (holistic, incremental, and retrieval-augmented generation) and measured performance at two granularities with five metrics: Completion@k, Execution@k, Pass@k, Requirement@k, and Architecture@k.

Results All LLMs performed substantially worse on repository-level generation than on standard function-level benchmarks, with significant gaps among models. Models were generally able to identify and scaffold modules defined in UML diagrams but produced poor-quality implementations due to grammar and logic errors, circular imports, and attribute mismatches. Generating the entire repository at once worked best for smaller codebases, while a module-by-module strategy outperformed alternatives on larger, more complex repositories. Ablation studies confirmed that providing detailed UML system design significantly improves generation quality.

Plain-language abstract R2C2-Coder is a system developed at Alibaba for improving and evaluating how well AI coding assistants can complete code at the level of an entire software repository, not just a single file. It introduces both a smarter prompt-construction method (R2C2-Enhance) and a new benchmark (R2C2-Bench) designed to reflect the complexity of real-world software projects.

Motivation Existing repository-level code completion methods rely on simple retrieval-augmented generation that breaks source files into small, disconnected code snippets. These fragments lose important project-wide context such as class hierarchies, cross-file dependencies, and shared utility definitions, and existing benchmarks cover only limited completion scenarios that do not faithfully represent how developers actually work.

Methodology R2C2-Enhance builds a two-tier retrieval pool from each repository: an abstraction context capturing coarse-grained, file-level structure using a parser-generator tool (Tree-sitter), and a snippet context providing fine-grained local code fragments. For each cursor position during completion, a retrieval query is formed and used to pull relevant entries from this pool; the retrieved context is combined with the current file to construct the prompt sent to a code LLM. Building on this, R2C2-Bench adds training, validation, and test splits along with a context perturbation strategy designed to test robustness to noisy or irrelevant retrieved context.

Results Extensive experiments across multiple benchmarks demonstrate the effectiveness of R2C2-Coder, showing that the richer retrieval pool and structured prompt construction improve repository-level code completion compared to baseline retrieval-augmented approaches.

Plain-language abstract RepoQA is a benchmark for testing how well large language models understand code in a realistic, long-context setting. The core task asks a model to find a specific function inside a large codebase using only a plain-English description of what that function does — success requires genuine code comprehension, not just pattern matching. The benchmark covers 500 tasks drawn from 50 real GitHub repositories across Python, C++, Java, TypeScript, and Rust, and was used to evaluate 33 general-purpose and code-specialized models.

Motivation Existing long-context benchmarks test models on plain text or synthetic retrieval tasks, but none specifically measured how well models understand code at repository scale. Meanwhile, prior code benchmarks such as RepoBench and CrossCodeEval kept input token counts small, and the widely cited SWE-Bench was too complex for model-only evaluation (GPT-4 achieved only a 1.31% pass rate using retrieval). RepoQA fills this gap by targeting the core code-understanding capability needed in real software development, where a developer must locate a function in thousands of lines of unfamiliar code.

Methodology The authors built an automated dataset curation pipeline over 50 popular GitHub repositories (10 per language) selected for topic diversity, code quality, and at least 100 GitHub stars. For each repository, source files were concatenated in topological dependency order, split into 64 equal chunks, and one unique function per chunk was selected as a candidate "needle." Ten needle functions per repository were chosen at evenly spaced positions, and GPT-4-Turbo was used to write structured four-part natural-language descriptions (purpose, input, output, procedure) without revealing function or variable names. During evaluation, each model received a 16k-token code context with the needle planted at a known depth, and retrieval success was measured using BLEU score against all functions in the context, requiring the output to be closest to the target with a similarity threshold of at least 0.8.

Results Among the 33 evaluated models, the top tier — claude-3-opus, gemini-1.5-pro, and gpt-4o — all achieved around 90.6% average retrieval accuracy. The best open-source models, DeepSeek-V2-Chat and Meta-Llama-3-70B-Instruct, reached 83.4% and 82.2% respectively, slightly ahead of some proprietary models like GPT-4-Turbo (76.4%). Performance varied notably across languages: most models scored highest on Java and TypeScript and lowest on Rust, likely reflecting training corpus size. A counterintuitive finding was that removing code comments improved retrieval accuracy for most models, suggesting LLMs can comprehend code semantics independently of inline documentation; Gemini models were the main exception, as they tended to lose track of the task when comments were replaced with synthetic line-number placeholders.

Plain-language abstract CodeAgent is an AI-based framework that helps large language models write code for real software repositories, not just isolated functions. It gives the model access to five programming tools — for searching documentation, finding existing code, and running tests — and uses four different strategies to guide how the model uses those tools. The authors also built a new benchmark, CodeAgentBench, with 101 real-world repository-level coding tasks to measure performance.

Motivation Most code-generation research focuses on producing short, standalone functions that do not depend on a surrounding codebase. Yet more than 70% of functions in real open-source projects are non-standalone, relying on repository-specific classes, utilities, and documentation. Existing large language models struggle with this complexity, creating a gap between benchmark performance and practical software development.

Methodology The authors built CodeAgent, an LLM-based agent framework that equips language models with five external programming tools enabling information retrieval, code-symbol lookup, and automated testing within a repository context. Four agent strategies — ReAct, Tool-Planning, OpenAIFunc, and a Rule-based approach — guide the model in selecting and sequencing tool use. They evaluated CodeAgent on nine open-source and closed-source LLMs ranging from 13B to 175B parameters, using both their new CodeAgentBench (101 repository-level functions and classes from real projects) and the established HumanEval function-level benchmark.

Results On CodeAgentBench, CodeAgent improved pass rates by 2.0 to 15.8 percentage points across all tested LLMs compared to direct generation without tools. On the HumanEval benchmark, CodeAgent also demonstrated competitive performance, outperforming commercial tools including GitHub Copilot in accuracy. These results held across models of different sizes and licensing types, indicating broad applicability of the approach.

Plain-language abstract RAMBO is a system that helps AI coding tools write complete function bodies in large software repositories. Rather than just finding code that looks similar to what is needed, RAMBO identifies the specific classes, methods, and variables that a new function actually needs to use, then feeds those ingredients—along with examples of how they are used—to a large language model so it can generate correct, project-aware code.

Motivation Completing entire method bodies in large repositories is much harder than completing a single line of code, because the generated code must correctly use project-specific APIs, follow inter-module dependencies, and respect custom conventions. Existing retrieval-augmented approaches like RepoCoder retrieve similar code snippets, but studies show that up to 80% of such retrievals do not help—and can actively hurt—because method bodies are usually too unique for similarity-based search to surface useful examples.

Methodology RAMBO uses a retrieval-augmented generation pipeline in which the retrieval step is redesigned to extract essential repository-specific code elements—declared classes, methods, and variables/fields—and their concrete usages relevant to the method being completed. These elements and usage examples are assembled into the prompt context for a code large language model. The approach was evaluated on 40 Java projects using several leading code LLMs.

Results RAMBO significantly outperformed state-of-the-art repository-level method body completion approaches, achieving improvements of up to 46% in BLEU score, 57% in CodeBLEU, 36% in Compilation Rate, and up to 3x in Exact Match. It also surpassed the RepoCoder Oracle method—which uses an idealized retrieval upper bound—by up to 12% in Exact Match, setting a new benchmark for this task.

Plain-language abstract RepoGenReflex is a framework for AI-assisted code completion that works at the repository scale, meaning it can draw on code from many files across an entire codebase rather than just the file being edited. It combines a retrieval step — finding relevant code snippets from the repository — with a feedback loop that iteratively refines both the retrieved context and the generated code, using natural-language feedback rather than retraining the model.

Motivation Existing code completion tools either rely on rigid rules or on large language models that are constrained by their context window size and their localized training knowledge. When a developer works in a large codebase, relevant context may span dozens of files, and prior retrieval-augmented approaches like RepoCoder used fixed iteration limits rather than dynamically stopping when results were good enough. There was no mechanism to learn from the quality of each generated attempt and redirect subsequent retrieval accordingly.

Methodology The framework consists of five components: a Retriever, an Actor (a generative language model, specifically CodeGen-Mono-6B), an Evaluator, a Reflector, and an Experience cache. In each iteration the Retriever fetches relevant code snippets using Jaccard similarity, the Actor generates a candidate completion, and the Evaluator scores it with Exact Match (EM) and Edit Similarity (ES) metrics. The Reflector — using Meta-Llama-3-8B, selected experimentally over a code-specialized model — then produces natural-language feedback including evaluation analysis, contextual analysis, and specific suggestions. This feedback is stored in Experience and used to construct new retrieval targets for the next iteration. The loop stops when ES improvement falls below 1% for three consecutive iterations or a perfect EM score is reached. Experiments used both the existing RepoEval benchmark and a new benchmark, RepoGenEval, built from 8 high-quality Python repositories on GitHub with 700–40,000 stars, requiring over 90% Python content and explicit unit tests, yielding 1,600 test samples.

Results RepoGenReflex consistently achieved higher EM and ES scores than four state-of-the-art baselines — CodeT5+ 2B, CodeLlama-7b-hf, StarCoder, and CodeGemma — on both the RepoEval and RepoGenEval benchmarks. On RepoGenEval, RepoGenReflex reached EM scores of approximately 0.438–0.455 and ES scores of approximately 0.735–0.740 across repositories, compared to the next-best model (CodeGemma) at roughly 0.421–0.441 EM and 0.700–0.728 ES. An ablation study confirmed that removing the Reflector and Experience components caused a significant drop in both metrics, and that removing the Evaluator also hurt performance, validating the contribution of each component.

Plain-language abstract This paper introduces RepoRift, a system that uses AI agents and Retrieval Augmented Generation (RAG) to improve semantic code search — the task of finding relevant code snippets from a codebase given a plain-English description. The agents enrich a user's query with context fetched from the relevant GitHub repository, then a multi-stream ensemble approach compares the augmented query against code embeddings to return the best-matching snippets.

Motivation Current semantic code search tools struggle when user queries are vague or use different terminology than the codebase — a problem called the Vocabulary Mismatch Problem. A user searching for 'model' might mean a machine-learning model, a database schema, or a design pattern, leading to poor results. Prior work improved the mapping between natural language and code but did not address the upstream problem of ambiguous or under-specified queries.

Methodology The system uses a CrewAI-based agent built on GPT-4 that performs an internet search scoped to a target GitHub repository and appends relevant technical context to the user's original query. The enriched query is then processed through a three-stream ensemble: one stream embeds the augmented query and ranks functions by cosine similarity; a second generates equivalent Python code via GPT-3.5-turbo and matches its embedding against repository functions and classes; a third decomposes that generated code into component functions and finds the closest match for each. A final GPT-4o re-ranking step selects the most relevant snippets from the combined candidate set. The system was evaluated on 101 manually processed queries drawn from the Python split of the CodeSearchNet dataset.

Results RepoRift achieved a 78.2% success rate at Success@10 and 34.6% at Success@1, outperforming the best prior baseline (PSCS) which scored 47.6% and 22.9% respectively on the Java split of the same dataset. The lower-bound accuracy at Success@10 was approximately 22.5 percentage points above the next-best method. The system also handled queries written in Russian, raw URLs, and vague conceptual descriptions without additional preprocessing.

Plain-language abstract MutaGReP is a system that helps AI coding assistants work with large code repositories without having to load the entire codebase into their context window. Given a user request, it searches for a step-by-step plan that identifies exactly which functions and classes from the codebase are needed, then hands that compact plan to any code-generation model to produce the final code.

Motivation When an AI model needs to write code using a large existing repository, the naive approach of dumping the whole repo into the prompt is wasteful and actively harmful: research shows LLM reasoning degrades sharply as context grows, and context windows are finite. The gap the paper addresses is the lack of a principled, execution-free method for automatically finding a focused, codebase-grounded plan that captures only the relevant symbols a model needs.

Methodology MutaGReP frames plan discovery as a neural tree search over a structured plan space. Each plan is a sequence of steps, where every step pairs a natural-language intent with a set of codebase symbols retrieved by a grounding function. Starting from the user query as a root node, an LLM-guided search expands nodes by mutating plans using either a monotonic successor (which only appends new steps) or an unconstrained successor (which can also modify or reorder steps). A scoring function ranks candidate plans, with two variants evaluated: a Likert-style quality scorer and a diversity scorer. The system is benchmarked on the LongCodeArena dataset of real repository-conditioned coding tasks.

Results MutaGReP plans occupy less than 5% of the 128K-token context window used by GPT-4o with full repository access, yet match its coding performance. Smaller models benefit substantially: Llama 3.2 3B improved from 7.4% to 32.9% symbol-overlap score, and Qwen 2.5 7B improved from 17.5% to 45.0%. Qwen 2.5 Coder 32B with MutaGReP plans (53.0% overlap) outperformed GPT-4o with full repository context (49.9% overlap) while conditioning on 95% less context. On the hardest LongCodeArena tasks, where GPT-4o with the full repository scores below 20% overlap, the tree-searched plans enabled dramatically better results, including 83.75% average overlap on one benchmark repository where full-context GPT-4o achieved only 12%.

Plain-language abstract This paper introduces SSCD, a tool for automatically finding duplicate or near-duplicate code fragments (called code clones) in very large software codebases. Instead of comparing every possible pair of code snippets — which becomes impossibly slow at scale — SSCD converts each snippet into a compact numerical representation and then uses a fast nearest-neighbour search to find similar ones, with GPU acceleration for extra speed.

Motivation Duplicate code is hard to maintain and a source of bugs, but finding inexact copies (Type 3 and Type 4 clones, which involve insertions, deletions, or functional similarity rather than textual identity) is extremely difficult. Existing neural-network approaches that compare code pairs one-by-one require O(n²) comparisons, making them impractical for industrial codebases of hundreds of millions of lines. An industrial partner needed a solution that could find these hard clones across their very large systems without taking days to run.

Methodology SSCD fine-tunes CodeBERT and GraphCodeBERT — pre-trained transformer models — using a contrastive loss objective so that each code fragment is mapped to an embedding where clones are close together in vector space. Clone detection then becomes a nearest-neighbour search over these embeddings rather than pairwise comparison, using GPU-accelerated approximate nearest-neighbour libraries. The approach was configured and evaluated empirically using BigCloneBench (BCB), a standard benchmark that includes a 320-million-line-of-code version, comparing against state-of-the-art tools SAGA and SourcererCC.

Results SSCD outperformed state-of-the-art approaches SAGA and SourcererCC in recall of Type 3 and Type 4 clones. In its optimal configuration — favouring shorter input lengths and text-only neural network models — it completed clone detection across the entire 320 million LOC BigCloneBench in just under three hours, compared to over 4.5 days required by SourcererCC on a 250+ million LOC version of the same benchmark. The configuration analysis showed that shorter inputs and text-only models improve efficiency with only a slight decrease in effectiveness.

Plain-language abstract This paper presents DB-SSCD, a system for finding duplicate or similar code segments across very large software repositories — think codebases with over a billion lines of code. It uses a neural network to convert each code fragment into a compact numerical representation, then searches for similar representations using disk storage instead of holding everything in RAM, making the approach practical for industrial-scale use.

Motivation Neural network-based code clone detection works well for moderate-sized codebases but breaks down at industrial scale because the search data structures must fit in memory. Existing tools like SSCD are limited by available RAM, leaving no scalable neural approach for repositories with hundreds of millions or billions of lines of code — a gap that matters for detecting vulnerabilities, license violations, and maintenance burdens in large industrial codebases.

Methodology The authors built DB-SSCD on top of the SSCD framework, replacing its in-memory nearest-neighbor index with disk-resident indexes using Faiss and Milvus. Each code fragment is encoded into a high-dimensional vector embedding via a transformer-based neural network (CodeBERT). Approximate k-nearest-neighbor search then finds similar embeddings directly on disk or solid-state drives, avoiding O(N²) pairwise comparison. They also introduced a de-duplication step that collapses sets of near-identical (Type-1) clone fragments to a single representative embedding before indexing, addressing a specific failure mode where large identical-code clusters degrade k-ANN search quality. Accuracy was evaluated on BigCloneBench (320 million lines of Java), and scalability was tested on the C subset of The Stack dataset (1.06 billion lines of code).

Results DB-SSCD achieves clone detection accuracy comparable to the equivalent in-memory approach, while scaling to codebases that exceed available RAM. On problems that fit within memory, the disk-based approach using a solid-state drive is approximately 2x slower than the in-memory baseline. The system was demonstrated to scale to over one billion lines of code, providing practical insights into the tradeoffs among indexing speed, query performance, and storage efficiency at industrial scale.

Plain-language abstract This paper asks whether giving an AI coding agent access to a team's product decisions — things like design choices, user personas, and competitive context — helps it write code that actually follows those decisions. The authors built a benchmark of realistic software tasks and compared a baseline AI agent (with only the codebase) against an augmented agent that could also retrieve recorded product context, finding a large improvement in how often the agent honored team-specific decisions.

Motivation AI coding agents can write working code, but they routinely ignore team-specific product, design, and engineering decisions that are not visible anywhere in the source code. There was no existing controlled benchmark for measuring this 'decision compliance' gap, nor a systematic study of whether providing external product context could close it.

Methodology The authors created a controlled benchmark with 8 realistic software engineering tasks containing 41 weighted decision points. They ran two configurations on identical prompts against the same repository: a baseline using Claude Code with codebase access only, and an augmented configuration that added a product-context retrieval system called Brief, which supplies spec generation, mid-build consultation, recorded decisions, persona pain points, customer signals, and competitive intelligence. All 16 resulting pull requests and the scoring harness were released for independent reproduction.

Results The augmented configuration achieved 95% decision compliance versus 46% for the baseline, a 49 percentage point improvement. Per-decision analysis showed the baseline reached 100% compliance on decisions visible in the codebase but only 0–33% on decisions requiring product context, indicating that product-context retrieval was the key driver of the improvement.

Plain-language abstract This paper presents K3 Trans, a system that helps large language models translate code between programming languages when the code is part of a larger software repository rather than a standalone function. The system enriches the translation prompt with three types of retrieved examples drawn from relevant codebases and past translation work, then uses the model to self-debug its own output.

Motivation LLMs perform well at translating isolated functions between programming languages, but their accuracy drops sharply when code belongs to a full repository with complex dependencies and project-specific conventions. One study found even the best model suffered a 30.8% drop in Pass@1 (from 74.3% to 43.5%) on repository-level tasks compared to isolated-function benchmarks, making LLM-based translation impractical for real industrial software projects.

Methodology K3 Trans builds a translation knowledge base by mining three sources: open-source target-language codebases (for syntax examples), the repository under translation (for dependency usage examples), and previously completed translation function pairs (for idiomatic pattern references). For each function to be translated, it retrieves relevant entries from all three knowledge types and assembles them into an augmented prompt fed to an LLM, which then generates the translated code. A self-debugging loop follows, where the same LLM is used to identify and fix errors in its own output. The system was evaluated on the RustRepoTrans benchmark.

Results K3 Trans outperformed the RAG-based baseline by 19.4% to 40.2% relative improvement in Pass@1 and improved CodeBLEU by 0.138. The proportion of correct-or-repairable translated functions reached 82.1% on the DSR@1 metric, with a Repairable Ratio of 44.6%. Ablation experiments showed that each of the three knowledge types contributes independently, with dependency usage examples providing the largest individual gain. The knowledge base also improved over time as the self-evolution process accumulated more successful translation pairs.

Plain-language abstract SWE-bench-java is a benchmark dataset for testing whether AI language models can automatically fix real bugs in Java software projects. The authors built the benchmark by collecting GitHub issues and their corresponding patches from 19 popular open-source Java repositories, then manually verified the quality of 91 final issue instances. They also ran several state-of-the-art AI models on the benchmark using the SWE-agent framework and released the dataset, a Docker-based evaluation environment, and a public leaderboard.

Motivation Existing benchmarks for automated software issue resolution, most notably SWE-bench, focused exclusively on Python. This left a gap for the many real-world domains — finance, cloud services, Android development — that rely heavily on Java, one of the world's most widely used programming languages. A multilingual benchmark was needed to evaluate whether AI models can handle issue resolution beyond Python.

Methodology The authors constructed SWE-bench-java-verified through a five-phase pipeline: collecting 70 candidate Java repositories from GitHub and the Defects4J database, crawling 1,979 issue instances from pull requests, determining runtime environments (Maven or Gradle build tools, JDK version), extracting fail-to-pass tests by comparing test results before and after ground-truth patches, and conducting questionnaire-based manual verification by 10 experienced Java developers. This filtering process reduced the dataset to a final 91 high-quality issues across 6 repositories. They then evaluated the SWE-agent framework paired with five large language models — GPT-4o, GPT-4o-mini, DeepSeekCoder-V2, DeepSeek-V2, and Doubao-pro — measuring the percentage of issues fully resolved.

Results Resolved rates across all tested model-agent combinations were low, ranging from 1.10% (GPT-4o-mini and Doubao-pro, 1 out of 91 issues) to 9.89% (DeepSeek-V2, 9 out of 91 issues). DeepSeek-V2 outperformed DeepSeekCoder-V2 on repositories with longer natural-language issue descriptions, while DeepSeekCoder-V2 performed better on repositories with minimal textual content, suggesting that natural language understanding capability is a key differentiator. Overall performance gaps among models confirm that the benchmark effectively distinguishes model capabilities and that Java issue resolution remains a substantially unsolved challenge.

Plain-language abstract This paper describes how Meta's AI-assisted code completion tool, CodeCompose, was extended from suggesting single lines of code to suggesting multiple lines at once. The authors explain the engineering challenges they solved and report on the measured impact after rolling out multi-line suggestions to tens of thousands of developers.

Motivation AI coding assistants had been limited to single-line completions, which help only with immediate, narrow context. Extending to multi-line suggestions could help developers discover APIs, best practices, and full implementation patterns, but doing so without disrupting the developer's workflow and without unacceptable latency required solving several unsolved problems at scale.

Methodology The team developed pre-processing and post-processing algorithms that use semantic understanding of programming language structure and cursor position to determine when and how to display multi-line suggestions without visually disrupting already-written code. They also invested in client and server-side optimizations to reduce suggestion latency, and added a UI loading indicator to reduce perceived wait time. The system was evaluated through online A/B experiments across tens of thousands of Meta engineers.

Results Server-hosting optimizations reduced multi-line suggestion latency by 2.5x. In A/B experiments, multi-line suggestions accounted for 42% of total characters accepted despite representing only 16% of displayed suggestions. The feature nearly doubled the share of keystrokes saved by users, from 9% to 17%. Multi-line CodeCompose was rolled out to all engineers at Meta, with fewer than 1% opting out.

Plain-language abstract This paper studies a class of software quality problems that arise specifically when deep learning frameworks are built using two programming languages together — typically Python for high-level interfaces and C/C++ for performance-critical internals. The authors define seven "inter-language design smells" (poor coding patterns that only appear at the boundary between languages), build an automated detection tool called CPSMELL, and measure how prevalent and persistent these smells are in five major deep learning frameworks.

Motivation Prior work on software quality in machine learning projects focused exclusively on single-language (Python) codebases, missing problems that emerge from the interaction between multiple programming languages. Because frameworks like TensorFlow and PyTorch mix Python and C/C++, bugs and maintainability issues can originate at language boundaries — where data, errors, and module bindings cross between the two languages — and standard single-language smell detectors cannot find them.

Methodology The authors surveyed Stack Overflow, technical documentation, research literature, and GitHub issues to identify candidate inter-language design smells, then required each candidate to be documented in at least two sources (one authoritative) and present in at least two frameworks. This process yielded seven smells covering three inter-language communication mechanisms (Python/C API, pybind11, ctypes). They implemented detection rules for all seven in the CPSMELL tool, manually validated its accuracy against five popular frameworks (TensorFlow, PyTorch, Chainer, PaddlePaddle, MindSpore), and ran an empirical study across ten version snapshots per framework to assess distribution, developer fix rates, and evolution over time.

Results CPSMELL achieved an overall detection accuracy of 98.17% across the five frameworks (individual smell accuracy ranged from 93.28% to 100%). The empirical study found that inter-language design smell instances are widespread and, in most frameworks, grow over time rather than being resolved — the number of smell instances showed a fluctuating but overall increasing trend in TensorFlow, PyTorch, PaddlePaddle, and MindSpore. Some smells such as Excessive Inter-Language Communication and Large Inter-Language Binding Class were more likely to be fixed, while Lack of Rigorous Error Check and Not Using Relative Path had fix rates of 0% across three years of version iterations. PaddlePaddle had the highest overall fix rate at 18.52%, while TensorFlow and PyTorch remained below 10%.

Plain-language abstract This paper asks whether AI-powered automatic bug-fixing agents—systems that can read a bug report, localize the fault, write a patch, and verify it—work as well inside a large software company as they do on popular research benchmarks. The authors built a Google-internal agent called Passerine and tested it on a curated set of 178 real bugs from Google's issue tracker, then compared the bug characteristics to the widely used SWE-Bench benchmark.

Motivation Agentic program-repair systems have been evaluated almost exclusively on SWE-Bench, a collection of Python bugs from open-source GitHub projects. It was unclear whether strong benchmark performance would carry over to an enterprise setting, where bugs span many languages, projects, and reporting mechanisms—and where the potential cost savings from automation are substantial.

Methodology The authors curated GITS-Eval, a benchmark of 178 bugs from Google's internal issue tracking system: 78 human-reported bugs and 100 machine-reported bugs (50 from automated sanitizers and 50 from a test-order-dependency analyzer). They then implemented Passerine, an agent inspired by SWE-Agent but adapted to Google's internal development environment, using a limited command inventory and Gemini 1.5 Pro as its underlying model. Each bug was given 20 sampled repair trajectories, and generated patches were assessed both automatically (plausibility via test passage) and manually (semantic equivalence to the ground-truth fix). The distribution of GITS bugs was also compared to SWE-Bench along dimensions such as language diversity, change size, and fault-localization difficulty.

Results With 20 trajectory samples, Passerine produced a plausible patch (one that passes the bug's tests) for 73% of machine-reported bugs and 25.6% of human-reported bugs. After manual review, 43% of machine-reported bugs and 17.9% of human-reported bugs had at least one patch semantically equivalent to the ground-truth fix—with the sanitizer-reported subset reaching 62% semantic correctness. The analysis also showed that GITS bugs differ meaningfully from SWE-Bench bugs in language diversity, change spread, and localization difficulty, so performance on one benchmark is not a reliable predictor of performance on the other.

Plain-language abstract This paper describes how Google adapted the Gemini large language model specifically for its own internal software engineering environment, producing a custom model called Gemini for Google (GfG). The team assembled a trillion-token dataset from Google's proprietary development ecosystem and trained the model through mid-training and fine-tuning to understand internal libraries, codebases, and workflows. The result is a deployed system that assists tens of thousands of engineers daily with tasks ranging from code completion to large-scale automated refactoring.

Motivation General-purpose AI coding assistants perform well on self-contained algorithmic challenges but struggle with the maintenance and evolution of large existing codebases full of proprietary libraries, internal tooling, and domain-specific patterns. Standard retrieval-augmented approaches yield inconsistent results, while lightweight adaptation methods like LoRA lack the representational capacity to absorb a trillion tokens of specialized data. There was a gap between frontier model capabilities and the specific demands of a hyperscale industrial software environment like Google's monorepo of billions of lines of code.

Methodology The authors curated a proprietary corpus of roughly one trillion tokens spanning six categories: code change generation (including comment-resolution tuples capturing critique-and-refine dialogues), internal knowledge bases and documentation, issues and bug fixes, code generation data in fill-in-the-middle format, production logs and efficiency edits, and developer activity timelines. They applied a mid-training intervention (continued pretraining) starting from a Gemini foundation checkpoint, using ablations on smaller models to determine optimal data mixture weights, followed by supervised post-training on internal high-quality datasets. A catastrophic-forgetting mitigation strategy was employed to retain general reasoning while injecting domain knowledge. The model was then evaluated both offline using static analysis and LLM-as-a-judge metrics, and online through a blind A/B study with over 29,000 Google developers.

Results In the large-scale blind A/B study, GfG reduced the mean number of conversational iterations per turn by 23% and decreased mean turn duration by 8.85%, while code survival rate in final submissions increased by 16.80% and hunk acceptance rate improved by 4.49% compared to the baseline Gemini model. In case studies, the model automated a migration of 5,359 JUnit3 test files (over 149,000 lines) in three months with 87% of AI-generated code committed without human modification. A code efficiency application deployed over 6,400 commits representing more than 25,000 lines of changed production code at a greater than 99.5% production success rate, saving an estimated 500,000 normalized CPU cores per quarter. The Smart Paste IDE feature built on GfG reached tens of thousands of daily users with a 45% acceptance rate.

Plain-language abstract This paper surveys 860 software developers at Microsoft to find out what AI tools they actually want built next. Rather than studying what AI tools developers currently use, it asks what systems should exist, cataloguing 22 distinct AI-assistant concepts that developers described, along with the conditions they placed on each one.

Motivation Existing AI coding tools focus almost entirely on code generation, yet writing code accounts for only about one-tenth of a developer's working day. The rest — debugging, code review, documentation, onboarding, incident response, and stakeholder communication — receives comparatively little AI support. Meanwhile, accelerating code generation is expanding the slower, more labour-intensive downstream work such as review and verification, creating what the authors call a 'right-shift burden.'

Methodology The study draws on open-ended survey responses from 860 Microsoft developers, using a human-in-the-loop, multi-stage thematic analysis (described in the text as 'council-based thematic' analysis) to surface and characterise the AI systems developers requested. For each of the 22 systems identified, the authors record the problem it addresses, what makes it technically difficult to build, and the explicit behavioural constraints developers placed on it.

Results Developers identified 22 desired AI systems spanning five task categories. The most prevalent single request, cited by 50.1% of development-block respondents, was a scoped pull-request builder for technical debt removal. The second most common (27.8%) was an embedded quality gate that provides diff-anchored bug-spotting and standards enforcement during authorship. Across all 22 systems, a consistent pattern emerged that the authors term 'bounded delegation': developers wanted AI to absorb assembly and coordination work, but explicitly not the core craft decisions. They consistently required authority scoping, provenance tracking, uncertainty signalling, and least-privilege access — stopping points rather than autonomous action.

Plain-language abstract This paper is a survey of how large language models handle code written in many different programming languages. It covers two main tasks: generating new code from a natural-language description across multiple target languages, and translating existing code from one language to another while keeping the same behavior. The authors review the methods researchers use, the benchmarks they test on, and the metrics they use to measure success.

Motivation Most research on AI-assisted coding focuses heavily on popular languages like Python and Java, leaving models much weaker on languages like Rust, OCaml, C++, or Lua. Real-world software systems routinely mix many languages, so a model that works only for Python is not practically useful. The survey addresses this gap by systematically mapping what has been done and what remains unsolved for truly cross-language code intelligence.

Methodology The authors organize the literature around four methodological paradigms: prompt engineering (eliciting multilingual behavior from frozen models via zero-shot or few-shot prompts), model pre-training and fine-tuning (embedding multilingual competence into model weights through large-scale data and synthetic generation), multi-agent collaborative frameworks (decomposing tasks into specialized roles with iterative feedback loops), and retrieval-augmented generation (injecting external language-specific knowledge at inference time). They also catalog benchmarks such as HumanEval-XL, McEval, mHumanEval, XLCoST, and RepoTransBench, and evaluation metrics including Pass@k, Computational Accuracy, and LLM-as-a-Judge approaches.

Results The survey finds that no single paradigm fully solves multilingual code intelligence. Prompt engineering improves surface-level alignment but is bounded by the model's pretraining distribution; fine-tuning embeds knowledge but is constrained by data imbalance; multi-agent frameworks add robustness through decomposition; and RAG supplies missing domain-specific knowledge at inference time. A key finding is that the primary barrier for models is no longer semantic understanding alone, but increasingly the mastery of precise, low-level implementation details in diverse and constrained programming environments, with the critical challenges being monolingual bias, weak cross-paradigm transfer, fragmented evaluation, and limited trustworthiness.

Plain-language abstract This paper is a systematic literature review of how 'breaking changes' — modifications to a software library that cause existing dependent code to stop working — arise, spread, and can be managed across modern software ecosystems. The authors analyzed 97 published studies covering five major ecosystems (Maven/Java, npm/JavaScript, Python, Web APIs, and Linux distributions) to produce the first comprehensive synthesis of this problem area.

Motivation Modern applications depend on large networks of third-party libraries, and a single incompatible change in a widely used package can cascade through the dependency graph and break downstream projects whose authors had no involvement in the change. Despite growing research on the topic, no comprehensive synthesis existed across ecosystems, leaving practitioners without a unified understanding of how breaking changes are classified, why they occur, how they are detected, and how they can be managed.

Methodology The authors followed the systematic literature review guidelines of Kitchenham and Charters, searching three major academic databases (IEEE Xplore, ACM Digital Library, and Springer) with structured keyword queries targeting breaking change terminology. They applied inclusion and exclusion criteria, then performed backward and forward snowballing, arriving at a final corpus of 97 primary studies. Data were extracted to answer four research questions covering types, reasons and impacts, detection approaches, and management strategies for breaking changes.

Results The review produced four main outputs: a four-dimensional taxonomy of breaking changes along the axes of nature, detectability, scope, and visibility; five reason categories and five impact dimensions, with maintenance and design improvements accounting for a larger share of breaking changes than new feature work; 43 detection approaches that achieve high accuracy on syntactic breaks but limited coverage on behavioral ones; and 66 management strategies organized by actor role. Three open challenges were identified — behavioral break detection at scale, the systematic failure of semantic versioning as a trust mechanism, and transitive dependency propagation under information asymmetry — along with three research opportunities involving LLM-assisted behavioral contract inference, ecosystem-level dependency graph intelligence, and domain-specific tooling for machine learning libraries.

Plain-language abstract This paper investigates whether widely-used code generation benchmarks have been contaminated by appearing in the training data of large language models (LLMs), and quantifies how much that contamination inflates measured performance. The authors build a pipeline to find benchmark problems and their solutions inside two major open training corpora, then test whether models score higher on the problems they have seen.

Motivation Benchmark contamination — where test examples leak into a model's training data — is a serious threat to reliable evaluation of LLMs, but prior work focused mostly on natural language tasks. Code is structurally different from prose: two programs can be semantically identical yet look very different on the surface due to variable renaming or whitespace, so standard text-similarity methods miss many contaminated examples. A rigorous, code-aware contamination study was needed.

Methodology The authors developed a two-stage matching pipeline applied to two popular code generation benchmarks, MBPP and HumanEval, against two large open pretraining corpora, the Pile and the Stack. Surface-level similarity was measured using character-level Levenshtein edit distance with a sliding window over the training data. Semantic similarity was measured using Dolos, a plagiarism-detection tool that canonicalizes programs into abstract syntax tree representations and computes k-gram overlap, making it robust to identifier renaming and whitespace differences. They then evaluated three model families — StarCoderBase, Pythia, and CodeGen-NL — comparing accuracy on contaminated versus clean subsets and analyzing the effect of model size and problem difficulty.

Results Substantial overlap was found between both benchmarks and the training corpora. Models performed significantly better on the subset of benchmark problems for which similar solutions appeared in their training data. After removing contaminated examples, the accuracy gap between StarCoderBase-15.5B and Pythia-12B shrank from 23.8% to 13.9%, suggesting that a large share of observed performance differences across models is attributable to contamination rather than genuine capability. Larger models within each family showed stronger memorization as well as better generalization, and the performance boost on seen questions could not be explained away by those questions being easier than unseen ones.

Plain-language abstract SWE-Explore is a benchmark that isolates one capability of coding agents — how well they explore a repository to find the code relevant to an issue — instead of scoring only the final pass/fail repair. Given an issue and a repository, an explorer returns a ranked list of line-level code regions under a fixed budget, scored against the regions that agents which actually solved the issue relied on.

Motivation Repository-level benchmarks like SWE-bench reduce each attempt to a single resolved/unresolved bit, which conflates exploration, localization, and patch synthesis and hides which step failed. Two distinct failure modes exist: an agent never explores the relevant code, or it finds the evidence but fails to synthesize a correct patch. Existing executable benchmarks capture the latter; the former is largely invisible, and no common, precise target lets classical retrievers, search agents, and long-context selectors be compared on exploration quality. File- or function-level localization only says an agent reached the right neighborhood, not which lines it consulted.

Methodology Repository exploration is formalized as a standalone task f:(issue, repo) -> ranked list of regions (file path + line range), requiring no patch and no repository interaction. Line-level ground truth is trajectory-grounded: read actions are extracted from independent agent trajectories that successfully solved the same issue (kept only when >=2 trajectories resolve it under the executable harness), deduplicated per file and aggregated by consensus into high-confidence core regions and lower-confidence optional regions, with human QA. Explorers are scored on coverage, ranking, and budget-efficiency metrics. A separate restricted-context repair bridge — feeding only the explorer's output to a fixed coding agent and checking whether the patch passes the original tests — validates that the metrics track repair, as a one-time external-validity check rather than part of the standard loop.

Results SWE-Explore spans 848 instances across 10 programming languages and 203 repositories (64.5% Python, the rest Go/JavaScript/Rust/Java/PHP/TypeScript/Ruby/C/C++), built from SWE-bench Verified, SWE-bench-Pro, and SWE-bench Multilingual. Across retrieval methods, general coding agents, and specialized localizers, agentic explorers form a clear tier above classical retrieval. File-level localization is already strong for modern methods, but line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers, and the exploration metrics strongly track downstream repair behavior in the restricted-context validation.

Plain-language abstract CORE-Bench is a benchmark for code retrieval as coding agents actually do it: instead of matching a natural-language query to one isolated snippet, it tests whether a retriever can, given a real development request and a snapshot of a repository, find the files and functions that need editing and the surrounding context required to make the change. It is organized in three levels — basic code understanding, issue-to-edit localization, and broader context retrieval — built largely from SWE-bench-series issues and their repositories.

Motivation In agentic and 'vibe' coding, retrieval is a step inside the agent's loop: given a bug report or feature request the agent must decide which files and functions to inspect before editing. Existing benchmarks such as CodeSearchNet and the code tasks in MTEB/CoIR only test decontextualized snippet matching under fixed corpora, missing five properties that matter in practice — a large intent-to-implementation gap, evidence scattered across code, configuration, and dependencies, long function-level chunks that dilute a single embedding, requests that need multiple edit locations, and dense in-repository distractors. Strong scores on those benchmarks can therefore overstate how useful a model is to a coding agent.

Methodology The benchmark has three levels. Level-1 reuses curated traditional retrieval tasks (text-to-code, code-to-code, hybrid). Level-2 (issue-to-edit localization) is built from SWE-bench-series instances: each repository is checked out to the commit immediately before the PR so the corpus matches the pre-edit state, source and documentation files are AST-chunked, and the PR patch is aligned back to those chunks to produce relevance labels, with an LLM filter (Qwen3.5-397B) removing answer-leaking or underspecified descriptions. Level-3 (broader context) reruns resolution with a code agent (Mini-SWE-Agent), extracts the files it browses from execution traces, judges each with a three-vote LLM ensemble (two Qwen votes plus one Claude Sonnet 4.6 vote), then verifies usefulness by re-running the agent under a closed allowlist of only the annotated files. Models are scored by NDCG@10 and Recall@100, and rewrite variants convert raw PR/issue text into concise assistant-style queries.

Results Retrieval models that look strong on traditional code search drop sharply on the agentic levels: Qwen3-Embedding-8B goes from 71.7 NDCG@10 / 96.9 Recall@100 on Level-1 to 20.3 / 48.0 on Level-2 and 34.4 / 41.5 on Level-3, and different models collapse into a similar low band, indicating Level-2/3 are not just scaled-up versions of code search. In-domain supervised fine-tuning on GitHub pull-request supervision improves performance across all difficulty regimes, but a clear gap remains and Recall@100 still falls as repositories grow denser and larger, leaving substantial headroom for requirement-driven repository retrieval.

Plain-language abstract LocAgent is a framework that helps AI systems find the exact places in a software codebase that need to be changed to fix a bug or add a feature. It works by first converting the codebase into a graph structure, then letting a language model navigate that graph to pinpoint relevant files, classes, and functions. A new benchmark called Loc-Bench was also introduced to evaluate code localization across a wider range of software tasks than existing benchmarks cover.

Motivation Developers spend a large fraction of debugging time just understanding where in a codebase a change needs to be made, and automated tools struggle with the same challenge. Existing code localization approaches rely on lexical or semantic text matching and fail to reason across the hierarchical structure and cross-file dependencies that real codebases have. Benchmarks like SWE-Bench also skew heavily toward bug reports and do not represent the full range of software maintenance tasks such as feature requests, security fixes, and performance improvements.

Methodology LocAgent parses a codebase into a directed heterogeneous graph capturing nodes at file, class, and function granularity and edges representing containment, import, inheritance, and invocation relationships. LLM agents then search and navigate this graph using unified retrieval tools that support multi-hop reasoning across dependencies. To make the approach cost-effective, open-source Qwen-2.5-Coder-Instruct models (7B and 32B variants) were fine-tuned on 768 training samples from SWE-Bench using LoRA, with training trajectories generated by Claude-3.5. A new benchmark, Loc-Bench (560 examples), was constructed with a balanced distribution of bug reports, feature requests, security issues, and performance issues to complement SWE-Bench-Lite.

Results Experiments on real-world benchmarks show LocAgent significantly improves code localization accuracy. The fine-tuned Qwen-2.5-Coder-Instruct-32B model achieves results comparable to state-of-the-art proprietary models at approximately 86% lower cost, reaching up to 92.7% accuracy at the file level. The fine-tuned Qwen2.5-7B model costs only $0.05 per example while remaining competitive with Claude-3.5, demonstrating that fine-tuned open-source models can serve as efficient alternatives to proprietary APIs for this task.

Plain-language abstract ARISE is a software engineering tool that helps AI agents find bugs and fix code across large multi-file software repositories. It builds a detailed graph of a codebase—down to individual statements and how data flows between them—and gives agents a structured API to query that graph. On a standard benchmark of 300 real GitHub bug reports, ARISE outperforms the baseline agent at both locating the faulty lines and producing correct patches.

Motivation Automated bug-finding and repair tools struggle at the scale of real repositories, where a fault may touch many files and depend on how data flows across functions. Existing graph-based code representations capture structural relationships (files, classes, functions) but stop short of modeling how variable values flow within procedures, leaving AI agents without the fine-grained, semantic information needed to pinpoint the exact faulty statement.

Methodology The authors built ARISE (Agentic Repository-level Issue Solving Engine), which constructs a multi-granularity program graph that extends structural repository relationships down to statement-level nodes connected by intra-procedural definition-use (data-flow) edges. This graph is exposed to a large language model agent through a three-tier tool API that makes data-flow slicing a directly queryable primitive—allowing the model to trace, in a single call, which statements define or consume a given variable. ARISE was evaluated on SWE-bench Lite (300 real GitHub issues across 11 Python repositories) using Qwen2.5-Coder-32B-Instruct as the backbone LLM.

Results Compared to the unmodified SWE-agent baseline, ARISE improves Function Recall@1 by 17.0 percentage points and Line Recall@1 by 15.0 percentage points for fault localization. These localization gains translate into better patch generation: ARISE achieves 22.0% Pass@1 (66 out of 300 issues resolved), a 4.7 percentage-point improvement over SWE-agent. Controlled ablations confirm that the data-flow graph drives the improvement rather than the tool schema alone, and that large code models can consume structured slice output directly without needing a natural-language summarization layer.

Plain-language abstract AOCI (AI-Oriented Code Indexing) is a system for helping large language models understand large software repositories by building a compact, structured index of the entire codebase before any task begins. Each entry in the index pairs a symbolic tag with a short semantic description covering a file's function, dependencies, and design constraints. The paper describes the indexing protocol, an automated platform that keeps the index in sync as code changes, and an evaluation showing AOCI outperforms retrieval, summarization, and agent-based alternatives on both academic benchmarks and real industrial tasks.

Motivation Large language models frequently fail on codebases with hundreds of thousands of lines because existing approaches — retrieval, natural-language summarization, and agent-based exploration — each construct a different, inconsistent view of the repository at query time. Larger context windows do not solve the problem: prior work has shown that performance degrades as irrelevant code fills the context (a phenomenon called Context Rot), and retrieval methods miss cross-file dependencies, leading to bugs such as using nonexistent fields or adding incorrect tables. The gap is a stable, LLM-readable file-level intent layer that captures each module's role, dependencies, and design constraints in one place.

Methodology The authors define an AOCI index as a structured file with encoding rules followed by per-file entries, each combining a multi-dimensional symbolic tag (carrying architectural coordinates) and a compact semantic description (covering function, relations, and interface). An AOCI Platform automates generation, incremental update, and consistency checking so the index stays aligned with the codebase over time. Evaluation was conducted on four open-source projects across three frontier LLMs and six context conditions (2,160 total evaluations), and on 19 end-to-end development tasks across five real industrial systems built in Go and Node.js. An ablation study isolated contributions of individual encoding components, and an LLM-as-judge scoring pipeline (aligned with human ratings at Spearman rho=0.90) was used for grading open-ended answers.

Results AOCI ranked first in overall accuracy among all deployable methods across the academic benchmark, achieving file-localization accuracy of 97.67%, only 0.66 percentage points below the Oracle upper bound. On the 19 industrial tasks, AOCI produced zero final-state defects, while three mainstream agent-based tools introduced defects in 12 of those tasks and consumed 4 to 130 times more tokens (p < 0.001). The advantage increased with task complexity, particularly on schema-sensitive and cross-module changes where token savings exceeded 100x.

Plain-language abstract This paper presents QVoG, a software analysis platform that automatically scans source code for bugs and security vulnerabilities without running the code. It represents programs as compressed graphs, runs structured queries over those graphs to find known vulnerability patterns, and optionally uses machine learning to catch more general cases. For very large codebases (over one million lines of code), QVoG completes its analysis in about 15 minutes.

Motivation Finding bugs and security vulnerabilities early in development is far cheaper than fixing them later, but existing graph-based static analysis tools struggle with large codebases because the graph structures they build grow enormous and consume too much memory. Additionally, the query languages these tools use are complex and hard to learn, and hand-crafted detection rules tend to be overly specific, causing missed detections or false alarms on similar but slightly different vulnerable patterns.

Methodology QVoG builds a compressed version of the Code Property Graph (CPG) by replacing each statement's full Abstract Syntax Tree with a single statement node that stores AST details as attributes, substantially shrinking graph size and reducing traversal steps. On top of this graph, the authors designed a declarative domain-specific query language to make writing detection rules simpler. They also integrated machine learning models trained on existing vulnerability datasets to generalize detection beyond hard-coded rules. Performance was evaluated by running QVoG, Joern, and CodeQL on the same datasets with semantically equivalent queries, measuring query accuracy (precision and recall on common CWE vulnerability types), execution time, and memory usage.

Results For projects with over one million lines of code, QVoG completed analysis in approximately 15 minutes compared to 19 minutes for CodeQL, demonstrating better scalability. The evaluation covered query accuracy on common CWE vulnerabilities and resource consumption metrics, with QVoG's compressed CPG representation reducing graph complexity and improving overall query efficiency relative to the comparison tools.

Plain-language abstract LibEvolutionEval is a benchmark and study that tests whether AI code-completion models can handle the fact that popular Python libraries change over time. The benchmark covers eight libraries—including PyTorch, Matplotlib, pandas, and scipy—across multiple versions, and evaluates whether models produce correct code for the specific library version a developer is using.

Motivation Code AI models are trained on large snapshots of open-source code, so they may have learned API usage patterns from one library version that are wrong or broken in another. Existing benchmarks focus on local file context and do not capture how rapidly-evolving public libraries affect model accuracy, leaving a gap in understanding how well these tools actually serve developers who work with different library versions simultaneously.

Methodology The authors built a dataset of version-specific code-completion tasks drawn from two sources: real GitHub repositories (where requirements files record the exact library version used) and controlled synthetic examples generated from official API documentation. Eight libraries were covered across their evolution over multiple years, with APIs annotated as introduced, deprecated, modified, or unchanged. Evaluations were run with StarCoder2, Mistral, and GPT-4o-mini using three context settings—in-file context only, version-aware prompting, and version-aware retrieval-augmented generation (RAG) with documentation retrieved using CodeSage embeddings.

Results Library evolution substantially affects model performance across all tested models and libraries, with accuracy fluctuating as APIs change across versions. Providing version-specific documentation via RAG consistently improved code completion accuracy, but did not fully eliminate version-based bias—both language models and embedding models showed persistent performance gaps tied to specific library versions, likely reflecting uneven representation of those versions in training data. Models performed better on indirect API completions (where they infer the API from an object reference) than on direct completions, and newly introduced or deprecated APIs posed the greatest difficulty.

Plain-language abstract AI coding agents spend a large fraction of their tool calls just finding their way around a codebase before they can do any real work. This paper asks whether giving an agent a compact, structured document that declares a project's module boundaries, function signatures, and design constraints can cut that exploration overhead — and if so, what format that document should take.

Motivation When an AI coding agent lacks a map of a codebase it must probe file after file to locate relevant code, wasting tool calls and increasing the chance of error. The paper addresses this navigational overhead and the open question of whether the format of an architecture descriptor (S-expression, JSON, YAML, or Markdown) meaningfully affects agent performance.

Methodology The authors ran three complementary studies using Claude Sonnet 4.6 at temperature 0. First, a controlled experiment with 24 code-localization tasks across 4 conditions tested whether architecture context reduces navigation steps. Second, a 20-question study across 4 descriptor formats measured accuracy, and a separate 15-task artifact-vs-process experiment isolated whether the descriptor file itself provides value independent of developer self-clarification. Third, an observational field study examined 7,012 Claude Code sessions, and a writer-side experiment ran 96 generation runs with 96 error injections across formats on 646,000 lines of production code.

Results Providing any architecture context reduced navigation steps by 33–44% regardless of format (Wilcoxon signed-rank p = 0.009, Cohen's d = 0.92). All four formats achieved identical 95% accuracy. An automatically generated descriptor with no human refinement reached 100% task accuracy versus 80% blind (p = 0.002, d = 1.04), proving the file itself has direct navigational value. Formal declaration correlated with a 52% reduction in agent behavioral variance across 7,012 sessions. The authors propose intent.lisp, an S-expression format, citing its non-atomic failure mode, compression density (22% shorter than JSON, with 5:1–64:1 compression ratios across the production code corpus), and syntactic enforcement of hierarchy as its key advantages over alternatives.

Plain-language abstract This paper introduces KGCompass, a system that helps large language models fix bugs in large software repositories more accurately and cheaply. It builds a knowledge graph linking issue reports, pull requests, and code entities, then uses that graph to guide the model to the right functions before generating a patch.

Motivation Fixing bugs automatically in real-world codebases is hard because bug reports are written in natural language while the code spans thousands of files. Most LLM-based repair systems either miss the buggy location entirely or flood the model with too much context, causing hallucinations and high cost. Only about 32.7% of bug reports in the SWE-bench Lite benchmark explicitly name the file or function containing the bug, so models need a smarter way to navigate the codebase.

Methodology KGCompass constructs a repository-aware knowledge graph that links repository artifacts—issues and pull requests—to code entities such as files, classes, and functions. Given a new bug report, it traverses this graph using multi-hop paths to narrow the search space down to the 20 most relevant candidate functions. A path-guided repair mechanism then feeds the traced entity paths and their context to an LLM, which generates a patch with an explanation. The system was evaluated on the SWE-bench Lite benchmark of 300 Python repository tasks.

Results KGCompass achieves a 45.67% repair resolution rate and 51.33% function-level fault localization accuracy on SWE-bench Lite, state-of-the-art among open-source single-LLM approaches, at a cost of only $0.20 per repair. Among successfully localized bugs, 69.7% required multi-hop traversal through the knowledge graph—paths that pure LLM approaches cannot reliably follow. When combined with stronger models, the graph-guided approach raised resolution rates by 50.8% over a Claude Sonnet baseline and by over 150% over weaker baselines.

Plain-language abstract RepoQA is a benchmark for testing how well large language models understand code in a realistic, long-context setting. The core task asks a model to find a specific function inside a large codebase using only a plain-English description of what that function does — success requires genuine code comprehension, not just pattern matching. The benchmark covers 500 tasks drawn from 50 real GitHub repositories across Python, C++, Java, TypeScript, and Rust, and was used to evaluate 33 general-purpose and code-specialized models.

Motivation Existing long-context benchmarks test models on plain text or synthetic retrieval tasks, but none specifically measured how well models understand code at repository scale. Meanwhile, prior code benchmarks such as RepoBench and CrossCodeEval kept input token counts small, and the widely cited SWE-Bench was too complex for model-only evaluation (GPT-4 achieved only a 1.31% pass rate using retrieval). RepoQA fills this gap by targeting the core code-understanding capability needed in real software development, where a developer must locate a function in thousands of lines of unfamiliar code.

Methodology The authors built an automated dataset curation pipeline over 50 popular GitHub repositories (10 per language) selected for topic diversity, code quality, and at least 100 GitHub stars. For each repository, source files were concatenated in topological dependency order, split into 64 equal chunks, and one unique function per chunk was selected as a candidate "needle." Ten needle functions per repository were chosen at evenly spaced positions, and GPT-4-Turbo was used to write structured four-part natural-language descriptions (purpose, input, output, procedure) without revealing function or variable names. During evaluation, each model received a 16k-token code context with the needle planted at a known depth, and retrieval success was measured using BLEU score against all functions in the context, requiring the output to be closest to the target with a similarity threshold of at least 0.8.

Results Among the 33 evaluated models, the top tier — claude-3-opus, gemini-1.5-pro, and gpt-4o — all achieved around 90.6% average retrieval accuracy. The best open-source models, DeepSeek-V2-Chat and Meta-Llama-3-70B-Instruct, reached 83.4% and 82.2% respectively, slightly ahead of some proprietary models like GPT-4-Turbo (76.4%). Performance varied notably across languages: most models scored highest on Java and TypeScript and lowest on Rust, likely reflecting training corpus size. A counterintuitive finding was that removing code comments improved retrieval accuracy for most models, suggesting LLMs can comprehend code semantics independently of inline documentation; Gemini models were the main exception, as they tended to lose track of the task when comments were replaced with synthetic line-number placeholders.

Plain-language abstract This paper asks whether giving an AI coding agent access to a team's product decisions — things like design choices, user personas, and competitive context — helps it write code that actually follows those decisions. The authors built a benchmark of realistic software tasks and compared a baseline AI agent (with only the codebase) against an augmented agent that could also retrieve recorded product context, finding a large improvement in how often the agent honored team-specific decisions.

Motivation AI coding agents can write working code, but they routinely ignore team-specific product, design, and engineering decisions that are not visible anywhere in the source code. There was no existing controlled benchmark for measuring this 'decision compliance' gap, nor a systematic study of whether providing external product context could close it.

Methodology The authors created a controlled benchmark with 8 realistic software engineering tasks containing 41 weighted decision points. They ran two configurations on identical prompts against the same repository: a baseline using Claude Code with codebase access only, and an augmented configuration that added a product-context retrieval system called Brief, which supplies spec generation, mid-build consultation, recorded decisions, persona pain points, customer signals, and competitive intelligence. All 16 resulting pull requests and the scoring harness were released for independent reproduction.

Results The augmented configuration achieved 95% decision compliance versus 46% for the baseline, a 49 percentage point improvement. Per-decision analysis showed the baseline reached 100% compliance on decisions visible in the codebase but only 0–33% on decisions requiring product context, indicating that product-context retrieval was the key driver of the improvement.

Plain-language abstract This paper investigates a specific failure mode in AI code completion tools that use retrieval-augmented generation (RAG): what happens when the retrieved code snippets are outdated? The authors run controlled experiments showing that stale repository context does not simply add noise — it actively pushes models to produce code that references obsolete function signatures, a distinct and dangerous failure pattern.

Motivation Code assistants routinely retrieve cross-file context from a project's codebase to help complete functions, but the retrieved snippets may come from older versions of the repository. Helper function signatures change over time as arguments are added, removed, or renamed, and a retriever or cache may return an old version. Prior work had not systematically isolated whether such temporally stale context is harmless noise or actively causes models to generate code that matches an obsolete repository state rather than the current one.

Methodology The authors curated a dataset of 17 production helper signature changes from five Python repositories (click, flask, httpx, requests, and rich), pairing each sample's stale parent-commit signature with its current child-commit signature. They tested two language models — Qwen2.5-Coder-7B-Instruct and gpt-4.1-mini — under five controlled retrieval conditions: current context only, stale context only, no retrieval, and two orderings of mixed current-and-stale context. Prompts deliberately omitted any indication of which snippet was current or what the expected signature was, so that implicit self-rescue from explicit anchors could not occur. Static call-pattern oracles classified model outputs as current-state matches, stale references, or fail-no-match.

Results Stale-only retrieval induced stale helper references on 15 of 17 samples for Qwen2.5-Coder-7B-Instruct and 13 of 17 for gpt-4.1-mini, representing increases of 88.2 and 76.5 percentage points over current-only retrieval respectively. No-retrieval produced zero stale references but also only 1 of 17 passing completions, confirming that stale retrieval actively redirects errors toward obsolete state rather than simply removing useful context. The two models shared 75% Jaccard overlap among samples that triggered stale outputs. When both current and stale snippets were provided together, the presence of valid current evidence largely eliminated stale failures, and retrieval rank order had negligible effect.

Plain-language abstract DocSync is an automated system for keeping software documentation accurate as code changes. It uses a combination of code structure analysis and a self-correcting AI loop to detect when documentation has drifted from the actual code behavior and to generate updated, factually grounded documentation drafts.

Motivation Software documentation routinely falls out of sync with code as codebases evolve, creating technical debt that causes API misuse, onboarding friction, and production incidents. Static analysis tools can only check whether documentation tags exist, not whether their content is semantically correct, leaving a critical gap that neither rule-based tools nor standard large language models alone can fill.

Methodology DocSync uses a multi-stage pipeline: an impact analysis step filters out trivial changes, then an Abstract Syntax Tree parser extracts function signatures and dependency graphs, and a Retrieval-Augmented Generation module fetches semantically relevant documentation context. A LoRA-adapted Phi-3 Mini language model generates updated documentation, which a critic module evaluates for consistency with the source code following the Reflexion paradigm; if the critic flags an issue, its natural-language feedback is fed back to the model for a revised draft. The system was evaluated on a proxy code-to-text task derived from the CodeXGLUE benchmark.

Results DocSync substantially outperformed a CodeT5-base baseline: an automated judge scored DocSync at 3.44 out of 5.0 versus 1.91 for the baseline. The Reflexion critic loop produced measurable improvements over the initial draft, raising the judge score from 3.25 to 3.44 and the summary-line exact match from 0.938 to 0.969, with negligible change in BLEU score, demonstrating that iterative semantic refinement adds value without requiring larger models.

Plain-language abstract GALA is a software engineering framework that helps automated bug-fixing tools work with bug reports that include screenshots, not just text. It builds graph representations of both the user interface shown in a screenshot and the code repository, then aligns them hierarchically to pinpoint exactly which files and functions need to be changed before generating a patch.

Motivation Modern bug reports, especially for front-end software, often include GUI screenshots that convey layout and rendering problems impossible to fully describe in words. Existing automated repair systems convert these images to plain text, which discards spatial relationships between UI elements and leads to imprecise, keyword-based localization that retrieves the wrong code files.

Methodology GALA operates in four stages on the SWE-bench Multimodal benchmark. First, a vision-language model guided by the issue description extracts UI elements and their spatial relationships into an Image UI Graph. Second, that graph is cross-referenced with repository-level file-reference structures to select candidate files. Third, function-call graphs within those files are used to ground specific visual elements to individual functions. Fourth, patch generation is conditioned on this hierarchically aligned context.

Results On SWE-bench Multimodal, GALA achieves a resolution rate of 35.40%, outperforming the next-best baselines GUIRepair (34.82%) and OpenHands-Versa (34.43%), and substantially surpassing earlier approaches such as Agentless Lite (25.34%) and SWE-Agent Multimodal (12.19%). At localization, GALA reaches 29.22% file-level recall and 17.14% function-level recall under a 122B-parameter model, exceeding SVRepair and GUIRepair at both levels. Ablation studies confirm that the performance gain comes specifically from cross-modal graph alignment rather than from model scale alone.

Plain-language abstract This paper surveys 860 software developers at Microsoft to find out what AI tools they actually want built next. Rather than studying what AI tools developers currently use, it asks what systems should exist, cataloguing 22 distinct AI-assistant concepts that developers described, along with the conditions they placed on each one.

Motivation Existing AI coding tools focus almost entirely on code generation, yet writing code accounts for only about one-tenth of a developer's working day. The rest — debugging, code review, documentation, onboarding, incident response, and stakeholder communication — receives comparatively little AI support. Meanwhile, accelerating code generation is expanding the slower, more labour-intensive downstream work such as review and verification, creating what the authors call a 'right-shift burden.'

Methodology The study draws on open-ended survey responses from 860 Microsoft developers, using a human-in-the-loop, multi-stage thematic analysis (described in the text as 'council-based thematic' analysis) to surface and characterise the AI systems developers requested. For each of the 22 systems identified, the authors record the problem it addresses, what makes it technically difficult to build, and the explicit behavioural constraints developers placed on it.

Results Developers identified 22 desired AI systems spanning five task categories. The most prevalent single request, cited by 50.1% of development-block respondents, was a scoped pull-request builder for technical debt removal. The second most common (27.8%) was an embedded quality gate that provides diff-anchored bug-spotting and standards enforcement during authorship. Across all 22 systems, a consistent pattern emerged that the authors term 'bounded delegation': developers wanted AI to absorb assembly and coordination work, but explicitly not the core craft decisions. They consistently required authority scoping, provenance tracking, uncertainty signalling, and least-privilege access — stopping points rather than autonomous action.

Plain-language abstract This paper is a survey of how large language models handle code written in many different programming languages. It covers two main tasks: generating new code from a natural-language description across multiple target languages, and translating existing code from one language to another while keeping the same behavior. The authors review the methods researchers use, the benchmarks they test on, and the metrics they use to measure success.

Motivation Most research on AI-assisted coding focuses heavily on popular languages like Python and Java, leaving models much weaker on languages like Rust, OCaml, C++, or Lua. Real-world software systems routinely mix many languages, so a model that works only for Python is not practically useful. The survey addresses this gap by systematically mapping what has been done and what remains unsolved for truly cross-language code intelligence.

Methodology The authors organize the literature around four methodological paradigms: prompt engineering (eliciting multilingual behavior from frozen models via zero-shot or few-shot prompts), model pre-training and fine-tuning (embedding multilingual competence into model weights through large-scale data and synthetic generation), multi-agent collaborative frameworks (decomposing tasks into specialized roles with iterative feedback loops), and retrieval-augmented generation (injecting external language-specific knowledge at inference time). They also catalog benchmarks such as HumanEval-XL, McEval, mHumanEval, XLCoST, and RepoTransBench, and evaluation metrics including Pass@k, Computational Accuracy, and LLM-as-a-Judge approaches.

Results The survey finds that no single paradigm fully solves multilingual code intelligence. Prompt engineering improves surface-level alignment but is bounded by the model's pretraining distribution; fine-tuning embeds knowledge but is constrained by data imbalance; multi-agent frameworks add robustness through decomposition; and RAG supplies missing domain-specific knowledge at inference time. A key finding is that the primary barrier for models is no longer semantic understanding alone, but increasingly the mastery of precise, low-level implementation details in diverse and constrained programming environments, with the critical challenges being monolingual bias, weak cross-paradigm transfer, fragmented evaluation, and limited trustworthiness.

Plain-language abstract This paper is a systematic literature review of how 'breaking changes' — modifications to a software library that cause existing dependent code to stop working — arise, spread, and can be managed across modern software ecosystems. The authors analyzed 97 published studies covering five major ecosystems (Maven/Java, npm/JavaScript, Python, Web APIs, and Linux distributions) to produce the first comprehensive synthesis of this problem area.

Motivation Modern applications depend on large networks of third-party libraries, and a single incompatible change in a widely used package can cascade through the dependency graph and break downstream projects whose authors had no involvement in the change. Despite growing research on the topic, no comprehensive synthesis existed across ecosystems, leaving practitioners without a unified understanding of how breaking changes are classified, why they occur, how they are detected, and how they can be managed.

Methodology The authors followed the systematic literature review guidelines of Kitchenham and Charters, searching three major academic databases (IEEE Xplore, ACM Digital Library, and Springer) with structured keyword queries targeting breaking change terminology. They applied inclusion and exclusion criteria, then performed backward and forward snowballing, arriving at a final corpus of 97 primary studies. Data were extracted to answer four research questions covering types, reasons and impacts, detection approaches, and management strategies for breaking changes.

Results The review produced four main outputs: a four-dimensional taxonomy of breaking changes along the axes of nature, detectability, scope, and visibility; five reason categories and five impact dimensions, with maintenance and design improvements accounting for a larger share of breaking changes than new feature work; 43 detection approaches that achieve high accuracy on syntactic breaks but limited coverage on behavioral ones; and 66 management strategies organized by actor role. Three open challenges were identified — behavioral break detection at scale, the systematic failure of semantic versioning as a trust mechanism, and transitive dependency propagation under information asymmetry — along with three research opportunities involving LLM-assisted behavioral contract inference, ecosystem-level dependency graph intelligence, and domain-specific tooling for machine learning libraries.

Plain-language abstract This paper investigates whether widely-used code generation benchmarks have been contaminated by appearing in the training data of large language models (LLMs), and quantifies how much that contamination inflates measured performance. The authors build a pipeline to find benchmark problems and their solutions inside two major open training corpora, then test whether models score higher on the problems they have seen.

Motivation Benchmark contamination — where test examples leak into a model's training data — is a serious threat to reliable evaluation of LLMs, but prior work focused mostly on natural language tasks. Code is structurally different from prose: two programs can be semantically identical yet look very different on the surface due to variable renaming or whitespace, so standard text-similarity methods miss many contaminated examples. A rigorous, code-aware contamination study was needed.

Methodology The authors developed a two-stage matching pipeline applied to two popular code generation benchmarks, MBPP and HumanEval, against two large open pretraining corpora, the Pile and the Stack. Surface-level similarity was measured using character-level Levenshtein edit distance with a sliding window over the training data. Semantic similarity was measured using Dolos, a plagiarism-detection tool that canonicalizes programs into abstract syntax tree representations and computes k-gram overlap, making it robust to identifier renaming and whitespace differences. They then evaluated three model families — StarCoderBase, Pythia, and CodeGen-NL — comparing accuracy on contaminated versus clean subsets and analyzing the effect of model size and problem difficulty.

Results Substantial overlap was found between both benchmarks and the training corpora. Models performed significantly better on the subset of benchmark problems for which similar solutions appeared in their training data. After removing contaminated examples, the accuracy gap between StarCoderBase-15.5B and Pythia-12B shrank from 23.8% to 13.9%, suggesting that a large share of observed performance differences across models is attributable to contamination rather than genuine capability. Larger models within each family showed stronger memorization as well as better generalization, and the performance boost on seen questions could not be explained away by those questions being easier than unseen ones.