Source-linked AI summary
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
Tianyang Liu, Canwen Xu, Julian McAuley
TL;DR
Existing benchmarks largely focus on single-file completion, leaving repository-level multi-file scenarios under-evaluated. RepoBench addresses this gap with retrieval, completion, and pipeline tasks across Python and Java, and its evaluations emphasize longer, more complex contexts while reporting a limitation in random-retrieval results.
Problem
Existing benchmarks predominantly evaluate single-file tasks, whereas real-world programming commonly involves code spanning multiple files and repositories.
Method
RepoBench evaluates repository-level code auto-completion through retrieval, next-line completion, and an end-to-end pipeline across Python and Java.
Results
Experiments analyze retrieval methods, completion models, and their combined pipeline performance, emphasizing the need to handle longer and more complex contexts.
Takeaways & Limitations
RepoBench provides a multi-task benchmark for comparing repository-level code auto-completion systems in settings closer to real-world programming.
Takeaways & Limitations
Random-retrieval results are indicative rather than conclusive because Codex rate limits prevented multiple runs to mitigate randomness.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have greatly advanced code auto-completion systems, with a potential for substantial productivity enhancements for developers. However, current benchmarks mainly focus on single-file tasks, leaving an assessment gap for more complex, real-world, multi-file programming scenarios. To fill this gap, we introduce RepoBench, a new benchmark specifically designed for evaluating repository-level code auto-completion systems. RepoBench supports both Python and Java and consists of three interconnected evaluation tasks: RepoBench-R (Retrieval), RepoBench-C (Code Completion), and RepoBench-P (Pipeline). Each task respectively measures the system's ability to retrieve the most relevant code snippets from other files as cross-file context, predict the next line of code with cross-file and in-file context, and handle complex tasks that require a combination of both retrieval and next-line prediction. RepoBench aims to facilitate a more complete comparison of performance and encouraging continuous improvement in auto-completion systems. RepoBench is publicly available at https://github.com/Leolty/repobench.
1 Introduction
RepoBench addresses the mismatch between single-file benchmarks and multi-file programming by evaluating repository-level code auto-completion through retrieval, completion, and pipeline tasks.
- Motivation: Existing code auto-completion benchmarks predominantly evaluate single-file tasks, unlike real-world development across multiple files and repositories.This creates a need for evaluation that reflects cross-file programming complexity.
- Benchmark Design: RepoBench evaluates repository-level code auto-completion through three interconnected tasks: retrieval, code completion, and an end-to-end pipeline.The tasks respectively assess context retrieval, next-line prediction, and their combination.
- Benchmark Design: RepoBench-R retrieves relevant code snippets as cross-file context for predicting the next line.
- Benchmark Design: RepoBench-C predicts the next code line from cross-file and in-file context.
- Benchmark Design: RepoBench-P combines retrieval and completion while requiring efficient selection of numerous candidates within processing limits.Its broader context is intended to represent the complete workflow of practical auto-completion systems.
2 Related Work
Prior code-completion research progressed from statistical and recurrent models to transformer-based systems, while benchmarks remained largely centered on single-file contexts despite the importance of cross-file evaluation.
- Code Completion Models: Code completion is an established IDE capability that predicts and automatically completes one or more upcoming code lines.
- Code Completion Models: Earlier code-completion systems used n-gram, recurrent neural network, and probabilistic grammar models.
- Code Completion Models: Transformer-based decoder-only models trained on large code datasets subsequently advanced code completion.Examples include GPT-C, CodeGPT, Codex, CodeGen, and related systems.
- Datasets and Evaluation: Public benchmarks such as CodeXGLUE, PY150, and the Github Java Corpus primarily evaluate single-file contexts.
- Datasets and Evaluation: Repository-level completion offers a more realistic setting for cross-file interactions and evaluating transfer to contexts absent from most pretraining.It also introduces longer prompts, which challenge many Transformer-based models.
3 The RepoBench Dataset
RepoBench combines repository-based data processing with retrieval and next-line completion tasks across Python and Java, including settings that vary cross-file dependence and context length.
- 3.1 Data Sources: RepoBench combines preprocessed sources serving different roles in benchmark construction, including training and evaluation data.
- 3.2 Data Processing: Tree-sitter parses imports to identify cross-file modules, usage lines, and snippets defining imported modules.
- 3.1 Data Sources: 10,345 Python and 14,956 Java historical repositories provide training data, while 1,075 Python and 594 Java newly crawled repositories provide test data.
- 3.3 Task Construction: XF-F masks the first cross-file usage, XF-R masks a later random cross-file usage, and IF masks a non-cross-file line.RepoBench-R excludes IF, whereas RepoBench-C and RepoBench-P include all three settings.
- 3.3 Task Construction: RepoBench-C combines parsed cross-file snippets with import statements and up to 30 preceding in-file lines for next-line prediction.It provides 2k and 8k subsets for different model context limits.
- 3.3 Task Construction: RepoBench-R retrieves one optimal gold snippet among candidate snippets, using easy subsets with 5–9 candidates and hard subsets with 10 or more.Retrieval is evaluated with Accuracy@k.
- 3.3 Task Construction: RepoBench-P first retrieves pertinent snippets and then uses them with in-file context to predict the next line.Its settings use minimum thresholds of 12,000 Python tokens and 24,000 Java tokens, with at least 10 retrieval candidates.
4 Experiments
The experiments evaluate retrieval, code completion, and end-to-end repository-level completion using cross-file and in-file context. Results identify semantic retrieval strengths and show that retrieved-context selection affects pipeline completion.
- RepoBench-R: RepoBench-R retrieves the most relevant snippets for next-line prediction by scoring candidate snippets against cropped in-file code.The retrieval formulation retains m lines of in-file code, computes similarity with each candidate, and returns the top k candidates.
- RepoBench-R: UniXcoder consistently outperforms other retrieval methods, while Jaccard Similarity generally exceeds Edit Similarity.The results attribute this pattern to UniXcoder’s semantic representation and the importance of shared tokens in lexical retrieval.
- RepoBench-R: Python retrieval typically achieves higher accuracy than Java across retrieval methods.The paper associates this difference with Python’s simpler syntax and local proximity between function arguments and calls.
- RepoBench-C: RepoBench-C predicts the next line from autoregressively modeled cross-file and in-file contexts using Codex, CodeGen, and StarCoder baselines.The benchmark includes CodeGen models ranging from 350M to 16B parameters and evaluates Exact Match and Edit Similarity.
- RepoBench-P: In RepoBench-P, retrieved cross-file context improves completion, effective retrieval outperforms random retrieval, and snippet placement influences effectiveness.These findings apply to cross-file and in-file next-line prediction settings; the pipeline first retrieves snippets and then predicts the next line.
5 Conclusion
RepoBench evaluates repository-level code auto-completion through three interrelated tasks for Python and Java. The paper positions the benchmark as an environment for assessing longer, more complex contexts and supporting continued code-intelligence innovation.
- 5 Conclusion: RepoBench comprises RepoBench-R for retrieval, RepoBench-C for code completion, and RepoBench-P for the complete autocompletion pipeline.The three tasks collectively provide a diverse evaluation environment for Python and Java.
- 5 Conclusion: The evaluation emphasizes models capable of handling longer and more complex contexts resembling real-world programming scenarios.The benchmark is designed to address repository-level settings rather than only single-file completion.
- 5 Conclusion: RepoBench aims to contribute to ongoing innovation in code intelligence.
A Ablation Study for Prompt Construction
The prompt-construction ablation combines cross-file context, import statements, and in-file context to support cross-file code completion. Using both import statements and cross-file context gives the strongest overall results, while also improving in-file completion.
- Prompt components: The study varies in-file context, import statements, and cross-file context as prompt components for cross-file completion.In-file context is tested with short and long preceding-line windows; import statements preserve dependencies, and cross-file snippets are prepended.
- Experimental design: Table 5 compares combinations of XFC, IS, and IFC-Short or IFC-Long using Codex across XF-F, XF-R, and IF settings.The All score averages Exact Match and Edit Similarity across the three settings, using 5,000 examples per setting for each language.
- Results: Integrating both IS and XFC produces the best overall cross-file completion results despite possible duplication between the two sources.The chosen prompt strategy combines XFC, IS, and short IFC.
- Results: Adding IS and XFC improves both cross-file and in-file completion, even when the snippets are not specifically aimed at in-file prediction.The improvement therefore extends beyond the completion setting directly targeted by the retrieved context.
- Results: For Python XF-R settings, longer IFC can help when the relevant module may have appeared earlier in the in-file context, while XFC+IS+IFC performs nearly comparably.This result highlights the contributions of extended same-file context, cross-file context, and import statements under different conditions.
B Ablation Study of Kept Lines for Retrieval
The retrieval ablation tests lexical and semantic retrievers while varying how many lines are retained from candidate snippets. Most retrievers perform best with short retained contexts, typically three or five lines.
- Methods: The study compares Jaccard Similarity, Edit Similarity, CodeBERT, and UniXcoder while retaining 3, 5, 10, 20, 30, 60, or 120 lines.These methods cover two lexical and two semantic retrieval approaches.
- Findings: As the number of retained lines increases, most retrievers tend to perform worse, with optimal performance typically at 3 or 5 lines.The analysis examines how retained-context length affects retrieval for code generation.
- Methods: The ablation uses separate tables for Jaccard Similarity, Edit Similarity, CodeBERT, and UniXcoder across different kept-line counts.The tables report retrieval-method performance for the evaluated line-retention settings.
C Experiment Settings
The experiments use model-specific inference and fine-tuning procedures, including quantized models for resource efficiency. Results include comparisons across Python and Java with specified generation settings and sampled training data.
- Inference: Non-Codex and non-fine-tuned models use CTranslate2 and Hugging Face model weights, while Codex is accessed through OpenAI’s API.Codex queries were subject to a rate limit of 20 queries or 40,000 tokens per minute.
- Inference: Limited resources led the experiments to rely on quantized models and fast-inference libraries.The authors warn that quantization or library bugs may cause discrepancies from the original models.
- Inference: Codex generation used temperature 0.2, produced 64 tokens per next-line prediction, and truncated output at the first non-comment line.
- Fine-tuning: Fine-tuning sampled 8,000 XF-F, 4,000 XF-R, and 4,000 IF training examples, plus 200 validation examples.CodeGen models with 350M and 2B parameters were trained on Nvidia A3090 GPUs with DeepSpeed.
D Performance Analysis of StarCoder vs Codex
The performance analysis compares StarCoder and Codex across context lengths using Exact Match and Edit Similarity for Python and Java. StarCoder shows a Python-specific medium-context dip, whereas both models decline with longer Java contexts, warranting cautious interpretation.
- Comparison setup: The analysis compares StarCoder and Codex using Exact Match and Edit Similarity across prompt lengths for Python and Java.The corresponding table uses these metrics for XF-F, XF-R, and IF settings, with All as a weighted average.
- Python: For Python, StarCoder underperforms at medium context lengths with a performance dip not observed for Codex.
- Python: Codex’s consistency across context lengths suggests more robust generalization, potentially related to exposure to varied code-length distributions.The passage presents this attribution as a possibility rather than an established cause.
- Java: For Java, both StarCoder and Codex show declining performance as context length increases.The shared trend may reflect differences in the intrinsic complexity of prompts at different lengths.
- Interpretation: The observed context-length trends should be interpreted cautiously because they may reflect training-data length distributions and prompt complexity.The authors call for further study of these factors across coding tasks and context lengths.
E StarCoder Performance on RepoBench-P
StarCoder is an unreliable base model for assessing retrieval strategies because its performance varies across metrics. It often performs best without retrieved code, suggesting limited repository-level generalizability and sensitivity to context length.
- StarCoder’s inconsistent performance across metrics makes it unreliable for evaluating retrieval strategies.
- StarCoder often performs best with the Baseline retrieval method, which provides no retrieved relevant code.
- The authors speculate that StarCoder struggles to generalize repository-level completion patterns from extensive cross-file snippets.
- Input sequence length appears important: allocating more in-file context does not consistently improve StarCoder’s performance.