PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeRAG-Bench: Can Retrieval Augment Code Generation?

Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, Daniel Fried

arXiv:2406.14497v2cs.SEcs.CL

TL;DR

Code generation with language models can be difficult when parametric knowledge alone is insufficient, while retrieval’s potential across coding tasks and sources remains under-explored. CodeRAG-Bench evaluates retrieval-augmented code generation using diverse tasks and document sources. The experiments report gains from high-quality contexts, while identifying limits in retrieval and context integration.

  • Problem

    How retrieval benefits diverse code-generation tasks remains under-explored, despite the need for external context when parametric knowledge alone is insufficient.

    Source: §1 · p. 1

  • Method

    The authors build CodeRAG-Bench with diverse coding tasks, canonical-document annotations, and five retrieval sources, then evaluate retrieved contexts in code generation.

    Source: §1 · pp. 1–2

  • Results

    Retrieving external documents can greatly benefit code generation, although current retrievers struggle to find useful documents and generators have limited context capacity and RAG abilities.

    Source: §6 · p. 8

  • Takeaways & Limitations

    CodeRAG-Bench provides a testbed for advancing retrieval-augmented code-generation research.

    Source: §6 · p. 8

    The initial exploration could be extended in task and language diversity, model choice, and methodology; the tested methods may not represent all model behaviors.

    Source: Limitations · p. 9

Abstract

from arXiv · show

While language models (LMs) have proven remarkably adept at generating code, many programs are challenging for LMs to generate using their parametric knowledge alone. Providing external contexts such as library documentation can facilitate generating accurate and functional code. Despite the success of retrieval-augmented generation (RAG) in various text-oriented tasks, its potential for improving code generation remains under-explored. In this work, we conduct a systematic, large-scale analysis by asking: in what scenarios can retrieval benefit code generation models? and what challenges remain? We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks, including basic programming, open-domain, and repository-level problems. We aggregate documents from five sources for models to retrieve contexts: competition solutions, online tutorials, library documentation, StackOverflow posts, and GitHub repositories. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources. While notable gains are made in final code generation by retrieving high-quality contexts across various settings, our analysis reveals room for improvement -- current retrievers still struggle to fetch useful contexts especially with limited lexical overlap, and generators fail to improve with limited context lengths or abilities to integrate additional contexts. We hope CodeRAG-Bench serves as an effective testbed to encourage further development of advanced code-oriented RAG methods.

1 Introduction

CodeRAG-Bench addresses the under-explored use of retrieval for diverse coding tasks by evaluating retrieval-augmented code generation across tasks and document sources. The experiments show benefits from useful retrieved contexts, alongside limitations in retrieval quality and context use.

1 Introduction

  • CodeRAG-Bench covers six programming tasks across four categories and pairs them with manually annotated canonical documents and five retrieval-source collections.The benchmark contains 9k coding tasks and 25 million retrieval documents.Source: §1 · p. 1
  • GPT-4o gains 27.4% on SWE-Bench and 6.9% on the harder ODEX subset when given canonical documents.In full retrieval-augmented settings, some models even surpass performance with gold documents, while current retrievers struggle especially on open-domain and repository-level tasks.Source: §1 · p. 1
  • Generation models with limited context windows show smaller improvements, leaving room for further advances.The benchmark’s scope spans basic programming, open-domain coding, repository-level problems, and code retrieval.Source: §1 · pp. 1–2
  • Open retrieval from StackOverflow or tutorials can improve generation, but gains on repository-level tasks are limited.Chunking documents to 200–800 tokens often gives the best results.Source: §1 · p. 2

2 The CODERAG-BENCH

CodeRAG-Bench combines diverse coding tasks, retrieval sources, canonical-document annotations, and a unified evaluation pipeline. Its design supports evaluating retrieval quality and execution-based code generation across canonical and open retrieval settings.

2 The CODERAG-BENCH

  • The benchmark is designed around diverse tasks, rigorous retrieval and execution-based evaluation, and a unified interface for retrieval, augmented generation, and evaluation.Its stated aim is to address the varied pipelines of existing datasets.Source: §2 · p. 2

2.1 Programming Problems

  • The benchmark includes code retrieval, basic programming, open-domain, and repository-level problems, drawing on multiple datasets for each category.It focuses on Python tasks; LiveCodeBench problems collected after the considered models’ training cutoff help decrease contamination risk, while RepoEval’s function split supports execution-based evaluation.Source: §2.1 · pp. 2–3, Table 1 · p. 3
  • The API and line splits are evaluated with lexical measures that have been shown ineffective for signifying functional correctness.This qualifies how retrieval quality is evaluated on those two CodeSearchNet splits.Source: §2.2 · p. 3

2.2 Retrieval Sources

  • Five retrieval sources include programming solutions, online tutorials, Python library documentation, StackOverflow posts, and GitHub repository files.Canonical retrieval uses only the canonical datastore, whereas open retrieval can use any datastore.Source: §2.2 · p. 3, Table 2 · p. 3

2.3 Canonical Document Annotation

  • Canonical documents provide the context needed to solve each problem and are annotated from the corresponding retrieval pool.The annotations use programming solutions for HumanEval and MBPP, library documentation for DS-1000 and ODEX, and canonical code for RepoEval and SWE-bench; LiveCodeBench has none.Source: §2.3 · p. 4

2.4 Evaluation Metrics

  • Retrieval is evaluated with NDCG, Precision, and Recall, using NDCG@10 as the primary metric; code generation uses pass@k for execution correctness.Final RAG performance is evaluated in canonical and open retrieval setups.Source: §2.4 · p. 4

3 Canonical RACG

The canonical and end-to-end evaluations show that retrieval can improve code generation, but its effectiveness depends on the task, retriever, and generator. The section also compares retrieval systems and evaluates their efficiency.

3. Canonical RACG

  • The evaluation covers 10 retrieval and 10 generation models, reporting retrieval, direct generation, and end-to-end RACG results on canonical sources.The setup includes sparse, dense, and proprietary retrievers, and code-specific and general text LMs; top-five documents are prepended to problems without other contexts such as few-shot examples.Source: §3 · p. 4, §3.1 · p. 4

3.2 Retrieval Results

  • Jina-v2-code exceeds GIST-base and BGE-base by 7.4 and 6.6 average NDCG@10, respectively; dense models frequently surpass BM25.SFR-Mistral retrieval is stronger than GIST-base but requires over 5× its index storage and adds nearly 100× document-encoding latency.Source: Table 3 · p. 5, §3.2 · p. 5, Table 4 · p. 5

3.3 Generation with Canonical Documents

  • Canonical contexts improve most setups: on open-domain tasks, most code-specific LMs gain up to 5.2 points, while GPTs show no gains with retrieval; on ODEX-hard, documents retrieved with most methods improve results by 20.3–40.1%.All models gain 7.5–17.2 points with canonical snippets on RepoEval, while only GPT-3.5-turbo and GPT-4o achieve non-trivial SWE-bench Lite results.Source: §3.3 · pp. 5–6

3.4 Retrieval-Augmented Code Generation

  • On MBPP, RACG beats the canonical setup by 15.6–17.8; StarCoder2 usually benefits on basic problems, while DeepSeekCoder does not and GPT-3.5-turbo improves.DeepSeekCoder’s lack of improvement is attributed to over-complicated, repetitive generations with added contexts.Source: §3.4 · p. 6
  • StarCoder2 benefits from retrieved library documentation across open-domain datasets, while DeepSeekCoder and GPT-3.5 mainly gain on ODEX-hard, where the best RACG matches canonical results despite low NDCG@10.This pattern aligns with canonical-document results indicating particular effectiveness on less popular libraries.Source: §3.4 · p. 6
  • On SWE-Bench, Retrieval-then-Rerank with GPT-4o improves 21 points over no retrieval but remains 9 points below gold; the best retriever varies by task and LM.The authors call for holistic evaluation because top retrieval performance does not always yield the best RACG outcome.Source: §3.4 · p. 6, Table 6 · p. 7

4 RACG with Open Retrieval

Open retrieval results vary by source and task: some external sources help on HumanEval and ODEX, whereas repository-level tasks favor local code. Chunking strategy also affects performance.

Open retrieval

  • On HumanEval, StackOverflow posts improve results by 1.8–4.3, while tutorials improve results by 2.1 only with GIST; mixed-source retrieval performs as well as gold documents.The retrieved posts and tutorials often concern the same programming problem and may hint at or disclose its answer.Source: §4 · p. 7
  • On ODEX, programming solutions bring 3.8–4.3 gains, and GPT-4o and StarCoder with OpenAI retrieval from solutions outperform their documentation-retrieval variants by 3.2 and 1.6 points, respectively.GIST-large is particularly good at retrieving tutorials, while BM25 and OpenAI embeddings find higher-quality programming solutions.Source: §4 · pp. 7–8
  • On RepoEval, open sources are less useful than local repository snippets; combining local and open contexts beats no retrieval but is only comparable with local context alone.The authors describe understanding local code contexts as crucial and irreplaceable by external resources.Source: Table 8 · p. 8, §4 · p. 8

Exploring optimal chunking strategies

  • Pre-retrieval chunking achieves the highest scores on almost all sources; reranking within the 200–800-token range greatly degrades results.First-N-token chunks from 200 to 1,500 tokens perform best at 800 tokens for most sources, except StackOverflow posts; the study compares first-N, reranked post-retrieval, and pre-retrieval chunks.Source: Table 9 · p. 8, §4 · p. 8, Figure 2 · p. 8

5 Related Work

Related work frames code generation as relying largely on natural-language queries and model parameters, while prior RAG research is mostly text-centric. The paper positions its benchmark as a way to study retrieval-augmented code generation across tasks and sources.

  • Most language models generate code from natural-language queries and parametric knowledge without external programming sources such as tutorials.Source: §5 · p. 8
  • RAG has been widely used in knowledge-intensive, mostly text-centric tasks with general-domain corpora (Lewis et al., 2020; Asai et al., 2024).Source: §5 · p. 8
  • Prior work used programming context from repositories or documentation, but the paper identifies a lack of study across varied coding tasks and knowledge sources (Ding et al., 2023; Yang et al., 2024; Zhou et al., 2023).The authors curate a unified benchmark and release its RACG codebase to support systematic study, following the role unified benchmarks played for text-centric tasks.Source: §5 · p. 8

6 Conclusion

The paper presents CODERAG-BENCH as a benchmark for retrieval-augmented code generation across coding tasks and retrieval sources. Its experiments find that external documents can help code generation, while retrieval and generation limitations leave results suboptimal.

  • CODERAG-BENCH is a benchmark for retrieval-augmented code generation with varied coding tasks and retrieval sources.Source: §6 · p. 8
  • Experiments show that retrieved external documents can greatly benefit code generation, but retrievers struggle to find useful documents and generators have limited context capacity and RAG abilities.The authors propose the benchmark as a testbed for future work on code-oriented RAG methods.Source: §6 · p. 8

Limitations

The paper frames CODERAG-BENCH as an initial exploration of retrieval-augmented code generation, with room to broaden its tasks, languages, models, and methods. The appendices illustrate canonical documents and helpful or distracting retrieval contexts, and describe retrieval-efficiency measurements.

Limitations

  • The benchmark’s initial exploration could be extended in task and language diversity, as well as through model and methodological improvements.The authors also note that code debugging remains under-explored, the benchmark focuses on Python, and extrapolating to other languages may bring additional challenges.Source: Limitations · p. 9

A.1 Example with Canonical Documents

  • The paper illustrates canonical-document examples from HumanEval, MBPP, DS-1000, and ODEX.The examples show canonical solutions for general programming and canonical library documentation for open-domain coding problems.Source: §A.1 · p. 12

A.2 RACG with Helpful and Distracting Documents

  • RACG examples demonstrate both gains from relevant retrieved documents and degradation from irrelevant retrieved documents.Source: §A.2 · p. 12

B Additional Details about Retrieval Efficiency

  • Retrieval efficiency is measured as query-embedding time and similarity-calculation time, with runtime for API models varying by organization, rate limits, and batch size.The experiment matches maximum context length to each original model, notably increasing SFR Mixtral encoding latency because of its longer context window.Source: B Additional Details about Retrieval… · p. 12
  • The evaluation used one A100 GPU for open-access models, with batch sizes of 64 for GIST base and large and 8 for SFR-Mistral.Source: B Additional Details about Retrieval… · p. 12

C Result Reproduction

The reproduction section uses shared zero-shot prompts to compare models and explains deviations from reported results through differences in prompts, datasets, and decoding. Reproduced scores vary from the original reports in several cases.

C Result Reproduction

  • The evaluation uses the same zero-shot prompt across models and datasets, with only the original problem description and no additional instructions or in-context examples.This choice is intended to avoid unknown effects from varying instructions or examples.Source: C Result Reproduction · p. 12
  • 31.7 versus 35.4 on HumanEval: the reproduced StarCoder2 result is lower, possibly because the original used test cases in its prompt.Source: C Result Reproduction · p. 12
  • MBPP reproduction differs in dataset scope: the original StarCoder2 evaluation used 399 of 427 examples with added test cases, while this evaluation used the full dataset.Source: C Result Reproduction · p. 12
  • DS-1000 results can vary because the original sampled 40 generations for pass@1, whereas this evaluation generated one program with greedy decoding.Source: C Result Reproduction · p. 12
  • CodeGemma reproduction was 3–5 points lower in pass@1, while CodeLlama’s MBPP result was 4 points lower; the authors cite prompting differences as possible explanations.CodeGemma’s report omits prompt details, while CodeLlama used 3-shot prompting compared with this evaluation’s zero-shot setting.Source: C Result Reproduction · p. 12
  • Llama3 reproduction results were about 4 points lower than the official blog’s HumanEval results, whose prompting and inference details were unspecified.Source: C Result Reproduction · p. 13

D Analysis on Open-Domain Coding Problems

The analysis tests whether library documentation helps strong models on examples requiring less common libraries. On this hard-library ODEX subset, retrieved documentation improves results across most retrieval methods.

D Analysis on Open-Domain Coding Problems

  • The analysis investigates why documentation brings limited benefits on the full ODEX set, especially for strong proprietary models such as GPT and Gemini.The authors hypothesize that these models are familiar with most common libraries and test that hypothesis on less common libraries.Source: D Analysis on Open-Domain Coding… · p. 13
  • The subset selects ODEX examples using the 20 least common libraries, including sqlite3, ftplib, and flask, based on the real-world library distribution.Source: D Analysis on Open-Domain Coding… · p. 13
  • 20.3–40.1%: augmenting with documentation retrieved by most methods improves results on the least-common-library ODEX subset.The authors say this split more clearly demonstrates the effectiveness of library-documentation augmentation than the full ODEX set.Source: Table 10 · p. 13, D Analysis on Open-Domain Coding… · p. 13

E How Many Documents to Augment?

The study compares RACG performance using different numbers of retrieved documents on HumanEval, ODEX, and RepoEval. Five documents produce the best results in most settings, though StarCoder2 on RepoEval is an exception.

E How Many Documents to Augment?

  • The study compares top-1, top-2, top-5, and top-10 document contexts on HumanEval, ODEX, and RepoEval.These datasets represent basic programming, open-domain coding, and repository-level tasks, respectively.Source: E How Many Documents to Augment? · pp. 13–14
  • Five retrieved documents yield the best results in most settings, except StarCoder2 on RepoEval, which performs best with eight.The five-document sweet spot holds despite the different context limits of StarCoder2 (16k) and DeepseekCoder (4k); adding lower-ranked documents may introduce noise and reduce generation quality.Source: E How Many Documents to Augment? · p. 14

F Does RACG Help Stronger Models?

After open retrieval improved StarCoder2, stronger models showed model- and task-dependent results: some benefited on HumanEval, ODEX gains were limited, and Claude models struggled on RepoEval.

F Does RACG Help Stronger Models?

  • RACG's benefits varied by model and task: stronger models improved on HumanEval, ODEX gains were limited, and Claude models struggled on RepoEval.Gemini-1.5-pro, unlike its weaker counterpart, improved substantially on RepoEval, while GPT-4o solved the task at a reasonable success rate.Source: F Does RACG Help Stronger Models? · pp. 14–15

Failure analysis

HumanEval

  • GPT-4 and Claude-3-sonnet consistently improved on HumanEval using all document sources, while weaker models needed multiple sources and Gemini-1.5-pro struggled with non-canonical sources.Claude-3-sonnet, despite performing worse than Claude-3-haiku without retrieval, benefited from all sources and surpassed haiku with canonical programming documents.Source: F Does RACG Help Stronger Models? · p. 14, Table 11 · p. 16

ODEX

RepoEval

  • On RepoEval, all Claude models scored below 10% pass@1 in most scenarios, while Gemini-1.5-pro gained about 10–25 points over its weaker variant.Claude models often explained the incomplete code instead of completing it; Gemini-1.5-flash also barely solved the task and frequently generated explanations.Source: F Does RACG Help Stronger Models? · p. 15, Table 13 · p. 16
Loading 2406.14497v2…