PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities

Zezhou Yang, Sirong Chen, Cuiyun Gao, Zhenhao Li, Xing Hu, Kui Liu, Xin Xia

arXiv:2501.13742v1cs.SE

TL;DR

Retrieval-augmented code generation has been used to address the semantic gap between natural-language requirements and code, but its effects and practical configuration have not been systematically explored. The paper evaluates retrieval and fusion choices across pre-trained code models and finds that retrieval augmentation improves performance, with BM25 and cost-effective fusion strategies among the practical options.

  • Problem

    Retrieval-augmented code generation lacks systematic study of its effects and framework usage, including how retrieval and fusion choices influence model performance.

    Source: §1 · p. 2, §2.3 · p. 6

  • Method

    The paper evaluates retrieval-augmented code generation across CodeGen, UniXcoder, and CodeT5, examining retrieval and fusion choices.

    Source: §1 · p. 2, §2.3 · p. 6

  • Results

    Retrieval augmentation improves existing pre-trained code models, and Sketch Filling Fusion improves original CodeT5 by an average of 14.83% in BLEU and 8.05% in CodeBLEU across three datasets.

    Source: RQ2: How do the retrieval techniques… · p. 3, §7 · p. 24

  • Takeaways & Limitations

    BM25 and Sequential Integration Fusion are practical options, while Sketch Filling Fusion can further improve performance at greater computational cost.

    Source: RQ2: How do the retrieval techniques… · p. 3

    The experiments use three datasets whose data remain distinct from real, specific development-environment contexts.

    Source: §6.4 · p. 24

Abstract

from arXiv · show

Code generation aims to automatically generate code snippets of specific programming language according to natural language descriptions. The continuous advancements in deep learning, particularly pre-trained models, have empowered the code generation task to achieve remarkable performance. One main challenge of pre-trained models for code generation is the semantic gap between natural language requirements and source code. To address the issue, prior studies typically adopt a retrieval-augmented framework for the task, where the similar code snippets collected by a retrieval process can be leveraged to help understand the requirements and provide guidance for the generation process. However, there is a lack of systematic study on the application of this framework for code generation, including the impact of the final generated results and the specific usage of the framework. In this paper, we choose three popular pre-trained code models, namely CodeGen, UniXcoder, and CodeT5, to assess the impact of the quality and utilization of retrieved code on the retrieval-augmented framework. Our analysis shows that the retrieval-augmented framework is beneficial for improving the performance of the existing pre-trained models. We also provide suggestions on the utilization of the retrieval-augmented code generation framework: BM25 and Sequential Integration Fusion are recommended due to their convenience and superior performance. Sketch Filling Fusion, which extracts a sketch of relevant code, could help the model improve its performance further. Additionally, we conduct experiments to investigate the influence of the retrieval-augmented framework on large language models for code generation, showing the effectiveness of the framework, and we discuss the trade-off between performance improvement and computational costs in each phase within the framework.

1 INTRODUCTION

Pre-trained code models have advanced code generation, but retrieval-augmented approaches lack systematic evaluation of how retrieval and fusion choices affect performance. The paper evaluates the framework across three models and datasets, finding performance gains, including substantial average improvements on HearthStone.

1 INTRODUCTION

  • The study addresses the limited exploration of how retrieval techniques and fusion strategies affect retrieval-augmented code generation.The authors identify noisy retrieved snippets as a possible source of reduced model performance.Source: §1 · p. 2
  • On HearthStone, retrieval augmentation yields average improvements of 41.60% in EM, 9.01% in BLEU, and 8.69% in CodeBLEU across three models.The study evaluates CodeGen, UniXcoder, and CodeT5 on CONCODE, CoNaLa, and HearthStone.Source: §1 · p. 2
  • The authors evaluate whether retrieval augmentation benefits different pre-trained code models and examine its performance and generalization.They use BM25 retrieval and concatenate retrieved snippets with the natural-language description as augmented input.Source: §1 · p. 2

RQ2: How do the retrieval techniques affect retrieval-augmented framework for code

The results indicate that retrieval and fusion choices shape retrieval-augmented code generation. BM25 performs strongly across datasets, while Sketch Filling Fusion can improve CodeT5 further but is resource-intensive; Sequential Integration Fusion offers a more cost-effective balance.

RQ2: How do the retrieval techniques affect retrieval-augmented framework for code

  • BM25 gives all three models their highest improvements on CONCODE and HearthStone; on CoNaLa, it is optimal for CodeT5 and suboptimal for CodeGen.These comparisons concern the retrieval techniques evaluated in the framework.Source: RQ2: How do the retrieval techniques… · p. 3
  • For original CodeT5, Sketch Filling Fusion averages 14.83% BLEU and 8.05% CodeBLEU improvement across three datasets, but its training is resource-intensive.Sketch Filling Fusion constructs sketches of relevant code; Sequential Integration Fusion is more cost-effective when balancing cost and performance.Source: RQ2: How do the retrieval techniques… · p. 3
  • Sequential Integration Fusion with retrieved results is described as a simple, effective fusion strategy.The study weighs its performance against the computational cost of more resource-intensive fusion.Source: RQ2: How do the retrieval techniques… · p. 3
  • The paper presents an empirical study of retrieval-augmented code generation and examines how retrieval techniques and fusion strategies affect performance.Its stated contributions include actionable insights into framework usage.Source: RQ2: How do the retrieval techniques… · p. 3

2 BACKGROUND AND RELATED WORK

The background covers pre-trained code models and code-generation research, then introduces retrieval-augmented generation and the need to study its phases systematically. It also situates the study among evolving datasets and large language models.

2.1 Pre-trained Models in Code Intelligence

  • Pre-trained code models use encoder-only, decoder-only, or encoder-decoder architectures, with UniXcoder and CodeT5 supporting multiple code-related tasks.UniXcoder combines code-comment semantics and AST information; CodeT5 uses token-type information from code.Source: §2.1 · p. 4

2.2 Code Generation

  • Code-generation research spans grammar- and AST-based methods, pre-training followed by fine-tuning, retrieval augmentation, and generation post-processing.The background also describes benchmarks ranging from semantic-parsing datasets to real-world-oriented and test-case datasets, alongside growing interest in LLMs for code generation.Source: §2.2 · pp. 4–5

2.3 Retrieval-Augmented Generation

  • Retrieval-augmented generation supplies retrieved results to improve generation, and the paper studies retrieval, fusion, and generation as distinct framework phases.Prior work often uses a single configuration, leaving retrieval techniques and fusion strategies without systematic exploration; the paper also examines the framework's impact on LLM code generation.Source: §2.3 · pp. 5–6

3 RETRIEVAL-AUGMENTED FRAMEWORK

The framework retrieves code related to a natural-language request, fuses it with the input, and generates code from the augmented input. It supports text-retrieval and code-search methods, alongside four fusion strategies.

3.1 Overview

  • Retrieval, fusion, and generation respectively find relevant snippets, integrate them with the natural-language description, and generate code from the augmented input.The model architecture remains unchanged across fusion strategies.Source: §3.1 · pp. 6–7

3.2 Retrieval Phase

  • The framework retrieves code either by matching natural-language descriptions in a database or by searching code directly, using two text retrievers and three code-search models.The evaluated retrievers are BM25 and RetroMAE; the code-search models are CodeBERT, UniXcoder, and CoCoSoDa.Source: §3.2 · p. 7, §3.2.1 · p. 7

3.2.1 Text Retrieval Algorithms.

  • BM25 scores database documents by summing query-token weights times token–document relevance, while RetroMAE is used directly for inference without labeled retrieval data.The database documents are natural-language descriptions paired with code snippets; the weight and relevance terms reflect token importance and token–document correlation.Source: §3.2.1 · pp. 7–8

3.2.2 Code Search Models.

  • CodeBERT, UniXcoder, and CoCoSoDa retrieve snippets by measuring relevance between natural-language inputs and code, with all three trained on each dataset’s training set.CodeBERT jointly encodes the input and code; UniXcoder encodes them separately, and CoCoSoDa compares their vector representations using cosine similarity.Source: §3.2.2 · pp. 8–9

3.3 Fusion Phase

  • The four fusion strategies differ in how they use retrieved snippets: SIF appends them to the input, SEF creates training instances, VDF combines vectors, and SFF supplies code sketches.VDF encodes expanded instances into vectors; SFF extracts a sketch intended to retain useful structure while filtering potentially irrelevant details.Source: §3.3 · p. 9

3.4 Generation Phase

  • During generation, SIF conditions on appended retrieved code, SEF uses an augmented training set, VDF feeds concatenated snippet vectors to the decoder, and SFF asks the model to edit a retrieved-code sketch.In VDF, the experiment uses the CodeT5 encoder and decoder; in SFF, the sketch acts as a template for the generated code.Source: §3.4 · pp. 10–11

4 EXPERIMENT STUDY SETUP

The experiments use three code-generation datasets, three pretrained code models, and multiple complementary evaluation metrics. The setup compares retrieval-augmented configurations with specified model, dataset, and implementation choices.

4.1 Dataset

  • The study evaluates on CONCODE, CoNaLa, and HearthStone, which respectively contain Java project examples, Stack Overflow question–solution pairs, and structured HearthStone card descriptions with Python code.CONCODE uses repository-based partitions; CoNaLa has 2,379 training and 500 test examples, with 200 training examples selected for validation.Source: §4.1 · pp. 11–12

4.2 Pre-trained Code Models

  • The generation models are CodeGen, UniXcoder, and CodeT5; CodeGen is decoder-only, while UniXcoder and CodeT5 use encoder-decoder architectures.The experiments use CodeGen-MONO, selected for its superior performance among the described CodeGen versions.Source: §4.2 · p. 12

4.3 Metrics

  • Evaluation combines exact-match accuracy, BLEU-4, edit distance, and CodeBLEU, which incorporates n-gram, AST-syntax, and data-flow similarity.Exact match measures identical predictions, while edit distance counts token edits; CodeBLEU combines BLEU-based scores with AST and data-flow similarity.Source: §4.3 · pp. 12–13

4.4 Implementation Details

  • The experiments retain the pretrained models’ original hyperparameters and use BM25 with Sequential Integration Fusion for RQ1, then vary retrieval methods and fusion strategies in later questions.For RQ2, five retrieval techniques are paired with SIF; for RQ3, the number of retrieved snippets is controlled before comparing four fusion strategies.Source: §4.4 · p. 14

5 EMPIRICAL STUDY RESULTS

The experiments find that retrieval augmentation improves code-generation performance across pretrained models and datasets. BM25 and Sequential Integration Fusion are recommended, while the best number of retrieved snippets and fusion strategy depend on dataset characteristics.

5.1 Research Questions

  • The study asks whether RAF improves pretrained code models, how retrieval techniques affect performance, and how retrieved results should be used.It evaluates CodeGen, UniXcoder, and CodeT5 across CONCODE, CoNaLa, and HearthStone, and uses CodeT5 for the retrieval-use experiments.Source: §5.1 · p. 14, §5.2 · p. 14, §5.4 · p. 17

5.2 RQ1: Effectiveness of Retrieval-Augmented Framework

  • RAF improves performance across pretrained code models and datasets; models with RAF outperform original models at significance level 0.05 (p-value 0.035).The reported improvements span multiple metrics, and the authors describe the results as generalizing across models and datasets.Source: §5.2 · pp. 14–15

5.3 RQ2: Impact of Retrieval Techniques for Retrieval-Augmented Framework

  • On CONCODE and HearthStone, all three models gain most with BM25; on CoNaLa, CodeT5 gains 25.69% in BLEU and 19.63% in CodeBLEU with BM25.CoCoSoDa can also maximize gains for CodeGen and UniXcoder on CoNaLa.Source: §5.3 · p. 15, Table 4 · p. 16
  • RetroMAE can reduce BLEU on CONCODE, while CodeBERT and UniXcoder require downstream fine-tuning and do not substantially improve over BM25 and CoCoSoDa.RetroMAE contributes the smallest improvement among the five techniques and is not always beneficial.Source: §5.3 · p. 16

5.4.1 The Impact of the Number of Utilized Retrieved Results.

  • With CodeT5 and Sequential Integration Fusion, more snippets do not always improve results; BLEU inflects between three and five on CONCODE and first improves then declines on CoNaLa.The experiments concatenate 1, 3, 5, 7, or 10 snippets; HearthStone metrics remain unchanged at five, with longer snippets causing truncation beyond five examples.Source: §5.4.1 · pp. 17–18, Figure 2 · p. 17

5.4.2 Comparison of Various Fusion Strategy.

  • SIF and SFF perform best overall; SIF fares better on CoNaLa's short, structurally inconsistent data, while SFF benefits from HearthStone's analogous code structure.SEF is second only to SFF on HearthStone and closely rivals SFF on the other datasets; SFF can further improve on SEF, while VDF performs worse than SEF.Source: Table 5 · p. 18, §5.4.2 · p. 18
  • SIF is the most time-efficient to train; SEF and SFF take 2 to 7 times longer, while SEF has the shortest inference time and SIF is recommended overall.VDF requires longer training than SIF and is the exception to the otherwise comparable inference times.Source: Table 5 · p. 18, §5.4.2 · p. 19

6.1 Implications of Findings

The retrieval-augmented framework improves results across existing pre-trained code models and LLMs, while retrieval and fusion choices shape performance and cost. The authors recommend BM25 and Sequential Integration Fusion (SIF), with Sketch Filling Fusion (SFF) offering potential gains at higher training cost.

6.1 Implications of Findings

  • The RAF improves performance across existing pre-trained code models without changing model architecture or size.The authors describe the framework as generalizable across models with different architectures and say stronger retrieval techniques and fusion strategies can further enhance a model.Source: §6.1 · p. 19, §6.1.1 · p. 20

6.1.1 Implications on the Effectiveness of the Retrieval-Augmented Framework.

  • All LLMs improve across all three datasets with the RAF; ChatGLM’s BLEU metric ratio reaches 198.67 on Hearthstone with BM25 versus original ChatGLM.On CONCODE, the BLEU metric ratio is 98.33 for ChatGLM prompted with CoCoSoDa-retrieved code versus ChatGLM directly.Source: §6.1.1 · p. 20
  • BM25 and CoCoSoDa perform best among five retrieval techniques, but BM25’s cost advantage diminishes on CONCODE and one retrieval takes about 4 seconds.The authors recommend BM25 for short-input or small datasets and suggest more efficient code search models for longer inputs or outputs.Source: §6.1.1 · p. 20, §6.1.2 · p. 21

6.1.2 Implications on the Utilization of Retrieval-Augmented Framework.

  • SIF is the recommended fusion strategy; SFF may improve performance on clearly structured datasets but requires 2–7 times the training time of direct fine-tuning.SFF’s fusion process requires training the model for k times the sample size.Source: §6.1.2 · p. 21
  • Ascending similarity order outperforms descending order when concatenating retrieved code, and the number of snippets should be chosen for model capacity and dataset attributes.The authors recommend a median number based on the data length in different datasets.Source: §6.1.2 · p. 21, Table 8 · p. 21

6.2 Case Study

The case studies illustrate how retrieved code can guide generation and how retrieval techniques return different candidate snippets. Their examples show both successful retrieval and cases where important requirements are missed.

6.2 Case Study

  • On CONCODE, BM25 retrieves code similar to the ground truth, and retrieval-augmented CodeT5 produces the same code as the ground truth where original CodeT5 does not follow the intended specification.The retrieved example serves as a reference for interpreting the request for singleton logout-request-sender instance methods.Source: §6.2 · p. 22

6.2.2 Analysis on the retrieved results by different retrieval techniques.

  • Different retrieval techniques identify different similar snippets for the subsequent fusion and generation phases.BM25 and RetroMAE use natural-language descriptions for retrieval, whereas the code search models compare natural language with code snippets directly.Source: §6.2.2 · pp. 22–23
  • For a request to obtain a filename without its extension, CoCoSoDa and CodeBERT retrieve suitable code, while RetroMAE misses the condition and UniXcoder returns an intermediate result.Text retrieval methods match natural-language descriptions to descriptions in the codebase; code search models compare the input directly with code snippets.Source: §6.2.2 · pp. 22–23

6.3 Future Work.

The authors identify active retrieval and stronger retrieval databases as future directions for retrieval-augmented code generation. They note that current experiments use training sets as retrieval databases, while evaluation settings may exclude those sets.

6.3 Future Work.

  • Irrelevant retrieved code may act as noise and reduce generation performance, motivating investigation of whether retrieval should be performed for a given input.The authors call this direction active retrieval and note that retrieval adds costs compared with using the generation model directly.Source: §6.3.1 · p. 23

6.3.1 Active Retrieval.

  • Because evaluation datasets such as HumanEval and MBPP may exclude training sets, retrieval-augmented generation needs comprehensive, diverse, and informative retrieval databases.The authors also identify fine-tuning deep learning-based retrieval models on code generation datasets as a worthwhile direction.Source: §6.3.2 · p. 23

6.4 Threats to Validity

The authors qualify their findings by noting limited model coverage, sensitivity to devices and parameter settings, and a gap between benchmark datasets and real development contexts.

6.4 Threats to Validity

  • The experiments cover three pre-trained models, leaving uncertainty about whether the findings apply to larger models or models with different architectures.The authors characterize their experimental exploration as constrained.Source: §6.4 · p. 23
  • Deep-learning code search and generation models can be affected by device and parameter settings, so the authors provide datasets and code to support replication.They say using the retrieval results, code, and parameter settings from previous work ensures consistency with their results only to some extent.Source: §6.4 · pp. 23–24
  • The evaluation uses three datasets whose content remains distant from specific development contexts, limiting how well the experiments represent real-world development scenarios.The authors note that CONCODE preprocessing makes its code difficult for humans to understand intuitively and its ground-truth code difficult to reproduce from natural-language descriptions.Source: §6.4 · p. 24

7 CONCLUSION

The study finds that retrieval augmentation improves existing pretrained code models and examines how retrieval techniques and fusion strategies affect performance.

7 CONCLUSION

  • Retrieval augmentation improves code-generation performance for existing pretrained models, including CodeGen, UniXcoder, and CodeT5.The study examines effectiveness and generalization on three different datasets.Source: §7 · p. 24
  • The study explores how different retrieval techniques and fusion strategies affect retrieval-augmented code generation.The authors summarize findings and offer implications for using the framework.Source: §7 · p. 24

Related papers

Loading 2501.13742v1…