PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Test Generation: How Far Are We?

Jiho Shin, Nima Shiri Harzevili, Reem Aleithan, Hadi Hemmati, Song Wang

arXiv:2409.12682v2cs.SEcs.AI

TL;DR

RAG-based unit-test generation for ML/DL APIs remains underexplored, including the effects of different knowledge sources. This study evaluates retrieval approaches using API documentation, GitHub issues, and StackOverflow Q&As, finding higher line coverage with RAG and particularly strong results from GitHub issues.

  • Problem

    RAG has advanced other software-engineering tasks, but its efficacy in unit-test generation and the effects of different knowledge sources remain underexplored.

    Source: §1 · p. 1

  • Method

    The study evaluates Basic and API-level RAG using API documentation, GitHub issues, and StackOverflow Q&As for APIs in five Python-based ML/DL libraries.

    Source: §1 · p. 1, §3.3 · p. 4

  • Results

    RAG improves line coverage by 6.5% on average, with API-level GitHub-issue RAG reaching an average improvement of 8.1%; the study reports 28 detected bugs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GitHub issues are particularly effective for code coverage, and combining RAG sources further improves results.

    Source: §6 · p. 11

    The study does not explicitly evaluate retrieval relevance, and Python-level coverage does not exercise core logic in C/C++ extensions.

    Source: §5 · p. 11

Abstract

from arXiv · show

Retrieval Augmented Generation (RAG) has advanced software engineering tasks but remains underexplored in unit test generation. To bridge this gap, we investigate the efficacy of RAG-based unit test generation for machine learning (ML/DL) APIs and analyze the impact of different knowledge sources on their effectiveness. We examine three domain-specific sources for RAG: (1) API documentation (official guidelines), (2) GitHub issues (developer-reported resolutions), and (3) StackOverflow Q&As (community-driven solutions). Our study focuses on five widely used Python-based ML/DL libraries, TensorFlow, PyTorch, Scikit-learn, Google JAX, and XGBoost, targeting the most-used APIs. We evaluate four state-of-the-art LLMs -- GPT-3.5-Turbo, GPT-4o, Mistral MoE 8x22B, and Llama 3.1 405B -- across three strategies: basic instruction prompting, Basic RAG, and API-level RAG. Quantitatively, we assess syntactical and dynamic correctness and line coverage. While RAG does not enhance correctness, RAG improves line coverage by 6.5% on average. We found that GitHub issues result in the best improvement in line coverage by providing edge cases from various issues. We also found that these generated unit tests can help detect new bugs. Specifically, 28 bugs were detected, 24 unique bugs were reported to developers, ten were confirmed, four were rejected, and ten are awaiting developers' confirmation. Our findings highlight RAG's potential in unit test generation for improving test coverage with well-targeted knowledge sources. Future work should focus on retrieval techniques that identify documents with unique program states to optimize RAG-based unit test generation further.

1 Introduction

RAG has advanced other software-engineering tasks, but its potential for unit-test generation remains underexplored. This study evaluates RAG for ML/DL APIs using different knowledge sources and reports improved line coverage.

1 Introduction

  • The study addresses the underexplored use of RAG in unit-test generation and examines how different document sources affect its practical benefits and limitations.RAG has been used in other software-engineering tasks, including code generation, search, completion, and automated program repair.Source: §1 · p. 1
  • The evaluation covers TensorFlow, PyTorch, Scikit-learn, Google JAX, and XGBoost, whose non-determinism and expansive API parameter spaces make unit testing challenging.The three knowledge sources are API documentation, GitHub issues, and StackOverflow Q&As.Source: §1 · p. 1
  • Four LLMs generate tests for 694 APIs, evaluated by parse, execution, and pass rates, API line coverage, and manual bug analysis.The issue and Q&A corpora contain 19K GitHub issues and 22K StackOverflow Q&As.Source: §1 · p. 1
  • RAG-based unit-test generation improves line coverage by 6.5% on average; API-level GitHub-issue RAG reaches an average improvement of 8.1%.The study also reports 28 detected bugs, including 24 unique bugs reported to developers; ten were confirmed.Source: §1 · p. 2
  • RAG helps cover unique lines by supplying program states that LLMs cannot generate directly.Among the reported bugs, ten were confirmed, four rejected, and ten awaited developer confirmation.Source: §1 · p. 2
  • The work introduces API-level RAG and a benchmark dataset, and reports a systematic comparison of knowledge sources for test coverage and bug detection.The authors also provide the experimental code and data as a replication package.Source: §1 · p. 2

2 Background and Related Works

The background presents RAG as retrieval-based augmentation whose effectiveness depends on retrieved knowledge quality. Prior LLM test-generation work leaves a gap in systematic comparisons of domain-specific knowledge sources, particularly for ML/DL libraries.

2.1 RAG Incorporation in ASE

  • RAG retrieves relevant external documents to augment an LLM query, and its effectiveness depends heavily on the quality of that knowledge.Prior software-engineering tasks using RAG include code generation, code search, and automated program repair.Source: §2.1 · p. 2

2.2 Unit Test Generation using LLMs

  • Prior LLM test-generation work includes iterative refinement, validation and repair, prompt design, and retrieval augmentation, but has not systematically compared domain-specific knowledge sources.The cited studies also report continuing challenges in correctness, coverage, and defect detection.Source: §2.2 · p. 2
  • Related ML testing approaches include API-level fuzzing, hierarchical test generation, and mining StackOverflow usage patterns for search-based tests.These studies differ from the paper’s focus on comparing knowledge sources for RAG-based test generation.Source: §2.2 · p. 2
  • The study positions its contribution as a systematic comparison of API documentation, GitHub issues, and community knowledge for unit-test generation.Earlier work had used limited RAG or execution-feedback prompting without systematically exploring these sources’ effects on coverage and bug detection.Source: §2.2 · p. 2

3 Methodology

The methodology evaluates LLM-based test generation for ML/DL APIs through baseline prompting and two RAG strategies, using domain-specific documents and multiple measures of test quality.

3.1 Research Questions

  • The five research questions cover baseline test generation, Basic RAG, API-level RAG, bug detection, and why knowledge sources affect generated tests.The baseline establishes a comparison for later questions.Source: §3.1 · p. 3

3.2.1 Studied ML/DL libraries.

  • The study focuses on TensorFlow, PyTorch, Scikit-learn, Google JAX, and XGBoost, selected for their popularity, active communities, and role in ML/DL.These systems also expose large parameter spaces and statistical, nondeterministic behavior.Source: §3.2.1 · p. 3

3.2.2 Experiment Data Collection.

  • The corpus combines API signatures, descriptions, and examples with closed GitHub issue threads and accepted StackOverflow answers, truncating each document to 5K tokens.Documents were filtered by API mentions; the final set included 694 APIs with at least ten GitHub issues and ten StackOverflow Q&As each.Source: §3.2.2 · pp. 3–4

3.3 Retrieval-Augmented Test Generation

  • Basic RAG retrieves globally relevant top-k documents, whereas API-level RAG uses API-specific databases and dense retrieval after filtering for the target API.The vector databases use ChromaDB and Sentence Transformers embeddings.Source: §3.3 · p. 4, §3.3.1 · p. 4

3.3.2 Prompting LLMs:

  • Prompts request executable unittest suites with imports and a main function, adding instructions to use retrieved documents to exercise more lines.Prompt wording was refined in pilot runs to encourage executable, diverse, nonredundant outputs.Source: §3.3.2 · p. 4

3.3.3 LLM Baselines.

  • The evaluated models were GPT-3.5-Turbo, GPT-4o, Mistral 8x22B Instruct, and Llama 3.1 405B Instruct.The set includes both open- and closed-source models.Source: §3.3.3 · p. 5

3.4 Evaluation Metrics

  • Evaluation measures parse, execution, pass, and line-coverage rates, plus bug detectability based on true bugs revealed by generated tests.The measures operate at different granularities, from test suites to individual test methods.Source: §3.4 · p. 5

4.1 RQ1: Performance of LLMs with Basic Instruction Prompting

With basic instruction prompting, GPT-4o generally performs best, but syntactic and execution correctness do not necessarily correspond to greater code coverage.

Result

  • Parse rates ranged from 65% to 100%, and most evaluated models generated considerable numbers of executable tests.Coverage reports exclude library back-end implementations such as C++.Source: §4.1 · p. 5
  • Mistral’s minimum execution rate was 38%, with failures linked to unstructured code fragments and missing imports, test functions, or other required structure.Some GPT-3.5-Turbo suites also failed after requiring unimported packages.Source: §4.1 · p. 5
  • GPT-4o ranked first in parse rate and coverage across the projects and usually led execution and pass rates, with exceptions on XGBoost.Llama had the lowest coverage across all projects, while GPT-3.5-Turbo and Mistral generally ranked in the middle.Source: §4.1 · p. 5
  • On XGBoost, Mistral achieved 52% coverage versus GPT-3.5-Turbo’s 48% despite executing 102 versus 106 tests.The authors note that more tests can exercise similar or identical code.Source: §4.1 · p. 5

4.2 RQ2: Impact of External Sources in Basic RAG

Basic RAG generally improves code coverage over basic instruction prompting, but the tested approaches were not statistically distinguishable in ranking.

Result

  • Basic RAG enhanced code coverage compared with basic instruction prompting, while lengthy retrieved documents sometimes reduced parse rates for GPT-3.5-Turbo and Mistral.The authors attribute the parse-rate effect to smaller models deviating from the instruction to generate runnable suites.Source: §4.2 · p. 6
  • Across 20 project-model cases, GitHub-issue Basic RAG beat basic instruction prompting 14 times, versus 13 for combined, 12 for API-documentation, and 11 for StackOverflow RAG.The combined approach did not consistently outperform individual sources.Source: §4.2 · p. 6
  • Combined and GitHub-issue Basic RAG each ranked 2.7 on average, but the Friedman test found no significant ranking differences (p-value 0.4809).GitHub issues may help by offering bug-reproduction examples and mitigation strategies in comments.Source: §4.2 · p. 6
  • Basic RAG improved unit-test generation, particularly coverage, but its ranking advantage was not statistically significant.The comparison included API documentation, GitHub issues, StackOverflow Q&As, and their combination.Source: §4.2 · p. 6

4.3 RQ3: Impact of External Sources in API-Level RAG

API-level RAG improves code coverage over basic instruction prompting, with GitHub-issue retrieval the strongest-performing source in the reported rankings. The section also compares win counts across API-level sources and notes parse-rate declines for two models.

4.3 RQ3: Impact of External Sources in API-Level RAG

  • API-level RAG improved code coverage over basic instruction prompting, while GPT-3.5-Turbo and Mistral showed a similar decline in parse rate.The API-level design restricts retrieval to documents mapped to the API under test.Source: §4.3 · p. 6
  • 15 of 20 API-level combined-RAG cases beat basic instruction prompting; the API-document, GitHub-issue, and StackOverflow variants won 13, 14, and 12 cases, respectively.These win counts were higher than Basic RAG for combined, API-document, and StackOverflow approaches, and equal for GitHub issues.Source: §4.3 · p. 7, Figure 4 · p. 7
  • The combined API-level approach beat API documents 17 times and StackOverflow Q&As 13 times, but beat GitHub issues only 8 times.Compared with Basic RAG, these counts rose by six for API documents and one for StackOverflow, but fell by two for GitHub issues.Source: §4.3 · p. 7
  • GitHub-issue API-level RAG ranked highest among the four API-level sources (2.3), followed by combined RAG (2.35), with significant rankings and a moderate effect size.The Friedman test had p-value 0.0056 and Kendall’s W = 0.38.Source: §4.3 · pp. 7–8
  • API-level GitHub-issue RAG ranked first overall (3.4), ahead of API-level combined RAG (3.7), and API-level RAG generally ranked above Basic RAG.The nine-approach Friedman ranking was significant (p-value 0.0127) with a small effect size (Kendall’s W = 0.23).Source: §4.3 · p. 8

4.4 RQ4: Bug Detection

The authors manually assessed whether generated failing tests exposed real ML/DL-library bugs, finding bugs in both API-documentation RAG and basic prompting. Many failures were not true bugs, in part because generated assertions could misrepresent documented behavior.

4.4 RQ4: Bug Detection

  • The analysis used GPT-4o tests on the same APIs as earlier research questions and compared API-documentation RAG with basic prompting to reduce potential issue/Q&A information leakage.Only failing tests were examined; two authors manually labeled failures and resolved disagreements through discussion.Source: §4.4 · p. 8
  • Despite bug-focused prompts, an average of 69.5% of generated tests passed, and only a few failed tests exposed real bugs.The authors interpret this as evidence that the libraries handle common exceptions robustly and that LLMs struggle to generate tests revealing true bugs.Source: §4.4 · pp. 8–9
  • 28 bugs were detected; 24 new bugs were reported, with ten confirmed, four rejected, and ten awaiting developer confirmation.API Documentation RAG found 19 true bugs, compared with nine from basic instruction prompting.Source: §4.4 · p. 9
  • One documented example uses a generated test to expose an assertion error because the implementation fails to validate that mask values are boolean.The documentation specifies boolean mask values.Source: §4.4 · p. 9, Figure 5 · p. 9
  • Documentation helped generate tests for invalid inputs and expected errors, while hallucinated behavior could produce incorrect assertions that did not match the API specification.Many detected errors involved implementation-documentation inconsistencies or inadequate validation.Source: §4.4 · p. 9

4.5 RQ5: Qualitative Analysis of RAG

The qualitative analysis links retrieved-document content to uniquely covered test lines and finds that GitHub issues contribute the most unique coverage. The sources differ in breadth, depth, and the kinds of test knowledge they provide.

4.5 RQ5: Qualitative Analysis of RAG

  • The analysis sampled 250 of 694 APIs, identified lines uniquely covered by each method, and used thematic coding to classify retrieved documents and their influence on tests.Two participants coded the documents independently, with two others resolving disagreements by consensus.Source: §4.5 · pp. 9–10
  • The first four information categories were bug-inducing documents, standard use cases, API usage sequences, and unique or complex inputs and parameters.These categories respectively concern defects or edge cases, representative interactions, dependent-call workflows, and uncommon configurations.Source: §4.5 · p. 10
  • The remaining categories were exception-handling patterns, integration patterns, and performance considerations.They concern expected failures, interactions with larger systems or external libraries, and efficiency or resource use.Source: §4.5 · p. 10
  • GitHub issues produced the most APIs with unique coverage and the highest average number of unique lines per API among the RAG sources.StackOverflow, GitHub issues, and combined RAG each covered 97 APIs; API documentation ranked second in average unique lines, while basic prompting uniquely covered 11 APIs and averaged 7.9 unique lines.Source: §4.5 · p. 10
  • StackOverflow showed breadth across API coverage, while detailed API-documentation parameters supported greater unique-line depth; basic prompting also contributed unique coverage.The authors conclude that the sources can be used complementarily.Source: §4.5 · p. 10
  • GitHub issues supplied the most identified documents influencing unique coverage and led in four of seven information categories; combined RAG followed in category variety and standard use cases.StackOverflow offered varied API sequences and integrations, whereas API documentation provided detailed parameters that could guide unique or edge-case inputs.Source: §4.5 · p. 10

4.6 Discussions

The discussion connects RAG’s coverage and bug-detection results to retrieval of precise, API-specific context, especially community-reported edge cases. It recommends fine-grained retrieval while noting infrastructure, model-capacity, and API-selection constraints.

4.6.1 Key Insights.

  • RAG achieved a 6.5% point increase in code coverage and identified ten additional real-world bugs compared with generation without RAG.The discussion attributes the gains to precise retrieved context, including edge cases from community issues that models may not recall.Source: §4.6.1 · pp. 10–11

4.6.2 Implications for Research and Practice.

  • The authors recommend exploring automated document selection, hybrid sparse-dense indexing, and mutation or property-based coverage, and suggest API-level retrieval for practice.They also recommend a lightweight GitHub-issue vector store queried alongside official documentation.Source: §4.6.2 · p. 11
  • Per-API vector databases add storage and query overhead and require refreshing; lengthy retrieved documents also coincided with parse-rate declines for smaller LLMs.Requiring at least ten documents per API excludes less-popular functions, limiting coverage of newly introduced APIs.Source: §4.6.3 · p. 11

5 Threats to Validity

The authors identify threats from evaluation choices and study scope, including limits to line-coverage measurement, retrieval assessment, and the projects and models examined.

5 Threats to Validity

  • Line coverage may not correlate strongly with actual bug detection, and mutation testing was infeasible, so the authors added manual bug analysis.They also note that LLM nondeterminism and prompt choices threaten internal validity; temperature was set to zero, prompts were optimized, and five-run checks found average metric differences below 2.5%.Source: §5 · p. 11
  • Retrieval relevance was not explicitly evaluated, and Python-level coverage excludes core logic in C/C++ extensions.The reported coverage therefore reflects the front-end API surface rather than comprehensive lower-level testing.Source: §5 · p. 11
  • The findings may not generalize beyond the sampled libraries and APIs, and closed-source LLM APIs can change without public visibility.The model versions used were preserved to support replication.Source: §5 · p. 11
  • The study covers five projects and a restricted dataset; requiring at least 10 GitHub issues and StackOverflow Q&As per API excluded substantial data.The authors also note possible omissions in mined information and API-document matching.Source: §5 · p. 11

6 Conclusion

The conclusion reports that GitHub issues are particularly effective for code coverage and that generated tests can find real bugs. It points to corner-case retrieval and broader knowledge sources as future directions.

6 Conclusion

  • GitHub issues are particularly effective for code coverage, combining RAG sources further improves results, and manual analysis found generated tests can find real bugs.The comparison includes Basic and API-level RAG using API documentation, GitHub issues, and StackOverflow.Source: §6 · p. 11
  • LLMs handle general cases without extra documents, while unique external examples help cover untested code lines.Source: §6 · p. 11
  • Future work includes improving retrieval for corner cases and exploring specification documents and technical blogs as additional knowledge sources.Source: §6 · p. 12
Loading 2409.12682v2…