PaperlayerView exact revision on arXiv

Source-linked AI summary

On Mitigating Code LLM Hallucinations with API Documentation

Nihal Jain, Robert Kwiatkowski, Baishakhi Ray, Murali Krishna Ramanathan, Varun Kumar

arXiv:2407.09726v1cs.CLcs.AIcs.LG

TL;DR

Underrepresented and changing APIs make API hallucinations a concern for Code LLMs. The paper introduces CloudAPIBench and combines API-index checks with confidence-based selective documentation retrieval; DAG++ outperforms DAG across Code LLMs on the benchmark while mitigating its high-frequency performance loss.

  • Problem

    Underrepresented or changing APIs are more prone to hallucinations, motivating systematic measurement of API hallucinations in cloud software engineering.

    Source: §1 · p. 1

  • Method

    The paper introduces CloudAPIBench and selectively triggers documentation retrieval using API-index validation or API-invocation confidence.

    Source: §1 · p. 2, §4.2 · p. 7, §4.3 · p. 8

  • Results

    DAG++ outperforms DAG across Code LLMs on CloudAPIBench, mitigating DAG’s 39.02% high-frequency API performance drop while maintaining or improving low-frequency gains.

    Source: §4.3 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    Selective API-documentation retrieval yields top CloudAPIBench performance across Code LLMs while addressing DAG’s high-frequency degradation.

    Source: §4.3 · p. 8, §6 · p. 9

    CloudAPIBench uses short synthetic Python prompts, which may not represent all real-world cloud API invocations, languages, or contexts.

    Source: §7 · p. 9

Abstract

from arXiv · show

In this study, we address the issue of API hallucinations in various software engineering contexts. We introduce CloudAPIBench, a new benchmark designed to measure API hallucination occurrences. CloudAPIBench also provides annotations for frequencies of API occurrences in the public domain, allowing us to study API hallucinations at various frequency levels. Our findings reveal that Code LLMs struggle with low frequency APIs: for e.g., GPT-4o achieves only 38.58% valid low frequency API invocations. We demonstrate that Documentation Augmented Generation (DAG) significantly improves performance for low frequency APIs (increase to 47.94% with DAG) but negatively impacts high frequency APIs when using sub-optimal retrievers (a 39.02% absolute drop). To mitigate this, we propose to intelligently trigger DAG where we check against an API index or leverage Code LLMs' confidence scores to retrieve only when needed. We demonstrate that our proposed methods enhance the balance between low and high frequency API performance, resulting in more reliable API invocations (8.20% absolute improvement on CloudAPIBench for GPT-4o).

1 Introduction

Code LLMs are especially prone to API hallucinations for underrepresented or changing APIs. The paper introduces CloudAPIBench and studies documentation retrieval, including selective DAG strategies to address cases where retrieval can hurt performance.

1 Introduction

  • Selective retrieval strategies mitigate DAG’s drawbacks and improve the reliability of Code LLM API invocations.The paper explores model-confidence checks and API-index validation before retrieval.Source: §1 · p. 2
  • DAG improves low-frequency API performance but can hurt high-frequency APIs when suboptimal retrievers add irrelevant documentation.High-frequency APIs may already be well covered by the model’s training data.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • CloudAPIBench evaluates real-world API hallucinations across major AWS and Azure cloud SDK APIs.The benchmark supports analysis of API frequency groups.Source: §1 · pp. 1–2
  • The study examines which documentation parts reduce hallucinations and how retrieval components affect model efficacy.Source: §1 · p. 2
  • The paper characterizes when DAG may degrade performance and presents selective retrieval methods for using documentation.Source: §1 · p. 2

2 API Hallucinations & CloudAPIBench

API hallucinations can propagate errors and security risks, motivating systematic measurement. CloudAPIBench evaluates cloud API invocation across frequency levels, where the results show that low-frequency APIs are harder for Code LLMs.

2.1 Impact of API Hallucinations

  • API hallucinations can propagate errors into later code, introduce security vulnerabilities, and increase developers’ work tracing and correcting affected code.A hallucinated API call can lead to hallucinated handling of its response in subsequent code segments.Source: §2.1 · p. 2

2.2 CloudAPIBench

  • CloudAPIBench comprises 622 synthetic Python tasks requiring specific AWS or Azure API calls, with prompts that stop just before the invocation.Its API frequency annotations use occurrence counts in The Stack v2 to support analysis across frequency levels.Source: §2.2 · p. 3
  • The benchmark groups APIs as low (0–10 occurrences), medium (11–100), or high (≥101), and reports the first granular analysis of a Code LLM’s pre-training corpus.The authors select diverse APIs within each class to reduce confounding factors such as invocation complexity.Source: §2.2 · p. 3
  • The benchmark’s API selection uses official specifications and The Stack v2 counts of function definitions and calls with matching API names.The construction process also aims to diversify providers, invocation complexity, and frequency.Source: §2.2 · p. 3
  • Validity is measured by whether a candidate call binds successfully to a dummy function mirroring the API’s signature.This checks API syntax without relying on string matching or the intricacies of static analysis.Source: §2.2 · p. 3, Figure 3 · p. 3

2.3 Evaluation & Results

  • All evaluated Code LLMs exhibit API hallucinations, and valid invocation rates are higher for high-frequency APIs than for low-frequency APIs.The benchmark’s results show this disparity even when models perform strongly on general programming tasks.Source: §2.3 · p. 4, Table 1 · p. 4

3 Documentation Augmented Generation (DAG)

Documentation Augmented Generation (DAG) retrieves API documentation from a model-generated candidate call, adds it to the prompt, and re-runs the model. Results vary by augmentation design and API frequency: specifications improve average performance, while retrievals can hurt high-frequency APIs.

3.1 Setup

  • DAG samples a candidate API invocation, uses it to retrieve documentation, appends the documentation to the prompt, and re-triggers the model.The retrieval query retains API-relevant keywords and excludes distractor prompt content.Source: §3.1 · pp. 4–5, Figure 4 · p. 5

3.2 Experiments & Results

  • 41.80% →86.82% average valid API invocation performance followed adding API Specification; Full Documentation used 685.24 tokens per augmentation.API Description + Specification was adopted for subsequent experiments as a balance between token efficiency and performance.Source: §3.2 · p. 5, Figure 6 · p. 6
  • At 50% retriever precision, valid API invocations for high-frequency APIs fell from 84.39% to 67.32%, while low-frequency APIs improved even with low-precision retrieval.High-frequency APIs required precision above 80% to match the base model’s performance.Source: §3.2 · pp. 5–6, Figure 7 · p. 6
  • With retriever precision fixed at 50%, one or more retrievals consistently improved low-frequency APIs, while high-frequency performance sharply declined with one retrieval and partially recovered with two or more.The authors attribute the high-frequency risk to irrelevant augmentations that may conflict with the model’s internal knowledge.Source: §3.2 · p. 6, Figure 7 · p. 6
  • DAG significantly reduces low-frequency API hallucinations, but irrelevant retrievals can hurt high-frequency performance; bypassing DAG for high-frequency APIs is suggested as a potential remedy.This discussion motivates allowing the model to use its internal knowledge when documentation retrieval is unnecessary.Source: §3.2 · p. 6

4 Improving DAG: When to Retrieve?

The section studies selective DAG triggering to limit regressions from suboptimal retrieval while using documentation when Code LLMs lack API knowledge. It evaluates index lookup, confidence-based triggering, and their combination in DAG++.

4.1 Index Lookup

  • Index lookup reduces DAG’s high-frequency regressions, sometimes improving over the base model, but can increase low-frequency hallucinations by suppressing retrievals.The method retrieves only when the first-iteration API name is absent from the index.Source: §4.1 · pp. 6–7, Table 2 · p. 7

4.2 API Invocation Confidence

  • API invocation confidence is the minimum probability of the predicted API-name subtokens, and API-use correctness correlates positively with this score.The minimum is intended to capture minor API-prediction uncertainty better than the mean.Source: §4.2 · p. 7, Figure 8 · p. 7
  • For StarCoder2-3B, confidence-triggered DAG performs best around a 0.7–0.8 threshold, reducing low-frequency hallucinations before high-frequency regressions emerge.The confidence scores are reported as well-calibrated across a range of Code LLMs.Source: §4.2 · p. 7, Figure 9 · p. 7

4.3 DAG++ & Discussion

  • DAG++ outperforms DAG on CloudAPIBench across models, mitigating high-frequency regressions while maintaining or improving low-frequency gains.DAG with suboptimal retrieval causes a 39.02% performance drop for GPT-4o on high-frequency APIs.Source: Table 3 · p. 8, §4.3 · p. 8
  • For GPT-4o, DAG++ triggers retrieval for 50.56% of low-frequency APIs and 3.41% of high-frequency APIs.DAG++ retrieves when the first-iteration API is absent from the index or its invocation confidence is below a fixed threshold.Source: §4.3 · p. 8
  • DAG++ shows similar enhancement trends across model families; average performance rises with model size, and larger models need fewer retrievals for optimal performance.Source: §4.3 · p. 8

5 Related Work

Related work spans API generation, code hallucinations, and retrieval-augmented generation; this paper focuses on documentation-based mitigation and frequency-aware evaluation.

5 Related Work

  • Prior API-generation evaluations do not capture public-domain API frequency, a granularity CloudAPIBench adds for targeted failure analysis.Source: §5 · p. 9
  • Where earlier work identifies code hallucinations, this study focuses on mitigating API hallucinations with documentation.Source: §5 · p. 9
  • The paper examines effective selective retrieval of API documentation, alongside prior fixed and adaptive RAG approaches.Source: §5 · p. 9

6 Conclusion & Future Work

The paper introduces CloudAPIBench and investigates documentation-based mitigation of API hallucinations across Code LLMs. It concludes that selective retrieval in DAG++ attains top CloudAPIBench performance across models and identifies directions for future work.

6 Conclusion & Future Work

  • CloudAPIBench measures API hallucinations across AWS and Azure APIs and categorizes APIs by low, medium, and high frequency.Source: §6 · p. 9
  • DAG supplies API syntax during inference, but irrelevant retrievals can degrade high-frequency API performance.Source: §6 · p. 9
  • DAG++ selectively triggers documentation retrieval using API-index lookup and API-invocation confidence, attaining top CloudAPIBench performance across Code LLMs.Source: §6 · p. 9
  • Future work could extend CloudAPIBench to long-context evaluation, study DAG beyond iterative generation, and improve robustness to irrelevant augmentations.Source: §6 · p. 9

7 Limitations

The limitations concern benchmark scope, heuristic construction, and the overhead of iterative DAG generation and retrieval.

7 Limitations

  • CloudAPIBench uses Python-only short synthetic prompts, which may not represent real-world cloud API invocations across languages and contexts.Source: §7 · p. 9
  • Some benchmark-construction steps use heuristics, including customized estimates of API frequency.The authors state that any impact from these approximations should be limited, given consistency with literature and expectations.Source: §7 · p. 9
  • Iterative DAG generates, retrieves, and regenerates; its overhead may make it unsuitable when latency is crucial.Source: §7 · p. 9

8 Ethics Statement

The paper identifies ethical risks from generated harmful content or content resembling training data, while assessing those risks as relatively low within its experiments.

8 Ethics Statement

  • Generated content should ideally be reviewed by human developers and executed in sandbox environments for real-world applications.The authors report that they trained no models and describe the computational footprint as relatively low; experiments used four NVIDIA A100 machines.Source: §8 · pp. 9–10

Supplementary Material: Appendices

The appendices describe CloudAPIBench’s API frequency composition and the process used to construct its AWS and Azure evaluation tasks.

A.1 Composition

  • CloudAPIBench annotates AWS and Azure APIs with frequencies proportional to their representation in public sources, grouped into low, medium, and high bins.Figure 10’s y-axis shows the number of tasks satisfying each respective criterion.Source: §A.1 · p. 13, Figure 10 · p. 13

A.2 Construction

  • The benchmark construction uses Python tasks for AWS and Azure APIs, with API specifications and documentation drawn from their respective SDK sources.AWS API stubs support API identification and specifications, while Azure API definitions provide docstrings; the AWS documentation is scraped from boto3 webpages.Source: §A.2 · p. 13
  • API frequencies are counted from source-specific code samples, with heuristics intended to reduce edge cases despite acknowledged residual noise.AWS and Azure source files are identified using package imports or filepath terms specific to each platform.Source: §A.2 · p. 13
  • Evaluation APIs are sampled to balance frequency classes and invocation complexity, and each API can appear in up to three prompts.The sampling aims to span easy-to-invoke APIs and APIs requiring careful syntax recall.Source: §A.2 · p. 14
  • Synthetic prompts end immediately before the API invocation and are built from imports, variable declarations, and an intent comment.When multiple APIs could satisfy a task, the authors manually enumerate possible targets to the best of their knowledge.Source: §A.2 · p. 14

A.3 Evaluation

The appendix describes how evaluation handles multiple possible target APIs, illustrates benchmark samples, and distinguishes three API hallucination types.

A.3 Evaluation

  • A candidate invocation is valid when it matches the syntax of any one of the identified target APIs.The evaluation appendix notes this rule for tasks with multiple target APIs.Source: §A.3 · p. 14

A.4 CloudAPIBench Samples

  • The appendix includes Azure and AWS benchmark samples, each showing its target API and frequency classification.The samples are presented for illustration in Figures 12 and 13.Source: §A.4 · p. 14

A.5 Hallucination Categorization & Illustration

  • The appendix distinguishes hallucinations involving an incorrect existing API, an incorrectly configured target API, or a non-existing API.Incorrect target usage includes unacceptable arguments or an invalid combination of required and optional arguments.Source: §A.5 · p. 14

A.6 Analyzing Low Frequency API Failures

The appendix reports that low-frequency API failures commonly involve non-existing APIs or incorrect target usage, then outlines documentation augmentation designs.

A.6 Analyzing Low Frequency API Failures

  • More than 50% of low-frequency API failures involve attempts to invoke non-existing APIs, while many others use the target API incorrectly.The analysis covers the largest model in each model family.Source: §A.6 · p. 14, Table 4 · p. 15

B.1 Augmentation Designs

  • The augmentation designs range from supplying only an API name to adding its description, specification, or both.The API specification gives the API name and required and optional arguments without argument descriptions.Source: §B.1 · p. 16
  • Full Docstring augmentation uses the entire collected documentation, right-truncated at 5,000 characters.This design assumes the information needed for invocation appears within those first 5,000 characters.Source: §B.1 · p. 16
  • Figure 15 illustrates the designs for an AWS API and omits the combined description-and-specification design because it combines the two individual designs.The figure’s full-documentation example is truncated to fit.Source: §B.1 · p. 16, Figure 15 · p. 18
Loading 2407.09726v1…