PaperlayerView exact revision on arXiv

Source-linked AI summary

DocPrompting: Generating Code by Retrieving the Docs

Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, Graham Neubig

arXiv:2207.05987v3cs.CLcs.AIcs.SE

TL;DR

Existing NL→code models struggle to use libraries and functions absent from training data. DocPrompting retrieves relevant documentation for an NL intent and conditions code generation on it, improving results across Bash and Python benchmarks, while remaining affected by retrieval-induced errors and requiring better document encoding or joint training.

  • Problem

    Existing NL→code models assume libraries and function calls are seen during training, despite continuously changing libraries and newly introduced functions.

  • Method

    DocPrompting retrieves relevant documentation from a shared pool and generates code using the NL intent together with the retrieved documents.

  • Results

    DocPrompting consistently improves NL→code models across two tasks, two programming languages, and multiple base models, including a 2.85% pass@1 improvement for CodeT5 on CoNaLa.

  • Takeaways & Limitations

    Documentation retrieval provides a practical way to support code generation involving unseen functions and libraries without retraining model components.

  • Takeaways & Limitations

    Case studies show retrieval can introduce incorrect arguments when documentation for related functions is also retrieved, and the authors identify cascading errors as a target for joint training.

Abstract

from arXiv · show

Publicly available source-code libraries are continuously growing and changing. This makes it impossible for models of code to keep current with all available APIs by simply training these models on existing code repositories. Thus, existing models inherently cannot generalize to using unseen functions and libraries, because these would never appear in the training data. In contrast, when human programmers use functions and libraries for the first time, they frequently refer to textual resources such as code manuals and documentation, to explore and understand the available functionality. Inspired by this observation, we introduce DocPrompting: a natural-language-to-code generation approach that explicitly leverages documentation by (1) retrieving the relevant documentation pieces given an NL intent, and (2) generating code based on the NL intent and the retrieved documentation. DocPrompting is general: it can be applied to any programming language and is agnostic to the underlying neural model. We demonstrate that DocPrompting consistently improves NL-to-code models: DocPrompting improves strong base models such as CodeT5 by 2.85% in pass@1 (52% relative gain) and 4.39% in pass@10 (30% relative gain) in execution-based evaluation on the popular Python CoNaLa benchmark; on a new Bash dataset tldr, DocPrompting improves CodeT5 and GPT-Neo1.3B by up to absolute 6.9% exact match.

1 INTRODUCTION

NL→code models typically rely on libraries and function calls seen during training, whereas DocPrompting retrieves relevant documentation before generating code. This lets models use documentation for unseen functions and libraries across general settings.

  • Motivation: Existing NL→code models assume test-time libraries and function calls were seen during training.New functions, libraries, and arguments can therefore fall outside the training distribution.
  • Motivation: Human programmers commonly consult manuals and documentation when using unfamiliar functions and libraries.DocPrompting is inspired by this documentation-based workflow.
  • Approach: DocPrompting retrieves relevant code documentation from a documentation pool before generating code from the intent and retrieved documents.The documentation pool can be updated with newly released library documentation without retraining model components.
  • Scope: DocPrompting is applicable across programming languages and underlying model architectures.The approach is demonstrated with multiple base models and retrieval methods on two NL→code benchmarks.

2 CODE GENERATION BY READING THE DOCS

DocPrompting formulates code generation as retrieval-conditioned generation: select a small relevant subset from the documentation pool, then condition code generation on that subset and the NL intent.

  • Assumption: DocPrompting assumes documentation is an exhaustive yet succinct resource for most libraries and programming languages.The method follows a retrieve-then-generate paradigm focused on documentation.
  • Retrieval-conditioned generation: The model retrieves a subset D_n of potentially relevant documents because only a few documents matter for each intent.Conditioning directly on the entire, unbounded documentation collection is computationally infeasible.
  • Retrieval-conditioned generation: The formulation decomposes code-generation probability into document-subset selection and code generation conditioned on the selected documents.The full probability marginalizes over possible document subsets.
  • Retrieval-conditioned generation: Because enumerating all document subsets is infeasible, DocPrompting approximates marginalization by using the most probable retrieved subset.The approximation conditions prediction on the selected documents and their intent.
  • DocPrompting pipeline: A retriever scores documentation against the intent and selects the top-k documents, which a generator uses with the intent to produce code.In the illustrated example, retrieved Pygments documentation supports code using PythonLexer and HtmlFormatter.

3 PRACTICAL INSTANTIATIONS OF DocPrompting

The practical system combines sparse or dense document retrieval with several generators, including language and code models. Training uses supervised and weak supervision for dense retrieval, while generator prompting varies by model input capacity.

  • System design: DocPrompting is model-agnostic and can be instantiated with any base retriever and generator.The experiments evaluate concrete retriever and generator choices selected for performance.
  • Retriever instantiations: The sparse retriever uses BM25 with word-frequency-based sparse features such as BM25 and TF-IDF.Elasticsearch provides the sparse retrieval implementation.
  • Retriever instantiations: The dense retriever trains neural representations contrastively by maximizing intent-positive-document similarity and minimizing in-batch negative similarity.Cosine similarity between neural encoder representations is used as the similarity function.
  • Retriever instantiations: Dense retrieval mixes supervised intent-document pairs with weak supervision from differently dropped-out representations of the same documentation sentence.The mixed supervision signal is intended to facilitate learning and reduce engineering effort.
  • Generator instantiations: Generators include GPT-Neo, Codex, T5, and CodeT5, with retrieved documents concatenated into prompts or encoded separately using fusion-in-decoder.Codex uses few-shot prompting because its parameters are unavailable for fine-tuning.

4 EXPERIMENTAL SETUP

The evaluation covers Bash and Python NL→code generation with shared documentation pools and oracle documents. It introduces the tldr benchmark and a CoNaLa split targeting unseen Python functions, using task-specific generation metrics.

  • Benchmarks: The experiments evaluate shell scripting and Python programming across retrieval-based code-generation benchmarks.Each benchmark provides a global documentation pool, with oracle documents used for retriever training.
  • 4.1 SHELL SCRIPTING: The tldr benchmark contains 1,879 unique Bash commands and 9,187 English NL→Bash pairs collected from a community-driven help project.The source covers commands ranging from common utilities such as cat and tar to uncommon commands such as toilet and faketime.
  • 4.1 SHELL SCRIPTING: For each tldr intent, the system retrieves an entire Bash manual and then the top-10 paragraphs from the top manual.Oracle documents are curated for each example using simple string matching.
  • Evaluation: The tldr evaluation measures command-name accuracy, exact match, token-level F1, and character-level BLEU while disregarding user-specific variable names.The benchmark examples pair NL descriptions with Bash command lines and argument flags.
  • 4.2 PYTHON PROGRAMMING: The re-split CoNaLa benchmark tests generalization to unseen Python functions by ensuring development and test examples contain functions absent from training.CoNaLa pairs StackOverflow questions with human-rewritten Python answers.
  • 4.2 PYTHON PROGRAMMING: The CoNaLa documentation pool contains 35,763 documents, each describing a single function from Python libraries available on DevDocs.The pool includes built-in and popular libraries such as numpy.
  • Evaluation: Python evaluation reports BLEU-4, function-name recall, unseen-function recall, and execution-based pass@k on manually tested examples.Unseen-function recall focuses on function calls absent from the training set.

5 RESULTS

Across shell scripting and Python programming, DocPrompting improves code-generation models by retrieving documentation, including when command names are known or examples are retrieved instead.

  • 5.1 SHELL SCRIPTING RESULTS: DocPrompting consistently improves base models on the tldr shell-scripting benchmark.T5+DocPrompting more than doubles command-name accuracy, adds over 16 charBLEU points, and gains almost 9 percentage points in exact match versus vanilla T5.
  • 5.1 SHELL SCRIPTING RESULTS: 6.7 charBLEU points: Codex+DocPrompting improves consistently across metrics in few-shot learning with only a few examples.The comparison is against a baseline that observes only NL-code pairs in its prompt.
  • 5.1 SHELL SCRIPTING RESULTS: 10 percentage points: providing Codex with the ground-truth command name raises exact match from 22.44% to 32.43% with DocPrompting.The oracle command is provided to both the baseline and the documentation-using model.
  • 5.1 SHELL SCRIPTING RESULTS: DocPrompting provides much higher gains than retrieving NL-code examples in the ExPrompting comparison.The paper notes that documentation may be available for newly released libraries when examples are not yet available.
  • 5.2 PYTHON PROGRAMMING RESULTS: 18.30 versus 9.03: DocPrompting substantially improves unseen-function recall over base CodeT5 on CoNaLa.CodeT5+DocPrompting also yields a 1.65 BLEU improvement over the CodeT5-initialized state-of-the-art baseline.
  • 5.2 PYTHON PROGRAMMING RESULTS: 2.85% pass@1 improvement: DocPrompting consistently outperforms CodeT5 across all evaluated pass@k values.The improvement reaches 4.45% at pass@5 and 8.38% when k = 200, indicating gains in functional correctness as well as surface quality.

6 ANALYSIS

DocPrompting improves retrieval and code generation by supplying documentation that bridges intent terminology and code terminology. Its effectiveness depends on retriever training, target-language pretraining, and avoiding documentation-induced argument errors.

  • Why does reading the documentation help generating more accurate code?: Unigram overlap in tldr increases from 12% to 24% when retrieved documentation is added to the NL intent.The authors connect this increase to documentation bridging the gap between intent terminology and code terminology.
  • Ablation study: Retrievers pretrained on the target programming language are generally stronger: CodeT5 outperforms RoBERTa on CoNaLa, while tldr benefits mostly from BM25 and RoBERTa.tldr is based on Bash, a language on which neither CodeT5 nor RoBERTa was explicitly pretrained.
  • Ablation study: Weak supervision on the documentation pool dramatically improves retriever recall, with removing it causing severe degradation on CoNaLa.The authors suggest weak supervision enables more effective domain adaptation.
  • Case study: DocPrompting corrects unseen function calls such as Image.open, but can transfer an argument from related documentation and generate skiprows=1 instead of header=False.The error arose because documentation for df.read_csv was retrieved alongside documentation for df.to_csv.

7 RELATED WORK

DocPrompting extends retrieve-then-generate ideas to code generation by retrieving documentation rather than relying only on training pairs or task-specific game manuals. Its motivation is that documentation remains available as libraries evolve and supports generalization across programming languages and datasets.

  • Code generation: Traditional NL→code models assume their training corpus covers all required libraries and functions, limiting generation of unseen functions.DocPrompting instead retrieves documentation for unseen functions at test time.
  • Code generation: Unlike code-example retrieval approaches, DocPrompting relies on documentation because newly released libraries commonly provide documentation before natural-language/code pairs.This is an assumption about the relative availability of documentation and paired examples.
  • Retrieval augmented generation: DocPrompting resembles retrieval-augmented generation in open-domain question answering, but targets documentation for frequently updated and newly introduced code libraries.The related-work comparison situates documentation retrieval within retrieve-then-generate methods.
  • Documentation conditioned generation: Earlier documentation-conditioned systems were tailored to specific games, whereas DocPrompting is applicable to multiple programming languages and datasets.The comparison distinguishes general-purpose code generation from game-specific environments.

8 CONCLUSION

The paper concludes that retrieving relevant documentation is a simple, effective way to improve NL→code generation across tasks, programming languages, and base models. It identifies richer document encoding and joint retriever-generator training as directions for further improvement.

  • Conclusion: DocPrompting consistently improves NL→code models across two tasks, two programming languages, and multiple strong base models.The conclusion presents this as the paper’s broad empirical finding.
  • Conclusion: 2.85% pass@1 improvement for CodeT5 on execution-based Python CoNaLa evaluation corresponds to a 52% relative gain.The reported result is on the popular CoNaLa benchmark.
  • Conclusion: On tldr, DocPrompting improves CodeT5 and GPT-Neo-1.3B by up to 6.9% exact match and Codex by 6.78 charBLEU score.These results concern the newly introduced Bash dataset.
  • Conclusion: The authors propose more structured encoding of long documents and joint retriever-generator training to reduce cascading errors and improve results.They also suggest applying the approach to additional code-related tasks and documentation-like resources.

A T L D R: A NEWLY CURATED SHELL SCRIPTING BENCHMARK

The paper introduces tldr, a shell-scripting benchmark built from natural-language descriptions and Bash commands, with documentation-based evaluation resources. Its construction separates commands across splits to test generalization and documents metric and pretraining considerations.

  • NL→Bash pairs: tldr contributors provide NL descriptions and Bash examples for commands, including varied flags and arguments covering common usages.The examples are mainly one-liners, and the benchmark includes commands such as cat.
  • NL→Bash pairs: The benchmark contains 9187 NL-code pairs, averaging 4.84 pairs per command, with completely different commands in training, development, and test sets.Commands lacking an available manual are discarded.
  • Documentation pool D: The documentation pool uses manuals for 1897 Bash commands, extracted from manned.org and split into paragraphs describing individual functions or flag usages.This paragraph-level decomposition supports retrieval of focused documentation pieces.
  • Documentation pool D: Oracle documentation is identified through command-name and flag-matching heuristics, including selecting paragraphs beginning with a relevant flag and the command-summary paragraph.These selected paragraphs form D* for the benchmark’s retrieval setup.
  • Evaluation metrics: Evaluation uses command-name accuracy, token-level F1, and exact match, while placeholder variation makes surface-form comparison of user-specific variable names less meaningful.The metric passage also describes BLEU-related evaluation context for tldr.
  • NL→Python pairs: CoNaLa is re-split so development and test examples contain at least one Python function absent from training, creating a separate unseen-function generalization setting.The split contains 2135 training, 201 development, and 543 test examples.
  • Human-annotated unit tests: Human-annotated unit tests provide execution-based evaluation of functional correctness on 100 randomly selected CoNaLa test examples.The tests check behavior of generated functions on concrete inputs.
  • Documentation pool D: The Python documentation pool contains 35763 manuals covering libraries available on DevDocs, including built-ins, numpy, and pandas.API signatures and corresponding documentation are extracted and indexed for retrieval.

C DENSE RETRIEVER TRAINING

The section describes dense-retriever and generator training choices, prompting setups, and analyses of retrieval quality on CoNaLa.

  • C DENSE RETRIEVER TRAINING: CodeT5 training uses 10 epochs, batch size 512, and learning rate 1e−5, with averaged final-layer hidden states as text representations.CoNaLa additionally uses its first 100k mined examples as supervised data and applies one search step.
  • C DENSE RETRIEVER TRAINING: Single-source generators train for 20 epochs, while FiD generators train for 10,000 steps with 200-token documents and batch size 8.Model selection uses token-level F1 on tldr and BLEU on CoNaLa.
  • C DENSE RETRIEVER TRAINING: Codex prompts compare a three-example baseline against DocPrompting prompts that append documentation to each in-context example and test intent.Test examples use the retriever’s top-five paragraphs, while examples use up to five oracle documents.
  • C DENSE RETRIEVER TRAINING: Figure 5 reports recall@k and corresponding BLEU scores for top-k retrieved documents on CoNaLa using CodeT5.The figure examines retrieval and generation performance as the document count changes.

F ADDITIONAL ANALYSIS

Additional analyses examine parameter efficiency, document-count effects, lexical overlap, retrieval latency, temperature sensitivity, and Codex prompting.

  • Parameter efficiency: FiD-based DocPrompting is more parameter-efficient than joint encoding: 220M-parameter T5+DocPrompting significantly outperforms 125M-parameter Neo-125M+DocPrompting.Neo+DocPrompting must scale to 1.3B parameters to match the 220M-parameter T5+DocPrompting, possibly because independent encoding better uses retrieved documents.
  • The impact of the number of documents: Increasing k consistently raises recall, but irrelevant documents limit generation gains; CodeT5 reaches its highest BLEU with 5 ≤k ≤10.With oracle documents, BLEU reaches 49.04, indicating that both document precision and recall matter.
  • Full n-gram overlap: Documentation significantly increases n-gram overlap recall between inputs and outputs on both tldr and CoNaLa.In CoNaLa, dense retrieval allows DocPrompting to work well without generally high n-gram overlap between the intent and retrieved documents.
  • Retrieval latency: Retrieval adds test-time computation, but pre-encoding documents and sub-linear top-k search keep latency increases non-prohibitive.The generator’s added document tokens increase memory consumption while causing only a small latency increase through parallel encoding; five documents may suffice in many cases.
  • Temperature analysis: Figure 6 plots pass@k on 100 test examples across different sampling temperatures.The main execution-based evaluation instead uses the best temperature for each model and k value.

H EXPERIMENTS WITH code-davinci-002

Experiments with code-davinci-002 find limited gains in non-oracle settings but significant improvements with oracle retrieval, alongside illustrative Bash and CoNaLa examples.

  • H EXPERIMENTS WITH code-davinci-002: Codex+DocPrompting does not improve over base Codex in non-oracle settings, possibly because CoNaLa and related data may have leaked into Codex’s training corpus.The authors also suggest that Codex’s capacity may require an equally strong retriever for improvement.
  • H EXPERIMENTS WITH code-davinci-002: Oracle retrieval produces significant DocPrompting improvements on both tldr and CoNaLa under code-davinci-002.The authors therefore identify stronger non-oracle retrieval as a route for improving the non-oracle results.
  • H EXPERIMENTS WITH code-davinci-002: On tldr examples, T5+DocPrompting retrieves correct Bash documentation and arguments when baseline T5 fails, while grounding reduces hallucinated flags in other cases.Table 10 contains the comparative prediction examples, and Table 11 provides additional CoNaLa examples involving unseen functions.
Loading 2207.05987v3…