PaperlayerView exact revision on arXiv

Source-linked AI summary

When Language Model Meets Private Library

Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, Jian-Guang Lou

arXiv:2210.17236v1cs.PLcs.CLcs.SE

TL;DR

Pretrained language models have not seen private APIs, although programmers commonly need to use private libraries. The paper retrieves APIs from documentation and generates code with them, reporting consistent gains from API information across three benchmarks.

  • Problem

    Pretrained language models have not seen private APIs, which are not publicly available, despite programmers commonly needing to write code with private libraries.

    Source: §1 · p. 1

  • Method

    The framework retrieves useful APIs from documentation and uses them to generate code, with both modules trained on public-library data.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across all three benchmarks, prompting API information consistently improves performance over No API for off-the-shelf models and CODEGENAPI.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    The paper provides TorchDataEval, MonkeyEval, and BeatNumEval as benchmarks for evaluating private-library code generation.

    Source: §1 · p. 2

    The three tested private libraries each have fewer than 200 APIs, and the authors surmise retrieval may become more challenging as API inventories grow.

    Source: Limitations · p. 9

Abstract

from arXiv · show

With the rapid development of pre-training techniques, a number of language models have been pre-trained on large-scale code corpora and perform well in code generation. In this paper, we investigate how to equip pre-trained language models with the ability of code generation for private libraries. In practice, it is common for programmers to write code using private libraries. However, this is a challenge for language models since they have never seen private APIs during training. Motivated by the fact that private libraries usually come with elaborate API documentation, we propose a novel framework with two modules: the APIRetriever finds useful APIs, and then the APICoder generates code using these APIs. For APIRetriever, we present a dense retrieval system and also design a friendly interaction to involve uses. For APICoder, we can directly use off-the-shelf language models, or continually pre-train the base model on a code corpus containing API information. Both modules are trained with data from public libraries and can be generalized to private ones. Furthermore, we craft three benchmarks for private libraries, named TorchDataEval, MonkeyEval, and BeatNumEval. Experimental results demonstrate the impressive performance of our framework.

1 Introduction

The paper addresses the difficulty of generating code with private APIs that pretrained models have not seen, proposing API retrieval followed by code generation. Experiments on three private-library benchmarks show improved performance.

1 Introduction

  • Private APIs are unseen during model training and unavailable on public code-hosting platforms, making direct private-library code generation challenging.Source: §1 · pp. 1–2
  • Codex 12B pass@1 drops from 18.88% on PandasEval to 1.47% on MonkeyEval.The paper presents this as evidence that existing models struggle with private-library code generation.Source: §1 · p. 2
  • The framework retrieves useful APIs with APIRetriever, then uses APICoder to generate code; both modules are trained on public-library data.The authors expect the modules to generalize to private libraries.Source: §1 · p. 2
  • The authors craft three private-library benchmarks: TorchDataEval, MonkeyEval, and BeatNumEval.TorchDataEval has 50 problems; MonkeyEval and BeatNumEval are adapted from PandasEval and NumpyEval, respectively.Source: §1 · p. 2
  • Experiments on TorchDataEval, MonkeyEval, and BeatNumEval show the framework improves pretrained models’ private-library code generation.The benchmarks were crafted to evaluate code generation for private libraries.Source: §1 · p. 2

2 Framework

The framework treats private-library code generation as retrieving relevant APIs from documentation and generating target code with them.

2 Framework

  • The task context x includes a natural-language problem description and code such as imports or a function header; target code y solves the problem.For private-library tasks, y contains calls to the corresponding private APIs.Source: §2 · p. 2
  • APIRetriever MR retrieves possible APIs from documentation given x, and APICoder MC generates code using the retrieved API set A.The process is formalized as A = MR(x) and y = MC(A; x).Source: §2 · p. 2
  • Each API information item includes its name, signature, and description; the implementation uses only the description’s first sentence.The authors say the first sentence is sufficient to summarize the API.Source: §2 · p. 3

3 Methodology

The methodology trains API retrieval and code generation components using public-library documentation and code, then applies them to private-library problems.

3.1 Data Collection

  • Data collection covers API documentation for 31 popular Python libraries and yields 4.54M preprocessed Python files containing calls to those libraries.The files were extracted from a 330GB GitHub corpus of 60.6M Python files.Source: §3.1 · p. 3

3.2 APIRetriever

  • APIRetriever’s dual encoder trains on problem descriptions paired with APIs from the same code block and randomly sampled unrelated APIs, producing 40.3M training sets.Independent BERT encoders embed descriptions and API information, which are compared by dot product.Source: §3.2 · pp. 3–4
  • At inference, APIRetriever indexes API embeddings with FAISS and recalls the top-k APIs closest to the problem-description embedding.Source: §3.2 · p. 4
  • The user interface presents the problem and top-five retrieved APIs for user selection; it shows names and descriptions because signatures reduced correct choices.Source: §3.2 · p. 4

3.3 APICoder

  • APICoder prepends API information to the problem context, using either off-the-shelf code models or a CODEGEN-MONO 350M model continually pre-trained on API-enriched code.The API information is formatted as name, signature, and description.Source: §3.3 · p. 4
  • CODEGENAPI training shuffles APIs and adds noise APIs to code blocks; its file-resampling strategy favors files with higher repository stars, unit-test function rates, and API rates.Source: §3.3 · p. 4

4 Benchmark Construction

The section introduces three benchmarks for evaluating code generation with private-library APIs. They combine a recently released real library with two pseudo-private libraries adapted from public-library benchmarks.

4 Benchmark Construction

  • The three benchmarks address the limited evaluation of code generation invoking private APIs: TorchDataEval, MonkeyEval, and BeatNumEval.Each problem includes context, target code, and corresponding test cases.Source: §4 · p. 5
  • TorchDataEval contains 50 problems based on TorchData API-documentation examples, checked by two experienced Python volunteers.Problem difficulty is controlled by the number of APIs in the target code, with a 6:3:1 distribution for one, two, and more APIs.Source: §4 · p. 5
  • MonkeyEval and BeatNumEval each contain 101 problems adapted from PandasEval and NumpyEval by manually changing library-related keywords.Their new API descriptions were manually paraphrased so pre-trained language models would not have seen them.Source: §4 · p. 5

5 Experiments

The experiments compare API-prompting settings and CODEGENAPI across three private-library benchmarks; API prompts generally helped, while closer analyses examined retrieval, noise, and task difficulty.

5.1 Experimental Setup

  • Evaluation compared No API, Perfect, Top-N (N = 1, 2, 3, or 5), and Human-selected APIs using pass@k (k = 1, 10, or 100), with n = 200 code samples per problem.Three programmer volunteers familiar with Python but unfamiliar with the benchmarks used APIRetriever to choose APIs for all problems; final APIs were determined by voting.Source: §5.1 · p. 5

5.2 Main Results

  • Across all three benchmarks, prompting APIs consistently improved off-the-shelf and CODEGENAPI performance over No API, with Top-N or Human sometimes exceeding Perfect.The gains were reported for both off-the-shelf models and continually pre-trained CODEGENAPI.Source: §5.2 · pp. 6–7, Table 1 · p. 6
  • CODEGENAPI beat CODEGEN on TorchDataEval and MonkeyEval but not BeatNumEval, where numerical calculation rather than API invocation was the key obstacle.The authors attribute the BeatNumEval result to numerical expressions such as a[:,None]+b*2.Source: §5.2 · p. 7

5.3 Closer Analysis

  • APIRetriever's top-5 recall was already high, and human-selected API accuracy dramatically exceeded top-1, top-2, or top-3 retrieval accuracy.The findings support offering five API candidates for user selection.Source: §5.3 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7
  • CODEGENAPI's advantage over CODEGEN was largest for one-API problems and persisted as the number of APIs increased.The analysis divides each benchmark into three parts according to the number of APIs in the target code.Source: §5.3 · pp. 7–8, Figure 7 · p. 7
  • A 5% noise rate was the default; alternatives of 0%, 10%, and 20% degraded performance, while removing resampling reduced results on all three benchmarks.The ablation table covers CODEGENAPI in the Top-2 setting.Source: §5.3 · p. 8, Table 2 · p. 8
  • On PandasEval and NumpyEval, CODEGENAPI's improvement over CODEGEN was less significant than on private libraries and depended heavily on APIRetriever performance.CODEGENAPI's advantage appeared with perfect APIs but not with top-2 retrieved APIs.Source: §5.3 · p. 8, Table 3 · p. 8

6 Related Work

Prior work largely targets standalone or public-library code, while this paper addresses private APIs through retrieval from private-library documentation.

6.1 Code Generation

  • Most code-generation models target standalone code; JigSaw and CERT address public-library code, whereas this work targets private APIs.JigSaw (Jain et al., 2021) and CERT (Zan et al., 2022a) are presented for public libraries.Source: §6.1 · p. 8
  • Existing benchmarks cover standalone code, mathematical problem solving, or public-library code; the paper proposes TorchDataEval, MonkeyEval, and BeatNumEval for private-library code.PandasEval and NumpyEval evaluate public-library code generation (Zan et al., 2022a).Source: §6.1 · p. 9

6.2 Retrieval-Based Generation

  • Unlike retrieval of public code or resources from GitHub or StackOverflow, APIRetriever retrieves private APIs from library documentation using code comments.The paper frames this design around private APIs not seen by pre-trained generative models.Source: §6.2 · p. 9

7 Conclusion

The paper concludes that its APIRetriever–APICoder framework and three benchmarks support private-library code generation, while identifying API-documentation use and real-world programming as future-work directions.

  • APIRetriever finds useful private APIs in documentation, and APICoder uses them to generate code for a programming problem.The framework is proposed for private-library code generation.Source: §7 · p. 9
  • TorchDataEval, MonkeyEval, and BeatNumEval are introduced to evaluate code generation for private libraries.Source: §7 · p. 9
  • Experiments and analysis support the framework's reasonableness and effectiveness; future work targets better use of API documentation and real-world programming.Source: §7 · p. 9

Limitations

The authors identify model-capacity dependence, small API inventories, benchmark realism, retrieval quality, and language scope as limitations.

  • API prompting depends heavily on the base model: stronger models benefit more, while weak models may gain nothing or perform worse.Source: Limitations · p. 9
  • The three private libraries contain fewer than 200 APIs each, and the authors expect retrieval to become more challenging as inventories grow.Source: Limitations · p. 9
  • Two benchmarks use pseudo-private libraries adapted from public ones, which may threaten fair evaluation despite efforts to approximate real private libraries.TorchDataEval is the other benchmark; the authors describe finding a real private library and crafting its benchmark as extremely challenging.Source: Limitations · p. 9
  • Most models using Top-N retrieval trail the same model with Perfect APIs, indicating substantial room to improve APIRetriever.Source: Limitations · p. 9
  • Experiments assess Python code generation; generalization to Java, C, or C++ may be limited by differences among language-library characteristics.Source: Limitations · p. 9

A Collection of API Documentation

The authors collect API documentation from popular public Python libraries to train models intended to generalize to private libraries.

B Resampling Strategy

The resampling strategy samples higher-quality Python files more frequently and assigns each file a weight based on repository stars, unit-test rate, and API counts.

B Resampling Strategy

  • Higher-quality Python files are sampled more frequently, and vice versa.The strategy uses a resampling weight for each Python file.Source: B Resampling Strategy · p. 11
  • Each Python file’s resampling weight reflects its repository’s star count, unit-test function rate, API-name count, and API count.The API count accounts for one API name matching multiple APIs.Source: B Resampling Strategy · p. 11
  • The stated clipping operation limits a value to [x, y].Source: B Resampling Strategy · p. 11

C Keywords Conversion from Public Library to Private Library.

The authors convert PandasEval and NumpyEval into MonkeyEval and BeatNumEval by manually modifying public-library-related keywords; Table 5 distinguishes original from converted keywords by background color.

C Keywords Conversion from Public Library to Private Library.

Loading 2210.17236v1…