PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolCoder: Teach Code Generation Models to use API search tools

Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, Zhi Jin

arXiv:2305.04032v5cs.SE

TL;DR

Code-generation models often select nonexistent or unsuitable APIs, especially for lesser-known and private libraries. ToolCoder teaches pretrained models to use API search tools through ChatGPT-based data annotation and tool-augmented decoding, achieving at least 6.21% average pass@1 improvement across five benchmarks.

  • Problem

    Current code-generation models frequently select nonexistent or unsuitable APIs, with errors especially pronounced for lesser-known and private libraries.

    Source: A. Limitations of current code… · p. 2

  • Method

    ToolCoder uses ChatGPT to annotate source-code data with tool calls, fine-tunes pretrained models, and integrates API search into decoding.

    Source: §I · p. 2, §III · p. 4, A. Automatic Data Annotation · p. 4

  • Results

    ToolCoder improves average pass@1 by at least 6.21% across five public and private-library benchmarks and is comparable to GPT-3.5.

    Source: §I · p. 2, §IX · p. 11

  • Takeaways & Limitations

    The results highlight the potential of incorporating programming tools into code-generation models for API selection.

    Source: §I · p. 2

    External validity is bounded by the five selected benchmarks and two API-library types, despite evaluation on both public and private libraries.

    Source: §VII · p. 10

Abstract

from arXiv · show

Automatically generating source code from natural language descriptions has been a growing field of research in recent years. However, current large-scale code generation models often encounter difficulties when selecting appropriate APIs for specific contexts. These models may generate APIs that do not meet requirements or refer to non-existent APIs in third-party libraries, especially for lesser-known or private libraries. Inspired by the process of human developers using tools to search APIs, we propose ToolCoder, a novel approach that integrates API search tools with existing models to assist in code generation and API selection. To teach our model to use tools, we introduce an automated data annotation method using ChatGPT to add tool usage information into the source code data and fine-tune code generation models. During inference, we integrate API search tools into the generation process so that our model can automatically use the search tool to get suggestions when selecting an API. Our experimental results demonstrate that ToolCoder exhibits excellent performance and generalization across five public and private library code generation benchmarks, with at least 6.21\% improvement on average pass@1 metrics and 9.64\% improvement on average pass@10 metrics compared to state-of-the-art methods. Furthermore, we show that our relatively small ToolCoder model is comparable to one of the current best models, GPT-3.5, highlighting the potential of incorporating programming tools into the code generation process.

I. INTRODUCTION

ToolCoder addresses API-selection failures in code generation by teaching models to use API search tools, improving generation across public and private-library benchmarks.

I. INTRODUCTION

  • Existing models generate non-existent or unsuitable APIs, with over 26% incorrect APIs on public benchmarks and over 90% failure on private libraries.These errors include nonexistent NumPy APIs, unqualified Pandas code, and fabricated private-library APIs.Source: §I · p. 1, Figure 2 · p. 2
  • ToolCoder integrates API search tools into pretrained code-generation models to assist API selection during decoding.The approach abstracts search tools into the generation process so models can retrieve API suggestions while producing code.Source: §I · p. 2
  • At least 6.21% average pass@1 improvement across five public and private-library benchmarks demonstrates stronger generalization than existing methods.ToolCoder also achieves at least 10.11%, 3.26%, and 1.39% pass@1 improvements on three public-library benchmarks, while remaining comparable to GPT-3.5.Source: §I · p. 2
  • The paper positions tool use as a novel capability for code-generation models and reports that it improves API-related code generation.The contribution is presented as the first incorporation of a programming tool into code-generation models.Source: §I · p. 2
  • ChatGPT automatically annotates source-code data with tool-usage information, reducing the cost of constructing training data.The annotated data is used to fine-tune code-generation models, and the annotation method is designed to transfer to other tasks.Source: §I · p. 2

II. MOTIVATING EXAMPLES

The motivating examples show that current models struggle with API correctness, while human-style search tools provide complementary routes to suitable public, lesser-known, and private-library APIs.

A. Limitations of current code generation models in selecting suitable APIs

  • More than 26% of generated APIs are incorrect on NumpyEval and PandasEval, while private-library API failure exceeds 90%.Observed failures include nonexistent APIs, wrong return types, and fabricated APIs caused by missing private-library knowledge.Source: A. Limitations of current code… · pp. 2–3

II. MOTIVATING EXAMPLES

  • Human programmers formulate API needs as queries and consult search or documentation tools before selecting an API.The motivating example maps removing a single dimension to the suggested API np.squeeze.Source: B. Existing search tools to aid API… · p. 3

B. Existing search tools to aid API selection

1) Online Search Engine Tool:

The paper uses online search for commonly discussed public libraries and documentation retrieval for lesser-known or private libraries, then exposes both through one model-facing API-search interface.

1) Online Search Engine Tool:

  • Online search engines retrieve API suggestions from community and tutorial content, making them useful for widely discussed public libraries.These sources are continually updated and cover varied programming scenarios.Source: 1) Online Search Engine Tool · p. 3

2) Documentation Search Tool:

  • Documentation search uses API comments and BM25 or semantic similarity to retrieve detailed suggestions for lesser-known and private libraries.Documentation offers broad library coverage, parameter explanations, and comparatively stable results.Source: 2) Documentation Search Tool · p. 3
  • Teaching models to navigate these search tools is intended to improve matching between API requirements and available implementations.The motivation is to use external information to address the large and changing space of library APIs.Source: 2) Documentation Search Tool · p. 3

III. API SEARCH TOOL

  • The method selects DuckDuckGo for common public libraries and BM25 retrieval over API documentation for lesser-known or private APIs.Search results from online communities are processed with string matching to extract mentioned APIs.Source: §III · p. 4
  • Both search sources are abstracted as APISearch(query) →answer and serialized with special API delimiters for model input.The model vocabulary includes ⟨API⟩, ⟨/API⟩, and → as special tokens.Source: §III · p. 4

IV. TOOLCODER

ToolCoder combines automatic tool-augmented data annotation, parameter-efficient fine-tuning, and tool-enhanced inference. Its annotation pipeline uses ChatGPT to create filtered training data, while LoRA adapts the model with few trainable parameters.

IV. TOOLCODER

  • ToolCoder consists of data annotation, fine-tuning, and inference modules arranged as a pipeline.The approach trains a model to select and use APIs based on partial code.Source: §IV · p. 4, Figure 3 · p. 5

A. Automatic Data Annotation

  • The authors randomly selected nearly 60k function-level source-code samples, used ChatGPT to generate one annotated datum per sample, filtered abnormal results, and obtained a final dataset of 53k.Filtering removed nested API Search calls, samples with five or more calls, samples lacking a public-library call, code changes after removing calls, and answers not reflected in generated code.Source: A. Automatic Data Annotation · p. 4
  • Three human-written numpy, pandas, and matplotlib examples guide ChatGPT’s automatic tool-augmented dataset annotation.One annotated example is generated for each base sample.Source: A. Automatic Data Annotation · p. 4, Figure 4 · p. 5

B. Parameter-efficient Fine-tuning

  • LoRA adapts transformer weights through trainable low-rank matrices while most pre-trained parameters remain frozen.For a weight matrix W, the update is represented as W + W_downW_up.Source: B. Parameter-efficient Fine-tuning · p. 5
  • Only 0.18% of CodeGen-350M and 0.09% of CodeGen-2B parameters are trained, enabling fine-tuning on an Nvidia GeForce RTX 2080.The strategy reduces computational burden while achieving results comparable to full-parameter training.Source: B. Parameter-efficient Fine-tuning · p. 5

C. Inference enhanced with Tools

During inference, ToolCoder detects generated API-search calls, queries the appropriate search backend, inserts the response, and resumes decoding. It removes the tool-call markup afterward to produce ordinary generated code.

C. Inference enhanced with Tools

  • ToolCoder interrupts decoding at an API token, searches the generated query, inserts the response, and resumes generation.The inserted sequence includes the API call, returned answer, and closing API token.Source: C. Inference enhanced with Tools · pp. 5–6
  • DuckDuckGo serves common public libraries, whereas BM25 retrieves documentation for lesser-known or private libraries.Search delay is controlled within 0.6s during generation.Source: C. Inference enhanced with Tools · p. 6
  • After decoding, ToolCoder removes text between the API delimiters so the final output contains generated code without search-call markup.This procedure targets API selection while preserving a regular code output format.Source: C. Inference enhanced with Tools · p. 6

V. EXPERIMENTAL SETUP

The experimental setup evaluates ToolCoder against state-of-the-art models on public and private library benchmarks, examines module contributions, and analyzes generated-code quality.

A. Research Question

  • The study compares ToolCoder with state-of-the-art models on three public-library benchmarks and tests generalization on two private-library benchmarks.These evaluations address public-library performance and private-library transfer separately.Source: A. Research Question · p. 6
  • An ablation study varies dataset, training, and inference-search settings to analyze the contributions of ToolCoder’s modules.The pipeline modules are data annotation, fine-tuning, and inference.Source: A. Research Question · p. 6
  • A case study evaluates generated-code quality and analyzes why the method works.This quality analysis is the fourth research question.Source: A. Research Question · p. 6

B. Datasets

ToolCoder is evaluated across three public and two private library benchmarks using pass@1 and pass@10, with GPT-3.5 included among the baselines.

B. Datasets

2) Private library benchmarks:

  • MonkeyEval and BeatNumEval conceal API names from training data and online search, so selection relies on documentation-based private-library search.MonkeyEval modifies PandasEval and BeatNumEval modifies NumpyEval.Source: 2) Private library benchmarks · pp. 6–7

C. Metrics

  • pass@k counts a problem as solved when any of k generated solutions passes all unit tests, with experiments using k = {1, 10}.The reported metrics are pass@1 and pass@10.Source: C. Metrics · p. 7

D. Baselines

  • The baseline set contains six series of recent code-generation models, including GPT-3.5 and both general and API-oriented models.The baselines include one of the most powerful models, GPT-3.5.Source: D. Baselines · p. 7

1) General Models:

ToolCoder is evaluated against general and API-oriented code-generation baselines using public-library benchmarks, with search-tool implementation details reported alongside the results.

1) General Models:

  • CodeT5, PyCodeGPT, CodeGen, and GPT-3.5 provide general-model baselines, while CERT and CodeGenAPI represent API-oriented approaches.CERT uses separate sketcher and generator modules; CodeGenAPI retrieves APIs before generation.Source: 1) General Models · p. 7, 2) API-oriented models · p. 7

E. Implementation Details

A. RQ1: Results for Public library API Code Generation

  • ToolCoder achieves the best results among the evaluated general and API-oriented baselines on public-library API generation.The results indicate that public-library API selection remains challenging for general code-generation models.Source: A. RQ1: Results for Public library API… · p. 8
  • 10.11%, 3.26%, and 1.39% pass@1 improvement over the best API-oriented baseline on three public-library benchmarks.ToolCoder also achieves comparable performance to GPT-3.5 on these benchmarks.Source: A. RQ1: Results for Public library API… · p. 8

B. RQ2: Results for Private library API Code Generation

On private-library benchmarks, ToolCoder uses documentation search and generalizes across new libraries, producing stronger aggregate results than API-oriented baselines.

B. RQ2: Results for Private library API Code Generation

  • The private-library evaluation switches from online search to documentation search because these libraries lack relevant online resources.The selected tool is therefore adapted to the library setting.Source: B. RQ2: Results for Private library… · p. 8
  • Documentation search gives ToolCoder stable generalization on MonkeyEval and BeatNumEval, where general code-generation models achieve low pass@1 and pass@10 scores.ToolCoder shows comparable performance to state-of-the-art API-oriented baselines on these private-library benchmarks.Source: B. RQ2: Results for Private library… · p. 8
  • 15.10% and 19.00% average pass@1 across five benchmarks exceed CodeGen-retrieval’s 8.89% by at least 6.21%.Average pass@10 also exceeds all API-oriented baselines by at least 9.64%.Source: B. RQ2: Results for Private library… · p. 8
  • ToolCoder-350M improves over CodeGen-350M by 2.03% and 1.58% on pass@1 and by 1.04% and 0.91% on pass@10.ToolCoder-2B shows similar improvement over CodeGen-2B, and some settings outperform GPT-3.5.Source: B. RQ2: Results for Private library… · p. 8

1) Dataset Setting:

Ablations show that annotated tool-use data, generated search queries, parameter-efficient training, and inference-time tools each contribute to ToolCoder’s performance.

1) Dataset Setting:

  • Removing annotation or generated queries causes substantial degradation, showing that tool-use annotation and query generation are essential to API selection.Using the problem description directly for search performs poorly because it is less specific than an API-focused query.Source: 1) Dataset Setting · pp. 8–9

2) Training Setting:

  • Training with LoRA keeps pass@1 within 0.4% of full training while reducing training time from 29h to 6h and trained parameters from over 350M to 0.65M.This trains 0.18% of CodeGen-350M parameters and 0.09% of CodeGen-2B parameters.Source: 2) Training Setting · p. 9

3) Inference Setting:

  • 1.88%, 2.57%, and 0.4% pass@1 gains for ToolCoder-350M and 2.87%, 0.29%, and 4.3% gains for ToolCoder-2B occur on public benchmarks when online search is enabled.Documentation search yields more significant gains on private libraries.Source: 3) Inference Setting · p. 9
  • On public benchmarks, NoTool can still perform relatively well because annotated tool calls provide API-selection reasoning during training.For private libraries, unseen library knowledge makes inference-time documentation search necessary for further improvement.Source: 3) Inference Setting · p. 9

D. RQ4: Qualitative analysis

The qualitative cases show how ToolCoder’s queries and search responses guide API selection and enable corrective code modifications, while validity is evaluated across multiple benchmark types.

D. RQ4: Qualitative analysis

  • Generated search queries provide fine-grained technical planning that bridges natural-language requirements and concrete APIs.In the NumpyEval case, the query decomposes dataframe summation by first focusing on summing arrays.Source: D. RQ4: Qualitative analysis · p. 9
  • Online search identifies APIs from relevant websites, while documentation search finds private-library APIs through API comments.Both tool responses contribute directly to the generated code.Source: D. RQ4: Qualitative analysis · p. 9, Figure 5 · p. 10
  • ToolCoder can modify a search response into the required API, producing np.cumsum when the online tool returns cumsum without the complete definition.This illustrates corrective use of retrieved suggestions rather than copying them unchanged.Source: D. RQ4: Qualitative analysis · pp. 9–10

VII. THREATS TO VALIDITY

  • External validity is assessed across five benchmarks covering public and private library API generation, while internal validity depends on architecture and hyperparameter choices.The study uses pass rates as its correctness metric.Source: §VII · p. 10

VIII. RELATED WORK

Prior code-generation research spans pretrained models and API-oriented pipelines, while tool-augmented language models had not yet incorporated programming tools. ToolCoder addresses this gap by integrating API search into decoding and reports improved benchmark performance.

A. Code Generation

  • Existing code-generation models include CodeGPT, CodeX, GPT-3.5, and CodeGen, with CodeGen selected as the base model for the experiments.The surveyed models are pretrained or adapted for generating source code from requirements, natural language descriptions, or pseudocode.Source: A. Code Generation · p. 10
  • API-oriented methods commonly use a two-stage search-then-generation pipeline, whereas ToolCoder integrates API search into decoding for closer alignment with the generated code fragment.The paper also claims that the integrated approach can adapt across API-related scenarios by changing the search tool.Source: A. Code Generation · p. 10

B. Tool-Augmented Large Language Models

  • Earlier tool-augmented language-model work used search, question-answering, translation, calculators, and other tools, but had not explored programming tools for code-related models.ToolCoder abstracts human programmers’ API-selection process into a programming tool for code generation.Source: B. Tool-Augmented Large Language Models · pp. 10–11

IX. CONCLUSION

  • ToolCoder integrates API search tools into code generation, using automatic tool-use annotation, parameter-efficient fine-tuning, and tool-enhanced decoding.The approach unifies online search engines and documentation search tools for API selection.Source: §IX · p. 11
  • ToolCoder outperforms state-of-the-art methods by at least 6.21% on average pass@1 across public and private library benchmarks.The conclusion identifies this result as evidence for incorporating programming tools into code generation.Source: §IX · p. 11
Loading 2305.04032v5…