PaperlayerView exact revision on arXiv

Source-linked AI summary

API Pack: A Massive Multi-Programming Language Dataset for API Call Generation

Zhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen, Rameswar Panda

arXiv:2402.09615v6cs.CLcs.AIcs.LG

TL;DR

Generating specific API-call code, particularly across languages with differing HTTP syntax, has received limited attention despite the time developers spend finding and adapting documentation examples. API Pack provides over one million instruction/API-call instances across 10 programming languages, with API and endpoint information. Fine-tuning CodeLlama-13B on 20,000 Python instances outperforms GPT-3.5 and GPT-4 on unseen API calls; scaling training data to 1 million improves generalization, and cross-language API generation is achievable.

  • Problem

    Generating specific API-call code, particularly across languages with differing HTTP syntax, has received limited attention despite the time developers spend finding and adapting documentation examples.

    Source: §1 · p. 1

  • Method

    API Pack provides over one million instruction/API-call instances across 10 programming languages, with API and endpoint information.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Fine-tuning CodeLlama-13B on 20,000 Python instances outperforms GPT-3.5 and GPT-4 on unseen API calls; scaling training data to 1 million improves generalization, and cross-language API generation is achievable.

    Source: §6 · p. 10

  • Takeaways & Limitations

    API Pack provides a basis for studying cross-language skill transfer and how training-data volume affects generalization to new APIs.

    Source: §1 · p. 1

    API Pack is not designed for multi-API calls and requires explicit API-name specification, which may limit use in complex software-development scenarios.

    Source: §6 · p. 10

Abstract

from arXiv · show

We introduce API Pack, a massive multi-programming language dataset containing over one million instruction-API calls for improving the API call generation capabilities of large language models. Our evaluation highlights three key findings: First, fine-tuning on API Pack enables open-source models to outperform GPT-3.5 and GPT-4 in generating code for entirely new API calls. We show this by fine-tuning CodeLlama-13B on 20,000 Python instances from API Pack. Second, fine-tuning on a large dataset in one language, combined with smaller datasets from others, improves API generation accuracy across multiple languages. Third, we confirm the benefits of larger datasets for API generalization, as increasing fine-tuning data to one million instances enhances generalization to new APIs. To support further research, we open-source the API Pack dataset, trained model, and code at https://github.com/zguo0525/API-Pack.

1 INTRODUCTION

The introduction motivates API-call code generation as a less-studied challenge than API intent detection and presents API Pack as a large, multilingual dataset for studying it. Experiments report gains on unseen APIs, cross-language generation, and data-scale generalization.

1 INTRODUCTION

  • API Pack contains over one million instances across 10 programming languages, addressing the limited attention to generating specific API-call code.The motivation is the time developers spend finding and adapting API examples in lengthy documentation.Source: §1 · p. 1
  • Fine-tuning CodeLlama-13B on 20,000 Python instances outperforms GPT-3.5 by 10% and GPT-4 by 5% on unseen API calls.The comparison demonstrates the result on API calls not seen during training.Source: §1 · p. 2
  • Cross-language API call generation can be enabled by extensive data in one language combined with smaller datasets from other languages.API Pack’s language diversity supports the paper’s first comprehensive study of cross-language skill transfer in API generation.Source: §1 · pp. 1–2
  • Increasing fine-tuning data from 0 to 1 million instances improves generalization to new APIs.The paper presents this as evidence of benefits from larger datasets for API generalization.Source: §1 · p. 2

2 RELATED WORK

Prior work automates instruction-data creation and studies API intent detection, while API-call code generation remains comparatively underexplored. API Pack’s related-work discussion distinguishes these approaches and identifies the narrower scope of prior code-generation projects.

2.1 METHODS TO GENERATE INSTRUCTION DATA WITH LLMS

  • Self-Instruct and Evol-Instruct automate instruction-response generation and filtering, while hybrid code-data pipelines combine human code samples with LLM-generated instructions.Human verification is described as impractical at scale, motivating automated LLM judging in recent work.Source: §2.1 · p. 2

2.2 LLMS FOR API CALL CODE GENERATION AND INTENT DETECTION

  • API intent detection maps natural-language tasks to endpoints, often leaving code generation to other components; strong multi-API intent detection remains challenging.The passage notes that this is likely how most function-calling LLMs have been trained to operate.Source: §2.2 · p. 3
  • API-call code generation is less explored than intent detection, with Gorilla [Patil et al., 2023] and ToolBench cited as prior projects.Gorilla targets calls to load pretrained machine-learning models from known hubs, while ToolBench covers tool manipulation for a few applications.Source: §2.2 · p. 3

3 API PACK

API Pack is built from API specifications and contains instruction/API-call examples with API and endpoint information. Its construction combines preprocessing, multilingual API-call generation, instruction creation, and validation.

3 API PACK

  • API Pack contains over one million instruction instances pairing task instructions with HTTP request code snippets and API and endpoint information.The data comes from four API hubs storing OpenAPI specification files; Table 2 reports final counts by source, with one API call per instance.Source: §3 · p. 3, Table 2 · p. 3
  • The dataset curation pipeline has four stages: preprocessing, API database creation, instruction generation, and data validation.Figure 1 illustrates the overall pipeline.Source: §3 · p. 3, Figure 1 · p. 4

3.1 DATA PRE-PROCESSING

  • Preprocessing filters out non-English or endpoint-free specifications and removes records missing information needed for API calls or instructions.The remaining records supply API-level and endpoint-level fields, including names, descriptions, methods, and paths.Source: §3.1 · p. 3

3.2 CREATE API DATABASES

  • The API database combines endpoint and API details with generated calls in 10 programming languages for supported sources.IBM API Hub calls are extracted directly from specifications; the language selection reflects StarCoder-report coverage and library support.Source: §3.2 · p. 4

3.3 INSTRUCTIONS GENERATION

  • Instruction generation starts from three selected endpoints per API database file, then uses refined examples as in-context prompts to generate five candidates per instance.Authors manually refine examples for API Gurus and IBM API Hub, while an LLM refines those for Swaggerhub and RapidAPI.Source: §3.3 · p. 4

3.4 DATA VALIDATION

  • Validation checks API-call consistency and instruction quality, then selects the best good instruction candidate using its likelihood of regenerating the input text.The final dataset contains 1,128,599 instances, each with a valid API call and at least two high-quality instructions.Source: §3.4 · pp. 4–5

4 EXPERIMENTS AND EVALUATION FRAMEWORK

The experiments compare fine-tuned models, retrieval settings, cross-language training mixtures, and dataset sizes. Evaluation tests API-call generation across three levels of API familiarity using a sequence-similarity measure, with privacy constraints limiting execution-based assessment.

4.1 EXPERIMENTAL SETTINGS

  • The study selects a base model by fine-tuning four candidates on 20,000 Python instances and evaluating their performance.The selected best-performing model is used in subsequent experiments.Source: §4.1 · p. 5
  • Retrieval experiments compare 0-shot and several 3-shot settings, including random, retrieved, re-ranked, and oracle examples.The re-ranked setting supplies five retrieved examples before selecting three for the final prompt.Source: §4.1 · p. 5
  • Cross-language generalization is tested by adding 1,000 instances from each of nine languages to a 100,000-instance cURL dataset.The same API functionality across languages allows assessment of whether models can translate the same invocation between languages.Source: §4.1 · pp. 5–6
  • The scaling experiment fine-tunes on datasets ranging from 0 to 1 million unique-API-call instances to test generalization to unseen APIs.The tested sizes are 0k, 10k, 20k, 40k, 80k, 100k, and 1 million.Source: §4.1 · p. 6

4.2 EVALUATIONS

  • Evaluation covers seen APIs and endpoints, new endpoints for seen APIs, and entirely unseen APIs and endpoints.Generated endpoint and API-call accuracy use a sequence similarity ratio, with outputs deemed correct at a threshold of 0.9.Source: §4.2 · p. 6
  • The evaluation measures structural and semantic correctness rather than exact syntax, and privacy-driven PII removal makes execution-based metrics difficult.Pass rate and hallucination rate are identified as potentially more robust alternatives.Source: §4.2 · p. 6

5 EXPERIMENTAL RESULTS

Fine-tuning on 20,000 Python API Pack instances lets open-source models match or exceed proprietary models on unseen API calls, with CodeLlama-13b strongest among the evaluated models. Retrieval, mixed-language training, and larger datasets each show benefits, though reranking and scaling gains have limits.

5.1 FINE-TUNED CODELLAMA EXCELS IN API CALL GENERATION

  • Fine-tuned open-source models match or exceed leading proprietary models on unseen API calls, although GPT-3.5 and GPT-4 are evaluated only at Level 3.Table 3 compares models fine-tuned on 20,000 Python instances; proprietary fine-tuning cost constrained their evaluation to Level 3.Source: §5.1 · p. 6, Table 3 · p. 7

5.2 RETRIEVAL AUGMENTATION IMPROVES API CALL GENERATION

  • Retrieved examples outperform random examples across evaluation levels, while generic reranking does not consistently improve results and can lower CodeLlama-13b Level 1 accuracy from 63.5% to 61.0%.Oracle results indicate that better example selection can improve performance, but the current reranker fails to capture those gains.Source: Table 4 · p. 7, §5.2 · p. 8
  • Retrieval-method performance gaps are largest for Level 1 and narrow for Levels 2 and 3; endpoint accuracy also exceeds full API-call accuracy across conditions.The paper attributes the latter gap to the additional need to specify parameters, their types, and formatting.Source: §5.2 · p. 8

5.3 CROSS-LANGUAGE GENERALIZATIONS IN API CALL PERFORMANCE

  • Without fine-tuning, models struggle to generalize to new languages in zero-shot settings, while three-shot performance improves significantly.The findings support cross-language generation using a large dataset in one language and smaller datasets in others, without extensive data in every target language.Source: §5.3 · pp. 8–9
  • A mixture trained on 100,000 cURL instances plus 1,000 instances from each of nine languages performs comparably to language-specific experts in three-shot tests across ten languages.Figure 3 presents three-shot performance across the ten languages; the compared instances invoke the same API functionality across languages.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 SCALING INSTRUCTION DATASET HELPS GENERALIZATION

  • As API Pack fine-tuning data grows from 0 to one million instances, zero-shot endpoint and API-call accuracy rise roughly linearly across all three levels, while three-shot gains are smaller.Figure 4 plots endpoint or API-call accuracy against the log-scale size of the fine-tuning dataset.Source: §5.4 · p. 9, Figure 4 · p. 9
  • With three-shot prompting, Level 1 improves more gently as data grows, while Levels 2 and 3 plateau near one million instances, most clearly at Level 3.The authors identify this pattern as indicating possible diminishing returns for the most difficult generalization task.Source: §5.4 · p. 9

5.5 ADDITIONAL EXPERIMENTS

  • With CodeLlama-13b, ToolBench-only training reduces performance in both 0-shot and 3-shot settings versus four-source API Pack; mixing 50,000 API Pack entries with Magicoder raises Level 3 3-shot API-call code-generation accuracy over 35.3%.The mixture still performs well on HumanEval+ and MBPP.Source: §5.5 · p. 10

6 CONCLUSION AND FUTURE WORK

API Pack contains over one million instruction–API call pairs across programming languages, and the paper reports gains in unseen-API generation, data scaling, and cross-language generation. The authors identify limitations and outline plans to address them.

6 CONCLUSION AND FUTURE WORK

  • Combining extensive training data in one language with smaller amounts from others supports cross-language API call generation.The authors also report that increasing training data to one million instances improves generalization to unseen APIs.Source: §6 · p. 10
  • Fine-tuning CodeLlama-13B on 20,000 Python instances enables it to outperform GPT-3.5 and GPT-4 on unseen API calls.The conclusion reports that model performance continues to improve as training data scales to one million instances.Source: §6 · p. 10
  • API Pack is not designed for multi-API call scenarios, and requiring explicit API names may limit use in complex software development.The authors also note that limited API-level license information may restrict proprietary use.Source: §6 · p. 10
  • Planned work includes privacy-preserving execution-based evaluation, multi-API and task-based instructions, broader language coverage, and improved API license documentation.Execution-based metrics such as pass rates and hallucination rates are difficult under the current focus on privacy protection and PII removal.Source: §6 · p. 10

A.1 ETHICAL AND CONSIDERATIONS

The authors discuss API currency, generated-code verification, ownership and licensing, and sensitive information as considerations for users of API Pack.

A.1 ETHICAL AND CONSIDERATIONS

  • APIs in the dataset may evolve, making some examples outdated or potentially deprecated.The authors recommend periodically updating the dataset or using retrieval-augmented generation to access current API specifications.Source: §A.1 · p. 15
  • Fine-tuned model outputs still require developer verification because hallucination remains a general risk of LLMs.The API Pack code itself was not generated by an LLM, which the authors say lowers the risk of dataset code errors due to LLM hallucination.Source: §A.1 · p. 15
  • Limited API license information may affect proprietary use, although the API data was sourced from public and official hubs.The authors say they will work toward complete and accurate license information in a future dataset version.Source: §A.1 · p. 15
  • The authors say API Pack data poses no sensitive-information release risk because its calls use argument placeholders rather than real argument values.Some calls include public or restricted API URLs released directly by API owners.Source: §A.1 · p. 15

A.2 BROADER IMPACT

The authors describe API Pack as a potential component of software development processes through LLMs specialized in API call generation. They discuss possible productivity benefits alongside risks and human oversight.

A.2 BROADER IMPACT

  • Automating routine coding tasks could accelerate software development, reduce costs, and potentially improve product quality.The authors also warn that reliance on generated code risks reducing developer accountability and control, so human review remains necessary.Source: §A.2 · p. 16
  • The authors frame API Pack as supporting human-in-the-loop use: developers decide whether to use generated API calls.They connect this design to concerns about job displacement, ownership, and digital inequality.Source: §A.2 · p. 16
  • The authors open-sourced the dataset, code, and models as a step toward reducing digital inequality.Source: §A.2 · p. 16

A.3 DATA FILTERING AT EACH STAGE OF THE PIPELINE

Table 7 presents instance counts by API source across stages of the data-filtering pipeline; its detailed stage descriptions are in Section 3.

A.3 DATA FILTERING AT EACH STAGE OF THE PIPELINE

A.4 DATA SPLITING FOR LEVEL 1, LEVEL 2, AND LEVEL 3 TESTING

The testing split creates three levels of held-out data using different entry ranges and API-selection criteria.

A.4 DATA SPLITING FOR LEVEL 1, LEVEL 2, AND LEVEL 3 TESTING

  • Level 1 selects up to 1,000 examples from the first 20,000 entries when the instruction test is not “None” and the API occurs more than four times.Source: §A.4 · p. 16
  • Level 2 selects examples from the first 20,000 entries until it has 1,000 unique APIs, then removes those APIs’ examples from training.Source: §A.4 · p. 16
  • Level 3 selects up to 1,000 examples from the 80,000th entry onward using APIs not selected for Level 2, then removes those APIs’ examples from training.Source: §A.4 · p. 16

A.5 HYPERPARAMETERS FOR TRAINING

This section documents API Pack’s instruction creation and quality assessment, API-instance representation, retrieval pipeline for testing, and workflows for instruction generation, backtranslation, annotation, few-shot fine-tuning, and evaluation.

A.6 TESTING PIPELINE

  • Testing retrieves semantically similar examples with cosine similarity, reranks them, and supplies none for 0-shot or the three most relevant for 3-shot inference.The training setup used HuggingFace Transformers on one node with eight NVIDIA H100 80GB GPUs and FSDP, with mixed precision, gradient checkpointing, and AdaFactor for efficiency.Source: §A.6 · p. 17, §A.5 · p. 17

A.7 API DB INSTANCE

  • API DB instances record API and endpoint metadata, HTTP request code, and its programming language; calls were generated from several API sources across 10 languages, with IBM Developer Hub providing eight.The instance fields include API name and description, provider, endpoint name and functionality, path, HTTP method, API call, and language.Source: §A.7 · p. 18

A.10 QUALITY ASSESSMENT OF COLLECTED INSTRUCTIONS

  • Most test instructions were rated High or Medium by GPT-4o-mini, and Level 1 had the largest proportion of High ratings.The evaluation judged naturalness and clarity; Figure 9 shows the distributions across Levels 1, 2, and 3.Source: §A.10 · p. 20, Figure 9 · p. 20

A.11.1 EXAMPLES REFINEMENT

  • The authors created three examples per API and refined them manually for IBM API Hub and APIs Gurus, using an LLM for examples from the other two sources.The refinement prompt targets clear, concise, grammatical, user-centered queries that include relevant API details without naming the endpoint.Source: §A.11.1 · p. 23

A.11.2 INSTRUCTION GENERATION

  • For instruction generation, the authors created five candidates per API Pack instance using a prompt emphasizing API name, specificity, concision, and excluding the endpoint.Candidates are illustrated for a single instance; the prompt aims to reflect realistic developer requests for an API call.Source: §A.9 · p. 19, Figure 8 · p. 19, §A.11.2 · p. 24
  • Instruction-generation prompts ask for realistic scenarios that show how a user benefits from a specific API, using precise and clear language.Source: §A.11.2 · p. 25

A.11.3 BACK TRANSLATION

  • Prompt 4 reverses Prompt 3 to regenerate API information from an instruction candidate, and inference saves the input-token mean of the returned API information.Source: §A.11.3 · p. 25

A.11.4 INSTRUCTION ANNOTATION

  • After manually reviewing 605 instructions to identify common errors, the authors crafted an LLM prompt to classify candidates as good or bad, with examples drawn from that review.Bad instructions may combine multiple directives, add unnecessary text, or misuse the API name or endpoint.Source: §A.11.4 · p. 26

A.11.5 FEW-SHOT FINE-TUNING AND EVALUATION

  • A shared 3-shot prompt template is used to post-process API Pack data and for random, retrieved, and retrieved-and-reranked few-shot inference.Source: §A.11.5 · p. 27
Loading 2402.09615v6…