PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating In-Context Learning of Libraries for Code Generation

Arkil Patel, Siva Reddy, Dzmitry Bahdanau, Pradeep Dasigi

arXiv:2311.09635v2cs.CL

TL;DR

Prior work leaves open whether novel libraries can be learned in context without demonstrations and by smaller, open models. This paper evaluates library learning across different scenarios and finds that smaller open models can learn libraries, including from descriptions or implementations, though constrained use remains difficult.

  • Problem

    It is unclear whether library learning in context requires demonstrations or is also possible for smaller, openly accessible models.

    Source: §1 · p. 1

  • Method

    The authors define novel libraries or a target language in context, pair specifications with task instructions, then execute generated code and evaluate correctness.

    Source: §1 · p. 2

  • Results

    Smaller open models also learn novel libraries, descriptions or implementations can match demonstrations, and constrained library use reduces performance.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    In-context library learning may support rapid adaptation without paired demonstration data.

    Source: §6 · p. 9

    Each scenario is evaluated in only one domain, library, or language.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Contemporary Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. A particularly promising area is their ability to interpret code modules from unfamiliar libraries for solving user-instructed tasks. Recent work has shown that large proprietary LLMs can learn novel library usage in-context from demonstrations. These results raise several open questions: whether demonstrations of library usage is required, whether smaller (and more open) models also possess such capabilities, etc. In this work, we take a broader approach by systematically evaluating a diverse array of LLMs across three scenarios reflecting varying levels of domain specialization to understand their abilities and limitations in generating code based on libraries defined in-context. Our results show that even smaller open-source LLMs like Llama-2 and StarCoder demonstrate an adept understanding of novel code libraries based on specification presented in-context. Our findings further reveal that LLMs exhibit a surprisingly high proficiency in learning novel library modules even when provided with just natural language descriptions or raw code implementations of the functions, which are often cheaper to obtain than demonstrations. Overall, our results pave the way for harnessing LLMs in more adaptable and dynamic coding environments.

1 Introduction

The introduction asks whether in-context library learning can use supervision other than demonstrations and extend to smaller open models and uncommon programming languages. It outlines an execution-based evaluation and reports that models learn libraries from descriptions or implementations, while constrained library use remains difficult.

  • The study asks whether easier-to-obtain supervision can replace demonstrations, whether smaller open models can use novel libraries, and whether models can learn uncommon languages in context.Source: §1 · p. 1
  • The evaluation supplies a library or language specification with each task instruction, executes generated code, and evaluates its correctness.The framework also tests different forms of in-context library supervision, including natural-language descriptions and code implementations.Source: §1 · p. 2
  • Open models such as Llama-2 and StarCoder learn novel libraries, while GPT-4 can learn from natural-language descriptions or raw implementations as effectively as from demonstrations.Source: §1 · p. 2
  • Models struggle when constrained to use in-context library functions and perform better when code generation is unconstrained.Source: §1 · p. 2
  • GPT-4 shows non-trivial in-context ability to learn a new programming language from scratch.Source: §1 · p. 2

2 Related Work

The related work distinguishes this study from general code-generation benchmarks and API/tool-use research by focusing on in-context learning of complex libraries and programming languages for reasoning tasks. It also examines supervision formats and capabilities that prior work had not covered.

  • Unlike general code-generation benchmarks, this work evaluates code generation based on novel libraries defined in context.Source: §2 · p. 2
  • Prior API and tool-use studies mostly address primitive calls, whereas this work studies complex libraries and languages for non-trivial reasoning through in-context learning.It focuses on in-context learning rather than retrieval-based or finetuning approaches.Source: §2 · p. 2
  • Related documentation studies provide context for library use, while this work also tests raw function implementations and learning a language from keyword descriptions.The authors distinguish their task-specific in-context approach from retrieval-based work and note that their findings do not require partial function-call examples.Source: §2 · p. 3

3 Learning Novel Libraries

Across four VisProg vision-language tasks, models generally learned the novel library from demonstrations, while GPT-4 performed comparably with descriptions or implementations. Results also varied with pretraining and function-name aliasing.

3.1 Experimental Setup

  • The evaluation covers four vision-language tasks using VisProg, a 20-module library for image and language operations.The tasks are GQA, NLVR, Knowledge Tagging, and Image Editing.Source: §3.1 · p. 3
  • The three supervision formats are paired demonstrations, natural-language function documentation, and Python implementations, with the latter two generally cheaper to obtain.For descriptions and implementations, models receive no example programs or domain-instruction information and must select needed functions from the full library.Source: §3.1 · pp. 3–4
  • Evaluation uses answer accuracy for GQA and NLVR, tagging F1 from precision and recall at IoU 0.5, and manual correctness checks for Image Editing.A code interpreter executes generated programs and fails when VisProg functions are used incorrectly.Source: §3.1 · p. 4

3.2 Results

  • Most models learned VisProg from demonstrations, and many also learned from descriptions or implementations; GPT-4 performed comparably with demonstrations, which remained best across tasks.Adding random programs without paired inputs to descriptions or implementations improved performance.Source: §3.2 · pp. 4–5
  • Pretraining data aligned with preferred supervision: LLaMA models leveraged descriptions more than implementations, whereas StarCoder leveraged implementations better despite its smaller size.StarCoder performed extremely poorly with natural-language descriptions.Source: §3.2 · p. 5
  • Aliasing function names produced model-specific effects: GPT-3.5 performance dropped even with synonyms, Llama-2 was similar with synonyms but dropped with random strings, and GPT-4 was robust to both.Table 2 reports performance for original (OG), synonymous (SYN), and random-string (RAN) names.Source: Table 2 · p. 4, §3.2 · p. 5

4 Constrained Generation Using Functions Defined In-Context

The section evaluates whether models can generate code under in-context library constraints, using a custom library extracted from existing solutions. Models’ execution accuracy falls under constraints, while supervision format yields similar performance and GPT-4 follows function-use instructions better than GPT-3.5-Turbo.

4.1 Experimental Setup

  • The evaluation covers natural-language-to-Python tasks drawn from 15,000 CodeContests and APPS examples, with execution accuracy and function usage measured separately.The task represents software development in which the model is constrained to use specified library functions.Source: §4.1 · pp. 5–6, §4 · p. 5
  • The study compares demonstrations, descriptions, and direct Python implementations as in-context supervision formats, using GPT-4 and GPT-3.5-Turbo.The authors restrict this scenario to the latest GPT models because preliminary experiments found the problems too difficult for smaller open models in context.Source: §4.1 · pp. 5–6

4.2 Creating a Library of Functions

  • The custom library combines aliased existing Python-library functions with user-defined functions extracted from candidate solutions.The dataset contains 12,000 examples constrained to existing-library functions and another 3,000 constrained to user-defined functions.Source: §4.2 · p. 6

4.2.1 Obtaining Specification Data

  • Function descriptions come from API documentation or GPT-4-generated documentation, while demonstrations are GPT-4-generated instruction-program pairs.Implementations come from available source code or existing user-defined code; GPT-4 supplies Python implementations when library source is in another language.Source: §4.2.1 · pp. 6–7

4.3 Results

  • Execution accuracy falls significantly for GPT-4 and GPT-3.5-Turbo when they must use specified functions, regardless of whether the generated code uses those functions.The authors hypothesize that prior familiarity with standard-library functions makes it harder for models to learn new functions with similar roles.Source: §4.3 · p. 7
  • Performance is similar with demonstrations, natural-language descriptions, or implementations, but GPT-4 follows function-use constraints while GPT-3.5-Turbo mostly ignores them.The Function Usage % difference between the models is described as large; no exact values are supplied here.Source: §4.3 · p. 7

5 Learning a New Programming Language

The study tests whether models can learn unfamiliar Isabelle for theorem proving from in-context supervision, including aliased keywords. Models show some learning from demonstrations despite aliasing, while description-only learning is most evident in GPT models and unpaired programs help up to a saturation point.

5.1 Experimental Setup

  • The study evaluates formal-proof generation for 140 algebra problems from miniF2F, using Isabelle, a language selected for its limited online paired data and small keyword set.Prompts provide language supervision, an Isabelle statement, and an informal natural-language proof sketch; correctness is checked with Isabelle and Sledgehammer.Source: §5.1 · pp. 7–8
  • The setup compares eight demonstrations with natural-language descriptions of Isabelle’s 12 keywords, and aliases keywords with random strings to reduce possible prior-familiarity effects.The models include GPT-4, GPT-3.5-Turbo, LLaMA-2 70B, CodeLlama, and StarCoder.Source: §5.1 · p. 8

5.2 Results

  • All models show substantial Isabelle-learning capability from demonstrations, and performance remains nonzero after keywords are aliased, though it decreases.The authors note that familiarity with Isabelle may partly explain performance without aliasing.Source: §5.2 · p. 8, Figure 7 · p. 8
  • Only GPT models show non-trivial performance from keyword descriptions alone; GPT-4 remains significant under aliasing, though performance diminishes.The authors caution that high zero-shot GPT-4 performance makes prior exposure to Isabelle during training likely.Source: §5.2 · p. 8, Table 4 · p. 8
  • Performance with synonymous keyword aliases is similar to performance with random aliases, suggesting learning depends little on keyword semantics.The comparison is presented in Figure 7.Source: §5.2 · p. 8, Figure 7 · p. 8
  • With description-based supervision and aliased keywords, exposure to unpaired Isabelle programs increases learning, then saturates after 5–7 programs, possibly due to overfitting.The unpaired programs are proofs without their corresponding inputs.Source: §5.2 · pp. 8–9, Figure 8 · p. 9

6 Discussion

The discussion reports that strong models can learn novel libraries from descriptions or implementations, and that this ability is not limited to proprietary models. It also highlights supervision-format sensitivity, difficulty obeying library constraints, and preliminary in-context learning of new languages.

6 Discussion

  • Strong LLMs learn novel libraries from natural-language descriptions and code implementations, while Llama-2 and StarCoder also show substantial in-context learning without fine-tuning.The authors present these findings as promising for adapting models without paired demonstrations.Source: §6 · p. 9
  • For smaller open models, the supervision format is especially consequential, informing which model or supervision type to choose in resource-constrained settings.The paper specifically contrasts natural-language descriptions with programming-language implementations.Source: §6 · p. 9
  • When models can generate code from their priors, constraining them to use in-context library functions degrades performance and exposes a bias toward familiar code patterns.The authors report that models find it difficult to use the provided functions in such cases.Source: §6 · p. 9
  • LLMs show preliminary but promising ability to learn new programming languages in context from demonstrations or language descriptions.Source: §6 · p. 9

Limitations

The evaluation covers only one domain or library/language per scenario. The authors also identify potential bias in manually crafted and automatically generated supervision, possible unsafe outputs, inherited dataset biases, and limited reproducibility of OpenAI results.

Limitations

  • Each of the three scenarios evaluates only one domain or library/language, limiting the breadth of the study’s coverage.The authors plan to consider more domains to strengthen the results.Source: Limitations · p. 9
  • Manually crafted description and implementation prompts may bias results; the study is exploratory rather than a proposal for general supervision approaches.One author crafted the prompts and the others verified them.Source: Limitations · p. 9
  • LLM-created descriptions and implementations may slightly bias the dataset, though the authors expect minimal effect for objective competitive-programming tasks.Source: Limitations · p. 9
  • Automatic data creation without checking every data point could produce unsafe responses, and source-dataset biases may transfer to the evaluation suite.Source: Limitations · p. 9
  • Results for OpenAI models may not be directly reproducible.Source: Limitations · p. 9

A Implementation Details

The implementation details specify model execution, prompt lengths, proof-checking settings, and the datasets used in the experiments.

A Implementation Details

  • GPT-3.5-Turbo and GPT-4 experiments used the OpenAI API; other experiments ran on eight NVIDIA A100 GPUs with 80 GB memory.The code uses PyTorch, HuggingFace Transformers, and, for certain open-source models, Huggingface Text-Generation-Inference.Source: A Implementation Details · p. 13
  • VisProg prompt contexts were approximately 2,000 tokens for demonstrations, 500 for descriptions, and 2,400 for implementations.Source: A Implementation Details · p. 13
  • Isabelle proofs were evaluated with Portal-to-ISAbelle on Isabelle2021 using default Sledgehammer settings and a 120-second timeout.Source: A Implementation Details · p. 13
  • The visual-question-answering datasets include GQA (1,460 test examples) and NLVR (6,967); Knowledge Tagging has 100 examples and Image Editing 107.GQA is compositional multi-step visual question answering; NLVR reasons over image pairs, while the other datasets involve image tagging or editing.Source: A Implementation Details · p. 13

B Cyclic Evaluation Process to Obtain Correct Descriptions and Implementations

The cyclic process checks generated function descriptions and implementations by inserting model-generated code into candidate solutions and testing correctness.

B Cyclic Evaluation Process to Obtain Correct Descriptions and Implementations

C.1 Zero-shot Baselines

The zero-shot baselines use task-specific alternatives when code generation cannot be evaluated directly. In a separate VisProg experiment, adding unpaired programs improves performance across models and both supervision formats.

C.1 Zero-shot Baselines

  • Because zero-shot code generations could not be executed reliably, GQA was tested by answering questions directly, NLVR used a 51.1% majority-class baseline, and complex tagging and editing tasks were assigned 0%.Execution problems included assumed file paths and library dependency issues.Source: §C.1 · p. 14
  • The zero-shot language baseline prompted models to generate Isabelle proofs for algebra problems in miniF2F; Table 6 reports proof correctness.Source: §C.1 · p. 14

C.2 Impact of Providing Unpaired Programs for VisProg

  • All tested models improved on GQA when given 10 unpaired VisProg programs with descriptions or implementations, especially where baseline performance was very low.The particularly noticeable cases were StarCoder with descriptions and Llama-2 with implementations.Source: Figure 10 · p. 14, Figure 11 · p. 14, §C.2 · p. 14

D Examples of Errors

The section illustrates GPT-4 errors when learning VisProg and Isabelle, while the example prompts show several ways to present novel libraries and tasks in-context.

D Examples of Errors

  • For the experiments in Section 4, the authors found no concrete error pattern; most failures came from incorrect application of the in-context function.Source: D Examples of Errors · p. 14

E Example Prompts

  • The code-generation examples require calling novel API functions, illustrated with heap insertion and functions described by their behavior and argument or return types.The prompt explicitly says the functions are already implemented and must be called in the solution.Source: E Example Prompts · pp. 15–16
  • Example prompts cover constrained generation from function demonstrations or descriptions, plus Isabelle learning from demonstrations or keyword descriptions, with and without aliasing.Source: E Example Prompts · p. 14, Figure 12 · p. 15, Figure 13 · p. 16

D Examples of Errors

  • GPT-4’s VisProg errors include misusing helper functions and passing OBJECT values to the wrong argument; its Isabelle errors include inventing a keyword and faulty calculation.The VisProg examples show failures to use LOC and CROP to focus on image regions, and a type mismatch involving the image argument.Source: D Examples of Errors · p. 14, Table 7 · p. 19, Table 8 · p. 19
Loading 2311.09635v2…