LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation

• Existing code-completion studies do not fully capture public-library use, where changing, deprecated, and newly introduced APIs make version-specific accuracy important.Source: §1 · p. 1
• LibEvolutionEval provides version-specific code-completion tasks spanning multiple years for eight libraries: torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas.Source: §1 · p. 2
• The benchmark analyzes torch and matplotlib using version-specific metadata, documentation retrieval, and code-completion tasks in realistic GitHub and controlled documentation-based scenarios.Source: §1 · p. 2
• Direct completions have clear library prefixes, while indirect completions use object-oriented references without a library-defined prefix; API introduction, modification, and deprecation pose additional challenges.Source: §1 · p. 2
• Code LLMs and embedding models vary substantially as public libraries evolve; version-specific documentation improves completion but does not remove version-based bias.Source: §1 · p. 2

2.1 Version-Specific Evaluation Creation

• The benchmark pairs real-world GitHub examples with controlled documentation-derived tasks to study version-specific API completion.The controlled examples isolate API evolution from coding-style variation; the realistic examples retain real-world confounding variables.Source: §2.1 · p. 3, §2 · p. 3
• GitHub examples are versioned from requirements files or package dependency matrices, while controlled examples inherit the documentation version.The benchmark also collects API signatures, types, parameters, and usage examples from public documentation.Source: §2.1 · pp. 3–4
• Rule-based and model-based filters standardize completion contexts, while selection rules remove repeated APIs from the same line or source.Contexts include target-library imports and the initial API-expression regex; comments are removed to reduce API leakage.Source: §2.1 · pp. 3–4

2.2 API Data Classification

• Direct completions use import-derived prefixes; indirect completions require inferring API relationships through previously instantiated objects.The benchmark also labels APIs as introduced, deprecated, modified, or unchanged and links those labels to evaluation examples.Source: §2.2 · p. 4

2.3 LLM Context Classification

• The benchmark compares in-file context with version-aware prompting and version-specific retrieved documentation, which is selected by matching intent queries to API entries.Retrieval selects the top three matching APIs and formats their signatures and parameters as commented code.Source: §2.3 · pp. 4–5

2.4 Dataset Statistics and Scope

• Including code to the right of each completion supports fill-in-the-middle evaluation, while documentation metadata supports retrieval-augmented evaluation.Source: §2.4 · p. 5

• The code-completion benchmark uses Mistral, StarCoder2, GPT-4o-mini, and CodeGen, with API-call F1; retrieval models are assessed by version-specific documentation MRR.Scaling experiments include several StarCoder and StarCoder2 sizes and CodeSage Small and Large.Source: §3 · p. 5
• Inference uses uniform hyperparameters, an 8K-token sequence limit, nearest 4K-token context, and greedy decoding, followed by AST-based API-expression extraction.Generation is limited to 128 tokens, and generated text that continues into following source code is truncated.Source: §3 · pp. 5–6
• Figure 5 reports F1 scores for StarCoder2, Mistral, and GPT-4o-mini and states that performance varies significantly as libraries evolve.Source: Figure 5 · p. 6

Library evolution impacts code LLM performance

• Performance varies significantly across models and libraries as the eight evaluated libraries evolve.The realistic benchmark evaluates StarCoder2, Mistral, and GPT4o-mini.Source: §4 · p. 6

Version aware contexts enhance code LLM performance

• Version-aware retrieval improves completions, but model predictions still show bias across library versions.Retrieved documentation is specific to the library version; the remaining bias is attributed as likely stemming from uneven training-data distribution across versions.Source: §4 · pp. 6–7, Figure 6 · p. 7

Library evolution impacts documentation retrieval using embedding models

• Documentation retrieval fluctuates with library evolution, and scaling or updating models improves performance without eliminating that fluctuation or version bias.The retrieval comparison uses CodeSage-Small and OpenAI Ada on Torch and Matplotlib; the scaling analysis suggests versioned-data fine-tuning or explicit temporal API information as specialized approaches.Source: §4 · p. 7, Figure 6 · p. 7

Direct vs indirect API completion

• Models perform better on indirect than direct API completions; indirect examples include a corresponding parent direct API call in the left context.The authors posit that this context helps the model recognize the version-specific relationship needed for completion.Source: §4 · p. 7, Figure 6 · p. 7

Impact of different APIs on code completion

• Deprecated APIs underperform the overall set, except in 2018, when newly introduced APIs dominate and are now widely adopted.The controlled PyTorch and Matplotlib comparison and the realistic PyTorch version analysis both report weaker deprecated-API performance.Source: §4 · pp. 7–8, Table 4 · p. 8

Temporal analysis of model performance on introduced and deprecated APIs

• StarCoder2 struggles with Matplotlib APIs deprecated in 2019 and 2020, while CodeGen-1.0 declines on APIs introduced in 2023 and 2024, consistent with its 2022 knowledge cutoff.The comparisons use controlled Matplotlib API tasks.Source: §4 · p. 8, Table 4 · p. 8

5 Related Works

• Many code-completion benchmarks use in-file or repository-level contexts rather than the public-library complexity of real software development.The related-work discussion cites in-file studies [Chen et al., 2021; Athiwaratkun et al., 2023; Lu et al., 2021] and repository-level studies [Ding et al., 2023b; Zhang et al., 2023].Source: §5 · p. 8
• Prior work on public-library code completion does not address the libraries’ rapidly evolving nature [Liao et al., 2023; Zan et al., 2022; Qin et al., 2024; Patil et al., 2023].Source: §5 · p. 8
• LibEvolutionEval evaluates LLM code completion across multiple versions of public libraries, capturing their evolution and reflecting scenarios involving different library versions.Source: §5 · p. 8

6 Conclusion

• LibEvolutionEval is a benchmark for assessing code LLM performance on code completion as public libraries evolve.Source: §6 · p. 8
• Model performance varies significantly with API version, highlighting challenges in handling library evolution.Source: §6 · p. 8
• The authors call for code-completion advances that consider dynamic public libraries to improve productivity and accuracy in real-world settings.Source: §6 · p. 8

A Limitations

• The evaluation is zero-shot; version-specific pretraining might reduce observed discrepancies, and prior exposure to evaluation code is possible.The authors advise considering potential overlap with the vast unlabeled code repositories used to train CodeLMs when interpreting results.Source: A Limitations · p. 12

B LIBEVOLUTIONEVAL Generation

• Realistic examples are extracted from Python files by parsing ASTs to identify direct and indirect API calls and their surrounding code context; controlled examples are synthesized from versioned documentation.The extraction tracks variable scopes and aliases for indirect calls, while the controlled template provides service name, API description, and mandatory arguments without revealing the API name.Source: B LIBEVOLUTIONEVAL Generation · pp. 12–13

C Generating Natural Language Instructions Using Claude

• The Claude prompt template instructs the model not to include the target completion’s name or required arguments in its natural-language instruction.Source: Figure 10 · p. 13

D Retrieval Performance vs Model Size on LIBEVOLUTIONEVAL

• The figure reports that larger CodeSage models perform better at documentation retrieval.Source: Figure 11 · p. 13

E Code Completions Performance vs Model Size on LIBEVOLUTIONEVAL

• The figure reports that larger StarCoder and StarCoder2 models perform better at code completion.Source: Figure 12 · p. 13

• Table 5 organizes library API examples by year.Source: Table 5 · p. 13

Matplotlib Library Ablation Data

• Table 6 compares the Matplotlib-focused dataset with the comprehensive dataset.Source: Table 6 · p. 14

PyTorch Library Data

• Table 7 presents PyTorch API usage data.Source: Table 7 · p. 14

F.1 PyTorch Documentation Data

• Table 8 presents Torch version changes over time.Source: Table 8 · p. 14

F.2 Matplotlib Documentation Data

• Table 9 presents API changes across versions.Source: Table 9 · p. 14

G Visualizing prompts given to CodeLMs in LIBEVOLUTIONEVAL

• Figure 13 visualizes prompts given to Code LLMs in LibEvolutionEval.Source: Figure 13 · p. 14