PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridge: Automatically Mining Ecosystem-Scale API Update Mappings and Client Update Instances

Kai Gao, Yu Sun, Chang-ai Sun

arXiv:2608.30497v1cs.SE

TL;DR

Automated library-update research lacks datasets that jointly represent API replacements, applicable version transitions, and real client adaptations across broad ecosystems. Bridge mines and validates update instances from client dependency commits, then derives grounded mappings; at scale it produces hundreds of thousands of instances, while replacement recommendation remains difficult for long-tail mappings.

  • Problem

    Existing datasets incompletely provide API update mappings, version transitions, and client update instances, and typically cover few third-party libraries.

  • Method

    Bridge mines candidate update instances from client dependency update commits, validates them with library-side evidence, and derives API update mappings from validated instances.

  • Results

    381,661 Java and 277,259 Python client update instances represent 18,900 and 4,456 API update mappings across 2,557 and 999 libraries, respectively.

  • Takeaways & Limitations

    Replacement API recommendation remains difficult for long-tail mappings, with evaluated LLMs performing substantially better on frequently observed mappings.

  • Takeaways & Limitations

    Recommendation accuracy may differ with other models, prompts, or input settings, and frequency–accuracy associations are not causal evidence.

Abstract

from arXiv · show

Library updates often require adapting client code to API changes. API update mappings that identify relations between legacy and replacement APIs, version transitions that these mappings apply, and client update instances that capture concrete API call changes are essential for developing and evaluating automated library update techniques. Existing library evolution datasets capture only subsets of this information and typically cover few third-party libraries. In this paper, we present Bridge, a client-driven framework for automatically constructing ecosystem-scale library update datasets that connect API update mappings, version transitions, and client update instances. Bridge first mines candidate update instances from client dependency update commits at scale, validates them using library-side evidence, and then derives API update mappings from validated instances. This design grounds each retained mapping in at least one client update instance. On a manually annotated ground truth dataset, Bridge achieves 91.6% precision and 88.7% recall for Java and 90.1% precision and 64.0% recall for Python. Applied to WoC V3, Bridge mines 381,661 Java and 277,259 Python client update instances, representing 18,900 and 4,456 API update mappings across 2,557 and 999 libraries, respectively. The mined mappings exhibit a pronounced long-tail distribution, with most appearing in only a few client update instances. As one application of the dataset, we evaluate four large language models on replacement API recommendation, a key step in library updates. The best recommendation accuracy reaches only 37.1% for Java and 44.4% for Python, and all evaluated models perform substantially better on frequently observed mappings than on mappings observed in only a few client update instances, highlighting the difficulty current LLMs face in recommending replacements for mappings in the long tail.

1 Introduction

Existing datasets incompletely connect API replacements, version transitions, and concrete client adaptations, while typically covering few libraries. Bridge reverses the workflow by mining client dependency updates, validating candidates with library evidence, and deriving grounded mappings at ecosystem scale.

  • Motivation: Existing library evolution datasets capture different subsets of API mappings, version transitions, and client update instances, and generally cover few third-party libraries.Their library-driven construction begins with predefined libraries and then seeks mappings and corresponding code examples.
  • Bridge: Bridge starts from client dependency update commits, mines candidate API call changes, validates them against library versions, and derives API update mappings.Each retained mapping is associated with a version transition and grounded in at least one client update instance.
  • Evaluation: 91.6% precision and 88.7% recall for Java, versus 90.1% precision and 64.0% recall for Python, were achieved on manually annotated ground truth.The corresponding F1 scores were 90.1% for Java and 74.9% for Python.
  • Contribution: Bridge is implemented for Java and Python and constructs an ecosystem-scale dataset with hundreds of thousands of client update instances and a pronounced long-tail distribution of API replacements.The contribution summary characterizes the dataset as spanning thousands of libraries in both ecosystems.
  • Application: LLMs have substantially greater difficulty recommending replacements for API mappings in the long tail than for frequently observed mappings.The recommendation study demonstrates the dataset’s utility for evaluating replacement API recommendation.

2 Background and Related Work

The paper defines the dataset entities and situates Bridge against library-driven approaches that incompletely connect update information and cover limited third-party ecosystems. It motivates a client-driven alternative using large-scale version-control data to locate real dependency updates.

  • 2.1 Terminology: An API update mapping relates a library’s legacy API signature to its replacement independently of a specific version transition.An API update pair adds the old and new library versions to that mapping.
  • 2.1 Terminology: A client update instance records legacy and replacement API calls observed when a client updates a library between two versions.Mappings express general replacement relations, whereas instances capture how clients perform those replacements in code.
  • 2.2 Library Evolution Datasets: Existing datasets differ in API information, coverage, and code sources, but none simultaneously connects mappings, version transitions, and real client update instances across many libraries.Their examples may be handwritten, LLM-generated, documentation-derived, or drawn from GitHub without systematic version-transition links.
  • 2.3 Library-Driven Workflow: Library-driven construction must identify mappings across releases and then locate projects that exercise affected APIs during applicable version transitions.Sparse client API usage means library-derived mappings may have few or no observable update instances.
  • 2.4 World of Code: WoC aggregates Git objects and cross-references commits, files, blobs, and projects, enabling ecosystem-scale queries without cloning hundreds of millions of repositories.The study uses WoC V3, containing more than 4.7 billion commits and 19.6 billion blobs across 234 million repositories.

3 The Bridge Framework

Bridge is a client-driven, four-phase framework that mines dependency-update commits, extracts and matches changed API calls, validates candidates against library versions, and derives connected update artifacts. Implemented for Java and Python, it uses ecosystem-scale version-control data and language-specific parsing to construct validated mappings and client instances.

  • Language-specific implementation: Java and Python implementations use language-specific dependency parsers, source analysis, and API-call metadata extraction to support the client-driven workflow.Java uses Maven pom.xml; Python handles requirements.txt, setup.py, pyproject.toml, and setup.cfg, including data-flow analysis for executable setup.py files.
  • Overview: Bridge starts from client dependency-update commits, using associated version changes and source edits to generate candidate library-update instances.The workflow reverses the usual library-driven order by beginning with observed client adaptations.
  • Overview: Four phases collect update commits, extract API-call metadata, mine similar legacy–replacement call pairs, and validate them with library-side evidence.Validated instances then support derivation of API update mappings and update pairs.
  • Phase 1: Update Commit Collection: Phase 1 identifies 753,902 Python update commits and 7.8 million unique Python source blobs after dependency parsing and source-change filtering.These commits involve 25,397 libraries and 236,292 library versions.
  • Validation: API-call similarity can produce false positives, so Bridge validates library association and API evolution before retaining mappings.Similarity alone may confuse shared import prefixes or semantically different calls.
  • Validation: Each validated instance yields a legacy-to-replacement API mapping, which is paired with its library version transition and client-side call change.This connects mappings, version transitions, and concrete update instances in one dataset.

4 Evaluation

The evaluation examines Bridge’s mining accuracy, design choices, mined-dataset characteristics, and dataset utility. It frames accuracy in terms of precision, recall, and F1 against manually annotated client update instances.

  • Evaluation: Bridge is evaluated across mining accuracy, key design choices, mined-dataset characteristics, and the utility enabled by its dataset.These dimensions are organized as four evaluation perspectives.

4.1 Ground Truth Dataset

The ground-truth evaluation uses manually annotated API-call change records, with diversity-oriented sampling across libraries and API calls. Independent annotation relies on external evidence to identify genuine replacements under observed version transitions.

  • Construction: The ground truth is constructed from Phase 3 API-call change records to evaluate candidate matching and library-side validation.The records cover the two design stages that determine which changes become retained client update instances.
  • Sampling: A two-stage sampling strategy improves diversity across libraries and API calls instead of allowing frequently updated libraries to dominate.Libraries are first sampled with probabilities proportional to their update-commit counts.
  • Annotation: A pair is labeled genuine only when external evidence supports the replacement relation under the observed library version transition.Annotators inspect all possible one-to-one pairings between old-side and new-side calls.
  • Evaluation: Table 4 reports GT, TP, FP, and FN alongside precision, recall, and F1 for Java and Python ground-truth performance.The caption defines the count abbreviations and states that the three metrics are percentages.
  • Annotation: Cohen’s κ reaches 0.92 across 2,283 Java and 1,988 Python API-call pairs annotated independently by two authors.Disagreements were resolved through discussion after consulting documentation, release materials, repositories, and, when needed, Stack Overflow.

4.2 Mining Effectiveness (RQ1)

Bridge mines client update instances with high precision in both ecosystems, although Python recall is lower and errors arise mainly from resolution and deprecation-checking limitations.

  • 305 of 344 annotated Java instances and 155 of 242 annotated Python instances are correctly identified.Java has 28 false positives and 39 false negatives; Python has 17 false positives and 87 false negatives.
  • Validation can retain incorrect candidates when similarity matching is wrong, particularly through removal for Java and deprecation checking for Python.Among false positives, 24 of 28 Java cases follow the removal path, whereas 13 of 17 Python cases follow the deprecation path.
  • Similarity thresholds, unresolved APIs, and unrecognized deprecation evidence produce false negatives in both ecosystems.For Java, 16 of 39 false negatives fall below the similarity threshold and 22 result from resolution failures; for Python, 22 of 87 fall below threshold and 51 result from resolution failures.
  • 91.6% precision, 88.7% recall, and 90.1% F1 are achieved for Java, while Python reaches 90.1% precision, 64.0% recall, and 74.9% F1.
  • Python’s lower recall primarily results from API signature resolution and deprecation-checking limitations.

4.3 Design Analysis (RQ2)

Similarity weights have limited influence on F1, while thresholds control the precision–recall tradeoff and library-side validation substantially improves precision at a recall cost.

  • Phase 4 Ablation: Library-side validation filters false candidates by requiring API resolution and evidence that the legacy API is removed or deprecated in the new version.
  • Weight Sensitivity: 89.9% is Java’s highest mean F1 across 37 weight configurations, while Python’s configurations span only 72.4% to 73.7%.The equal Java configuration reaches 89.9% mean F1; the equal Python configuration reaches 73.1%.
  • Threshold Sensitivity: 91.6% precision and 88.7% recall are obtained for Java at threshold 0.45, yielding the highest F1 score of 90.1%.Increasing the threshold from 0 to 0.45 raises precision from 84.0% to 91.6% while recall falls from 91.3% to 88.7%.
  • Threshold Sensitivity: 90.1% precision is obtained for Python at threshold 0.35, while recall is 64.0% and F1 decreases by only 0.1 points from its maximum.The threshold is selected because precision is particularly important when mined instances are released as a reusable dataset.
  • Phase 4 Ablation: 40.3 percentage points for Java and 43.3 points for Python are the precision gains from library-side validation.F1 increases by 23.4 points for Java and 13.1 points for Python, while recall decreases in both languages.

4.4 Dataset Characteristics (RQ3)

Bridge produces a large but highly uneven dataset: most libraries and mappings have few observations, while a small number account for many instances.

  • Commits per Library: 49.0% of Java libraries and 46.9% of Python libraries appear in only one update commit, while 91.9% in both languages appear in at most ten.
  • Mappings per Library: 37.5% of Java libraries and 49.1% of Python libraries contain only one API update mapping, while 87.4% and 93.2% contain at most ten.
  • Instances per Library: 92.6% of Java libraries and 95.1% of Python libraries appear in at most 100 instances, despite maxima of 132,630 and 124,226 instances.
  • Instances per Mapping: 42.1% of Java mappings and 37.1% of Python mappings appear in only one client update instance, while 89.0% and 84.4% appear in at most ten.The most frequent mappings appear in 33,678 Java and 58,780 Python instances.
  • 381,661 Java and 277,259 Python client update instances represent 18,900 and 4,456 API update mappings across 2,557 and 999 libraries.

4.5 Dataset Utility (RQ4)

Bridge’s dataset supports evaluating replacement API recommendation across LLMs, input settings, and mapping frequencies. Accuracy remains limited overall and substantially lower for sparsely observed mappings than for frequent ones.

  • Approach: The evaluation uses four LLMs to recommend replacement APIs from either API signatures alone or signatures plus focused client call context.The context includes relevant imports and the legacy call; Java additionally includes receiver and argument variable declarations.
  • Overall recommendation accuracy: GPT-5.4 achieves the highest accuracy in every language and input setting.For Java, full signature accuracy ranges from 21.2% to 27.5% with signature input and 20.6% to 26.5% with context; Python FQN accuracy ranges from 30.0% to 42.5% and 29.1% to 44.4%.
  • Overall recommendation accuracy: 37.1% for Java and 44.4% for Python are the highest reported FQN accuracies, indicating that replacement API identification remains challenging.Java FQN accuracy exceeds full signature accuracy by 3.0 to 9.7 percentage points because some errors involve formal parameter types.
  • Effect of client call context: Focused client call context does not consistently improve recommendation accuracy.Java full signature accuracy decreases for all four models by 0.6 to 2.0 percentage points, while only GPT-5.4 shows a statistically significant Python FQN increase after Holm correction.
  • Recommendation performance by mapping frequency: 15.4%–21.1% for Java and 23.9%–36.9% for Python with one-instance mappings rise to 48.5%–55.8% and 73.8%–79.4%, respectively, for mappings exceeding 100 instances.The same frequency-related pattern appears under context input, linking the dataset’s long tail to lower recommendation accuracy.
  • Recommendation performance by mapping frequency: All evaluated models perform substantially worse on sparsely observed mappings than on recurrent mappings.The dataset’s frequency distribution therefore exposes a practical difficulty for replacement API recommendation in the long tail.

5 Discussion

Bridge combines client-side candidate generation with library-side validation to construct a reusable dataset for automated library updates. The discussion emphasizes its practical utility while qualifying recall, coverage, mined-relation correctness, reference replacements, sampling, and generalizability.

  • Dataset utility: Bridge connects API mappings, version transitions, and client update instances across thousands of libraries for developing and evaluating automated update techniques.Retained dependency update commits can also support project-level benchmarks with further curation and build-based validation.
  • Design implications: 40.3 percentage points for Java and 43.3 points for Python are the precision gains from library-side validation at selected thresholds.Similarity weights have limited impact on F1, supporting client-side candidate generation followed by library-side validation.
  • Implications: Replacement API recommendation remains challenging for current LLMs, particularly for mappings in the long tail.The discussion suggests separating replacement selection from code adaptation and verifying selected APIs before subsequent compilation, type checking, and testing.
  • Threats to validity: RQ1 recall measures candidate matching and library-side validation rather than end-to-end recall of the complete Bridge pipeline.Earlier missed dependencies, API calls, or excluded changes cannot enter the ground truth dataset.
  • Threats to validity: Bridge assumes dependency updates and associated code adaptations occur within the same commit.Some co-occurring code changes may be unrelated, although candidates are restricted to the updated library and validated against library versions.
  • Threats to validity: Candidate matching and library-side validation can introduce false positives and false negatives.Removal or deprecation does not establish the intended replacement, while resolution failures and unrecognized deprecation mechanisms can discard genuine instances.
  • Threats to validity: The reference replacements used for RQ4 have high but imperfect precision, and exact-match accuracy may count other valid replacements as incorrect.A strict majority rule retains one replacement, although a legacy API may have multiple valid replacements for a version transition.
  • Threats to validity: The reported precision, recall, and F1 scores are not population-weighted estimates over all API call change records.The ground truth covers 100 sampled libraries per language, with sampling designed to promote diversity across libraries and old-side APIs.

6 Conclusion

Bridge automatically constructs ecosystem-scale library update datasets by mining and validating client dependency updates, connecting API mappings, version transitions, and concrete client changes. Its large-scale results expose long-tail mapping coverage and limited replacement-API recommendation accuracy, especially for rarely observed mappings.

  • 6 Conclusion: Bridge mines candidate update instances from client-side changes, validates them against involved library versions, and derives API update mappings from validated instances.The framework connects mappings, version transitions, and client update instances while grounding each retained mapping in at least one client update instance.
  • 6 Conclusion: 91.6% precision for Java and 90.1% precision for Python were achieved on a manually annotated ground truth dataset.Library-side validation improved precision by more than 40 percentage points for both languages.
  • 6 Conclusion: 381,661 Java and 277,259 Python client update instances were mined across 2,557 and 999 libraries, respectively.These instances represented 18,900 Java and 4,456 Python API update mappings.
  • 6 Conclusion: Most mined API mappings appeared in only a few client update instances, forming a pronounced long-tail distribution.A small number of mappings appeared substantially more frequently.
  • 6 Conclusion: Replacement API recommendation remained difficult for current LLMs, which performed substantially better on frequently observed mappings than on rarely observed mappings.The result identifies long-tail replacement recommendation as a challenge for current LLMs.
  • 6 Conclusion: Bridge provides an automated approach to connecting what APIs are replaced, between which library versions, and how clients perform the updates.Future work will extend the framework to additional dependency configurations and programming language ecosystems.
Loading 2608.30497v1…