Source-linked AI summary
An Empirical Study on the Impact of Refactoring Activities on Evolving Client-Used APIs
Raula Gaikovina Kula, Ali Ouni, Daniel M. German, Katsuro Inoue
TL;DR
Clients adopting newer library versions depend on previously used APIs remaining compatible, while evolving libraries also address bug fixes and maintainability concerns. This study uses client usage to identify client-used APIs and examines their breakages and refactorings across popular libraries. It finds that fewer than 37% of client-used API breakages relate to refactoring, with many others tied to maintenance issues.
Problem
The study asks how refactoring-related API changes affect clients, for whom compatibility matters when migrating to newer library versions.
Source: §1 · p. 1, §4.1 · p. 8Method
The authors use actual client usage to identify client-used APIs and conduct a tool-based empirical study across eight popular Java libraries.
Source: §1 · p. 2, §3.2 · p. 5Results
Less than 37% of client-used API breakages were related to refactoring, and maintainers were less likely to break client-used APIs.
Source: §1 · p. 2, §8 · p. 23Takeaways & Limitations
Many other client-used API breakages address bug fixes or new features and are likely to involve more complex refactorings.
Source: §1 · p. 2, §8 · p. 23The client projects and selected popular libraries may not represent the real world, so findings may not apply to less popular libraries.
Source: §6.4 · p. 22
Abstract
from arXiv · showhide
Context: Refactoring is recognized as an effective practice to maintain evolving software systems. For software libraries, we study how library developers refactor their Application Programming Interfaces (APIs), especially when it impacts client users by breaking an API of the library. Objective: Our work aims to understand how clients that use a library API are affected by refactoring activities. We target popular libraries that potentially impact more library client users. Method: We distinguish between library APIs based on their client-usage (refereed to as client-used APIs) in order to understand the extent to which API breakages relate to refactorings. Our tool-based approach allows for a large-scale study across eight libraries (i.e., totaling 183 consecutive versions) with around 900 clients projects. Results: We find that library maintainers are less likely to break client-used API classes. Quantitatively, we find that refactoring activities break less than 37% of all client-used APIs. In a more qualitative analysis, we show two documented cases of where non-refactoring API breaking changes are motivated other maintenance issues (i.e., bug fix and new features) and involve more complex refactoring operations. Conclusion: Using our automated approach, we find that library developers are less likely to break APIs and tend to break client-used APIs when addressing these maintenance issues.
1. Introduction
As libraries evolve through client requests, bug fixes, and maintainability work, API changes can threaten compatibility for clients adopting newer versions. This study examines how refactoring relates to breakages in APIs actually used by clients, finding that such APIs are less often broken and that fewer than 37% of their breakages relate to refactoring.
1. Introduction
- The study investigates whether refactoring breaks APIs used by clients, whose migration to newer library versions depends on previously invoked external APIs continuing to work.The authors describe refactoring as a way to improve software design, readability, and reusability, while unaddressed maintenance concerns can increase complexity and degrade quality.Source: §1 · pp. 1–2
- Less than 37% of client-used API breakages were related to refactoring; maintainers were less likely to break client-used APIs, and simple refactorings were less frequent on them.The study also reports that many other breakages address maintenance issues likely to involve more complex refactorings.Source: §1 · p. 2
- Using actual client usage to distinguish APIs, the authors analyze over 9,700 breaking classes and around 12,900 refactoring operations across eight popular Java libraries.The study covers around 10–38 consecutive releases per library.Source: §1 · p. 2
- The paper contributes an automated large-scale study of API breakages and refactorings involving APIs used by actual clients, plus a publicly available replication dataset.The replication package is identified as publicly available by the authors.Source: §1 · p. 2
2. Basic Concepts & Definitions
The paper defines API compatibility and refactoring concepts, then classifies library classes using client usage as an approximation of intended external API use.
2.1. Backward Compatibility of APIs
- API compatibility is framed as preserving binary linkage and the contract of accessible members and constructors; renaming or removing a method can break it, while adding members usually does not.The paper also notes that compatibility depends partly on Java binary compatibility.Source: §2.1 · p. 2
- A library can also break a client indirectly when an environmental change, such as a JDK update, breaks a method within the library.Source: §2.1 · p. 2
2.2. Refactoring Activities and API breakages
- Refactoring restructures existing code without changing external behavior; a Refactoring Operation (Ropt) is an atomic refactoring change between two library versions.Source: §2.2 · p. 3
2.3. API Categorization Based on Client Usage
- A public class may still be designated internal: Guava’s Finalizer class is public for reflective access across class loaders but is not part of its published API.The paper notes that internal APIs may nevertheless be used by clients.Source: §2.3 · p. 3
- The study approximates external APIs by mining client usage, but usage does not perfectly distinguish intended external APIs from internal APIs.Internal APIs can be used by clients, and distinguishing the two types is difficult without explicit documentation.Source: §2.3 · pp. 3–4
- A client-used API class is used by at least one client, while a non-client-used API class is used by none; classes are categorized as API or non-API and as breaking or non-breaking.An API class has at least one public entity accessible to clients; a breaking class changes API compatibility, whereas a non-breaking class does not.Source: §2.3 · p. 4
- A Ref class is a changed API class with at least one refactoring operation on an element, and Ropt density counts operations per class.Source: §2.3 · p. 4
3. Approach
The approach mines client projects to identify client-used API classes, detects binary incompatibilities and refactorings across selected library versions, then maps their overlap.
3.1. Subject Libraries
- Eight libraries span 183 consecutive versions, selected for client usage, evolution history, domain diversity, and prior study.The study used consecutive official releases and binaries with available source code.Source: Table 1 · p. 4, §3.1 · p. 5
3.2. Client-Used API Extraction Method
- Compiling client projects reveals the fully qualified library classes they load, approximating the external APIs actually used.Client projects were collected until added projects revealed no more client-used API classes; Guava reached saturation after 98 clients, covering 184 API classes.Source: §3.2 · p. 5, Table 2 · p. 5, Figure 2 · p. 6
3.3. API Breakage Detection Method
- Japi-cmp compares library binaries for compatibility and accessibility, identifying over 9,700 breaking classes; binary-compatible semantic incompatibilities may be missed.Detected breaking changes are definite breaks, but some semantically incompatible APIs may remain undetected.Source: §3.3 · p. 7
3.4. Refactorings Detection Method
- Ref-Finder detects structural refactorings through mechanical transformations, yielding 12,900 operations across the eight libraries.The tool does not detect changes requiring restricted conditions or additional developer specifications it cannot infer.Source: §3.4 · p. 7
3.5. Mapping Refactorings to API Breakages
- Mapping Ref classes against breaking classes left 10,800 refactoring operations after 2,100 false positives were discarded.One detected example was removal of Guava's ImmutableMultiset.of(E[]) method, classified as a remove-method operation.Source: §3.5 · p. 7, Figure 3 · p. 7
4. Empirical Study
The empirical study asks how often client-used APIs break and how much breakage overlaps refactoring, then examines non-refactoring causes and the types of refactoring involved.
4.1. Research Questions
- Four research questions address client-used API breakage over time, its relation to refactoring, non-refactoring changes, and breaking refactoring types.The study uses an automated approach to reduce manual inspection and heuristic errors and enable large-scale analysis.Source: §4.1 · p. 8
4.2. Research Method for RQ1
- The RQ1 breakchange metric is the per-version share of breaking classes among all changed classes, assessed across class categories.The study compares categories using non-parametric tests and estimates effect size with Cohen’s d and Bonferroni correction.Source: §4.2 · p. 9
4.3. Research Method for RQ2
- RQ2 measures refactoring-related breaks both among Ref classes and as a share of all breaking classes.The breaking–to–Ref rate approaches 100% when more refactorings break a class category; the Ref–to–breaking rate approaches 1 when more breaks relate to refactoring.Source: §4.3 · p. 9
4.4. Research Method for RQ3
- For RQ3, researchers manually mapped documented Guava and HttpClient breakages to bug fixes or features; Xerces was excluded because references were ambiguous.Three researchers checked changelogs using keywords or issue links, without project-specific knowledge.Source: §4.4 · pp. 9–10
4.5. Research Method for RQ4
- RQ4 aggregates refactoring operations by type and compares their breaking overlap in client-used versus non-client-used API classes using prsv.The study hypothesizes that prsv below 1 indicates fewer operations applied to client-used classes, while a value of 1 indicates equal application.Source: §4.5 · p. 10
5. Results
Across four research questions, maintainers break and refactor client-used API classes less often than non-client-used classes. Refactoring-linked breaks account for less than 37% of client-used API breakages, while manual cases associate other breaks with maintenance needs or complex refactorings.
5.1. Findings for RQ1
- Across releases, most libraries had more non-client-used than client-used API classes, while client-used counts stayed stable; log4j was the exception.Source: §5.1 · p. 10, Figure 4 · p. 11, Figure 5 · p. 12
- Incompatible API code changes were statistically more likely in non-client-used than client-used API classes; for comparisons across all class categories, the post-hoc test found a medium effect (p<0.01, r=0.54).Source: §5.1 · p. 15
5.2. Findings for RQ2
- Up to 75% of refactored API classes broke client-used APIs, yet these breakages accounted for less than 37% of all client-used API breakages.Source: §5.2 · p. 16
5.3. Findings for RQ3
- Two documented cases illustrate non-Ref breaks: one addressed a complex defect triggered by JDK incompatibility, and another involved complex refactoring missed by automation.Source: Figure 9 · p. 17, §5.3 · p. 18
- 82% of Guava and 58% of HttpClient non-refactoring client-used API breakages mapped to API documentation, and many reflected bug fixes or new features.Source: §5.3 · p. 16, Table 6 · p. 18
5.4. Findings for RQ4
- For client-used API classes, leading breaking refactorings were remove parameter in guava (30 Ropt), move method in httpclient (3 Ropt), and rename method in xerces (146 Ropt).Source: Table 7 · p. 19, §5.4 · p. 20
- Maintainers were less likely to apply frequent Ropt types to client-used classes for 5/10 types in guava, 3/5 in httpclient, and 16/17 in xerces.Source: §5.4 · p. 20
6. Discussion
Library maintainers apply incompatible changes to external API classes less often than to other classes. The authors suggest this reflects awareness of client update effort and may relate to stable-interface design.
6.1. Implications
- Maintainers apply incompatible changes to external API classes less often than to other classes, consistent with awareness of client update effort.The authors connect API stability with client trust and adoption latency, and suggest protected variation may help explain library evolution.Source: §6.1 · p. 20
6.2. Comparison to Literature
- Ref-Finder identified 23 of Fowler’s 70 refactoring types, while this study excludes behavioral breakages and analyzes consecutive releases unlike prior work.The study also found API changes absent from change logs.Source: §6.2 · p. 21
6.3. Challenges of the Automated Approach
- Automated accuracy is threatened by missed refactorings, unreported API changes, and false positives, including generic type-parameter changes that may not affect binary compatibility.Figure 10 illustrates a reported API break that Java documentation says does not itself affect binary compatibility.Source: §6.3 · p. 22
6.4. Threats to Validity
- Random inspection checked tool results, but recall remains difficult to assess without ground truth; class-level analysis can also produce false positives and possible underestimates.The authors manually checked path consistency and judged the client sample sufficient to identify popular APIs in client-used classes.Source: §6.4 · p. 22
- The selected clients and popular Java libraries may not represent the real world, limiting applicability to less-popular libraries and other languages.The authors consider the library diversity sufficient for generalization, while identifying broader library and language coverage as future work.Source: §6.4 · p. 22
7. Related Work
Related work covers client API-usage analysis, support for migrating client code, and empirical studies of library evolution and API breakage.
API usage
- Prior API-usage studies examine popularity and usage patterns, use API usage as a stability measure, and study client effects across several ecosystems.The cited examples include Stack Overflow, Android, Pharo, and Smalltalk.Source: §7 · p. 22
Library Migration Support
- Migration research includes change-specification languages, automated replacement recommendations, code analysis, and studies of how maintainers balance compatibility with evolution.Examples include SemDiff’s recommendations for replacements for framework methods accessed by clients.Source: §7 · p. 23
Library Evolution
- Related library-evolution research ranges from manual analyses of change complexity to empirical studies of library migration and API breakage.The cited empirical studies examine libraries in Maven Central.Source: §7 · p. 23
8. Conclusions and Future Work
The study finds that library developers are less likely to break APIs, while many client-used API breakages relate to maintenance needs beyond refactoring. It calls for further research into automated refactoring detection.
8. Conclusions and Future Work
- Library developers are less likely to break APIs, and refactoring operations relate to at most 37% of client-used API breakages.Source: §8 · p. 23
- Many client-used API breaking changes relate to bug fixes and new features rather than refactoring.Source: §8 · p. 23
- The authors call for further research into automated refactoring detection to improve understanding of refactoring activities on API breakages.Source: §8 · p. 23