Source-linked AI summary
Who Broke Me? Execution-Guided Repair of Behavioral Dependency Breaks
Juntao Chen, Tingting Bi, Yanlin Wang, Patanamon Thongtanunam
TL;DR
Behavioral dependency breaks leave interfaces intact, so failing tests may not identify the upstream API responsible, and compiler feedback is absent while documentation is often unreliable. BBCFixer uses differential execution to identify a candidate root API and filter the library diff, with BBCBench providing 100 behavioral breaks for evaluation. On BBCBench, BBCFixer raises average pass rate by up to 17% over Pure Agent and needs up to 27% fewer agent steps per successful repair.
Problem
Behavioral breaks can leave the interface intact while failing tests do not identify the responsible upstream API, and compiler feedback is absent while documentation is often unreliable.
Source: §1 · p. 2Method
BBCFixer runs failing tests under old and new library versions, ranks calls with differing return values to select a candidate root API, and filters the library diff with it.
Source: §1 · p. 2Results
BBCFixer raises average pass rate by up to 17% over Pure Agent and needs up to 27% fewer agent steps per successful repair on BBCBench.
Source: §1 · p. 3, §8 · p. 18Takeaways & Limitations
The evaluation indicates that failure-specific evidence is a useful complement to improving the repair agent itself.
Source: §1 · p. 3The evaluation uses one agent framework and two open models, so results by library, language, or model rest on small samples and gains are not measured on higher-priced proprietary models.
Source: §6 · p. 17
Abstract
from arXiv · showhide
Dependency upgrades can break downstream projects without changing the library interface. Such behavioral breaking changes are difficult for developers to fix, because the failing test does not always point to the root API, the upstream API that causes the break. Existing LLM-based repair methods obtain evidence for the repair from compiler feedback or library documentation. However, a behavioral break produces no compiler feedback and is often undocumented. Agents that receive no upgrade evidence also usually do not retrieve upstream evidence themselves, and most of their failed repairs do not identify the root API. The library diff provides useful upstream evidence, but the root API must first be identified to select the relevant part of the diff. We present BBCFixer, a repair method that runs the failing test under the old and new library versions, ranks the calls whose return value differs to identify a candidate root API, and filters the library diff with that candidate. To evaluate repairs of behavioral breaks, we further introduce BBCBench, a benchmark of 100 behavioral dependency breaks in Python and JavaScript. We compare BBCFixer on BBCBench with two baselines: (1) Pure Agent, an agent that receives no upgrade evidence, and (2) BDUpdater, a method that mines library documentation. BBCFixer raises the average pass rate by up to 17% relative to Pure Agent and needs up to 27% fewer agent steps per successful repair. BBCFixer thus provides a way to select the upstream evidence behind a behavioral break, so that an LLM agent can repair behavioral breaking changes more effectively and efficiently.
1 Introduction
Behavioral dependency breaks preserve the library interface, leaving failing tests that may not identify the upstream cause; compiler feedback is absent and documentation is often unreliable. BBCFixer selects failure-specific evidence from differential execution and the library diff, and BBCBench supports evaluation of repairs against reference fixes and reproducible harnesses.
1 Introduction
- Behavioral breaks leave the interface intact, so failing tests may not identify the upstream API, while compiler feedback is absent and only 27.7% of detected behavioral incompatibilities are documented.A studied sample found that 18% of client bugs caused by behavioral incompatibilities remained unfixed, although most developers tried to fix them.Source: §1 · pp. 1–2
- Agents often fail to retrieve upstream evidence themselves, while documentation mining can raise successful-repair cost by up to 132%.The paper frames selecting evidence before the agent begins as a distinct challenge from generating a patch.Source: §1 · p. 2
- BBCFixer runs the failing test under old and new library versions, ranks calls with differing return values to propose a root API, and filters the library diff with that candidate.The first divergence and filtered diff are supplied as repair evidence.Source: §1 · pp. 2–3
- BBCFixer raises average pass rate by up to 17% over Pure Agent and up to 53% over BDUpdater, with up to 27% and 58% fewer agent steps per successful repair, respectively.It achieves a higher pass rate under both baselines in every round.Source: §1 · p. 3
- BBCBench contains 100 confirmed behavioral dependency breaks in real Python and JavaScript projects, each with a reproducible harness and reference fix.The reference fix's test changes encode intended post-upgrade behavior.Source: §1 · pp. 2–3
2 Background and Motivation
The section defines behavioral dependency breaks and the repair task, then explains why selecting a relevant fragment from the upstream library diff is necessary. The diff is too large to use directly, and a failing test can touch several candidate APIs.
2.1 Behavioral Breaking Changes and the Repair Task
- A behavioral breaking change alters library behavior without changing its interface, so downstream code can still run while tests fail without naming the library or API responsible.In the lodash example, _.sample() changes from returning an array of three elements to a single element, leaving the downstream project with six failing tests.Source: §2.1 · p. 3
- The repair task requires modifying downstream source to build and pass tests on the upgraded version; downgrading, removing, or switching the dependency is not allowed.The section introduces the background and motivates filtering the library diff before an agent uses it.Source: §2.1 · pp. 3–4, §2 · p. 3
2.2 Filtering the Library Diff
- The library diff can provide evidence about implementation changes and revised tests specifying new behavior, but it must be filtered to retain the fragment relevant to the break.Selecting only changed APIs still leaves several candidates, so the relevant API must be identified using evidence tied to downstream usage.Source: §2.2 · p. 4
- Across benchmark cases, the library diff has a median of 115,866 lines across 168 files, while one failing test touches a median of five upstream APIs, three of which the diff changes.The differing return value of lib.load() identifies it as the root API in the illustrative case.Source: §2.2 · p. 4
3 Research Design
The research design links failure-specific upstream evidence, a benchmark of real behavioral dependency breaks, and evaluation of the repair workflow under two models.
3 Research Design
- BBCFixer transforms behavioral differences observed in downstream tests into compact upstream repair evidence.Figure 2 illustrates narrowing the upstream APIs touched by a failing test to the root API.Source: §3 · p. 4, Figure 2 · p. 5
- BBCBench provides 100 behavioral dependency breaks in real projects, with reproducible intact, broken, and fixed states and an automated evaluation script.The benchmark supports evaluation of the repair task.Source: §3 · p. 4
- The experimental setup evaluates the complete repair workflow and its evidence configurations under two models.The evaluation is the third component of the research design.Source: §3 · p. 4
3.1 Our Approach: BBCFixer
BBCFixer constructs compact repair evidence before the agent begins by identifying a candidate root API, filtering the library diff, and supplying failure-specific evidence to the repair agent.
3.1 Our Approach: BBCFixer
- BBCFixer compares the failing test under intact and broken dependency versions, filters the library diff using the candidate API and failure symptoms, then presents the resulting evidence to the repair agent.The pipeline has three stages, as summarized in Figure 3.Source: §3.1 · p. 5, Figure 3 · p. 6
3.1.1 Differential Execution.
- The first divergence is the highest-ranked call whose return value differs across versions, with downstream calls and calls from files in the failing stack trace ranked higher.Execution order resolves remaining ties; the selected call's API becomes the candidate root API.Source: §3.1.1 · p. 5
3.1.2 Library Diff Filtering.
- BBCFixer filters changed library lines using the candidate root API and symptom symbols from the failure; if no divergence is found, it falls back to upstream symbols used by downstream code.The retained fragments include surrounding context and can combine implementation changes with revised library tests specifying expected behavior.Source: §3.1.2 · p. 6
3.1.3 Evidence-Guided Repair.
- Before repair, BBCFixer mounts the error message, first divergence when available, and filtered diff as read-only evidence for the agent.The opening prompt presents the first divergence and selected diff fragments; the agent can inspect the remaining mounted evidence.Source: §3.1.3 · p. 6
3.2 BBCBench
BBCBench addresses the lack of behavioral-break benchmarks with confirmed cases, reference fixes that encode intended behavior, and reproducible harnesses. It contains 100 cases across Python and JavaScript, each evaluated against post-upgrade tests.
3.2.1 Why Existing Benchmarks Are Insufficient.
- None of six existing dependency-upgrade benchmarks meets all three requirements: confirmed behavioral breaks, reference fixes, and public harnesses.Only TimeMachine-bench Verified has a confirmed behavioral subset, with 19 cases in the authors’ audit.Source: §3.2.1 · p. 7
3.2.2 BBCBench Construction and Composition.
- BBCBench candidates come from benchmark golden patches, downstream adaptation commits, and documented fixes applied to real projects.Candidates are manually screened as behavioral breaks by examining the reference fix and confirming an upgrade-induced test failure on a wrong value or runtime error.Source: §3.2.2 · pp. 7–8
- Every admitted case passes a shared audit and review, then is packaged with intact, broken, and fixed states in a reproducible harness.The reference fix includes test changes encoding intended post-upgrade behavior; two independent reviewers rerun the gates and states.Source: §3.2.2 · p. 8
- BBCBench contains 100 cases—41 Python and 59 JavaScript—with 11 from TimeMachine-bench Verified, 29 from maintainer adaptations, and 60 from documentation.The agent works in the broken state; evaluation discards agent test edits, applies the reference test edits, and requires the upgraded dependency and full post-upgrade test suite to pass.Source: §3.2.2 · pp. 8–9, §3.2.3 · p. 9
3.3 Experimental Setup
The experiments compare BBCFixer with two baselines and two ablations using a shared agent, models, and run budget. Each setting is repeated across the benchmark, with pass rate as the primary metric.
Agent, Models, and Environment
- All settings use mini-swe-agent 2.4.6 with qwen3-coder and GLM-4.5-Air, under a per-run limit of 100 steps, 30 minutes, or US$4.Runs use the default sampling temperature of 0.7 and execute in Docker containers on one Linux virtual machine.Source: §3.3 · p. 9
Baseline Settings
- Pure Agent receives only the broken repository and error message, while BDUpdater mines and verifies upstream breaking-change records for the agent.Both settings allow inspection of the installed library; Pure Agent receives no upgrade evidence.Source: §3.3 · p. 9
Ablation Settings
- The two ablations remove either first-divergence evidence or the filtered library diff from BBCFixer’s repair procedure.Without the diff, the agent sees which API’s return value changed, but not what changed inside that API.Source: §3.3 · pp. 9–10
Root-API Labels
- Each case has a labeled root API based on its reference fix and upstream change, with two authors independently agreeing on 92.0% of labels.Agreement was Cohen’s kappa 0.83.Source: §3.3 · p. 10
Runs, Metrics, and Statistical Tests
- Each setting runs three full rounds of 100 cases per model; average pass rate is primary, with robustness views and paired case-level statistical tests also reported.The tests include a paired bootstrap, Wilcoxon signed-rank test, and exact McNemar test; p-values are Holm-corrected within each model’s result table.Source: §3.3 · p. 10
4 Results
BBCFixer improves repair outcomes and effort against the baselines, while results suggest its combined evidence is more effective than either evidence component alone. Pure Agent runs often do not consult upgrade-specific information, and diagnosis failures dominate their unsuccessful repairs.
4.1 RQ1: Pure Agent Behavior and Failure Patterns
- Failure to identify the root API accounts for 66.9% of qwen3-coder and 69.2% of GLM-4.5-Air failing runs.Incorrect adaptation accounts for most remaining failures, while execution failures are rare.Source: Table 3 · p. 11, §4.1 · p. 11
- 61.3% of qwen3-coder runs and 83.0% of GLM-4.5-Air runs consult neither textual nor code-level information about the upgraded library.Among runs that consult information, code is read more often than text; no qwen3-coder run compares versions through source code or tests.Source: §4.1 · pp. 10–11
4.2 RQ2: Effectiveness of BBCFixer
- 17% higher average pass rate than Pure Agent with qwen3-coder and 12% with GLM-4.5-Air; BBCFixer also passes more cases than both baselines in all three rounds.On the strictest view—cases passing in all three runs—the gains are 27% and 33%, respectively.Source: §4.2 · pp. 11–12, Table 4 · p. 12
- BBCFixer uses 15% to 27% fewer agent steps per successful repair than Pure Agent and 29% to 58% fewer than BDUpdater.It also uses 27% to 58% less cost per successful repair than BDUpdater.Source: Table 6 · p. 12, §4.2 · p. 13
4.3 RQ3: Effect of Different Evidence Configurations
- Only BBCFixer, which uses the first divergence to filter the library diff, improves over Pure Agent under every view with both models.Neither evidence ablation consistently improves on Pure Agent, and each falls below it under at least one view.Source: §4.3 · pp. 13–14, Table 7 · p. 13
4.4.1 RQ4.1: Agent Identification of the Root API.
- Both evidence-based settings increase root-API identification by 20% to 22% relative to Pure Agent; BBCFixer’s increase is 22% under both models.The evidence-based settings raise identification rates, with BDUpdater improving slightly less than BBCFixer under both models.Source: Table 9 · p. 14, §4.4.1 · p. 14
4.4.2 RQ4.2: Association between Root-API Identification and Repair.
- Root-API-identifying runs have pass rates 52% to 251% higher than non-identifying runs across all six groups.This association holds in every setting; BBCFixer also has the highest pass rate among runs that do not identify the root API under both models.Source: Table 10 · p. 15, §4.4.2 · p. 15
4.4.3 RQ4.3: Gain on Cases Whose Evidence Points to the Root API.
- On the 51 cases whose evidence points to the labeled root API, BBCFixer’s gain over Pure Agent is 20% versus 17% under qwen3-coder and 16% versus 12% under GLM-4.5-Air.The evidence points to the labeled root API in 51 of the 100 benchmark cases.Source: Table 11 · p. 15, §4.4.3 · p. 16
5 Discussion
BBCFixer sometimes misses the labeled root API, but recorded behavior changes can still suggest the needed adaptation. The discussion identifies recording and ranking limits and points to broader uses for the evidence.
5 Discussion
- BBCFixer identifies the labeled root API in 51 of 100 cases; in the other 49, missing recorded calls account for more than half of misses.The remaining cases split evenly between another divergence ranking higher and the change altering no recorded return value.Source: §5 · p. 16
- The recorder observes return-value changes in calls entering the library, but misses some calls and changes mediated by operators, settings, or callbacks.The discussion proposes recording calls through loaders or wrappers, improving ranking, and adding operator results and attribute reads.Source: §5 · p. 16
- Among runs that miss the root API, BBCFixer still has the highest pass rate under both models, and the first divergence can suggest an adaptation.It shows the call’s return value under both versions; the example’s format_date output changes from ’11/08/2026’ to ’2026-08-11’.Source: §5 · p. 16
- Differential execution inputs already exist in dependency-upgrade pull requests, making evidence preparation compatible with continuous integration.The inputs are the project before and after the upgrade and the failing test.Source: §5 · p. 17
6 Threats to Validity
The validity discussion qualifies the repair measurements through test coverage, evaluation design, and the scope of the benchmark and models.
6 Threats to Validity
- The pass rate is an upper bound on repair correctness because reference-fix tests do not cover every behavior, and heuristic root-API labels are not guaranteed causes.RQ4 therefore measures whether evidence points to the labeled API and how that relates to repair outcomes, not root-cause localization accuracy.Source: §6 · p. 17
- Single-run outcomes vary, and the comparison with Pure Agent measures BBCFixer’s complete workflow rather than isolating its evidence artifacts.A controlled comparison under an identical prompt remains necessary to attribute gains solely to evidence selection.Source: §6 · p. 17
- BBCBench has 41 Python and 59 JavaScript cases, while evaluation uses one agent framework and two open models, limiting subgroup samples and model scope.The gains were not measured on higher-priced proprietary models; some reference fixes may also have appeared in model training data.Source: §6 · p. 17
7 Related Work
Related work covers empirical studies and detection tools for behavioral breaks, alongside methods for repairing breaking changes.
7.1 Behavioral Breaking Changes
- Studies report behavioral-change prevalence and downstream effects, including 68.1% of breaking changes marked by npm maintainers and changed defaults affecting 35% of analyzed Scikit-Learn clients.Other studies inspect 126 incompatibilities in 15 Java libraries and find behavioral breaks reaching client tests as exceptions or assertion failures.Source: §7.1 · p. 17
- Detection tools range from static API comparisons to dynamic execution, but stop at detection; BBCFixer filters the library diff and gives it with the first divergence to the repair agent.GILESI also records and compares library-call inputs and outputs across versions.Source: §7.1 · p. 18
7.2 Repairing Breaking Changes
- Pre-LLM repair tools apply adaptations or rules from frameworks, clients, pull requests, or semantic patches, so undescribed behavioral changes lack a source for those rules.The methods discussed are SemDiff, APIfix, MELT, and JSFIX.Source: §7.2 · p. 18
- LLM repair methods use compile errors, maintainer documentation, or library diffs, while BBCFixer starts from behavior observed by running the failing test under both versions.The reviewed benchmarks and version-conditioned completion tasks differ in whether they reproduce and repair a broken downstream project.Source: §7.2 · p. 18
8 Conclusion
BBCFixer uses differential execution to select upstream evidence for behavioral dependency breaks, and BBCBench evaluates it on real Python and JavaScript cases. The reported results show higher pass rates and fewer agent steps than Pure Agent.
8 Conclusion
- BBCFixer raises average pass rate by up to 17% over Pure Agent and needs up to 27% fewer agent steps per successful repair.Source: §8 · p. 18
- BBCFixer runs the failing test under old and new library versions, ranks calls with differing return values, and filters the library diff using a candidate root API.Source: §8 · p. 18
- BBCBench contains 100 behavioral dependency breaks from Python and JavaScript projects.Source: §8 · p. 18
9 Data Availability
The replication package provides BBCFixer and BBCBench, allowing the results to be reproduced directly.
9 Data Availability
- The replication package contains the implementations of both BBCFixer and BBCBench.Source: §9 · p. 19
- The package is available at https://github.com/Elendil3703/BBCFixer.Source: §9 · p. 19
- The package allows the results to be reproduced directly.Source: §9 · p. 19