PaperlayerView exact revision on arXiv

Source-linked AI summary

Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?

Zijian Luo, Runzhi He, Pengfei Gao, Yu Kang, Zeqi Lin, Minghua Ma, Qingwei Lin, Saravan Rajmohan, Yongqiang Tian

arXiv:2608.30300v1cs.SE

TL;DR

Dependency upgrades can require hidden repository-wide source adaptations that existing update tools do not perform. This paper introduces DEPBENCH, an oracle-validated benchmark for evaluating coding agents on such repairs, and finds that the best completed configuration solves only 104/203 tasks, with substantial variation across models, harnesses, and ecosystems.

  • Problem

    Dependency upgrades can break downstream assumptions about signatures, types, APIs, build constraints, or runtime semantics, while existing update tools typically do not perform the required source repair.

  • Method

    DEPBENCH separates dependency changes, source repairs, and held-out tests from real bot pull requests and validates tasks with an executable four-state oracle.

  • Results

    104/203 tasks (51.2%) is the best completed configuration’s result, with substantial variation across models, harnesses, and ecosystems.

  • Takeaways & Limitations

    Effective upgrade agents must propagate changed dependency contracts through repository-wide wrappers, artifacts, types, and runtime behavior, while benchmarks need causal validation and held-out tests.

  • Takeaways & Limitations

    The benchmark’s held-out tests and oracle reduce gaming risk, but future versions should add non-regression checks for downgrades, test deletion, and validation bypass.

Abstract

from arXiv · show

Modern software systems rely heavily on third-party dependencies, but upgrading those dependencies remains a costly maintenance activity. Dependency upgrades do not always preserve the function signatures, type systems, APIs, or runtime semantics assumed by existing code. Consequently, developers often need to perform source code adaptations to accommodate dependency-induced changes. However, such code-level changes are often not explicitly communicated to project maintainers, posing a significant challenge to software reliability. Meanwhile, coding agents have emerged as a new form of software development tool and are increasingly adopted by developers due to their automation capabilities. In this paper, we introduce DEPBENCH, a benchmark consisting of 203 real-world dependency-upgrade tasks across five package ecosystems spanning five language communities, each involving hidden code-level changes that require source code adaptation. We evaluate mainstream coding agents on DEPBENCH. The best completed configuration solves only 104/203 tasks (51.2%), with substantial variation across agent harnesses, models, and ecosystems, highlighting an important gap between current agent capabilities and real-world software maintenance needs.

I. INTRODUCTION

Dependency upgrades can create hidden source-level breakage that existing update automation does not repair. DEPBENCH isolates this maintenance problem and shows that current coding agents remain far from reliably solving it.

  • Dependency bumps can break assumed signatures, types, APIs, build constraints, or runtime semantics, causing build failures, test failures, or silent misbehavior.
  • Update tools can propose safer or newer dependency versions, but they typically do not perform the repository repair required after upgrade-induced breakage.
  • The benchmark evaluates coding agents from a base repository and upgrade instruction while hiding the developer repair and held-out tests until verification.
  • DEPBENCH is an executable repository-level benchmark isolating dependency-upgrade repair through patch decomposition, held-out tests, and a four-state oracle.
  • The paper combines dependency-update bot mining, hunk classification, auditing, leakage controls, and oracle validation with reproducible agent evaluation.

II. BACKGROUND

Dependency-upgrade repair requires benchmarks that establish upgrade causality, avoid leaking the intended fix, and verify that the repair is necessary and sufficient. The paper motivates these requirements by auditing limitations in existing benchmark data.

  • Upgrade-induced breakage may be explicit or implicit, including compile-visible interface changes and runtime semantic changes that leave code compiling but misbehaving.
  • Existing repository, dependency-update, and migration benchmarks were designed for different goals and do not consistently isolate dependency-upgrade repair.
  • Some candidate tasks are unrelated to dependency upgrades because source changes or test expectations may instead reflect bug fixes, refactoring, or test cleanup.
  • Reverse-generated problem statements can leak replacement APIs, target files, triggering paths, or the intended repair direction.
  • A code-patch-level fail-to-pass contract is needed to distinguish necessary and sufficient repairs from unrelated tests, weak oracles, or unreplayable environments.

III. METHOD

DEPBENCH is constructed from real dependency-update bot pull requests and filtered, decomposed, audited, and oracle-validated into reproducible repair tasks. The released benchmark spans five ecosystems and retains only causally grounded upgrade failures.

  • The construction pipeline separates dependency-manifest changes, developer repairs, test changes, and unrelated maintainer churn from merged bot pull requests.
  • Mining retains merged bot-authored pull requests that modify dependency manifests or lockfiles and contain non-trivial human source or test changes.
  • The crawl identified 4,660 candidate pull requests spanning npm/yarn, Maven, Go, Cargo, and Python.
  • A representative js-wacz pull request upgrades glob from 8.1.0 to 10.3.3 and adds human changes for glob’s v10 ESM-style export surface.

C. Patch Decomposition

Patch decomposition separates dependency changes, source repairs, and test changes so DEPBENCH can evaluate whether an upgrade causes a failure and whether the repair resolves it. The js-wacz example illustrates an import-boundary migration from glob v8 to v10.

  • Patch Decomposition: Each pull-request diff is split into disjoint manifest, repair, and test patches.
  • Patch Decomposition: Path-based classification routes manifests and lockfiles to the manifest patch, test-related paths to the test patch, and remaining files to the repair patch, with ambiguous cases manually reviewed.
  • Patch Decomposition: The workflow converts dependency-update bot pull requests into causally validated tasks before evaluating agents from the base repository and upgrade instruction.
  • Patch Decomposition: For js-wacz#41, glob changes from 8.1.0 to 10.3.3, replacing the v8 default-export object and .sync property with the v10 named export.
  • Patch Decomposition: The source repair migrates imports and calls from glob.sync to globSync, while the test patch updates its expected fixture-count computation to the same API.

D. Four-State Oracle Validation and Release

DEPBENCH uses an iterative four-state oracle, followed by human and LLM-assisted audit, to retain dependency-upgrade tasks whose repairs and tests are causally tied to the upgrade.

  • Oracle Validation: The four-state oracle checks base health, upgrade-induced failure, repaired-upgrade success, and failure without the dependency change.The checks distinguish dependency-induced breakage from unrelated repair or test changes.
  • Oracle Validation: Candidates are decomposed into manifest or lockfile changes, developer repairs, and held-out tests before validation.The decomposition uses m for dependency changes, c for repairs, and t for tests, fixtures, mocks, and expected outputs.
  • Iterative Validation: Validation failures are separated from infrastructure exceptions, with affected checks rerun when infrastructure prevents completion.Verifier failures include build, runtime, or test failures; exceptions include unavailable images or expired package repositories.
  • Audit and Release: Oracle-clean candidates undergo human and LLM-assisted inspection of ambiguous diffs and weak tests before release.Reviewers may discard candidates or remove unrelated test hunks and then revalidate the revised patches.
  • Audit and Release: The audit remains partly manual because determining whether test hunks are semantically tied to the upgraded dependency may limit scalability.Repeated oracle validation does not fully automate this semantic judgment.

A. Protocol and Environment

The evaluation runs coding agents in a controlled full-upgrade setting across models, harnesses, ecosystems, and task categories. Results show substantial variation, with the strongest configuration solving only 104/203 tasks and harness, model, and ecosystem choices affecting performance.

  • Protocol and Environment: The agent receives the base repository and upgrade instruction but not the developer repair, hidden tests, or oracle scripts.The verifier overlays the same hidden test patch for every agent, while upstream repository and pull-request pages are blocked.
  • Protocol and Environment: Agents run from pinned repository images under a fixed 30-minute agent budget and five-minute verifier budget.The verifier applies hidden tests and computes the final pass or non-pass outcome.
  • Overall Difficulty: 104/203 tasks are solved by the best completed configuration, while GPT-5.5 configurations range from 71/203 to 104/203 across harnesses.The benchmark remains unsaturated, and pass counts exclude exceptions unless later verification returns reward 1.
  • Harness Effects: 33 tasks, or 16.3 percentage points, separate Codex and Claude Code when GPT-5.5 is held fixed.The corresponding spreads are 2 tasks for Gemini 3.5 Flash and 4 tasks for Claude Opus 4.8.
  • Ecosystem Effects: Maven/Java is easiest for the strongest rows, while npm/yarn and Cargo are much harder across completed configurations.With GPT-5.5, Codex solves 46/65 Maven tasks, compared with 32/65 for OpenCode and 26/65 for Claude Code.

C. RQ2: What Behavior Do the Held-Out Tests Encode?

Held-out tests encode concrete, upgrade-induced behavior rather than arbitrary test churn. They assess API changes, semantic changes, compilation and namespace migrations, generated artifacts, and regression behavior while filtering out weak repair signals.

  • Held-out tests serve as behavioral oracles and are audited for direct dependency-induced assertions and resistance to trivial no-op or test-deletion solutions.The audit evaluates whether tests directly assert or correct upgraded behavior and whether they prevent superficial passes.
  • 142/203 tasks modify existing expectations, mocks, fixtures, or snapshots, while 60/203 add new tests or test cases.
  • 156/203 tasks (76.8%) in the primary partition correct existing behavior, add direct coverage, or combine both with smaller deletions.These tasks force agents to reproduce observable upgraded-dependency behavior rather than merely preserve compilation.
  • API-surface tests require coordinated changes across client construction, mocks, adapters, and downstream method calls.OpenAI and PostHog examples show why import-only or constructor-only repairs are insufficient.
  • Semantic-output tests check changed runtime behavior, generated artifacts, compile paths, namespaces, and focused regressions beyond simple API renaming.Examples include Gremlin overflow behavior, pnpm metadata, Javax-to-Jakarta imports, and syn parser contracts.
  • Fixture/setup-only patches may be legitimate but provide less-local contract evidence, whereas deletion-heavy patches risk passing without positive replacement behavior.The four-state oracle requires both the upgrade and source repair to be necessary for validation success.

D. RQ3: Which Non-Pass Modes Dominate?

Incomplete repository-wide migration is the dominant non-pass mode: agents often locate the affected dependency surface but fail to propagate its changed contract through all relevant code and artifacts. Failures span type propagation, API surfaces, semantic outputs, compilation and namespaces, and fixture or setup updates, with difficulty varying by configuration and ecosystem.

  • Repository-consistent source repair is the main obstacle across completed non-pass runs.
  • Incomplete migration is the largest adjudicated agent-side category in every configuration, often leaving wrappers, generated state, fixtures, or call sites inconsistent.The analysis used independent annotation and adjudication of non-pass trajectories, patches, and verifier outcomes.
  • 77/203 GPT-5.5, 114/203 Gemini, and 102/203 Claude Opus 4.8 tasks are consensus-hard across included configurations.The remaining tasks are configuration-asymmetric and therefore solvable by some configurations but not others.
  • Cargo is proportionally hardest, with 14/20 GPT-5.5 and 17/20 tasks under both Gemini and Claude Opus 4.8 unsolved by all included configurations.npm/yarn contributes the largest absolute number of consensus-hard tasks, while Maven/Java shows a different pattern.
  • API-surface failures commonly fix one removed symbol while missing wrappers, mocks, factories, return objects, or secondary call sites.
  • Semantic-output failures can compile yet preserve obsolete overflow behavior or generated package-manager snapshots after the dependency declaration changes.
  • Compile-time, namespace, fixture, and setup migrations often repair the first visible error but leave another repository path inconsistent.Examples include Javax-to-Jakarta imports, syn derive paths, generated sources, lockfiles, snapshots, and bootstrap files.
  • The GitLab case requires propagating int to int64 through abstractions and filtering logic, illustrating a type-contract migration rather than a manifest-only update.A partial repair left the merge-request list as []int instead of []int64, producing a hidden-test semantic mismatch.

E. RQ4: What Makes a Dependency-Upgrade Task Hard?

Dependency-upgrade difficulty reflects interacting ecosystem, hidden-test, propagation, and agent-behavior factors rather than a single cause. Hard tasks require repository-wide semantic migration, not merely local edits.

  • Task difficulty: Hard tasks combine ecosystem and tooling friction, rich hidden behavioral oracles, repository-wide propagation requirements, and incomplete agent reasoning.The paper explicitly rejects a single-factor explanation for task difficulty.
  • Ecosystem: 14/20 Cargo tasks and 34/68 npm/yarn tasks are unsolved by all four GPT-5.5 configurations.The same zero-pass pattern affects 17/20 and 45/68 tasks for Gemini, and 18/20 and 43/68 for Claude Opus 4.8.
  • Held-out tests: Mixed direct-behavior tests have 31/55 zero-pass tasks for GPT-5.5, while corrects-only tests have 35/87.Adds-only and fixture/setup-only tests have lower zero-pass counts: 3/14 and 8/47.
  • Agent behavior: Documentation lookup varies sharply, whereas local API/source inspection and test-running evidence appear frequently across configurations.For GPT-5.5, documentation evidence ranges from 20 to 75 trajectories, local inspection from 141 to 170, and test-running from 170 to 197.
  • Propagation depth: The GitLab case requires type-contract propagation through interfaces and helpers, while the Gremlin case links numeric semantics to comparator compatibility and overflow behavior.Both cases show why a plausible local edit is insufficient.

A. Dependency-Upgrade Benchmarks and Repair

Prior benchmarks cover issue resolution, breaking updates, migrations, and environment repair, but DEPBENCH isolates executable dependency-upgrade repair across ecosystems. It separates upgrade changes from source repairs and validates hidden post-upgrade failures.

  • Prior benchmarks: BUMP defines breaking dependency updates as version changes that turn a successful build into a failing one.DEPBENCH extends this setting across ecosystems and treats repair as an agent task.
  • Prior benchmarks: SWE-Chain studies chained release upgrades, whereas DEPBENCH isolates independent downstream-client upgrades across five ecosystems.The benchmarks therefore target different upgrade structures.
  • Prior benchmarks: DI-BENCH infers dependencies needed to execute a repository, rather than repairing code after a known upgrade.This distinguishes dependency discovery from dependency-upgrade repair.
  • Repair approaches: Existing repair approaches address compilation failures, Java projects, multi-step upgrade workflows, update assessment, or dependency-related build repair in narrower settings.DEPBENCH complements these methods with executable evaluation over broader hidden-upgrade failures.
  • Repository-level benchmarks: Repository-level benchmarks span GitHub issue resolution, Java and Python migration, full-CI repair, and workflow tasks involving environment setup and dependency resolution.These provide evaluation templates but do not isolate the same task definition.
  • DEPBENCH: DEPBENCH addresses construction risks through manifest/code/test decomposition, manual audit, held-out tests, and a four-state oracle.BeyondSWE identifies dependency-driven migration as a first-class category, while DEPBENCH operationalizes it with stronger isolation.

VI. THREATS TO VALIDITY

DEPBENCH reports strong task coverage and a substantial performance gap, but its validity depends on reproducibility controls, test-oracle assumptions, source selection, leakage boundaries, and anti-gaming safeguards.

  • Reproducibility: Pinned commits, recorded container configurations, and pre-release validation mitigate changes in registries, base images, package mirrors, and upstream Git references.These controls address reproducibility threats but do not eliminate environmental change over time.
  • Test oracle: The test suite serves as the behavioral oracle, but passing tests do not guarantee semantic correctness and tests may be incomplete or flaky.An oracle-completability gate removes tasks whose tests are too unstable for reliable validation.
  • Selection bias: Dependabot- and Renovate-derived pull requests may not represent manual-upgrade failure modes, motivating a broader future source pool.This is a selection-bias boundary on the benchmark’s task population.
  • Solution leakage: Public upstream pull requests may appear in model pretraining data, so DEPBENCH supports controlled comparison rather than out-of-distribution generalization claims.The benchmark explicitly limits its interpretation because of possible solution leakage.
  • Oracle weakness and gaming: Held-out tests and oracle validation reduce gaming risks, but future versions should add checks against downgrades, test deletion, and validation bypass.These safeguards address agents that might otherwise weaken the evaluation target.
  • Scope and results: DEPBENCH contains 203 oracle-validated, container-native dependency-upgrade tasks across five ecosystems, with the best completed configuration solving 104/203 tasks (51.2%).The benchmark separates version changes, source repairs, and held-out post-upgrade tests.
  • Scope and results: Failures commonly involve incomplete propagation across wrappers, generated artifacts, lockfiles, types, and runtime behavior.The benchmark therefore emphasizes repository-wide semantic consistency and post-upgrade tests rather than visible-test success alone.
Loading 2608.30300v1…