PaperlayerView exact revision on arXiv

Source-linked AI summary

How Quickly Do Development Teams Update Their Vulnerable Dependencies?

Imranur Rahman, Ranindya Paramitha, Nusrat Zahan, William Enck, Laurie Williams

arXiv:2403.17382v4cs.SEcs.CR

TL;DR

Dependency-update metrics need to capture recent practices and vulnerable-dependency updates, but externally reported vulnerability data is unavailable for many packages. This paper introduces MTTUdep and MTTRdep and analyzes them across npm, PyPI, and Cargo. Its findings indicate that MTTUdep can serve only partially as a proxy for MTTRdep when vulnerability information is unavailable.

  • Problem

    MTTRdep cannot be computed for over 99% of packages in each studied ecosystem because they had no vulnerable direct dependency during their lifetime.

    Source: §I · p. 2

  • Method

    The study introduces MTTUdep and MTTRdep and analyzes dependency-update behavior and package characteristics across ecosystems.

    Source: §I · p. 2

  • Results

    The study concludes that MTTUdep can serve partially as a proxy for MTTRdep when vulnerability information is unavailable.

    Source: §IX · p. 13

  • Takeaways & Limitations

    MTTUdep may be used with caution as a proxy for MTTRdep when vulnerability data is unavailable.

    Source: §IX · p. 13

    The OSV advisory dataset may be incomplete, and the study does not account for vulnerability reachability or severity differences.

    Source: §VIII · p. 13

Abstract

from arXiv · show

Industry practitioners are increasingly concerned with software that contains vulnerable versions of third-party dependencies that are included both directly and transitively. To address this problem, projects are encouraged to both (a)~quickly update to non-vulnerable versions of dependencies and (b)~be mindful of the update practices of the dependencies they choose to use. To this end, researchers have proposed metrics to measure the responsiveness of the development teams of the packages in keeping their dependencies updated: Mean-Time-To-Update (MTTU) and Mean-Time-To-Remediate (MTTR). While MTTU covers all dependencies, MTTR quantifies the time needed for a package to update its vulnerable dependencies. However, existing metrics fail to capture important nuances, such as considering floating versions and prioritizing recent updates, leading to inaccurate reflections of a development team's update practices. \textit{The goal of this study is to aid practitioners in understanding how quickly packages update their dependencies.} We propose two novel metrics, Mean-Time-To-Update for dependencies (MTTU) and Mean-Time-To-Remediate for vulnerable dependencies (MTTR), that overcome the limitations of existing metrics. We conduct an empirical study using $163,207$ packages in npm ($117,129$), PyPI ($42,777$), and Cargo ($3,301$) and characterize how the ecosystems differ in MTTU and MTTR, as well as what package characteristics influence MTTU and MTTR. We found that most packages have a relatively fast dependency update practice. We further study whether MTTU can be used as a proxy for MTTR when sufficient vulnerability data is not available. As we did not find enough statistical evidence for a strong proxy, our findings suggest that MTTU could only be partially used (may be used but with caution) as a proxy for MTTR when vulnerability data is not available.

I. INTRODUCTION

Packages need measures of how quickly teams update dependencies, but existing metrics miss important update practices and vulnerable-dependency data is sparse. The study introduces dependency-update metrics and examines them across three ecosystems, including whether MTTU can serve as a proxy for MTTR.

I. INTRODUCTION

  • 86% of codebases contain at least one vulnerable open-source dependency, and 81% contain high- or critical-risk externally reported dependency vulnerabilities.Many codebases had vulnerable dependencies for over two years despite a fixed version being available; fear of breaking changes and update costs are cited as reasons for delay.Source: §I · p. 1
  • MTTU measures time to update all dependency versions, while MTTR measures time to update vulnerable dependency versions to a fixed version.These metrics provide a historical view of update responsiveness, unlike a check focused on recent project activity.Source: §I · p. 1
  • Existing dependency metrics may mishandle floating constraints, fail to localize vulnerabilities, fragment results across package versions, and ignore recency; vulnerable-dependency metrics are also not publicly available.These limitations make it difficult to assess a package’s overall update practice or compare its current maintenance with that of packages with different lifespans.Source: §I · p. 1
  • The study proposes MTTUdep and MTTRdep algorithms designed to address existing metric-calculation challenges and make measurements responsive to recent update practices.The stated goal is to aid practitioners in understanding how quickly packages update dependencies.Source: §I · p. 2
  • The empirical analysis computes the proposed metrics for 163,207 npm, PyPI, and Cargo packages using package-version release information and dependency relations.The study also examines ecosystem differences, package characteristics, and whether MTTUdep can proxy for MTTRdep.Source: §I · p. 2
  • MTTRdep cannot be computed for 99.51% of npm, 99.17% of PyPI, and 99.95% of Cargo packages because they had no vulnerable direct dependency during their lifetime.The paper tests whether MTTUdep can provide a practical proxy for packages without externally reported vulnerability data.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Prior work studied technical lag, outdated dependencies, and vulnerability remediation, but generally treated general updates and security updates separately. This paper focuses on both dependency-update measures and their relationship.

II. BACKGROUND AND RELATED WORK

  • Technical-lag research examined outdated dependencies in npm and other ecosystems, reporting lag growth over package lifetimes and identifying missed updates and pinned dependencies as contributors.Studies also reported a median 3.5-month technical lag in npm and a median time lag over a year for community Docker images.Source: §II · pp. 2–3
  • Studies of outdated or vulnerable dependencies reported widespread staleness and delayed security updates, but focused on general updates or security updates rather than both together.Examples include 81.5% of projects continuing to use outdated dependencies and critical vulnerabilities persisting over a year on average in most programming languages.Source: §II · p. 3
  • MTTR research often concerns vulnerabilities in a package itself, while this study measures how packages update vulnerable dependencies; some industry metric procedures are proprietary or undisclosed.Sonatype’s 2024 Maven report could not be reproduced because its methodology was unavailable.Source: §II · p. 3

III. CHALLENGES APPLYING EXISTING UPDATE METRICS

The section introduces time lag as a measure of dependency outdatedness and illustrates how it is calculated for a package’s dependency. The example compares the dependency version allowed at each package release with the latest version then available.

III. CHALLENGES APPLYING EXISTING UPDATE METRICS

  • The section uses technical lag to motivate design gaps in existing dependency-update metrics.It presents tLag as an example of an existing update metric before discussing why new metrics are needed.Source: §III · p. 3

A. Technical Lag Using tLag

  • Time lag measures the difference between the release dates of the latest dependency version and the version allowed by the package’s constraint when that package version was released.The example computes tLag by subtracting the last-allowed version’s release date from the latest version’s release date.Source: A. Technical Lag Using tLag · p. 3
  • For five codemod-cli releases, the reported tLag values for simple-git were 707, 78, 78, 100, and 0.Table I and Figure 1 illustrate the computation for this package-dependency relationship.Source: Table I · p. 3, A. Technical Lag Using tLag · p. 4, Figure 1 · p. 4

B. Design Gaps

The design-gap discussion argues that dependency-update metrics should accommodate floating constraints, aggregate package-level behavior, isolate vulnerable updates, and emphasize recent practice. The following section introduces design choices intended to fill those gaps and define MTTUdep and MTTRdep.

B. Design Gaps

  • Floating constraints can permit automatic dependency updates, but tLag does not model updates that become allowable after a package release, including automatic vulnerability remediation.The paper identifies this as a concern when measuring exposure to vulnerabilities.Source: B. Design Gaps · p. 4
  • tLag reports values per package version and dependency, while a package-level metric would combine those data into one value that developers can more readily understand and compare.The paper notes that interpreting many values across a package’s releases and dependencies is difficult.Source: B. Design Gaps · p. 4
  • A security-oriented metric should isolate updates to vulnerable dependencies, whereas tLag combines security fixes with bug fixes and feature updates.Prior work had not measured tLag specifically for vulnerable-dependency update practices.Source: B. Design Gaps · p. 4
  • tLag does not weight recent updates more heavily, although the paper argues that recent practice better reflects ongoing maintenance and helps compare packages with different lifespans.For example, the paper says update practice from ten years ago should not count as heavily as current practice.Source: B. Design Gaps · p. 4
  • The paper compares its metrics with technical-lag measures and dependency freshness using version sequence number in Table II.The comparison includes time lag and version lag from technical lag and dependency freshness using VSN.Source: B. Design Gaps · p. 4

IV. RQ1 NOVEL DEPENDENCY UPDATE METRICS

  • The next section presents design choices intended to fill the identified gaps and develop MTTUdep and MTTRdep.It also previews a case study and a detailed methodology for the metrics.Source: §IV · p. 4

A. Intuition

The metrics evaluate direct dependencies over release-based intervals, accounting for floating constraints and weighting recent update behavior more heavily. A running example illustrates the calculations; transitive dependencies remain outside the metrics’ scope.

A. Intuition

  • Splitting direct-dependency histories at package or dependency releases captures floating constraints without favoring pinning over floating, then downweights older intervals exponentially.The design resolves constraints using versions available at each interval’s start.Source: A. Intuition · pp. 4–5
  • An interval is updated when its resolved version is the highest available, and remediated when it is not vulnerable despite a fix being available.The metrics aggregate these interval labels for each package-dependency relationship.Source: A. Intuition · p. 5
  • The metrics assess developer responsiveness only for direct dependencies and do not penalize vulnerable transitive dependencies.Developers influence transitive versions indirectly through their direct-dependency choices.Source: A. Intuition · p. 5

B. An Example With Our Metrics

  • In the codemod-cli/simple-git example, codemod-cli moved to fixed version 3.4.0 after a vulnerability was found in version 2.48.0.The example considers CVE-2022-24433, fixed in version 3.3.0.Source: B. An Example With Our Metrics · p. 6
  • TTUdep is 2.4 days versus a naive 614 days, while TTRdep is 3.67 days versus a naive 11 days.The example attributes the lower weighted values to reduced emphasis on older intervals.Source: B. An Example With Our Metrics · p. 6

C. Metrics Definitions

The definitions aggregate outdated direct-dependency time for MTTUdep and outdated, vulnerable direct-dependency time for MTTRdep. Exponential weighting emphasizes recent intervals, with a two-year half-life in the study’s configuration.

C. Metrics Definitions

  • MTTUdep is the weighted aggregate of the time a package uses outdated direct-dependency versions.The package metric aggregates across its direct dependencies.Source: C. Metrics Definitions · p. 6
  • The exponential weight gives recent intervals near-full weight and older intervals weight that decays toward zero, using a two-year half-life.The study sets the decay factor using τ = 2 years.Source: C. Metrics Definitions · p. 6
  • Exponential weighting was chosen over linear and inverse alternatives because it is more responsive to recent data and configurable through the half-life.The authors also describe it as more robust than inverse weighting.Source: C. Metrics Definitions · p. 6
  • MTTRdep is the weighted aggregate of the time a package uses outdated and vulnerable direct-dependency versions.The metric is defined for a package’s direct dependencies.Source: C. Metrics Definitions · pp. 6–7

V. EMPIRICAL STUDY METHODOLOGY

The study gathered vulnerability, package-version, dependency, and package-characteristic data for npm, PyPI, and Cargo. It filtered the initial dataset to packages meeting age, recent-activity, and dependency criteria.

A. Data Collection

  • Vulnerability data came from OSV on 2024-09-12, while package versions and dependency data came from deps.dev on 2024-08-20.The selected ecosystems were npm, PyPI, and Cargo.Source: A. Data Collection · p. 7
  • Package characteristics were downloaded from libraries.io and included contributors, dependencies, dependents, releases, SourceRank, forks, stars, major-version counts, and package age.The researchers manually inspected a sample to check the downloaded characteristics.Source: A. Data Collection · p. 7

B. Package Inclusion and Exclusion Criteria

  • 163,207 packages met the selection criteria: at least two years old, activity in the last two years, and at least one dependency.The final set comprised 117,129 npm, 42,777 PyPI, and 3,301 Cargo packages; the criteria may exclude younger or inactive packages that are not abandoned.Source: B. Package Inclusion and Exclusion… · p. 7

C. Metrics Implementation

The implementation labels resolved dependency intervals as updated or remediated, then applies statistical methods to assess MTTUdep as a proxy for MTTRdep and examine package-characteristic associations.

C. Metrics Implementation

  • For each resolved interval, the implementation flags whether the selected dependency is current and whether it remains vulnerable despite an available fix.The data are stored in PostgreSQL, with indexes added to support metric calculations on the large dataset.Source: C. Metrics Implementation · p. 7

D. Proxy Analysis

  • Proxy analysis combines TOST with Mann–Whitney tests, OLS regression with a Wald test, sensitivity indices, and Pearson and Spearman correlations.The criteria include equivalence, significant slope, fit indices above 0.90, and correlation strength.Source: D. Proxy Analysis · p. 8

E. Package Characteristics Analysis

  • The characteristic analysis tests associations between nine package-level variables and MTTUdep or MTTRdep.The tested variables include contributor, dependent, and dependency counts; the hypotheses expect fewer contributors or dependents to align with less-updated dependencies, but fewer dependencies with more-updated ones.Source: E. Package Characteristics Analysis · p. 8
  • The hypotheses also predict update associations for version and major-version counts, package age, SourceRank, forks, and stars.They predict more-updated dependencies for packages with fewer major versions or lower age, and less-updated dependencies for those with fewer releases, lower SourceRank, fewer forks, or fewer stars.Source: E. Package Characteristics Analysis · p. 8

F. Regression Analysis

Across npm, PyPI, and Cargo, dependency-update times are generally low but have long tails, while small interquartile ranges indicate consistency within ecosystems. The regression analysis uses package characteristics to model MTTUdep and MTTRdep, with a Bonferroni-adjusted significance threshold.

F. Regression Analysis

  • The regression models relate package characteristics to MTTUdep or MTTRdep, using p < 0.0055 as the significance threshold after Bonferroni correction.The correction accounts for testing nine characteristics; the models report coefficients and p-values for the characteristics.Source: F. Regression Analysis · pp. 8–9

A. RQ2: How do packages in npm, PyPI, and Cargo differ in MTTU and MTTR?

  • 50% of npm packages have MTTUdep below 51 days, while the maximum MTTUdep is 2653 days.The distributions are right-skewed, with comparable interquartile ranges: Cargo 1–39 days, npm 4–45 days, and PyPI 11–54 days.Source: A. RQ2: How do packages in npm, PyPI,… · p. 9
  • MTTRdep was computed for 22,513 packages, with interquartile ranges of 6–24 days in Cargo, 10–42 in npm, and 12–45 in PyPI.The MTTRdep distributions are also right-skewed and have long tails.Source: A. RQ2: How do packages in npm, PyPI,… · p. 9
  • Weighting recent updates yields lower MTTRdep values than prior work reported, although some packages still took months or longer to remediate known vulnerabilities.Older dependency-update practices have less impact on the weighted metrics.Source: A. RQ2: How do packages in npm, PyPI,… · p. 9
  • Most packages have relatively fast dependency-update practices, but long tails show that some lag for months or longer across ecosystems.The small interquartile ranges indicate consistency within each ecosystem; the authors note that update challenges are not specific to one ecosystem.Source: A. RQ2: How do packages in npm, PyPI,… · p. 9

B. RQ3: Can MTTU serve as a proxy for MTTR?

The analysis finds that MTTUdep is only a partial proxy for MTTRdep and should be used with caution. Package characteristics generally have weak associations with update times, and regression models explain little of their variation.

Proxy analysis

  • TOST results showed statistical equivalence except in Cargo, while correlation analyses found moderate positive relationships between MTTUdep and MTTRdep.The MTTUdep coefficient was positive and significant in regression, but it explained 30.5% of MTTRdep variance and did not provide an acceptable fit.Source: B. RQ3: Can MTTU serve as a proxy for… · pp. 9–10
  • MTTUdep is only a partial proxy for MTTRdep: three criteria were strongly met, three moderately, and three failed, so use requires caution.The authors encourage further research to identify better proxies.Source: B. RQ3: Can MTTU serve as a proxy for… · p. 10

Correlation analysis

  • Contributor, dependent, dependency, age, major-version, and SourceRank measures showed weak associations with MTTUdep or MTTRdep.The reported correlations include contributors at -0.07 for both metrics, dependency count at -0.10 and -0.11, and age at 0.10 and 0.07.Source: B. RQ3: Can MTTU serve as a proxy for… · pp. 10–11
  • Version count had moderate negative correlations with MTTUdep (-0.55) and MTTRdep (-0.41), while the tested characteristics overall showed weak or negligible correlations.The paper’s takeaway is that higher version counts are associated with lower MTTUdep and MTTRdep.Source: B. RQ3: Can MTTU serve as a proxy for… · pp. 10–11
  • Forks and stars were not significantly related to MTTUdep or MTTRdep, despite positive correlations with maintainer count of 0.86 and 0.85, respectively.The authors note that more forks or stars do not directly indicate faster dependency updates.Source: B. RQ3: Can MTTU serve as a proxy for… · p. 11

Regression analysis

  • The regression models explained 3.9% of MTTUdep variation and 2.6% of MTTRdep variation, despite being statistically significant.The low R2 values indicate that other factors also matter to update times.Source: B. RQ3: Can MTTU serve as a proxy for… · p. 11
  • In the MTTUdep model, older package age had a positive coefficient, while SourceRank and major-version count had negative coefficients.The model found all listed independent variables except dependent count and forks statistically significant at p < 0.0055.Source: B. RQ3: Can MTTU serve as a proxy for… · p. 11
  • In the MTTRdep model, package age had a small positive coefficient, while major-version count, dependency count, contributors, and SourceRank had negative coefficients.Dependent count, forks, and stars were not statistically significant in this model.Source: B. RQ3: Can MTTU serve as a proxy for… · pp. 11–12

A. Practical Implications For Developers

Use MTTRdep when vulnerability data is available and MTTUdep as a cautious, partial proxy otherwise; the discussion also recommends routine dependency updates and further study of factors shaping update practices.

A. Practical Implications For Developers

  • 0.49% of npm, 0.83% of PyPI, and 0.05% of Cargo packages had directly depended on a vulnerable package, limiting where MTTRdep could be computed.MTTUdep met six of nine proxy criteria, so the authors advise using it cautiously when MTTRdep is unavailable.Source: A. Practical Implications For Developers · p. 12
  • Allowing automatic minor and patch updates alongside manual major updates could keep MTTUdep and MTTRdep minimal, provided developers use SEMVER correctly.The authors caution that floating versions may be infeasible when breaking changes are a concern.Source: A. Practical Implications For Developers · p. 12

B. Practical Implication For Researchers

  • Packages with many contributors or dependents did not have better MTTUdep or MTTRdep, and both regression models had R2 values below 10%.The authors identify team dynamics, dependency constraints, testing effort, and codebase size as possible unmodeled factors warranting further study.Source: B. Practical Implication For Researchers · p. 12

C. Practical Implication For Tool Builders

The paper proposes incorporating its dependency-update metrics into developer and security-risk tools, while noting trade-offs and scope limits. Its conclusion reports partial proxy value for MTTUdep when vulnerability information is unavailable.

C. Practical Implication For Tool Builders

  • Dependency-management tools and security-risk assessments such as OpenSSF Scorecard could incorporate the metrics as accessible, actionable package measures.The authors suggest the metrics could help developers assess package security risk.Source: C. Practical Implication For Tool… · p. 12

D. Gaming Metrics

  • Floating constraints can minimize time spent with outdated versions and enable automatic security fixes, but can also propagate malicious package updates.The paper frames this as a trade-off, citing the xz incident as an example of the risk.Source: D. Gaming Metrics · pp. 12–13

VIII. THREATS TO VALIDITY

  • The analysis covers runtime dependencies only, treats vulnerabilities equally, and does not assess whether vulnerable versions are exploitable or reachable.The authors also note limits to generalizing findings to other ecosystems and possible incompleteness in OSV.dev advisories.Source: §VIII · p. 13

IX. CONCLUSION AND FUTURE WORKS

  • MTTUdep can serve partially as a proxy for MTTRdep when vulnerability information is unavailable, based on analysis across npm, PyPI, and Cargo.The study also analyzed relationships between package characteristics and dependency-update behavior.Source: §IX · p. 13

DATA AVAILABILITY

Loading 2403.17382v4…