PaperlayerView exact revision on arXiv

Source-linked AI summary

A Novel Approach for Estimating Truck Factors

Guilherme Avelino, Leonardo Passos, Andre Hora, Marco Tulio Valente

arXiv:1604.06766v1cs.SE

TL;DR

The work targets empirically supported truck-factor estimates for a metric used to assess concentration of team knowledge. It proposes an automated heuristic for estimating truck factors. Among 87 systems, 65% have TF ≤2; in valid survey answers, 84% agreed or partly agreed that the TF authors were the systems’ main authors, and 53% gave positive or partly positive responses about estimated truck factors.

  • Problem

    The work targets the need for truck-factor estimates backed by empirical evidence, for a metric used to assess knowledge concentration in teams.

    Source: §I · p. 1

  • Method

    An automated heuristic estimates truck factor from developer-file authorship mappings, removing top authors until remaining authors cover less than 50% of current files.

    Source: §I · p. 1, §III · p. 2, A. Realization · p. 3

  • Results

    87 systems (65%) have TF ≤2; among valid survey answers, 84% agreed or partly agreed that TF authors were the systems’ main authors, and 53% gave positive or partly positive responses about estimated truck factors.

    Source: §X · p. 10

  • Takeaways & Limitations

    Truck-factor estimates can help managers and maintainers assess personnel-loss risks and identify knowledge silos for timely action planning.

    Source: §I · p. 1, §II · p. 2

    The findings cannot be directly generalized to other systems, mainly closed-source ones, and do not address all development-environment factors that may affect truck-factor results.

    Source: §VIII · p. 9

Abstract

from arXiv · show

Truck Factor (TF) is a metric proposed by the agile community as a tool to identify concentration of knowledge in software development environments. It states the minimal number of developers that have to be hit by a truck (or quit) before a project is incapacitated. In other words, TF helps to measure how prepared is a project to deal with developer turnover. Despite its clear relevance, few studies explore this metric. Altogether there is no consensus about how to calculate it, and no supporting evidence backing estimates for systems in the wild. To mitigate both issues, we propose a novel (and automated) approach for estimating TF-values, which we execute against a corpus of 133 popular project in GitHub. We later survey developers as a means to assess the reliability of our results. Among others, we find that the majority of our target systems (65%) have TF <= 2. Surveying developers from 67 target systems provides confidence towards our estimates; in 84% of the valid answers we collect, developers agree or partially agree that the TF's authors are the main authors of their systems; in 53% we receive a positive or partially positive answer regarding our estimated truck factors.

I. INTRODUCTION

The literature lacks a formal, empirically supported way to estimate truck factor (TF). This paper proposes an automated approach and evaluates its estimates using GitHub projects and developer surveys.

I. INTRODUCTION

  • Most literature lacks a formal TF definition or estimation method; Zazworka et al. and follow-ups, lack real-world reliability evidence.The paper frames its approach as addressing both gaps.Source: §I · p. 1
  • The automated approach is applied to 133 GitHub systems; surveys across 67 systems report 84% agreement on authors and 53% positive responses on estimated TFs.The data and estimates are made publicly available for external validation.Source: §I · p. 1

II. TRUCK FACTOR: AN EXAMPLE FROM THE EARLY DAYS OF PYTHON

  • A 1994 mailing-list discussion framed Python’s dependence on Guido as an adoption risk, with concentrated knowledge threatening continuity or causing delays.The paper uses the example to motivate measuring TF so managers and maintainers can identify knowledge silos and plan timely action.Source: §II · p. 2

III. PROPOSED APPROACH

  • The proposed workflow lists source files, resolves developer aliases, traces file histories, determines authorship, and estimates system TF.Authorship includes creators and other developers whose later contributions indicate they can maintain a file.Source: §III · p. 2

A. Realization

The realization processes a Git repository snapshot and its history to identify file authors, then estimates TF by iteratively removing the author responsible for the most files.

A. Realization

  • The latest master snapshot supplies source files; documentation, examples, ignored files, and identified third-party code are excluded.Linguist is used to identify third-party files.Source: A. Realization · p. 2
  • Git users are merged by shared email or similar names, allowing at most one character edit under the Levenshtein-distance rule.The result maps Git users to a single developer name.Source: A. Realization · p. 3
  • File histories are traced across commits, recording each file path, mapped developer, and change type: addition, modification, or rename.The history is collected with git log --find-renames.Source: A. Realization · p. 3
  • Authorship is measured with normalized degree-of-authorship (DOA),, combining first authorship, the developer’s deliveries, and other developers’ changes.The model treats first authorship as its strongest predictor; deliveries raise DOA and others’ changes lower it more slowly.Source: A. Realization · p. 3
  • An author qualifies when normalized DOA > k and absolute DOA ≥ m; the reported settings are k = 0.75 and m = 3.293.Both thresholds are configurable.Source: A. Realization · p. 3
  • The greedy TF algorithm removes the current author with the most files until remaining authors cover less than 50% of files; ties go to the first found.The procedure starts at TF zero and stops when coverage drops below 0.5 or the author list is empty.Source: A. Realization · p. 3

IV. VALIDATION METHODOLOGY

The validation methodology selects GitHub systems, estimates their truck factors, and surveys developers to assess the estimates and gather further insights.

IV. VALIDATION METHODOLOGY

  • The validation selects 133 GitHub systems and estimates the truck factor for each target system.The section describes corpus selection and setup of the estimation approach.Source: §IV · pp. 3–4
  • When multiple authors tie for the most files, the algorithm selects the first one found.This tie rule is specified for the greedy estimation procedure.Source: §IV · p. 3
  • Developer surveys are used to validate the estimates and gather further insights.The section discusses how developers are surveyed as part of the validation methodology.Source: §IV · p. 4

A. Selection of Target Subjects

The study selects popular GitHub repositories across six languages, applies size and history filters, and retains 133 distinct target systems. The resulting corpus spans more than 373,000 files and 2 million commits.

A. Selection of Target Subjects

  • Repositories were selected from the 100 most-starred projects in each of six languages, filtered by developer, commit, and file counts, then intersected.The filters removed each language’s bottom quartile on all three metrics.Source: A. Selection of Target Subjects · p. 4
  • Filtering 135 candidates for migration history and duplicate subjects left 133 systems for analysis.Repositories with over 50% of files added in fewer than 20 commits were removed as likely incomplete migrations.Source: A. Selection of Target Subjects · p. 4
  • The 133 targets contain over 373K files, 41 MLOC, more than 2 million commits, and over 60K developers.The corpus includes 33 Ruby systems and 17 PHP systems.Source: A. Selection of Target Subjects · p. 4

B. Setting up Inputs

The approach requires a manually prepared list of ignorable files and configurable authorship thresholds. The paper describes how it excludes external or optional code and calibrates the thresholds.

B. Setting up Inputs

  • The approach takes an ignorable-source-file list and configurable DOA thresholds k and m as inputs.The section explains how both inputs were set.Source: B. Setting up Inputs · p. 4
  • The ignorable-file list targets third-party libraries and plugin code, which may distort a system’s truck factor.For Linux, the authors excluded driver-related code because drivers are generally optional features.Source: B. Setting up Inputs · p. 4
  • Homebrew’s estimated TF falls from 250 to 2 when its Library/Formula directory is excluded, illustrating sensitivity to external code.The authors also ignore the Casks directory in homebrew-cask.Source: B. Setting up Inputs · p. 5
  • Thresholds k = 0.75 and m = 3.293 were selected after inspecting 120 files from six popular systems and comparing authorship with git-blame rankings.With these thresholds, 64% of selected authors ranked first and 91% ranked among the top three in git-blame.Source: B. Setting up Inputs · p. 5

C. Survey Design and Application

The authors designed and piloted a developer survey to assess authorship and truck-factor estimates and solicit practices for mitigating developer loss. They posted it through GitHub issues and analyzed responses from participating projects.

C. Survey Design and Application

  • Three authors iteratively reviewed the survey questions and piloted them to identify and fix communication problems.The questions were finalized after the pilot study.Source: C. Survey Design and Application · p. 5
  • The survey asked whether top-ranked authors were the main developers, whether losing TF-responsible developers would put projects in trouble, and which practices could attenuate that loss.The authors distinguish main developers from file creators in describing the first question.Source: C. Survey Design and Application · p. 5
  • Survey issues were opened in 114 of 133 projects; after excluding five issues without a top-10 contributor’s answer, 62 systems remained.The study collected 170 messages from 106 respondents across 67 systems before exclusions.Source: C. Survey Design and Application · p. 5
  • Two authors independently classified responses to the first two questions into four positions and crosschecked their coding.The categories were agree, partially agree, disagree, and unclear.Source: C. Survey Design and Application · pp. 5–6
  • Answers to the third question were categorized using principles of grounded-theory open coding.Source: C. Survey Design and Application · p. 6

V. TRUCK FACTOR ESTIMATES

Most target systems have low truck factors, while a small proportion of developers account for authorship in many systems. The section also reports file filtering and examples of concentrated contributions.

A. Preceding Output

  • Source-file filtering leaves 243,660 files (33 MLOC), 34% fewer than the original set, and reduces developers by 28% and commits by 39%.Source: A. Preceding Output · p. 6
  • The author proportions’ first, second, and third quartiles are 16%, 23%, and 36%, respectively.Four of the top ten systems by author ratio have private-organization support.Source: A. Preceding Output · p. 6, Figure 4 · p. 6
  • Backbone’s six authors account for 67% of commits, while two Sprockets authors account for 95%.Sprockets has 61 developers associated with its history; 27 others each contributed a single commit changing one line.Source: A. Preceding Output · p. 6

B. Results

  • 45 systems (34%) have TF = 1 and 42 systems (31%) have TF = 2; the truck-factor quartiles are 1, 2, and 4.The distribution also includes outliers, led by torvalds/linux at TF = 57.Source: B. Results · p. 6, Figure 5 · p. 6

VI. TF VALIDATION: SURVEYING DEVELOPERS

The survey finds substantial but incomplete agreement with both the ranked-author lists and estimated truck factors. Respondents also identify changing activity, succession, documentation, and community support as relevant to interpreting or mitigating knowledge concentration.

VI. TF VALIDATION: SURVEYING DEVELOPERS

  • The analyzed responses came from 106 respondents, including 84 top-10 contributors, across 62 participating systems.The analysis filtered issue discussions to include only issues with at least one message from a top-10 project contributor.Source: §VI · p. 7
  • 31 systems (50%) fully agreed that the listed developers were their projects’ main developers, while 18 (29%) partially agreed.The main partial-agreement concern was that older developers’ accumulated authorship could outweigh newer developers’ current activity.Source: Table III · p. 7, §VI · p. 7
  • Nine systems (15%) disagreed with the author lists; six respondents said other contributors had taken responsibility, while other disagreements involved generated code.One example involved a developer who was no longer active and two suggested contributors with more recent commits; their authored-file shares were 41% and 26% for the listed authors versus 9% and 17% for the suggested contributors.Source: §VI · p. 7, Figure 6 · p. 7
  • 24 systems (39%) agreed with the truck-factor results, while 27 disagreed; 21 of those said others could take over the project.Some respondents described succession after a truck-factor author had already left, with documentation and willing developers helping transitions.Source: Table IV · p. 7, §VI · pp. 7–8
  • Documentation was mentioned 36 times as a practice to attenuate truck-factor loss, followed by active communities (15), automatic tests (10), and code legibility (10).The survey also notes active communities and paid developers as reasons some respondents were less concerned about losing top-ranked authors.Source: Table V · p. 8, §VI · p. 8

VII. DISCUSSION

The discussion presents lessons from the study and directions for future research on truck-factor measurement and applications.

VII. DISCUSSION

  • The discussion sets out study lessons and future research directions for truck-factor measurement and applications.Source: §VII · p. 8

A. DOA Results

Developers of 49 systems (84% of valid answers) agreed or partially agreed with the DOA results, while some respondents questioned its emphasis on first authorship. They suggested considering more recent development history.

A. DOA Results

  • 49 systems (84% of valid answers) received developer agreement or partial agreement with the DOA results.Source: A. DOA Results · p. 8
  • Six developers disagreed because the model can retain first authorship when an original bulk-code contributor later becomes less active, such as after switching to a leadership or mentoring role.Source: A. DOA Results · p. 8
  • Some developers suggested computing DOA from only recent history, such as commits from the last year; one noted old code may be unfamiliar even to its original author.Source: A. DOA Results · p. 8

B. Challenges on Computing Truck Factors

Survey responses support the heuristic’s estimates, while respondents and validity checks identify limits in automatic authorship-based measurement. The findings are bounded by uncertain authorship weights, incomplete development histories, and the study’s open-source setting.

B. Challenges on Computing Truck Factors

  • Respondents cited inaccessible institutional memory as an automation challenge and usually named documentation as the best practice for a truck-factor event.Developers also recognized possible effects on systems’ public reputations.Source: B. Challenges on Computing Truck Factors · p. 9
  • 30 systems (53% of valid answers) received positive or partially positive assessments of the estimated truck factors.The survey received answers for 67 of 114 systems, a 59% response ratio.Source: B. Challenges on Computing Truck Factors · pp. 8–9
  • The heuristic treats files equally and ignores when they were last changed, although file criticality can be unclear and stable files may not need their original authors.The authors discard some files when they produce highly skewed results and suggest considering recently changed files where stable files are common.Source: B. Challenges on Computing Truck Factors · p. 9

VIII. THREATS TO VALIDITY

  • Authorship weights derived from other systems may not be optimal for GitHub, though the authors cite robustness evidence and compare 162 author-file pairs against git-blame.The comparison suggested the degree-of-authorship values were reliable.Source: §VIII · p. 9
  • Estimates can be sensitive to lost development history; migration detection and rename handling mitigate some cases, but the impact of copies and file splits needs further investigation.The approach considers changes in target files to identify their authors.Source: §VIII · p. 9
  • The findings cannot be directly generalized to other systems, especially closed-source ones, and development policies or processes may affect truck-factor results.The study selected real-world systems across six programming languages.Source: §VIII · p. 9

IX. RELATED WORK

Prior truck-factor approaches had limits in scale and real-world validation; this study distinguishes itself through its authorship model, broad system sample, and developer validation. Its conclusion reports low truck factors in many sampled systems and identifies documentation as the most effective practice named by surveyed developers.

IX. RELATED WORK

  • Earlier work evaluated small student or research-group projects, and Zazworka’s algorithm scaled only to projects with 30 or fewer developers, while 122 of this study’s 133 systems exceeded 30.The cited work also describes that algorithm’s implementation as NP-hard.Source: §IX · p. 9
  • The study differs from prior work by using the DOA model for file authorship, evaluating real-world systems in six languages, and validating results with expert developers.Source: §IX · pp. 9–10

X. CONCLUSION

  • 87 systems (65%) had TF ≤2, and developer surveys from 67 systems yielded 84% agreement or partial agreement on main authors and 53% positive or partially positive assessments of estimates.The conclusion presents the heuristic-based approach as an estimate of knowledge concentration among team members.Source: §X · p. 10
  • Surveyed developers identified documentation as the most effective practice for overcoming a truck-factor event, followed by an active community and automatic tests.Source: §X · p. 10
  • Future work will evaluate suggested heuristic improvements, including recent-file filtering and line-level authorship, and compare results with industrial and closed systems.Source: §X · p. 10
Loading 1604.06766v1…