PaperlayerView exact revision on arXiv

Source-linked AI summary

Categorizing Bugs with Social Networks: A Case Study on Four Open Source Software Communities

Marcelo Serrano Zanetti, Ingo Scholtes, Claudio Juan Tessone, Frank Schweitzer

arXiv:1302.6764v2cs.SEcs.LGcs.SInlin.AOphysics.soc-ph

TL;DR

Bug triage can be laborious in large OSS projects, where many resolved reports are duplicates, invalid, or incomplete. The paper uses nine reporter-network measures with an SVM to classify report validity; across the studied communities, the scheme achieved up to 90.3% precision and a 0.92 F-score, with performance dependent on the project.

  • Problem

    Bug triage is laborious in large OSS projects, and many resolved reports are duplicates, invalid, or incomplete.

    Source: §1 · pp. 1–2

  • Method

    The authors apply an SVM to nine topological measures of bug reporters’ positions in collaboration networks to classify whether reports are valid or faulty.

    Source: §7 · p. 21

  • Results

    Depending on the project, the classifier achieved up to 90.3% precision and an F-score of up to 0.92; valid-report centrality was significantly higher in three studied projects.

    Source: §7 · p. 21

  • Takeaways & Limitations

    The authors describe the network-measure classifier as sufficiently accurate yet simple enough to be practically relevant to support-infrastructure design.

    Source: §7 · p. 21

    The network analysis uses a fixed 30-day window; the authors are investigating whether tuning window size separately for each community could improve performance.

    Source: §6 · p. 20

Abstract

from arXiv · show

Efficient bug triaging procedures are an important precondition for successful collaborative software engineering projects. Triaging bugs can become a laborious task particularly in open source software (OSS) projects with a large base of comparably inexperienced part-time contributors. In this paper, we propose an efficient and practical method to identify valid bug reports which a) refer to an actual software bug, b) are not duplicates and c) contain enough information to be processed right away. Our classification is based on nine measures to quantify the social embeddedness of bug reporters in the collaboration network. We demonstrate its applicability in a case study, using a comprehensive data set of more than 700,000 bug reports obtained from the Bugzilla installation of four major OSS communities, for a period of more than ten years. For those projects that exhibit the lowest fraction of valid bug reports, we find that the bug reporters' position in the collaboration network is a strong indicator for the quality of bug reports. Based on this finding, we develop an automated classification scheme that can easily be integrated into bug tracking platforms and analyze its performance in the considered OSS communities. A support vector machine (SVM) to identify valid bug reports based on the nine measures yields a precision of up to 90.3% with an associated recall of 38.9%. With this, we significantly improve the results obtained in previous case studies for an automated early identification of bugs that are eventually fixed. Furthermore, our study highlights the potential of using quantitative measures of social organization in collaborative software engineering. It also opens a broad perspective for the integration of social awareness in the design of support infrastructures.

1 Introduction

Bug triage in large OSS projects can consume substantial resources, while many resolved reports are duplicates, invalid, or incomplete. This study tests whether reporters’ collaboration-network positions can indicate report quality and support automated classification.

1 Introduction

  • More than 50,000 of Mozilla Firefox’s 64,000-plus resolved reports (about 78%) were duplicates, invalid, or incomplete.Invalid reports describe user error rather than a software issue; incomplete reports lack information needed to reproduce the alleged bug.Source: §1 · p. 2
  • For the four OSS communities studied, reporter centrality in the collaboration network is indicative of bug-report quality.The paper examines this relationship and uses quantitative measures of network position for automated classification.Source: §1 · pp. 2–3
  • The network-measure classifier achieved up to 90.3% precision with 38.9% recall for valid bug reports across the four studied communities.The approach extends previous studies of automated classification of bugs eventually fixed, using a more comprehensive data set and more sophisticated methods.Source: §1 · p. 2
  • The study combines quantitative measures of reporters’ positions in evolving community structures with a support vector machine.The paper presents the approach as practical for integration into bug-tracking platforms.Source: §1 · pp. 2–3

2 Social Aspects in Bug Report Processing

Prior work has applied report content, language processing, handling dynamics, coordination patterns, and reporter reputation to bug classification. This paper situates network centrality as a social signal related to individual report outcomes.

2 Social Aspects in Bug Report Processing

  • Earlier network research linked developer degree to bug-fixing rate, while this study examines whether reporters’ centrality relates to whether reports concern actual software issues.Other work examined communication structures in relation to code failure proneness and build failures,.Source: §2 · pp. 3–4
  • Automated bug-triage models used submission-time and post-submission report data from the first hours and days, increasing predictive power 5% over pure chance; Eclipse models achieved 62.9% precision and 84.5% recall predicting bugs reopened after closure.Source: §2 · p. 4
  • Machine-learning studies also addressed developer assignment, bug tossing, and bug-category prediction from descriptions and comments.The cited work on bug tossing defines it as simultaneous assignment of bugs to multiple developers.Source: §2 · p. 4
  • Reporter reputation, measured partly by the fraction of prior reports eventually fixed, yielded 68% precision and 64% recall for predicting fixed bugs.The same approach was also reported as successful for predicting reopened bugs.Source: §2 · p. 4

3 Study Design and Methodology

The study asks whether reporters’ positions in evolving collaboration networks relate to report quality and can predict valid bugs. It analyzes resolved Bugzilla reports from four OSS projects using time-stamped interactions and network measures.

3 Study Design and Methodology

  • The study asks whether reporters’ network positions relate to report quality and whether quantitative position measures can predict valid bugs.Its methodology uses nine topological measures and a machine-learning classifier, restricted to information available when a report is submitted.Source: §3 · p. 5

3.3 Network Measures

  • The analysis quantifies reporter position with nine network measures, including centrality, degree, clustering, k-coreness, and largest-component membership.The study uses these measures to describe structural features of bug-processing communities.Source: §3 · p. 5, §3.3 · p. 8

3.1 Data Retrieval

  • The data comprise roughly 715,000 reports and 5.8 million change events from January 1999 through June 2012 across four OSS projects.The study focuses on 287,540 resolved reports; Eclipse’s INCOMPLETE category is excluded because its status practice does not permit straightforward classification.Source: §3.1 · pp. 6–7, §3 · p. 6

3.2 Network Construction

  • Monthly collaboration networks use time-stamped ASSIGN and CC interactions as proxies for collaboration and communication.The authors omit comment sequences because inferring direct interactions from them is more difficult and error-prone; they use 30-day windows rather than aggregating more than a decade of interactions.Source: §3.2 · pp. 7–8

3.3.1 Centrality measures

  • Degree counts connections, closeness is the inverse sum of distances to other nodes, and betweenness counts shortest paths passing through a node.Eigenvector centrality recursively reflects the centrality of direct neighbors; clustering coefficient and k-coreness capture local ties and position in network shells.Source: §3.3.1 · pp. 8–9

3.3.2 Analysis of Largest Connected Component

  • The analysis restricts monthly networks to the largest connected component because several measures, including eigenvector centrality, are not well-defined across disconnected components.The average share of users in that component ranged from 0.53 for Mozilla Thunderbird to 0.96 for NetBeans.Source: §3.3.2 · p. 9

4 User Centrality and Bug Report Quality

The section tests whether bug reporters’ position in collaboration networks relates to bug outcomes across four OSS communities. Centrality before a report is significantly associated with whether it is eventually fixed, but the direction differs in Eclipse.

4 User Centrality and Bug Report Quality

  • The analysis tests five resolution categories using eigenvector centrality before and after reports; H3 proposes that pre-report centrality indicates eventual handling outcome.The categories are FIXED, DUPLICATE, INVALID, WONTFIX, and INCOMPLETE; the later classifier uses additional network measures beyond eigenvector centrality.Source: §4 · p. 11, §4.1 · p. 12

4.1 Analysis

  • In Firefox, Thunderbird, and Netbeans, reporters of eventually fixed bugs have significantly higher pre-report centrality than reporters in other outcome categories; Eclipse shows the opposite relation.The analysis reports strong evidence for H3 across the comparisons, with centrality significantly smaller for fixed-bug reporters in Eclipse.Source: Table 2 · p. 13, §4.1 · p. 14
  • For fixed-bug reporters, centrality significantly increases after reports in Firefox, decreases in Thunderbird, and shows no significant change in Eclipse or NetBeans.The analysis compares eigenvector-centrality distributions before and after reports using the Wilcoxon-Mann-Whitney test.Source: §4.1 · pp. 12, 14
  • Duplicate-reporters’ centrality decreases after submission in all four projects, while invalid-reporters’ centrality changes significantly only in Thunderbird.For invalid reports, the analysis cannot reject the no-change hypothesis in Firefox, Eclipse, or NetBeans.Source: §4.1 · p. 14

5 Classification of Bugs with Social Network Analysis

This section tests whether reporters’ network positions predict whether bug reports are Valid or Faulty. It compares LCC membership, an eigenvector-centrality threshold, and a nine-measure SVM.

5 Classification of Bugs with Social Network Analysis

  • The workflow progresses from prior-month LCC membership to an eigenvector-centrality threshold and then an SVM using nine node-level measures, with 5.0% of samples used for SVM training.The measures include LCC membership, centrality measures, degree, clustering coefficient, and k-coreness.Source: §5 · pp. 15, 17, 18
  • Classifying reports as VALID when reporters belong to the prior-month LCC performs better than random sampling against the actual VALID-report proportion.The difference in LCC membership between VALID and Faulty reporters is more pronounced for Firefox and Thunderbird than for Eclipse and NetBeans.Source: §5 · pp. 15–16
  • Adding eigenvector centrality to LCC membership generally increases precision while decreasing recall and F-score, with a particularly pronounced F-score drop for Thunderbird.The threshold is tuned individually for each community; Thunderbird’s drop aligns with the reported negative relation between its eigenvector centrality and report quality.Source: §5 · p. 17
  • The nine-measure SVM reaches 82.5% precision and 0.58 F-score for Firefox, and 90.3% precision and 0.54 F-score for Thunderbird.These projects have VALID-report fractions of 21% and 23.3%, respectively.Source: Table 4 · p. 17, §5 · p. 18
  • Faulty-report classification yields F-scores of 0.92 and 0.91 and precision of 86.9% and 84.9% for Firefox and Thunderbird, versus F-scores of 0.69 and 0.67 and precision of 73.6% and 73.1% for Eclipse and NetBeans.The paper proposes this reversed target when most reports are VALID.Source: §5 · p. 18, Table 5 · p. 18

6 Discussion, Threats to Validity and Implications for Future Work

The study’s findings are a proof of concept across four varied OSS communities, but their generality and the social mechanisms behind the network signal remain uncertain. The authors identify limits in the data, network construction, and 30-day window, while noting that machine learning requires enough community history.

Scope and generalizability

  • The findings are a proof of concept from four Bugzilla OSS communities; accuracy outside the studied settings, including commercial or smaller and less diverse teams, requires further study.The projects span different levels of user technical proficiency and other community characteristics.Source: §6 · p. 19

Social mechanisms

  • The exact social mechanisms linking reporter network position and report outcomes remain unclear; survey replies supported a possible reputation effect in Eclipse and NetBeans, but Mozilla replies were absent.Managers cited reproducibility or reporter reputation as factors in triage decisions.Source: §6 · p. 19

Network construction

  • The network measures use CC and Assign relations as proxies for social organization, excluding other possible interactions such as bug-comment sequences because their collaboration signal is less clear.Source: §6 · p. 19

Future data sources

  • Mailing lists and threaded forums are potential additional data sources for augmenting the network perspective in subsequent analyses.Source: §6 · p. 20

Data requirements

  • The method can quantify social embeddedness after report submission, but applying machine learning in newly born communities remains challenging because they may lack enough accumulated samples.Source: §6 · p. 20

Window choice

  • The analysis uses a fixed 30-day network window; although results were accurate with this choice, whether community-specific tuning improves performance remains under investigation.Source: §6 · p. 20

Overfitting risk

  • The authors limited randomly chosen training data to 5.0% to reduce the risk of overfitting.Source: §6 · p. 20

7 Conclusions

The study finds that reporter centrality is associated with bug-report quality across the four OSS communities, while patterns vary by project and report outcome. Using nine network measures, an SVM classifies valid reports with precision up to 90.3% and F-score up to 0.92.

Centrality over time

  • For Mozilla Firefox, reporter eigenvector centrality increases after valid reports, while Mozilla Thunderbird shows the opposite relation.The analysis uses a 30-day time resolution over ten years.Source: §7 · p. 20

Duplicate reports

  • Eigenvector centrality decreases significantly after duplicate reports in all four projects.Source: §7 · p. 20

Report quality

  • For Mozilla Firefox, Mozilla Thunderbird, and NetBeans, reporters of valid bugs have significantly higher eigenvector centrality than reporters of faulty bugs.Source: §7 · p. 21

Automated classification

  • The SVM using nine reporter-level topological measures achieves precision up to 90.3% and F-score up to 0.92, depending on the project.Source: §7 · p. 21

Implications

  • The authors present the classifier as simple enough for practical support infrastructures and suggest that combining it with features from earlier studies may further improve accuracy.Source: §7 · p. 21
Loading 1302.6764v2…