PaperlayerView exact revision on arXiv

Source-linked AI summary

Untangling Fine-Grained Code Changes

Martín Dias, Alberto Bacchelli, Georgios Gousios, Damien Cassou, Stéphane Ducasse

arXiv:1502.06757v1cs.SE

TL;DR

Tangled commits combine unrelated tasks, complicating review, reversion, integration, and historical analysis. This paper builds a fine-grained change-clustering approach and dataset, then evaluates the tool with developers. It reports over 88% worst-case accuracy on historical data and a 91% median deployment success rate.

  • Problem

    Developers bundle unrelated changes into commits, making review, reversion, integration, and historical analysis harder.

    Source: §I · p. 1

  • Method

    The paper records fine-grained IDE changes, builds a developer-labeled dataset, models change-pair relationships, and uses the model to propose clusters for developer review.

    Source: §I · p. 1, §VIII · p. 10

  • Results

    Historical-data testing achieved over 88% accuracy in the worst case, and deployment with new developers achieved a median success rate of 91%.

    Source: §VIII · p. 10

  • Takeaways & Limitations

    Temporal proximity, intervening changes, and same-class edits are especially important features for clustering fine-grained code changes.

    Source: §I · p. 1

    The models and feature selection rely on data from two developers, so findings may be biased toward their working habits.

    Source: B. Threats to Validity · p. 9

Abstract

from arXiv · show

After working for some time, developers commit their code changes to a version control system. When doing so, they often bundle unrelated changes (e.g., bug fix and refactoring) in a single commit, thus creating a so-called tangled commit. Sharing tangled commits is problematic because it makes review, reversion, and integration of these commits harder and historical analyses of the project less reliable. Researchers have worked at untangling existing commits, i.e., finding which part of a commit relates to which task. In this paper, we contribute to this line of work in two ways: (1) A publicly available dataset of untangled code changes, created with the help of two developers who accurately split their code changes into self contained tasks over a period of four months; (2) a novel approach, EpiceaUntangler, to help developers share untangled commits (aka. atomic commits) by using fine-grained code change information. EpiceaUntangler is based and tested on the publicly available dataset, and further evaluated by deploying it to 7 developers, who used it for 2 weeks. We recorded a median success rate of 91% and average one of 75%, in automatically creating clusters of untangled fine-grained code changes.

I. INTRODUCTION

Tangled commits combine changes from different tasks, making review, reversion, integration, and historical analysis harder. This paper introduces a fine-grained approach for untangling changes and evaluates it using developer-labeled data and live use.

I. INTRODUCTION

  • Developers often bundle unrelated changes in one commit, making review, reversion, integration, and historical analysis more difficult.The example commit combined a refactoring with code formatting; integrating the formatting change separately would itself be demanding.Source: §I · p. 1
  • The paper expands prior untangling work to an untyped setting, uses fine-grained change information, and evaluates the approach with manually labeled data and developers doing real-world work.The approach considers information such as when each line changed and its versions over time.Source: §I · p. 1
  • The study collected four months of manual change clustering from seven developers, validated data from two, modeled change-pair relationships, and evaluated EpiceaUntangler in a two-week deployment.The approach uses machine learning to predict whether two fine-grained changes belong together, then proposes clusters.Source: §I · p. 1
  • With new developers during daily tasks, EpiceaUntangler recorded 75% average and 91% median success rates; Random Forest accuracy was 95% with same-developer training and testing, and over 88% when tested on a different developer.A set of 200 manually clustered fine-grained code changes, equivalent to a few days of work, was sufficient to reach good performance.Source: §I · p. 1
  • Time between changes, intervening changes, and whether changes affect the same class were especially important clustering features.Random Forests achieved 95% accuracy with same-developer training and testing, and over 88% when tested on a different developer.Source: §I · p. 1

II. PROBLEM DESCRIPTION

Tangled commits make several software-development activities harder, while existing approaches face constraints in dynamically typed settings, incomplete line histories, and evaluation based on researcher-labeled commits.

II. PROBLEM DESCRIPTION

  • Tangled commits make review, reversion, integration, and historical analysis more difficult because each activity must be handled together.Even disciplined developers may need to split changed code into several commits, and doing so requires frequent commits and workflow interruptions.Source: §II · p. 2, A. Existing Solutions for Tangled… · p. 2

A. Existing Solutions for Tangled Changes

  • Line-based tools can expose incomplete code, produce invalid partial commits, and obscure modification order by displaying changes in file order.Repeated edits to one line can shadow earlier versions that might belong in a separate commit.Source: A. Existing Solutions for Tangled… · p. 2
  • Herzig et al.’s voters include file distance, package distance, call-graph differences, change coupling, and shared data dependencies.Their method aggregates voter scores and uses multilevel graph partitioning to generate clusters.Source: A. Existing Solutions for Tangled… · p. 2
  • Herzig et al.’s approach uses confidence voters to group lines, but some voters depend on static analysis that may be unavailable or weaker for dynamically typed languages.Its input can also omit earlier versions of repeatedly changed lines, and its validation uses researcher-classified commits without project-expert feedback.Source: A. Existing Solutions for Tangled… · p. 2

B. Addressing the Current Limitations

The proposed approach addresses prior limitations by working in a dynamically typed setting, retaining fine-grained changes, and validating clusters against developer decisions. Epicea records IDE events to provide the change history used by the approach.

B. Addressing the Current Limitations

  • EpiceaUntangler aims to alleviate prior limitations through support for dynamically typed environments, fine-grained changes collected during development, and developer-approved validation.These choices respond to limited static analysis, lost earlier versions in line-based histories, and validation that does not use developer feedback.Source: B. Addressing the Current Limitations · p. 2
  • The approach targets dynamically typed environments, where some static analyses, such as accurate call-graph analysis, are not possible and cannot be relied upon.The stated setting therefore rules out reliance on those analyses.Source: B. Addressing the Current Limitations · p. 2
  • Recording each saved code change preserves multiple versions of an artifact, unlike tools that show only the latest version of a changed line.For example, changing and saving a method three times produces three fine-grained changes.Source: B. Addressing the Current Limitations · p. 2

III. PROPOSED SOLUTION

  • The proposed workflow presents automatically computed clusters for developer review and adaptation before generating one commit per cluster.Developers can add or remove clusters and move changes between them before validating the result.Source: §III · p. 3

A. Epicea: Event Modeler with Fine-Grained Changes

  • Epicea listens to IDE actions and records timestamped events, including code changes and test-run outcomes, when the user saves code.It records additions, modifications, and deletions of classes and methods; the tool is described as invisible to users with no performance impact.Source: A. Epicea: Event Modeler with… · p. 3

B. Voters

The approach represents relations between fine-grained change pairs as voters, uses machine learning to estimate whether pairs belong together, and clusters changes by merging high-scoring groups.

B. Voters

  • Voters describe relations between pairs of changes across six dimensions: code structure, content, testing, spread, message sending, and variable access.Structure includes whether changes occur in the same package, class, or method; spread measures elapsed time and intervening changes.Source: B. Voters · p. 3

C. Machine Learning Approaches

  • The study computes voter values for change pairs less than three days apart and evaluates logistic regression, naïve Bayes, and random forests.The algorithm choice is based on empirical data; the considered algorithms differ in their assumptions about the data and model.Source: C. Machine Learning Approaches · p. 3
  • The machine-learning step outputs the probability that two changes belong to the same cluster.The model takes the pair’s voter values as input.Source: C. Machine Learning Approaches · p. 3

D. Clustering

  • Clustering starts with each change in its own cluster, then successively merges cluster pairs with maximum scores to build a dendrogram.A similarity threshold cuts the dendrogram into final clusters; thresholds that are too low yield many small clusters, while thresholds that are too high yield a single cluster.Source: D. Clustering · p. 4
  • The resulting independent clusters of fine-grained changes are displayed to the user.The table presents the different voters tested in the investigation.Source: D. Clustering · p. 4, Table I · p. 4

IV. RESEARCH METHOD

The study examines which features and machine-learning approach best group fine-grained changes, how well that model works on historical data across developers, and how an untangling tool performs in developers’ daily work.

A. Research Questions

  • The study identifies significant voters for untangling changes and compares three machine-learning approaches for modeling whether two changes belong in the same cluster.The question focuses on the untyped setting.Source: A. Research Questions · p. 4
  • The study evaluates how well a model using significant voters untangles historical changes, including when training and testing data come from different developers.The researchers examine the effect of individual working styles on prediction performance.Source: A. Research Questions · p. 4
  • The study tests whether a tool using the best voters and machine-learning approach can effectively cluster changes when deployed with developers on their usual tasks.Participants were not to have contributed training data.Source: A. Research Questions · p. 4

B. Research Settings

The study uses Pharo to collect developer-labeled fine-grained changes, then analyzes data retained from two developers to select and evaluate a prediction model and its voters.

B. Research Settings

  • The researchers chose Pharo because its community welcomed research tools and its integrated language, environment, and versioning system supported data collection and rapid prototyping.The setting also supported deploying the resulting tool with more participants.Source: B. Research Settings · pp. 4–5

1) Fine-grained data generation and collection:

  • Seven Pharo contributors manually grouped fine-grained changes into tasks at commit time over a four-month collection period.The ETC interface displayed changes since the previous commit for developers to cluster during normal work.Source: 1) Fine-grained data generation and… · p. 5

2) Data analysis and evaluation of voters:

  • Only data from two of seven developers was retained because the other five users’ data was sparse and inconsistent after they could not afford the review time.The retained dataset is called 2devs.Source: 2) Data analysis and evaluation of… · p. 5
  • For 2devs, the researchers framed change-pair grouping as binary classification, balanced training samples at a 2:1 false-to-true ratio, and used cross-validation and voter trimming to build the final model.They evaluated models with standard metrics and exposed the resulting classifier as a web service for EpiceaUntangler.Source: 2) Data analysis and evaluation of… · p. 5

3) Deployment and evaluation with developers:

EpiceaUntangler predicts whether fine-grained changes belong together, clusters them automatically, and presents the result for developer review. The section describes the deployment evaluation and a method for measuring clustering success.

3) Deployment and evaluation with developers:

  • The tool scores change pairs, aggregates those scores into hierarchical clusters, and chooses a dendrogram cut using the largest similarity gap below 0.25.Developers see the precomputed clusters at commit time and can reorganize incorrect groupings.Source: 3) Deployment and evaluation with… · p. 6
  • Six participants used EpiceaUntangler for two weeks and confirmed or rearranged its automatic clusters to provide the expected clustering for evaluation.The resulting data was used to evaluate the approach’s accuracy.Source: 3) Deployment and evaluation with… · p. 6
  • The comparison matrix reports Jaccard indexes for computed–expected cluster pairs, where 1 means equal sets and 0 means no overlap.Figure 4 illustrates four computed clusters and five expected clusters.Source: 3) Deployment and evaluation with… · p. 6, Figure 4 · p. 6
  • Success rate counts changes whose computed and expected clusters occur in the same matched pair; in the Figure 4 example, 5 of 6 changes succeed (0.83).Cluster pairs are matched by maximizing the sum of their Jaccard indexes.Source: 3) Deployment and evaluation with… · p. 6, Figure 4 · p. 6

V. RESULTS

Across the evaluated approaches, Random Forests performed best on all metrics for predicting whether two fine-grained changes belong together. The analysis then identifies the most influential voters for that classifier.

A. What Are the Dominant and Significant Voters?

  • Random Forests delivered the best classification results across all metrics on 320,000 change pairs from the 2devs dataset.The binlogreg classifier’s high recall came with equally low precision: it labeled most file changes as belonging together, but few of those decisions were correct.Source: A. What Are the Dominant and… · p. 7, Table IV · p. 7
  • The three most important voters were time between changes, ordered distance between changes, and whether the changed code belonged to the same class.The study used mean decrease in accuracy over 50 random-forest runs to estimate voter importance; it could not determine whether each voter’s effect was positive or negative.Source: A. What Are the Dominant and… · p. 7

B. How Effective Is Random Forests with the Dominant Voters?

Random Forests remained effective with the three dominant voters, though performance varied with how developers’ data were used. The reported results also show that useful models can be trained from relatively few changes.

B. How Effective Is Random Forests with the Dominant Voters?

  • With only the three dominant voters, the Random Forest model stayed within 3% of the full model’s performance across all metrics.The reduced-voter model is labeled ‘ranforest-trimmed’ in Table IV.Source: B. How Effective Is Random Forests… · p. 7
  • Training and testing on the same developer’s data gave the best results; cross-developer results were worse, with performance dropping as training size increased.The authors attribute the cross-developer drop to overfitting individual developers’ working habits.Source: B. How Effective Is Random Forests… · p. 7
  • Combining data from both developers produced high, stable results, with no apparent overfitting.The combined dataset draws training and testing data from both developers’ samples.Source: B. How Effective Is Random Forests… · pp. 7–8
  • With 200 changes, combined and same-developer models averaged only 2% lower accuracy than models trained with 800 changes.The authors suggest a multi-developer pre-trained model could serve as a starting point for later adaptation to an individual developer’s habits.Source: B. How Effective Is Random Forests… · p. 8
  • 95% accuracy was achieved by a Random Forest model for one developer using about 200 fine-grained changes and a few easy-to-calculate voters.The 200 changes were equivalent to a few days of work.Source: B. How Effective Is Random Forests… · p. 8

C. How Effective Is EpiceaUntangler for Developers?

In a two-week deployment with developers on their usual tasks, EpiceaUntangler’s proposed clusters were compared with participants’ judged clusters. Developers reported both positive experiences and concerns about the volume of fine-grained changes.

C. How Effective Is EpiceaUntangler for Developers?

  • The median success rate was 0.915 and the average was 0.753 (standard deviation 0.30) when proposed clusters were compared with participants’ judged clusters.The authors report the median because the results were not normally distributed.Source: C. How Effective Is EpiceaUntangler… · p. 8, Figure 7 · p. 8
  • Most developers gave positive feedback, but several found the fine-grained information noisy and painful to review, including intermediate method modifications.One participant said the tool guessed clusters correctly even in a large commit, while another said it worked well especially for smaller change sets.Source: C. How Effective Is EpiceaUntangler… · p. 8
  • Participants suggested UI additions for deleting tasks and naming them as reminders while clustering.Source: C. How Effective Is EpiceaUntangler… · p. 8

A. Results

The discussion reports that code spread and structure were the most significant voter dimensions, and that Random Forests performed well with a small set of changes. It also identifies dataset, task-definition, and platform boundaries on these findings.

A. Results

  • Spread had the greatest impact among voter dimensions, while the only significant code-structure voter measured whether two changes were in the same class.The authors suggest these voters may apply to other object-oriented languages, whether typed or untyped.Source: A. Results · p. 8
  • The voter analysis used changes from only two developers, so a larger training-data study could confirm or alter the findings.Source: A. Results · p. 8
  • Random Forests with the most significant voters worked best on same-developer data; cross-developer results were promising, combined-developer results showed no overfitting, and about 200 changes reached most of the effectiveness.The authors say training on more developers is needed for a more general approach and that individual users could train on their own behavior with a few days of work.Source: A. Results · pp. 8–9
  • The deployment exposed developers to all recorded fine-grained changes, including intermediate states, which contributed to information overload; UI evaluation was outside the paper’s scope.The authors propose investigating whether some changes can be omitted without losing relevant information.Source: A. Results · p. 9

B. Threats to Validity

  • Because the models and feature selection relied on two developers’ data, the findings may reflect those developers’ working habits.Source: B. Threats to Validity · p. 9
  • Task granularity was ambiguous across participants, while the success metric measured agreement with each participant’s expected clustering.The study used Pharo and Smalltalk; the authors caution that results may not generalize to all languages or working environments.Source: B. Threats to Validity · p. 9

VII. RELATED WORKS

Prior work established the impact of tangled changes and explored fine-grained change models and tools, while leaving automatic clustering of dependent changes open. EpiceaUntangler extends this work with fine-grained development data and reports strong results in historical-data testing and developer deployment.

VII. RELATED WORKS

  • At least 16.5% of source files in Herzig et al.’s datasets were incorrectly associated with bug reports when tangled change sets were ignored.Tao et al. reported that developers value decomposing changes into individual development issues but lack tool support; tangled changes can also lower review quality or prolong reviews.Source: §VII · p. 9
  • Herzig et al.’s automatic untangling work inspired this paper, which addresses its stated limitations of static-analysis dependency, incompleteness, and artificiality.The authors distinguish their approach by using fine-grained timing and IDE events, working in a dynamically typed language, and evaluating with developers.Source: §VII · p. 9
  • Robbes et al.’s fine-grained change model and logical-coupling measures inform a possible future extension of this paper’s voters.The cited model records IDE changes as they happen and represents program states as abstract syntax trees.Source: §VII · p. 9
  • CoExist supports change navigation, reversion, test-failure localization, and manual reassembly, but leaves automatic clustering of dependent changes as future work.The authors describe their work as extending CoExist in this direction despite an unrelated implementation.Source: §VII · p. 9
  • Wloka’s committable-change analysis motivates richer change context, while this paper also uses unit-test execution beyond its Same Test Run voter.Wloka’s technique identifies changes that can be released early without causing existing tests to fail.Source: §VII · p. 9

VIII. CONCLUSION

  • Over 88% worst-case accuracy on manually labeled historical data and a 91% median deployment success rate support EpiceaUntangler’s clustering approach.The authors identify time between changes, intervening modifications, and whether changes affect the same class as especially important features.Source: §VIII · p. 10
  • The paper contributes a publicly available, manually clustered dataset and EpiceaUntangler, evaluated with developers over two weeks.The conclusion also lists feature/voter evaluation, a tool implementation, and analysis of improvement opportunities in existing untangling work.Source: §VIII · p. 10
Loading 1502.06757v1…