PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Software Quality Through Refactoring Commit-Impact Analysis

Md Safayat Hossen, Sheikh Moonwara Anjum Monisha, Shadid Intisar Quasem

arXiv:2610.12890v1cs.SE

TL;DR

This study asks how refactoring commits relate to software quality as projects grow more complex and code can depart from accepted standards. It compares pre- and post-commit metrics in GitHub projects using static analysis and repository mining. Refactoring impacts were mostly positive, though negative and neutral impacts also occurred, and positive impacts increased in middle-to-later development stages.

  • Problem

    The study asks whether refactoring commits affect quality metrics and code smells, against a backdrop of growing software complexity and difficulty maintaining coding standards.

    Source: §I · p. 1

  • Method

    The authors compare software metrics before and after refactoring commits in GitHub projects using static code analysis and repository mining.

    Source: §I · p. 1, B. Refactor commit selection · p. 3

  • Results

    Refactoring impacts were mostly positive: 46% and 50% in two Python projects, and 57% and 70% in two Java projects; negative impacts also occurred.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The authors conclude that positive refactoring impacts increase from the middle to later stages of development, while refactoring can also have negative effects.

    Source: §VII · p. 8

    Data collection and analysis depend on tool results and default configurations, although the selected tools are popular and Refactoring Miner’s average precision is 96.4%.

    Source: Internal validity · p. 7

Abstract

from arXiv · show

The aim of this study is to conduct an empirical analysis of the correlation between code modifications and software quality. This will be achieved by examining the effects of changes made before and after each refactoring commit, using software metrics such as size, complexity, code smells, vulnerability, and bug detection. The objective is to conduct static code analysis on open-source software from GitHub repositories using various repository mining and static code analysis tools. Our approach involves formulating research-based questions and utilizing the specified tools to obtain results that will enable us to resolve the case study. The obtained results have been summarised and further analyzed using tables and graphs to achieve the objective.

I. INTRODUCTION

As software grows, complexity and departures from coding standards can make maintenance difficult, and refactoring offers a way to improve internal structure. This study examines pre- and post-refactoring-commit changes in software metrics using static analysis and repository mining, guided by four research questions.

Motivation

  • Software tends to grow and become more complex, while evolution can lead code away from accepted standards and make maintenance difficult.The introduction presents refactoring as something that can be applied later to address such code.Source: §I · p. 1

Background

  • Refactoring is framed as altering source code to improve a project’s internal structure and measurable attributes such as cohesion, coupling, complexity, inheritance, and size.The passage attributes this definition to Fowler and the measurable-attributes framing to.Source: §I · p. 1

Motivation

  • Version-control commits record code changes and their authors and timestamps, providing a historical source for studying software development and maintenance.The passage cites for commits as a source for understanding software development.Source: §I · p. 1

Study objective

  • The analysis compares code before and after refactoring commits using size, complexity, code-smell, vulnerability, and bug-detection metrics.It uses static code analysis on open-source GitHub projects together with repository mining.Source: §I · p. 1

Research questions

  • The study examines refactoring commits’ impact on quality metrics, code smells, novice versus mature projects, and projects over time.The questions ask whether refactoring affects quality metrics and code smells, whether impacts differ by project maturity, and how impacts change over time.Source: §I · p. 1

II. RELATED WORK

Prior work has studied refactoring’s relationship to software quality through commit analysis, quality attributes, prediction techniques, and empirical evaluations. The paper also describes static-analysis and repository-mining tools used to assess software quality and changes.

II. RELATED WORK

  • Software evolution and repository-mining research informed the case study’s focus on refactoring and internal software quality.Source: §II · p. 1
  • Prior studies examined commit-level quality changes, internal quality attributes, refactoring-impact prediction, mobile-app refactoring, and refactoring in continuous integration.The cited work reports that effects vary: some metrics improve, others change little, and LCOM can degrade; one study found improvements in 65% of cases and no change in 35%.Source: §II · pp. 1–2

III. TOOLS

  • The study uses three static code-analysis tools and two repository-mining tools; static analyzers provide quality metrics without executing source code.The passage also notes that static analysis can reveal defects associated with design flaws that compilers may not trace.Source: §III · p. 2

Refactoring Miner 2.1:

The paper describes tools for detecting refactorings and analyzing source code, metrics, and code issues. Their stated functions range from mining project histories to measuring complexity and identifying common programming errors.

Refactoring Miner 2.1:

  • Refactoring Miner detects 81 refactoring types in project histories and supports Java, with Python- and Kotlin-supported versions also available.The passage lists examples including Extract Method, Inline Method, Rename Method, and Move Method.Source: Refactoring Miner 2.1 · p. 2

PYREF:

  • PYREF extracts method-level refactoring operations from Python Git repositories, following an approach similar to RefactoringMiner.The listed operations include Rename Method, AddParameter, ExtractMethod, MoveMethod, PullUpMethod, and PushDownMethod.Source: PYREF · p. 2

PMD (Programming Mistake Detector):

  • PMD detects common programming errors and includes CPD, a detector for duplicated code across multiple programming languages.The passage names examples such as redundant code, unused variables, and empty catch blocks.Source: PMD (Programming Mistake Detector) · p. 2

Source Monitor:

  • Source Monitor measures source-code metrics, including cyclomatic complexity, lines of code, classes, methods per class, and statements per method.It also supports historical comparison of saved measurements.Source: Source Monitor · p. 2

SonarQube

SonarQube reports code-quality metrics including complexity, non-commented lines of code, classes, bugs, vulnerabilities, and code smells. The study selected four GitHub projects—two Java and two Python—using repository filters.

SonarQube

  • SonarQube reports complexity, NCLOC, class counts, bugs, vulnerabilities, and code smells.The passage describes it as a code-quality assurance tool that gathers and evaluates source code.Source: SonarQube · pp. 2–3

A. Project Selection

  • The project set comprised Java projects IutAssistance (69 commits, 24 refactoring commits) and Alligator (198, 113), plus Python projects you-tube-search (168, 18) and mini-shop-server (506, 90).The authors selected two projects in each language and searched using filters including language, commit count, issues, and contributors.Source: A. Project Selection · p. 3

B. Refactor commit selection:

The study selected refactoring commits and neighboring revisions for before-and-after analysis, then applied static-analysis tools to Java and Python projects.

B. Refactor commit selection:

  • Java RefactoringMiner identified 24 of 69 and 113 of 198 commits as refactorings; the study manually selected 16 commits (8 pairs) and 18 commits (9 pairs).Source: B. Refactor commit selection · p. 3
  • PyRef identified 13 and 80 refactoring commits in the two Python projects; project 1 analysis examined 26 commits (13 pairs), and SonarQube ran on 18 randomly chosen pairs.Source: B. Refactor commit selection · p. 3

C. Tool application

  • PMD was used as an extensible cross-language static analyzer, but this study applied it only to Java with its default ruleset across five categories.The categories were Best practices, Code style, Design, Documentation, and Error-prone.Source: C. Tool application · p. 3
  • For before-and-after impact checks, PMD was run on the previous revision and then the refactored revision for comparison.Source: C. Tool application · p. 3
  • SourceMonitor produced 15 metric types for Java; Python SonarQube reports for these projects covered code smell and technical debt.SonarQube can report other metrics, including vulnerabilities, bugs, duplications, and security hotspots.Source: C. Tool application · p. 3

D. Metrics selection

The study selected basic size measures and major quality measures, then collected tool outputs for analysis across Java and Python revisions.

D. Metrics selection

  • Basic metrics cover lines of code, statements, and classes; major metrics include average complexity, error-prone, code smell, and bugs.Table I is identified as the inventory of metrics used in the study.Source: D. Metrics selection · p. 3, Table I · p. 4
  • The study counted code smells, defined technical debt as effort to fix them in time units, and described bugs as demonstrably wrong or likely-to-be-unexpected-behavior code.Source: D. Metrics selection · pp. 3–4

E. Data Collection

  • PMD and SourceMonitor produced 21 metrics for Java, while Python SonarQube produced code smell and technical debt results stored for analysis in Tables II–V.Source: E. Data Collection · p. 4

F. Additional data collected from Source monitor and PMD:

V. RESULT

The results report changes in software metrics after refactoring commits and classify code-smell outcomes as positive, negative, or unchanged. Code-smell outcomes were mixed across the two Python projects, while quality metrics also changed in different directions.

RQ1: Do refactoring commits have any impact on quality metrics?

  • Method/class and average statements/method generally decreased after refactoring, although some cases reversed; lower method/class values are treated as better quality.The reported interpretation links fewer methods per class to easier maintenance and preservation of the Single Responsibility Principle.Source: RQ1: Do refactoring commits have any… · p. 4
  • Average depth and complexity changed after refactoring, and the authors associate increased complexity with maintainability difficulties.They note that greater depth may indicate increased dependency and coupling.Source: RQ1: Do refactoring commits have any… · p. 4
  • Software size generally increased after refactoring, measured through file, size, and statement attributes.The text points to Tables IV and V and Figure 1 for these size observations.Source: RQ1: Do refactoring commits have any… · p. 4

RQ2: What is the impact of refactoring on code smells?

  • The code-smell analysis covered 62 revision files, including 13 pairs from Python project 1 and 18 from project 2, and compared smell counts before and after commits.The reported value difference is the number of smells before refactoring minus the number after refactoring.Source: RQ2: What is the impact of refactoring… · pp. 4, 6
  • Python project 1 had 46% positive, 15.3% negative, and 38% neutral code-smell impact; project 2 had 50% positive, 16.67% negative, and 33.33% neutral impact.Positive impact denotes fewer smells after refactoring; negative impact denotes more, and unchanged impact denotes the same count.Source: RQ2: What is the impact of refactoring… · pp. 4–5
  • The authors report that code smells tended to be higher where less refactoring was applied, and that more smells entail more effort to fix.They report refactoring shares of 7% for the first project and 17.7% for Python project 2; they define debt as the technical effort to fix smells.Source: RQ2: What is the impact of refactoring… · p. 6

RQ3: Does the refactoring commit impact the same way on novice and mature projects?

The study compared one novice project with three mature projects by classifying commit-pair impacts across quality metrics. Positive impact was similar across project types, while mature projects had fewer negative and more neutral impacts.

RQ3: Does the refactoring commit impact the same way on novice and mature projects?

  • The comparison used three mature projects and one novice project, with maturity distinguished using stars, forks, and tags.Mature projects had more than 250 for these features; the novice project's values were near zero.Source: RQ3: Does the refactoring commit… · p. 6
  • Commit pairs were scored across quality metrics as positive, negative, or neutral, then classified by the sum of their metric scores.Individual metric impacts received scores of 1, -1, or 0.Source: RQ3: Does the refactoring commit… · p. 6
  • Positive impact was quite similar for novice and mature projects, but mature projects had fewer negative and more neutral impacts.The novice result was taken directly, while the mature-project result averaged three projects.Source: RQ3: Does the refactoring commit… · p. 6

RQ4: How does refactoring commit impact projects over time?

To examine change over time, the study ordered commit pairs by date and categorized their quality-metric impacts as positive, negative, or neutral. It plotted the resulting pattern for two Java projects; the supplied size figure separately identifies file, line, and statement measures by project.

RQ4: How does refactoring commit impact projects over time?

  • Commit pairs were ordered by date and their quality-metric impacts classified as positive, negative, or neutral for the time analysis.The study used these categories to examine whether impact changed over the projects' development periods.Source: RQ4: How does refactoring commit… · p. 6
  • The time-series graphs cover two Java projects and use positive, negative, and neutral impact categories.The passage describes plotting graph 8, but supplies no outcome values or trend for that graph.Source: RQ4: How does refactoring commit… · p. 6
  • The size figure distinguishes file, line, and statement measures, with Java Project 1 in column 1 and Java Project 2 in column 2.The caption identifies the measures and project columns but gives no comparative values.Source: Figure 1 · p. 7

VI. LIMITATIONS

The study's findings are bounded by tool-dependent data collection, four selected systems, and a small sample with manually selected pre-refactoring commits.

Internal validity:

  • Tool results and default configurations constrain data-collection and analysis accuracy, although Refactoring Miner has 96.4% average precision and low false-positive rates.The authors relied mostly on static-analysis and mining tools.Source: Internal validity · p. 7

External validity:

  • Because the study covers four conveniently selected open-source systems, its findings are difficult to generalize to a larger context.Source: External validity · p. 7

Conclusion validity:

  • Time constraints limited the sample, and manually selecting the pre-refactoring commit leaves a chance of human error.Source: Conclusion validity · p. 7

VII. CONCLUSION

The study used static-analysis tools on four open-source Java and Python projects to examine refactoring's relation to software quality. Impacts were mostly positive but sometimes negative, and the reported positive impact increased in later development stages.

VII. CONCLUSION

  • Python refactoring impacts were mostly positive in Project 1 (46%) and Project 2 (50%), with negative impacts of 15.3% and 16.6%, respectively.The study classified impacts by changes in code-smell counts.Source: §VII · p. 8
  • The authors report that code-smell counts are directly proportional to the effort to fix them, and that the same number of smells takes less effort to fix when identified earlier.Between the two Python projects, the one with more frequent refactoring tended to have fewer code smells.Source: §VII · p. 8
  • Java impacts were mostly positive in Project 1 (57%) and Project 2 (70%), with negative impacts of 28% and 30%, respectively.Java Project 1 also had 15% neutral impacts.Source: §VII · p. 8
  • The positive impact of refactoring increased in the middle-to-later stages of development.Source: §VII · p. 8
  • The authors conclude that refactoring commits affect quality metrics mostly positively, while noting that their analysis covered four projects and plan to expand data collection.Planned extensions include more projects, languages, metrics, dynamic analysis, and regression testing.Source: §VII · pp. 7–8
Loading 2610.12890v1…