PaperlayerView exact revision on arXiv

Source-linked AI summary

Fault Localization with Code Coverage Representation Learning

Yi Li, Shaohua Wang, Tien N. Nguyen

arXiv:2103.00270v1cs.SE

TL;DR

Fault localization needs to distinguish faulty code when conventional coverage-derived scores cannot exploit full execution patterns or distinguish lines shared by passing and failing tests. DeepRL4FL treats localization as image pattern recognition, combining coverage, dependency, and source-code representations in a CNN. It improves top-1 results over statement-level and method-level baselines across the reported comparisons, while remaining limited by test quality, uncommon faults, short methods, and evaluation scope.

  • Problem

    Existing fault-localization approaches can fail when faulty and non-faulty statements receive indistinguishable coverage-derived scores.

  • Method

    DeepRL4FL combines ordered code-coverage representation learning, statement-dependency learning, and source-code representation learning in a CNN classifier.

  • Results

    DeepRL4FL improves top-1 results over statement-level baselines by 173.1%–491.7% and over method-level baselines by 15.0%–206.3%.

  • Takeaways & Limitations

    The evaluation supports combining coverage, dependency, and code representations for fault localization at both statement and method levels.

  • Takeaways & Limitations

    Performance depends on adequate test cases and training examples, and the comparisons were conducted only on Defects4J for the reported baselines.

Abstract

from arXiv · show

In this paper, we propose DeepRL4FL, a deep learning fault localization (FL) approach that locates the buggy code at the statement and method levels by treating FL as an image pattern recognition problem. DeepRL4FL does so via novel code coverage representation learning (RL) and data dependencies RL for program statements. Those two types of RL on the dynamic information in a code coverage matrix are also combined with the code representation learning on the static information of the usual suspicious source code. This combination is inspired by crime scene investigation in which investigators analyze the crime scene (failed test cases and statements) and related persons (statements with dependencies), and at the same time, examine the usual suspects who have committed a similar crime in the past (similar buggy code in the training data). For the code coverage information, DeepRL4FL first orders the test cases and marks error-exhibiting code statements, expecting that a model can recognize the patterns discriminating between faulty and non-faulty statements/methods. For dependencies among statements, the suspiciousness of a statement is seen taking into account the data dependencies to other statements in execution and data flows, in addition to the statement by itself. Finally, the vector representations for code coverage matrix, data dependencies among statements, and source code are combined and used as the input of a classifier built from a Convolution Neural Network to detect buggy statements/methods. Our empirical evaluation shows that DeepRL4FL improves the top-1 results over the state-of-the-art statement-level FL baselines from 173.1% to 491.7%. It also improves the top-1 results over the existing method-level FL baselines from 15.0% to 206.3%.

I. INTRODUCTION

DEEPRL4FL treats fault localization as image pattern recognition, learning from full code-coverage matrices, statement dependencies, and source-code representations to identify faulty statements and methods. On Defects4J and ManyBugs, it improves top-1 localization over established statement- and method-level baselines.

  • Problem setting: Fault localization identifies suspicious code using test executions and a code coverage matrix whose rows are statements and columns are test cases.Cells indicate whether each statement executes in each test case.
  • Existing limitations: SBFL assigns suspiciousness from coverage patterns, but gives the same score to lines executed in both failing and passing tests.MBFL enriches coverage through mutations but remains ineffective for bugs requiring fixes more complex than a mutation.
  • Approach: DEEPRL4FL applies CNN-based pattern recognition directly to enhanced coverage matrices instead of reducing each row to a suspiciousness score.It orders test-case columns so nearby statements with shared coverage are positioned near one another.
  • Approach: The approach combines coverage evidence, similar buggy code, and data dependencies among statements to represent both dynamic execution information and related code context.Dependency-aware suspiciousness can incorporate statements that exhibit consequences of a bug through execution or data flows.
  • Results: DEEPRL4FL improves method-level top-1 results over MULTRIC, FLUCCS, TraPT, and DeepFL by 206.3%, 53.1%, 57.1%, and 15.0%, respectively.On ManyBugs, it localizes 27 faulty statements and 98 faulty methods using top-1 results.

2. DEEPRL4FL: Novel DL-based fault localization ap-

DEEPRL4FL treats fault localization as image-pattern recognition by preserving code-coverage details and organizing test cases for CNN analysis. It addresses limitations of existing approaches that summarize coverage into suspiciousness scores.

  • DEEPRL4FL incorporates code coverage, similar buggy code, and statement data dependencies as three information sources for fault localization.The dependency view accounts for statements that may exhibit consequences of a buggy line through execution or data flows.
  • The paper evaluates the approach against recent statement- and method-level FL models in within-project and cross-project settings for C and Java.
  • Existing spectrum-, mutation-, and deep-learning-based FL approaches summarize each statement’s coverage into scores, limiting their use of full coverage details.In the motivating examples, these approaches fail when buggy and non-buggy lines receive identical scores.
  • Test cases are ordered so executions covering nearby statements become adjacent, helping CNN filters detect localized patterns in the matrix.The ordering begins with the test case covering the most statements and then selects cases sharing as many executed statements as possible with the previous case.

III. EXPLORATORY STUDY

The exploratory study tests whether ordering coverage-matrix columns improves CNN feature recognition, then outlines DEEPRL4FL’s representation-learning architecture. The architecture combines enhanced coverage, dependency, and source-code representations.

  • Exploratory study: The exploratory experiment compares CNN inputs using the original spectrum-based coverage matrix and an ordered coverage matrix.Both executions produce outputs marking buggy rows with ones and other rows with zeros.
  • Exploratory study: In testing 10 randomly selected faults, the CNN identified the correct buggy statement in 2 cases.
  • Exploratory study: After ordering, the coverage matrix’s left side becomes darker, and a feature map shows a darker spot at the buggy statement than at non-buggy statements.The study visualizes matrices and feature maps as grayscale images, with rows representing statements and columns representing test cases.
  • DEEPRL4FL architecture: DEEPRL4FL has three representation-learning processes: code coverage, statement dependencies, and source code.These correspond to dynamic coverage information, relationships among statements, and structural representations of buggy training code.

V. CODE COVERAGE REPRESENTATION LEARNING

DeepRL4FL constructs spectrum- and mutation-based coverage matrices from test executions, marks error-exhibiting statements, and orders test cases to expose coverage patterns for CNN-based fault localization.

  • Coverage-matrix construction: GZoltar supplies per-method coverage matrices, error messages, and recorded statement execution paths for test-case executions.The approach additionally uses PIT-1.1.5 to mutate statements and generate mutation-based coverage matrices.
  • Coverage-matrix construction: 47?
  • Coverage-matrix construction: A coverage-matrix cell is 1 for passing coverage, 0 for non-coverage, and -1 for an error-exhibiting statement in a failing test.The -1 values are assigned using class names, method names, and line numbers from failing-test error messages.
  • Test-case ordering: The test-case ordering algorithm places failing tests first, grouping tests that share error-marked statements, then orders remaining tests by shared covered statements.When ties remain, the resulting local visual effects are described as similar.
  • Coverage-matrix enhancement: The same error-marking and test-case-ordering enhancements are applied to both spectrum-based and mutation-based coverage matrices.

VI. STATEMENT-DEPENDENCY REPRESENTATION

DeepRL4FL represents statement dependencies through execution-order and data-flow embeddings, combines them with enhanced coverage matrices, and organizes resulting statement and method representations for CNN processing.

  • Execution-order representation: Execution paths are embedded with word2vec to encode relations among statements within a method.Each statement receives a word2vec-generated vector from the execution paths of test cases.
  • Data-dependency representation: Data-dependency representation supplements execution order because loops can dominate paths and dependent statements may be nonadjacent.WALA-generated data-flow graphs and node2vec are used to model these dependencies.
  • Data-dependency representation: Hadamard multiplication combines execution-order and program-dependency vectors into a statement-dependency vector.The resulting vector models dependencies and/or execution orders among statements.
  • Combining dependencies and coverage: Each coverage value is multiplied by its statement-dependency vector to enrich the spectrum- or mutation-based coverage matrix.The enhanced matrix associates statements and test cases with dependency-aware vectors.
  • CNN-ready representations: A statement is a 2-D vector and a method a 3-D matrix in spectrum-based matrices, whereas mutation-based matrices use 3-D statements and 4-D methods.A fully connected layer reduces mutation-based method representations from 4-D to 3-D before CNN processing.

VII. SOURCE CODE REPRESENTATION LEARNING

DeepRL4FL learns source-code representations at statement and method levels by embedding tokens and structural program elements.

  • Statement representations: Statement source code is tokenized as a sentence, with word2vec producing token vectors that are reduced to one statement vector.The token-vector matrix is compressed using a fully connected layer.
  • Method representations: At the method level, code2vec represents abstract-syntax-tree structure through word2vec embeddings of long paths.A long path runs between leaf nodes through the AST root and is represented as an ordered list of node vectors.
  • Method representations: ASTNN represents methods by splitting their ASTs into statement-level subtrees and modeling subtree sequences with bidirectional GRUs.The method combines long-path-based and tree-based representations through a fully connected layer.

VIII. FAULT LOCALIZATION WITH CNN MODEL

DeepRL4FL combines coverage, dependency, and source-code vectors into CNN inputs for statement- and method-level faulty-code classification.

  • Statement-level fault localization: Statement-level coverage and source-code vectors are combined with Hadamard products after dimension expansion and broadcasting.The three statement vectors are spectrum-based, mutation-based, and source-code-based representations.
  • Statement-level fault localization: A CNN with softmax classifies each combined statement representation as faulty or non-faulty.The softmax output is standardized between 0 and 1, and training uses the corresponding faulty/non-faulty label.
  • Method-level fault localization: Method-level localization combines spectrum-based, mutation-based, source-code, and error-message similarity vectors before CNN classification.The method representation includes a fourth similarity vector with 15 features.
  • Method-level fault localization: The same Hadamard-product combination and CNN softmax classification framework is applied to methods.Training follows the statement-level fault-localization procedure.

IX. EMPIRICAL EVALUATION

The evaluation compares DEEPRL4FL with statement- and method-level fault-localization baselines using benchmark datasets, cross-validation, and established ranking metrics. It also analyzes matrix-enhancement and representation-learning techniques, cross-project performance, and C-code performance.

  • Evaluation scope: The study evaluates DEEPRL4FL through research questions covering baseline comparisons, matrix enhancements, representation learning, cross-project analysis, and C-code performance.Experiments use Defects4J and ManyBugs, with within-project and cross-project settings.
  • Metrics: Recall at Top-K counts faults with at least one correctly predicted faulty statement within the top K ranked statements.The reported values are Top-1, Top-3, and Top-5.
  • Metrics: Mean Average Rank averages the ranks of all faulty elements across faults, while Mean First Rank averages the first faulty element’s rank.MFR emphasizes locating at least one faulty element early when faults contain multiple faulty elements.
  • Baselines: Statement-level comparisons include Ochiai, Dstar, MUSE, Metallaxis, RBF, and DeepFL, while method-level comparisons include MULTRIC, FLUCCS, TraPT, and DeepFL.The experiments use leave-one-out cross validation within projects.
  • Ablation analyses: The matrix-enhancement analysis compares a base model with variants using test-case ordering and statement dependencies, while representation analysis tests enhanced matrices, source-code representation, and textual similarity.These experiments isolate the contributions of ordering, dependencies, and learned representations.

C. Experimental Results

DEEPRL4FL improves statement- and method-level fault localization over the evaluated baselines, with ordering, dependencies, and learned representations contributing to performance. Its results also extend across project settings and C code, although performance is stronger within projects.

  • Statement-level comparison: 317.6% to 491.7%: DEEPRL4FL improves statement-level Recall at Top-1 over Ochiai, Dstar, MUSE, Metallaxis, RBF, and DeepFL.The respective improvements are 317.6%, 273.7%, 173.1%, 195.8%, 491.7%, and 82.1%.
  • Method-level comparison: 206.3% to 15.0%: DEEPRL4FL improves method-level Recall at Top-1 over MULTRIC, FLUCCS, TraPT, and DeepFL.Its MFR is 10.4% lower than DeepFL’s, although its MAR is 3.6% higher.
  • Matrix enhancements: Ordering improves every evaluated metric, localizes 53 additional bugs using Top-1, and improves MFR and MAR by 20.1% and 12.7%.The analysis attributes this benefit to CNN detection of features along buggy statements after neighboring coverage patterns are organized.
  • Matrix enhancements: Adding statement dependencies improves Top-1, MFR, and MAR by 8.4%, 9.6%, and 4.5%, respectively.In one example, the buggy method ranks fifth without both enhancements, second with ordering, third with dependencies, and first with both.
  • Representation learning: Representation learning adds 23 localized bugs through mutation-based matrices, nine through code representation, and further positive contributions from textual similarity.For statement-level localization, code representation raises Top-1 localization from 65 to 71 bugs, a 9.2% increase.
  • Project settings: DEEPRL4FL detects 217 bugs at Top-1 cross-project versus 207 for the best baseline, while within-project it detects 230.The paper reports stronger within-project results because training and testing data come from the same project.
  • C-code performance: On ManyBugs, DEEPRL4FL localizes 27 faulty statements and 98 faulty methods at Top-1, with percentages similar to its Java results.The statement-level percentages are 14.6% for C versus 18.0% for Java; method-level percentages are 53.0% versus 62.0%.

7) Threats to Validity:

The comparative evaluation is limited by the use of implemented baselines and comparisons conducted only on Defects4J. Further evaluation on other datasets is identified as necessary.

  • Threats to validity: Baseline implementations may constrain the comparative study, and the reported baseline comparisons were conducted only on Defects4J.The paper calls for further evaluation on other datasets to assess result generalization.

8) Limitations:

DeepRL4FL has several scope and data requirements that constrain when it can localize faults effectively.

  • The approach requires high-quality test cases to learn useful representations for fault localization.With only a few passing tests or when the crash is far from the faulty method, the learned representation may be ineffective.
  • DeepRL4FL does not work well for faults requiring statement additions, a limitation also shared by the evaluated baselines.
  • Short methods provide fewer statement dependencies, reducing the approach’s effectiveness.
  • As a deep-learning approach, DeepRL4FL may fail on uncommon faults absent from its training data.

X. RELATED WORK

Related fault-localization research includes spectrum-, mutation-, learning-to-rank, neural-network, and code-representation-learning approaches. DeepRL4FL extends this landscape by treating fault localization as image recognition, modeling statement and failing-test relations, and combining coverage and code representations.

  • Fault Localization: SBFL ranks code elements using coverage patterns, while MBFL additionally evaluates mutated code.SBFL methods share the insight that elements mainly executed by failed tests are more suspicious; MBFL adds mutated-code information.
  • Fault Localization: Learning-to-rank methods combine suspiciousness values with additional information such as invariants, code complexity, or mutation-based scores.
  • Code Representation Learning: Neural networks and deep learning have been applied across fault localization and broader program-analysis tasks, including code representation learning.
  • DeepRL4FL: DeepRL4FL treats fault localization as image recognition, models relations among statements and failing tests, and combines coverage, statement-dependency, and code representations.
Loading 2103.00270v1…