PaperlayerView exact revision on arXiv

Source-linked AI summary

Test Case Purification for Improving Fault Localization

Jifeng Xuan, Martin Monperrus

arXiv:1409.3176v1cs.SE

TL;DR

Fault localization must help developers find bugs despite limited evidence from failing tests and aborted assertions. The paper purifies failing tests into single-assertion, sliced variants and combines their spectra with existing rankings. Across six Java projects and 1,800 seeded faults, purification generally improves existing techniques, while the authors identify test-case dependencies, rank-refinement design, and slicing-tool limitations as important boundaries.

  • Problem

    Fault localization must identify faulty source-code locations from execution traces, but failing tests can omit assertions after aborting, limiting available oracle evidence.

  • Method

    The approach atomizes failing tests into single-assertion tests, dynamically slices unrelated statements, and refines an existing fault-localization ranking with purified-test spectra.

  • Results

    Across 1,800 seeded faults in six open-source Java projects, purification improves existing techniques on 18 to 43% of faults and worsens results on only 1.3 to 2.4%.

  • Takeaways & Limitations

    The reported benefits extend across six fault-localization techniques, showing that manipulating existing test cases can improve fault localization without replacing the original techniques.

  • Takeaways & Limitations

    Negative results are mainly attributed to dependencies between test cases, while the method uses a simple rank-refinement formula and JavaSlicer has known slicing limitations.

Abstract

from arXiv · show

Finding and fixing bugs are time-consuming activities in software development. Spectrum-based fault localization aims to identify the faulty position in source code based on the execution trace of test cases. Failing test cases and their assertions form test oracles for the failing behavior of the system under analysis. In this paper, we propose a novel concept of spectrum driven test case purification for improving fault localization. The goal of test case purification is to separate existing test cases into small fractions (called purified test cases) and to enhance the test oracles to further localize faults. Combining with an original fault localization technique (e.g., Tarantula), test case purification results in better ranking the program statements. Our experiments on 1800 faults in six open-source Java programs show that test case purification can effectively improve existing fault localization techniques.

1. INTRODUCTION

The paper proposes manipulating failing test cases, rather than only developing new suspiciousness metrics, to improve spectrum-based fault localization. Its three-phase approach is evaluated across 1,800 seeded faults in six Java projects and generally improves existing techniques.

  • Spectrum-based fault localization ranks program entities using test-execution traces and failing-test oracles to help developers locate bugs.
  • Test case purification generates single-assertion failing tests, removes unrelated statements, and combines their spectra with an existing technique such as Tarantula.The phases are test case atomization, test case slicing, and rank refinement.
  • 18 to 43% of faults obtain better localization depending on the subject program, while only 1.3 to 2.4% become worse.These figures summarize the reported comparison across the subject programs.
  • Tarantula-Purification performs better than Tarantula on 43.28% of faults, with an average improvement of 36.44 statements.The paper reports Tarantula-Purification as the best-performing considered technique.
  • The study evaluates six fault-localization techniques with and without purification on 1,800 seeded faults across six real-world open-source Java projects.The compared techniques are Tarantula, SBI, Ochiai, Jaccard, Ochiai2, and Kulczynski2.

2. BACKGROUND AND MOTIVATION

The background frames fault localization as ranking source-code entities from execution spectra and test oracles. A motivating Apache Commons Lang example shows how aborted assertions obscure discriminating evidence that purified tests can expose.

  • A test case combines program input with executable assertions that act as test oracles for expected behavior.
  • A spectrum records which program entities a test case executes, providing coverage data for spectrum-based fault localization.
  • Motivation: The motivating example injects a fault into Apache Commons Lang and uses Tarantula to analyze the resulting failing test case.The example focuses on a conditional expression in the maximum/minimum calculation code.
  • Motivation: Because the failing test aborts at an unsatisfied assertion, a later assertion is not executed and eleven statements receive the same suspiciousness.This makes the faulty statement difficult to distinguish from the other executed statements.
  • Motivation: Purification creates copies with one valid assertion each, producing distinct failing-test spectra that better discriminate suspicious statements.The example generates a1, a2, and a3, followed by purified tests p2 and p3.

3. TEST CASE PURIFICATION

Test case purification transforms each failing multi-assertion test into smaller, assertion-focused tests, slices irrelevant statements, and refines an existing fault-localization ranking with the resulting spectra.

  • Framework: The framework has three phases: test case atomization, test case slicing, and rank refinement.It preserves the usual subject-program and test-suite input and statement-ranking output of fault localization.
  • Test case atomization: Atomization replaces a failing test with k single-assertion copies by converting the other k−1 assertions into non-interrupting statements.A try-catch structure prevents assertion exceptions from reaching the testing framework.
  • Test case atomization: The method handles assertion failures during atomization, while unexpected errors can still abort a single-assertion test.
  • Test case slicing: Dynamic slicing removes statements irrelevant to the broken assertion and its variables, yielding purified tests whose spectra are then collected.The slicing criterion is the broken assertion together with all variables at that statement.
  • Rank refinement: Rank refinement combines normalized original suspiciousness with a purified-test ratio and re-ranks candidate statements.Statements not covered by any purified test receive ratio(s) = 0, and the combined score is used for the final ranking.

4. EXPERIMENTAL SETUP

The experiments use six open-source Java programs and mutation-generated faulty versions to evaluate six spectrum-based fault localization techniques and test case purification. The setup combines automated execution, source transformation, dynamic slicing, and wasted-effort measurement.

  • 4.1 Subject Programs: Six open-source Java libraries with large JUnit test suites form the experimental subjects.The researchers verify that each original program and its dependent libraries pass the complete test suite.
  • 4.1 Subject Programs: Mutation testing generates faulty program versions using six mutant operators, with each mutant applying a single program change.Equivalent mutants can produce the same observable output as the original program.
  • 4.2 Techniques in Comparison: The evaluation compares Tarantula, Statistical Bug Isolation, Ochiai, Jaccard, Ochiai2, and Kulczynski2.These techniques calculate statement suspiciousness from failing and passing test-case execution information.
  • 4.2 Techniques in Comparison: Spectrum-based fault localization ranks statements using suspiciousness values computed from failing and passing test cases that do or do not execute each statement.The four execution counts are αef(s), αnf(s), αep(s), and αnp(s).
  • 4.2 Techniques in Comparison: Wasted effort is measured as the faulty statement's rank, using the average rank when multiple statements share the same suspiciousness value.The candidate set is S, the faulty statement is s∗, and set size is denoted by |·|.
  • Implementation: The implementation automatically runs three purification phases using Java, Spoon for test-case transformation, JavaSlicer for dynamic backward slicing, and GZoltar-based fault localization.The experiments use Ant and JUnit for automated test execution, while slicing is applied selectively because it can require substantial time and resources.

5. EXPERIMENTAL RESULTS

Across six subject programs and 1800 seeded faults, test case purification generally improves existing fault-localization techniques, especially for faults initially ranked beyond the top 10 statements. The evaluation also measures wasted effort, detailed fault categories, negative-result causes, and computation time.

  • Overall comparison: Test case purification improves existing fault-localization techniques across six subject programs and 1800 seeded faults.The evaluation compares six techniques with and without purification.
  • Overall comparison: 43% of Tarantula-localized faults improve with purification, while 2.44% worsen.For Tarantula, 779/1800 faults achieve lower wasted effort after purification.
  • Overall comparison: Over 50% of faults show neutral results across all six techniques, partly because many faults are already easy or near-optimal to localize.The approach is reported to work particularly well for faults originally localized beyond the top 10 statements.
  • Overall comparison: 35.62 statements is the average wasted effort with Tarantula-Purification, versus 72.06 statements with Tarantula.Tarantula-Purification is best among 12 compared techniques for three subject programs; Ochiai-Purification is best for the other three.
  • Overall comparison: 18.22% better performance is reported for Tarantula-Purification than the best original technique, whose average wasted effort is 43.56 statements.The summary reports up to 43% positive results at the cost of 2.4% worsened faults.
  • Detailed comparison per fault category: For faults with soriginal > 10, Tarantula improves 524/687 faults (76.27%) and saves over 65 statements on average, with 30 worsened faults (4.37%).For soriginal = 1, 99.66% of faults retain an optimal rank; test-case dependency is identified as a major reason for negative results.

6. THREATS TO VALIDITY

The paper identifies threats involving experiment construction and method construction, including limited generality, mutation choices, test-case dependencies, and slicing-tool omissions.

  • Experiment construction: The evaluation covers six fault localization techniques and six Java subject programs, so its generality beyond these techniques and programs remains to be studied.Faults are generated through mutation testing, with 300 randomly selected faulty versions used in the experiments.
  • Experiment construction: Mutation-operator choice may affect fault localization effectiveness, but the paper does not analyze results by mutant type.The authors leave this issue for future work.
  • Method construction: The rank-refinement method uses a simple formula, while alternative combinations such as average or weighted average remain unexplored.The authors plan to design methods that use purified-test-case spectra more effectively.
  • Method construction: Test-case dependencies can produce negative results because a failure may propagate from a faulty statement absent from the failing test case’s spectrum.The paper assumes test cases execute independently, although dependencies sometimes occur.
  • Method construction: JavaSlicer may miss traces from native methods and Java standard-library classes, and the implementation may still omit some statements.The paper uses JavaSlicer because it is considered easy to use for Java 1.6.

7. RELATED WORK

Related work localizes faults through spectrum-based, mutation-based, slicing-based, and other approaches. This paper instead manipulates existing test cases to improve multiple fault localization techniques.

  • Novelty: The authors claim this is the first work to directly manipulate test cases for improving fault localization.Its stated contrast with existing work is the direct operation on test cases rather than subject programs.
  • Spectrum-based fault localization: Spectrum-based techniques such as Tarantula, Ochiai, and Jaccard rank faulty program locations using execution coverage and test outcomes.These techniques are described as state-of-the-art spectrum-based fault localization methods.
  • Other fault localization approaches: Prior work also uses code coverage, dynamic dependencies, noise reduction, semantic diagnosis, test-input reduction, and test-case prioritization for fault localization.These approaches target different aspects of coverage, diagnosis, inspection cost, labeling, and prioritization.
  • Test-suite and test-generation approaches: Some studies maximize diagnostic ability by changing test suites or generate test cases for fault localization, including approaches for invalid HTML programs.The cited HTML approach does not require test oracles because browser crashes report failures.
  • Contrast with prior work: Mutation-based and slicing-based approaches modify subject programs to identify faulty parts, whereas this work modifies test cases and reuses existing test oracles.The paper positions test case purification as a generally applicable framework for existing approaches.

8. CONCLUSION

The paper proposes test case purification to improve fault localization by extracting discriminating spectra from small test-case fractions. Experiments show benefits across six techniques, while future work targets broader evaluation, ranking methods, slicing cost, and other software problems.

  • Test case purification manipulates existing test cases into small fractions that collect discriminating spectra for assertions.The approach aims to make better use of existing test oracles.
  • The benefits of test case purification extend across six fault localization techniques.
  • Future experiments will assess the approach on additional Java projects.
  • Planned extensions include new ranking methods, lower test case slicing cost, and applications to regression testing or automatic software repair.
Loading 1409.3176v1…