PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Study of Pseudo-tested Methods

Oscar Luis Vera-Pérez, Benjamin Danglot, Martin Monperrus, Benoit Baudry

arXiv:1807.05030v2cs.SE

TL;DR

The paper asks whether pseudo-tested methods indicate poorly tested code and whether they identify test gaps developers consider worth improving. The authors replicate prior detection work and combine quantitative analysis with developer-informed examination of pseudo-tested methods, including a manual sample of 101. All 21 studied mature Java projects contained pseudo-tested methods, at rates from 1% to 46%, and these methods had significantly lower mutation scores than required methods.

  • Problem

    The paper asks whether pseudo-tested methods indicate poorly tested code and whether they identify test gaps developers consider worth improving.

    Source: §1 · p. 2

  • Method

    The authors replicate prior detection work and combine quantitative analysis with developer-informed examination of pseudo-tested methods, including a manual sample of 101.

    Source: §1 · p. 2, §2.2 · p. 6

  • Results

    All 21 studied mature Java projects contained pseudo-tested methods, at rates from 1% to 46%, and these methods had significantly lower mutation scores than required methods.

    Source: §7 · p. 29

  • Takeaways & Limitations

    Less than 30% of pseudo-tested methods in a sample of 101 were actual hints for test-suite improvement; targeted automatic test generation could improve adequacy and free developers for core features.

    Source: §7 · pp. 29–30

    Return-value choices may change method classifications, and requiring only one detected value can overestimate required methods.

    Source: §5 · p. 27

Abstract

from arXiv · show

Pseudo-tested methods are defined as follows: they are covered by the test suite, yet no test case fails when the method body is removed, i.e., when all the effects of this method are suppressed. This intriguing concept was coined in 2016, by Niedermayr and colleagues, who showed that such methods are systematically present, even in well-tested projects with high statement coverage. This work presents a novel analysis of pseudo-tested methods. First, we run a replication of Niedermayr's study with 28K+ methods, enhancing its external validity thanks to the use of new tools and new study subjects. Second, we perform a systematic characterization of these methods, both quantitatively and qualitatively with an extensive manual analysis of 101 pseudo-tested methods. The first part of the study confirms Niedermayr's results: pseudo-tested methods exist in all our subjects. Our in-depth characterization of pseudotested methods leads to two key insights: pseudo-tested methods are significantly less tested than the other methods; yet, for most of them, the developers would not pay the testing price to fix this situation. This calls for future work on targeted test generation to specify those pseudo-tested methods without spending developer time.

1 Introduction

The paper investigates whether pseudo-tested methods indicate poorly tested code and whether they identify test gaps developers consider worth fixing. It replicates prior work and combines quantitative comparisons with developer-informed qualitative analysis.

1 Introduction

  • Pseudo-tested methods occur in all studied projects, including projects with high test coverage, and are covered even though tests do not assess their effects.The body can be removed without a test failing.Source: §1 · p. 2
  • Across 21 open-source Java projects and 28K+ methods, the study replicates prior work with a different detection tool and 17 additional projects.The authors say this addresses internal and external validity threats.Source: §1 · p. 2
  • Pseudo-tested methods have significantly lower fault-detection ratios than other covered methods.The study compares method-level mutation scores for pseudo-tested and required methods.Source: §1 · p. 2
  • The study also identified test-suite issues in seven projects, and all submitted pull requests were merged to improve the suites.The authors’ contribution summary describes a quantitative comparison and a manual analysis involving developers.Source: §1 · pp. 2–3
  • In a manual analysis of 101 methods, developers considered only 30 worth additional testing effort.The analysis involved developers from three projects.Source: §1 · p. 3

2 Pseudo-tested Methods

The paper defines covered, pseudo-tested, and required methods in terms of test execution and whether tests assess method effects. It detects pseudo-tested methods by applying extreme body transformations and checking whether the test suite fails.

2.1 Definitions

  • A method is covered when a test case executes at least one path; a pseudo-tested method is covered but has no effects assessed by the suite.A required method is covered and has at least one effect assessed.Source: §2.1 · pp. 3, 5
  • The paper treats state changes to the receiver or other objects and returned values as possible method effects.Tests specify expected effects through assertions.Source: §2.1 · pp. 3–4
  • In the example, a test calls incrementVersion but does not check its version-field change, so removing that method body would go unnoticed.The example also illustrates that testability issues can make a method harder to test.Source: §2.1 · p. 5

2.2 Tool for Finding Pseudo-tested Methods

  • The detection algorithm selects covered methods, strips void bodies or substitutes predefined return values, and labels a method pseudo-tested if no variant causes a test failure.Variants are applied to the original program and analyzed separately; selection criteria exclude some likely false positives.Source: §2.2 · p. 6
  • Descartes implements the detection procedure for Java programs with JUnit suites as an extension of PITest.It also reports covered methods and uses PITest for test running and selection.Source: §2.2 · p. 6

3 Experimental Protocol

The experimental protocol tests the prevalence and testing quality of pseudo-tested methods, as well as their relevance to developers. It uses selected Java projects and method-level mutation metrics, alongside developer feedback.

3 Experimental Protocol

  • For methods returning values, the protocol adds transformations that return null or an empty array to expand the analysis scope.Table 1 summarizes extreme transformations by return type.Source: §3 · p. 6, Table 1 · p. 7

3.1 Research Questions

  • The protocol asks how frequent pseudo-tested methods are, whether they are the weakest-tested code, and whether developers consider them worth additional testing.It also investigates concrete test-suite issues and consults developers about a sample of 101 methods.Source: §3.1 · pp. 7–8

3.2 Study Subjects

  • The study selects 21 active Java projects using Maven and JUnit, drawing on prior-study subjects, industry partners, testing-literature projects, and projects selected for size or history.The subjects vary widely in project size, with some test codebases as large as or larger than application code.Source: Table 2 · p. 8, §3.2 · p. 9

3.3 Metrics

  • The analysis excludes uncovered methods and several categories, including simple getters and setters, deprecated or empty methods, constant-returning methods, and compiler-generated methods.Constructors and static initializers are excluded because the extreme transformations cannot target them.Source: §3.3 · pp. 9–10
  • The study measures covered-method and pseudo-tested-method ratios and compares method-level mutation scores for pseudo-tested and required methods using a Wilcoxon test.The mutation score is the ratio of detected mutants; the two method-specific scores use mutants generated within each method group.Source: §3.3 · p. 10

4 Experimental Results

The passage introduces the paper’s experimental results section without reporting a result.

4 Experimental Results

  • The following sections present the experimental results in depth.Source: §4 · p. 11

4.1 RQ1: How frequent are pseudo-tested methods?

Pseudo-tested methods appear in every study project, with prevalence ranging from 1% to 46%. Project-specific testing practices and domains qualify how those rates should be read.

4.1 RQ1: How frequent are pseudo-tested methods?

  • 1%-46% of analyzed methods were pseudo-tested across the projects, confirming their presence even in very well-tested projects.This independent replication reported a lower range than Niedermayr et al.’s 6%-53%; excluding deprecated methods and adding two mutation operators changed the targeted methods.Source: §4.1 · p. 11, §4.1.3 · p. 12

4.1.1 Analysis of outliers

  • Authzforce’s parameterized tests exhaustively test covered methods, yielding a lower PS RATE than projects with similar coverage.Its test suite has low method coverage, but those covered methods are tested exhaustively.Source: §4.1.1 · p. 11
  • Domain-specific testing helps explain high rates: 62 of scifio’s 72 pseudo-tested methods concern DICOM metadata whose presence tests do not check, while pdfbox needs visual oracles.Specifying some PDF functionality through an automated, fine-grained oracle is difficult.Source: §4.1.1 · pp. 11–12
  • Commons-cli and jopt-simple each had only two pseudo-tested methods despite 97% and 98% line coverage, respectively.Three of the four methods create and return exception messages; the other is a toString implementation.Source: §4.1.1 · p. 12

4.1.2 Relationship between pseudo-tested methods and coverage

  • Coverage and PS RATE have a moderate negative relationship (Pearson coefficient -0.67, p < 0.01), but coverage alone does not determine PS RATE.Authzforce and xwiki-rendering have comparable coverage but different PS RATE values; the authors attribute the general correlation to well-tested projects also tending to have higher coverage.Source: §4.1.2 · p. 12

4.2 RQ2: Are pseudo-tested methods the weakest points in the program, with respect to the test suite?

Mutation analysis finds pseudo-tested methods significantly less tested than required methods across all 21 projects. Exception-raising mutants can nevertheless be detected, and method-level scores vary widely.

4.2 RQ2: Are pseudo-tested methods the weakest points in the program, with respect to the test suite?

  • 52% above that of pseudo-tested methods was the average mutation score of required methods; the difference was significant across all 21 projects (p < 0.01; effect size 1.5).The smallest gap was in pdfbox (32% versus 68%); in scifio, 1% of pseudo-tested-method mutants were killed versus 84% in required methods.Source: §4.2 · pp. 13–14, Figure 1 · p. 14, §4.2.2 · p. 16

4.2.1 Analysis of interesting examples

  • Some mutants in pseudo-tested methods are killed because they make the program throw an exception, not because assertions check method behavior.In authzforce, two of five mutants in checkNumberOfArgs were detected through exceptions; in jopt-simple, all nine killed mutants caused crashes. Scifio’s metadata mutants were not detected because test oracles do not specify those values.Source: §4.2.1 · pp. 14–15

4.2.2 Distribution of method-level mutation score

  • Required-method mutation-score distributions tend rightward, while pseudo-tested-method distributions tend leftward and usually span a wide range.Scifio’s pseudo-tested methods are notably left-skewed because none of their metadata mutants were detected; the violin plots omit commons-cli and jopt-simple because they have too few pseudo-tested methods.Source: §4.2.2 · pp. 15–16
  • 63 pseudo-tested methods had a 100% mutation score, but 34 had only one or two trivial mutants.In jsoup, all 69 mutants in the pseudo-tested load method caused exceptions, although the method’s effects could be removed without detection.Source: §4.2.2 · p. 16

4.3 RQ3: Are pseudo-tested methods relevant for developers to improve the quality of the test suite?

The qualitative study examined pseudo-tested methods across projects, finding recurrent oracle and input flaws and obtaining developer acceptance of several test improvements. Case studies show how misplaced or missing assertions and weak checks can let tests pass despite suppressed behavior.

4.3 RQ3: Are pseudo-tested methods relevant for developers to improve the quality of the test suite?

  • The study examined accessible void and boolean pseudo-tested methods and identified eight test issues across seven projects: seven oracle problems and one missing test input.The authors prepared pull requests or emailed developers to solicit feedback on each issue.Source: §4.3 · p. 16

4.3.1 Feedback from aws-sdk-java

  • In aws-sdk-java, a misplaced assertion inside the socket mock was never evaluated; moving it after prepareSocket made the method no longer pseudo-tested, and the developer merged the change.The four covering tests shared the same pattern.Source: §4.3.1 · pp. 16–17

4.3.2 Feedback from commons-collections

  • In commons-collections, tests for unsupported iterator add and remove operations could pass without checking for the expected exception; the developer merged a pull request adding that check.The report identifies two pseudo-tested add implementations and four remove implementations.Source: §4.3.2 · p. 18

4.3.3 Feedback from commons-codec

  • In commons-codec, isEncodeEqual lacked an assertion; adding one exposed an incorrect input pair, and developers accepted the fixture change, which increased code coverage by 0.2%.The method was covered by one test case, and the test inputs were expected to make it return true.Source: §4.3.3 · pp. 19–20

4.3.4 Feedback from commons-io

The case studies show that pseudo-tested methods expose varied test weaknesses, from weak oracles to unverified behavior. Developers recognized the issues, but test repairs could be deferred when resources were directed to higher-priority work.

4.3.4 Feedback from commons-io

  • In commons-io, write methods were pseudo-tested because the oracle checked that two outputs matched, so two empty streams still passed; an accepted change compared both against expected output.The accepted change slightly increased code coverage by 0.07%.Source: §4.3.4 · p. 20

4.3.5 Feedback from spoon

  • In spoon, developers agreed that indirectly covered visitCtAssert did not have its effects assessed and should receive stronger verification or a new test; they opened a general issue but took no immediate action.The method handles assert expressions in an Abstract Syntax Tree.Source: §4.3.5 · p. 21

4.3.6 Feedback from flink-core

  • In flink-core, developers said configure’s test intentionally checked that a field remained unchanged, while the authors noted it did not verify configuration loading; developers took no further action.With the method body removed, the assertion about the field’s initial value still passed.Source: §4.3.6 · p. 21

4.3.7 Feedback from sat4j-core

  • In sat4j-core, a developer added an assertion for removeConstr and a more complex test for learn, whose optimization ran but produced the same result with or without the optimization.The learn method was covered indirectly by 68 test cases.Source: §4.3.7 · p. 22

4.3.8 Discussion

  • Across the eight issues, seven involved oracles; developers recognized the flaws and accepted supplied fixes, but did not always prioritize repairs when only given the problem.The discussion says limited resources were directed to new features and bug fixes rather than improving existing tests.Source: §4.3.8 · pp. 22–23

4.4 RQ4: Which pseudo-tested methods do developers consider worth an additional testing action?

Developers judged whether pseudo-tested methods warranted more testing based on their role, structure, and project context. In their sample, they considered 30 of 101 methods worth additional testing, while some criteria for prioritization conflicted.

4.4 RQ4: Which pseudo-tested methods do developers consider worth an additional testing action?

  • In sat4j-core and spoon, 23% of sampled methods were worth additional testing; in authzforce, 46% were, but its absolute count of six was close to sat4j-core’s eight.The authors suggest that many pseudo-tested methods may belong to functionality considered less important or not a priority.Source: §4.4 · pp. 23–24
  • Developers considered methods less worth testing when they were generated, debug-related, little-used, migrating, trivial, placeholders, or delegating with little custom logic.They also cited methods created to complete an interface implementation.Source: §4.4 · p. 24
  • Methods were considered worth testing when they supported core or widely used functionality, external clients, incomplete features, unique access to class functionality, or operation preconditions.Developers also valued methods whose behavior affected a class’s main responsibility.Source: §4.4 · pp. 25–26
  • Worth-testing criteria can conflict, and developers’ decisions rely on internal codebase knowledge, making automatic determination difficult.The study selected three projects for developer discussions and used random samples of about 25% of pseudo-tested methods in sat4j-core and spoon, but all 13 in authzforce.Source: §4.4 · pp. 23, 26
  • 30 of 101 pseudo-tested methods (30%) were judged worth additional testing by lead developers from three mature projects, whose decisions drew on domain and design knowledge.The authors conclude that an absolute absence of pseudo-tested methods should not be prescribed.Source: §4.4 · p. 26

5 Threats to validity

The quantitative findings are subject to limits involving mutation equivalence, method classification, and the Java-only study sample. The qualitative assessment also depends on project and developer perspectives.

5 Threats to validity

  • Equivalent mutants may affect mutation-score values, and methods with empty bodies or constant returns are excluded from analysis.The equivalent-mutant issue also applies to traditional mutation testing.Source: §5 · pp. 26–27
  • The chosen return values can change whether non-void, non-boolean methods are classified as pseudo-tested or required, potentially overestimating required methods.The study reports 916 such pseudo-tested methods, 36% of the total; void and boolean methods tend to produce more pseudo-tested methods in these projects.Source: §5 · p. 27
  • Because all study subjects are Java projects, the findings cannot be generalized to other programming languages; treating unit and integration tests equally could also affect pseudo-tested-method counts.The authors did not distinguish unit from integration tests because doing so would require an arbitrary threshold.Source: §5 · p. 27
  • The qualitative results depend on the researchers’ and consulted developers’ insight, codebase characteristics, and the methods presented; developer interest and opinions varied.Source: §5 · p. 27

6 Related work

The paper builds on work about pseudo-tested methods, mutation testing, and test oracles while distinguishing its study through new subjects and tools, adequacy analysis, and developer interactions.

6 Related work

  • Niedermayr et al. introduced pseudo-tested methods to examine covered code whose behavior is poorly assessed and whether code coverage indicates test quality.Source: §6 · p. 27
  • The study extends Niedermayr et al.’s work with novel study objects and a different detection tool, then examines test-suite adequacy and developers’ views of additional testing effort.Nineteen of the 21 projects had not been analyzed in the earlier study.Source: §6 · pp. 27–28
  • Prior oracle research includes checked coverage, which Schuler and Zeller found more realistic than usual coverage in manual checks of seven projects.Source: §6 · p. 28
  • Jahangirova et al. combine mutation testing to improve oracle fault detection with automated unit-test generation to reduce correct executions rejected by assertions.Their approach increased the mutation-score approximation of fault-detection ratio by 48.6% on average across five projects.Source: §6 · p. 28
  • Prior work studies oracle completeness and soundness formally, while another study reports that one in ten test inputs fail to propagate to observable points.Staats et al. frame oracles as predicates over tests and programs; Androutsopoulos et al. analyze propagation across 30 programs and more than 7M test cases.Source: §6 · p. 28
  • Mutation-testing literature evaluates fault-detection assumptions and tools, while statement-deletion approaches create fewer mutants but can remain time-consuming to assess for equivalence.The cited deletion approaches do not use the complete-method-body transformations used here, which the authors say are easier for developers to understand.Source: §6 · pp. 28–29

7 Conclusion

Across 21 mature Java projects, pseudo-tested methods were present and had lower mutation scores than required methods. The qualitative analysis found weak oracles, but fewer than 30% of sampled methods warranted further testing in developers’ view.

7 Conclusion

  • All 21 studied mature Java projects contained pseudo-tested methods, comprising 1% to 46% of analyzed methods; their mutation scores were systematically and significantly lower than required methods’ scores.Source: §7 · p. 29
  • Developers confirmed weak test-oracle issues identified through pseudo-tested methods and accepted proposed pull requests to fix them.Source: §7 · p. 29
  • Less than 30% of 101 sampled pseudo-tested methods were considered actual hints for further test-suite improvement.Source: §7 · p. 29
  • Among the methods not considered worth additional testing were little-used, generated, trivial, helper, debugging, and delegation-receiver methods.Source: §7 · pp. 29–30
  • Methods revealing test-suite interaction issues involved core functionality, broad codebase use, external clients, or input-precondition checks.Source: §7 · p. 30
  • The authors propose investigating targeted automatic test generation for pseudo-tested methods, alongside better static-analysis filters to reduce methods developers deem unworthy of testing.They also identify comparisons of statement coverage and validation of extreme transformations’ fault-detection capabilities as future work.Source: §7 · p. 30

A Source code for the study subjects

The appendix identifies the study projects and describes the source-code and revision information recorded for them.

A Source code for the study subjects

Related papers

Loading 1807.05030v2…