Source-linked AI summary
Metamorphic Testing: A New Approach for Generating Next Test Cases
T. Y. Chen, S. C. Cheung, S. M. Yiu
TL;DR
Conventional testing often lacks practical test oracles and leaves the value of successful tests for later error discovery underexplored. This paper introduces Metamorphic Testing, which derives new tests from successful ones, and reports that these tests are feasible and useful for revealing some otherwise undetected errors.
Problem
The value of successful test cases for revealing further errors, production-phase error detection, and testing without practical oracles remain inadequately addressed.
Method
Metamorphic Testing generates new test cases from previous successful cases to expose errors potentially missed by those cases.
Results
The authors report that the approach is feasible and that newly constructed test cases are useful for revealing some software errors.
Takeaways & Limitations
Metamorphic Testing can augment existing test-selection strategies and support error detection when practical test oracles are unavailable.
Takeaways & Limitations
A fully developed methodology supporting the approach has not yet been established.
Abstract
from arXiv · showhide
In software testing, a set of test cases is constructed according to some predefined selection criteria. The software is then examined against these test cases. Three interesting observations have been made on the current artifacts of software testing. Firstly, an error-revealing test case is considered useful while a successful test case which does not reveal software errors is usually not further investigated. Whether these successful test cases still contain useful information for revealing software errors has not been properly studied. Secondly, no matter how extensive the testing has been conducted in the development phase, errors may still exist in the software [5]. These errors, if left undetected, may eventually cause damage to the production system. The study of techniques for uncovering software errors in the production phase is seldom addressed in the literature. Thirdly, as indicated by Weyuker in [6], the availability of test oracles is pragmatically unattainable in most situations. However, the availability of test oracles is generally assumed in conventional software testing techniques. In this paper, we propose a novel test case selection technique that derives new test cases from the successful ones. The selection aims at revealing software errors that are possibly left undetected in successful test cases which may be generated using some existing strategies. As such, the proposed technique augments the effectiveness of existing test selection strategies. The technique also helps uncover software errors in the production phase and can be used in the absence of test oracles.
Preliminaries
The approach constructs new test cases from an existing test case and its output to reveal errors that remained undetected. It also imposes practical constraints on construction and output checking, especially during production use.
- Preliminaries: The approach constructs new test cases based on an input–output pair.
- Preliminaries: New test cases are designed to reveal errors that go undetected by the original test case.
- Preliminaries: Construction and corresponding-output checking are assumed to take strictly less time than program execution.
- Preliminaries: In production, newly constructed test cases should not modify the production database, although such modifications are allowed during testing.
Examples
The examples show how metamorphic test cases expose errors in binary search, kth-occurrence, and shortest-path programs when outputs appear correct or test oracles are impractical. They also illustrate production-oriented checking through related inputs and expected-result relationships.
- Example 1: Binary Search on Sorted Array: Binary-search examples derive related tests to expose array overwriting and splitting errors that an apparently correct output may not reveal.The construction compares neighboring or boundary elements and reruns the program with modified inputs.
- Example 2: kth Occurrence of x in Unsorted Array: For kth-occurrence search in an unsorted array, related tests check false nonexistence reports, overwriting, and returning the qth occurrence when q differs from k.Some constructions may be abandoned when scanning is too long or a suitable element cannot be found.
- Example 3: Shortest Path in an Undirected Graph: For shortest paths in nontrivial weighted graphs, verifying only that consecutive output vertices are edges is insufficient because a practical test oracle may not exist.The example states that the listed related test cases together reveal a program error.
Conclusion
The paper proposes Metamorphic Testing, which generates additional test cases from apparently successful cases to expose errors that may remain undetected. The approach augments existing testing, applies in testing and production without test oracles, but still lacks a fully developed methodology.
- Conclusion: Metamorphic Testing generates additional test cases from the input-output pair of an apparently successful test case selected by established criteria.The new cases target errors possibly left undetected by the previous successful case.
- Conclusion: The approach enables practitioners to design additional tests, improve software reliability, and augment existing testing strategies by reusing corresponding successful test cases.The newly constructed tests are described as feasible and useful for revealing some common errors.
- Conclusion: Metamorphic Testing is applicable in both testing and production phases and does not rely on the assumption of a test oracle.This makes its use in situations without test oracles particularly interesting.
- Conclusion: The approach does not depend on any particular test case selection strategy, so it can be used with and combined with other selection strategies.The paper also notes that it can be combined with Blum's program checker.
- Conclusion: A fully developed methodology supporting Metamorphic Testing remains an open limitation, with the ultimate goal of pragmatically constructing next test cases.Because the approach generally requires problem-domain knowledge, the authors expect a domain-specific methodology and are studying criteria for classifying programs into appropriate domains.