Jartege: a Tool for Random Generation of Unit Tests for Java Classes

Re9sume9

• Jartege generates random test cases as sequences of constructor and method calls for Java classes specified in JML.JML specifications help reject irrelevant tests and serve as the test oracle.Source: R´esum´e · p. 2
• Users can parameterize generation by weighting classes and operations and controlling the number of instances created for each class.Source: R´esum´e · p. 2

Mots-cle9s

• The report indexes the work under unit testing, random test-case generation, Java, and JML.Source: Mots-cl´es · p. 2, Keywords · p. 2

1 Introduction

• Testing seeks faults because exhaustive testing is generally impossible for large or infinite input domains, and its cost is estimated at about 40% of software-development cost.Source: §1 · p. 3
• Unlike systematic strategies that partition the input domain, random testing generally selects tests without using the program or its specification, though an operational profile may guide it.Source: §1 · pp. 3–4
• Random testing is presented as cheap and easy to implement, able to produce very large test sets and detect substantial errors at low cost.Source: §1 · pp. 3–4
• Partition testing can be much more effective when it concentrates on small subdomains likely to cause failures.Source: §1 · p. 4
• Manual unit tests can exceed the class's code volume and may also need changes during refactoring, making continuous testing and refactoring hard to reconcile.Source: §1 · p. 4
• The report proposes random test generation to facilitate continuous testing and refactoring in extreme programming.The report's outline says later sections describe a bank-account case study and errors detected by generated tests.Source: §1 · p. 4

2 Approach

• The JML compiler translates specifications into runtime assertions and distinguishes a method's entry precondition from internal preconditions of operations it calls.Source: §2 · p. 4
• Jartege generates random test cases for Java classes specified in JML, using the specifications as a test oracle in the JML-JUnit style.Source: §2 · p. 5
• Jartege rejects test sequences containing calls that violate entry preconditions, treating them as uninteresting tests that would expose faults in the test program.Source: §2 · p. 5
• The approach trusts the specification rather than testing whether entry preconditions are too strong, because including such calls would prevent long sequences; internal precondition violations remain detectable.Source: §2 · p. 5
• Violations of invariants, postconditions, or internal preconditions count as errors, which may indicate a Java-program fault or a specification fault.Source: §2 · p. 5

3.1 Informal Specification of the Bank Accounts

• The case study's informal account model includes a balance and minimum, credit and debit operations, cancellation of recent operations, and changing the minimum balance.Source: §3.1 · pp. 5–6

3.2 Bank Account Modeling

• Account stores the balance and minimum and provides credit, debit, and cancel; a linked history of previous balances supports cancellation.Source: §3.2 · p. 6, Figure 1 · p. 6

3.3 JML Specification and Java Implementation

• The Java classes use lightweight public JML specifications, with Account's invariant requiring its balance to remain at least its minimum.Pure accessors permit use in JML assertions; History has no JML assertions.Source: §3.3 · pp. 6, 8
• Credit requires a nonnegative amount, increases the balance, and records the former balance in a new history node; debit additionally requires the resulting balance to exceed the minimum.Source: §3.3 · p. 7
• Cancel requires a nonempty history and restores the prior balance and preceding history node.Source: §3.3 · p. 8

4 Jartege

• Jartege generates random sequences of calls for JML-specified classes and supports tests spanning several classes when their dependencies make isolated testing unsuitable.Generated programs can be rerun after corrections or for regression testing.Source: §4 · p. 8

4.1 Practical Use of Jartege

• The generate method creates a test class with a chosen number of cases and call attempts, using accessible constructors to create argument objects.Calls are executed during generation, and those violating operation preconditions are rejected; strong preconditions can leave fewer calls than attempted.Source: §4.1 · p. 9

4.2 Test Programs Produced by Jartege

• Jartege produces a class whose main method runs generated test cases sequentially, each comprising constructor and method calls.When a call violates its entry precondition, the generated program can mark the test inconclusive; such cases may follow later code or specification changes.Source: §4.2 · pp. 9–10
• 71 errors were reported across 100 tests, with zero inconclusive tests in the example run.The first reported error followed a credit operation that violated Account's invariant.Source: §4.2 · p. 10

5.1 Weights

• Weights determine the probability of choosing classes and operations; setting an operation's weight to zero prevents its selection.Weights default to 1 and can be changed for all methods in a class or for selected methods.Source: §5.1 · pp. 10–11

5.2 Creation of Objects

• Creation-probability functions control whether Jartege creates a new object or reuses an existing one, allowing users to limit or encourage instances.A threshold function permits at most s instances of a class.Source: §5.2 · p. 11

5.3 Parameter Generation of Primitive Types

• For primitive parameters, users can provide generators that target values satisfying an operation's precondition.This addresses cases where unrestricted random values are unlikely to meet a strong precondition; the example generates debit amounts within a balance-dependent range.Source: §5.3 · p. 12

5.4 Fixtures

• A test fixture lets multiple generated test cases operate on a particular set of objects, with optional setUp and tearDown methods.The fixture also contains attributes corresponding to the objects used by a test.Source: §5.4 · p. 12

• 100 generated tests revealed three errors: one involving credit and two involving cancel; the cancel errors required three calls in a specific order with particular values.The authors noted these errors were not entirely obvious and could have been missed in a manually developed test suite.Source: §6 · p. 13
• Credit can reduce the balance below its previous value when integer overflow occurs.The example overflows on a large credit, producing a negative balance below the minimum.Source: §6 · p. 13
• Cancel can produce an incorrect result when setMin raises the minimum above the balance before cancellation.The example performs credit, raises the minimum, then cancels and restores a balance below that minimum.Source: §6 · p. 13
• The third error combines a debit overflow with the cancel error involving a changed minimum balance.The report characterizes the debit overflow as similar to the credit overflow.Source: §6 · p. 13

• Unlike JML-JUnit, which requires users to supply the invoking object and parameter values and tests one method call at a time, Jartege generates sequences automatically.The report notes that JML-JUnit may miss interesting values and requires users to construct test data.Source: §7 · p. 14
• Korat automatically constructs bounded-size objects and exhaustively tests a method, but each test case contains only one object construction and one method invocation.Source: §7 · p. 14
• Tobias generates tests from abstract test patterns, while Jartege is designed to generate long sequences without combinatorial explosion.The report identifies combinatorial explosion as Tobias's main problem when sequences exceed a couple of method calls.Source: §7 · p. 14

• The authors describe Jartege as being in its infancy, with substantial work still remaining.Source: §8 · p. 14
• Primitive method-parameter values are generated with manually written methods; deriving them automatically from JML precondition range constraints is proposed future work.Source: §8 · p. 14
• Jartege can generate tests with hundreds of calls, motivating a future tool to extract a minimum call sequence that reproduces a failure.Source: §8 · p. 14
• Jartege produced tests more easily than writing unit tests with JUnit or JML-JUnit and detected a few errors in its case study.The authors plan to continue specifying Jartege in JML and testing its classes with the tool itself.Source: §8 · p. 14
• No comparison with other testing strategies has yet been done; Jartege is not claimed to replace systematic methods or expert-designed test sets.The authors expect boundary-testing methods such as BZTT to produce more interesting test cases.Source: §8 · p. 14
• The initial goal is debugging unstable Java classes; evaluating the reliability of stable components before release is another proposed use.Source: §8 · p. 14
• Jartege offers operational-profile features for statistical testing, but defining a correct profile and relating generated tests to reliability remain difficult, especially when component state matters.Source: §8 · p. 15

• Jartege generates numerous random unit tests for JML-specified Java classes, aiming to find errors at low cost and support continuous testing and code refactoring.Source: §9 · p. 15
• JML specifications filter irrelevant test cases and serve as runtime test oracles; the authors argue this cost can be worthwhile for intensive testing.Working with specifications alongside the program can also help developers build confidence in them and reuse them in other tools.Source: §9 · p. 15
• The authors characterize their statistical approach as complementary to deterministic test generation, with a combination of both potentially fruitful.Source: §9 · p. 15
• JML's Java-based syntax and placement in source-code comments make it accessible to Java programmers and convenient to develop alongside a program.The authors also cite automatic test oracles and test-case generation as reasons to use specification languages such as JML.Source: §9 · p. 15