PaperlayerView exact revision on arXiv

Source-linked AI summary

A3Test: Assertion-Augmented Automated Test Case Generation

Saranya Alagarsamy, Chakkrit Tantithamthavorn, Aldeida Aleti

arXiv:2302.10352v1cs.SE

TL;DR

Test generation is laborious, and AthenaTest has limited assertion knowledge. A3Test transfers assertion-generation knowledge to test generation and verifies names and signatures; on Defects4J it improved correctness and coverage over AthenaTest.

  • Problem

    AthenaTest pretraining uses natural language and source code but not assertion knowledge, although assertions assess expected unit-function behavior.

    Source: §4.1 · p. 7

  • Method

    A3Test pre-trains on assertion generation, fine-tunes for test generation, and verifies generated tests’ naming consistency and signatures.

    Source: §1 · p. 2, §4 · p. 7

  • Results

    A3Test generated 147% more correct test cases and achieved 15% more method coverage, with fewer generated tests than AthenaTest.

    Source: §1 · p. 3, §9 · p. 16

  • Takeaways & Limitations

    Assertion pre-training and verification each contributed to improvement, and their combination performed best.

    Source: §1 · p. 3

    Reported accuracy is a lower bound because the study did not seek optimal hyperparameter settings.

    Source: §8 · p. 15

Abstract

from arXiv · show

Test case generation is an important activity, yet a time-consuming and laborious task. Recently, AthenaTest -- a deep learning approach for generating unit test cases -- is proposed. However, AthenaTest can generate less than one-fifth of the test cases correctly, due to a lack of assertion knowledge and test signature verification. In this paper, we propose A3Test, a DL-based test case generation approach that is augmented by assertion knowledge with a mechanism to verify naming consistency and test signatures. A3Test leverages the domain adaptation principles where the goal is to adapt the existing knowledge from an assertion generation task to the test case generation task. We also introduce a verification approach to verify naming consistency and test signatures. Through an evaluation of 5,278 focal methods from the Defects4j dataset, we find that our A3Test (1) achieves 147% more correct test cases and 15% more method coverage, with a lower number of generated test cases than AthenaTest; (2) still outperforms the existing pre-trained models for the test case generation task; (3) contributes substantially to performance improvement via our own proposed assertion pre-training and the verification components; (4) is 97.2% much faster while being more accurate than AthenaTest.

1 INTRODUCTION

A3Test augments test generation with assertion knowledge and verification, and outperforms AthenaTest on correctness, method coverage, and speed in an evaluation spanning about 5,000 methods from five projects.

1 INTRODUCTION

  • Random- and search-based generators can achieve good code coverage but may produce unreadable tests, limiting their fit with industrial developers’ needs.Source: §1 · p. 1
  • The authors’ initial AthenaTest implementation produced 0% correct tests; after modifications, it reached 18.08%, similar to the reported 16.21%.Source: §1 · p. 2
  • AthenaTest pretraining omits assertion knowledge; 26.71% of its generated tests had incorrect assertions, and 9.49% were syntactically incorrect.Source: §1 · p. 2
  • A3Test adapts assertion-generation knowledge to test generation, then verifies naming consistency and test signatures by revising names and adding missing signature keywords or annotations.Source: §1 · pp. 2–3
  • A3Test generated 147% more correct test cases and achieved 15% more method coverage, with fewer generated tests than AthenaTest.Source: §1 · p. 3
  • A3Test outperformed PLBART, CodeGPT, CodeBERT, and CodeT5 for test case generation, although not all pre-trained models were effective for that task.Source: §1 · p. 3
  • Assertion and verification components contributed 35.30% and 23.7% of relative improvement over basic PLBART, while their combination performed best.Source: §1 · p. 3
  • A3Test took 2.9 hours to generate test cases in one attempt, 97.2% faster while being more accurate than AthenaTest.Source: §1 · p. 3

2 BACKGROUND & RELATED WORK

The background contrasts unit testing and conventional automated generation with learned test generation. It presents AthenaTest as a transformer approach, while noting its low correct-test rate and the need for better performance.

2.1 Unit Testing in a Nutshell

  • Unit testing checks individual software units or components, with test methods using assertions to verify a focal method’s behavior.The example uses a Calculator focal method and a corresponding test method.Source: §2.1 · p. 3
  • Random- and search-based tools generate tests differently, but their test cases are criticized for poor quality and understandability.Randoop generates random method-call sequences; EvoSuite uses an evolutionary search targeting code coverage.Source: §2.1 · p. 4

2.2 DL-based Test Case Generation

  • AthenaTest uses a BART transformer to learn from focal methods and developer-written tests, aiming for realistic, accurate, readable outputs.It is pretrained on English and Java code and fine-tuned on Methods2Test.Source: §2.2 · pp. 4–5
  • 16% of AthenaTest-generated tests are reported correct, motivating further improvement in deep-learning test generation.The background contrasts learned generation with methods that optimize coverage but rely on manually created rules or heuristics.Source: §2.2 · pp. 4–5

3 A REPLICATION OF ATHENATEST

The replication modification achieved 18.08% correct tests, close to the original AthenaTest’s 16.21%. The authors also report that their initial replication produced no correct tests and that the study did not systematically tune parameters.

3 A REPLICATION OF ATHENATEST

  • The initial replication produced 0% correct tests; the authors attribute the difficulty to missing implementation details and settings.Their manual analysis found outputs skewed toward natural-language text rather than test cases.Source: §3 · pp. 5–6
  • The authors suspect unequal pretraining epochs contributed to natural-language-biased outputs: English used 40 epochs, versus 10 for Java.Their modification reduced English pretraining to 8 epochs, below Java pretraining, and changed other settings to obtain comparable results.Source: §3 · p. 6
  • 18.08% correct tests in the modified replication was similar to the original AthenaTest’s 16.21%.The modification followed an initial replication that generated 0% correct tests.Source: §3 · pp. 5–6
  • The replication did not systematically tune parameters, and the authors say tuning could further improve its results.They trialed a few possible settings and report that the modification performed 1.8% better than the original AthenaTest.Source: §3 · p. 6

4 A3TEST - ASSERTION AUGMENTED AUTOMATED TEST CASE GENERATION

A3Test transfers assertion-generation knowledge to test-case generation, then verifies generated tests for naming and signature consistency. Its pipeline combines PLBART-based assertion pre-training, test-case fine-tuning, decoding, and automated corrections.

4 A3TEST - ASSERTION AUGMENTED AUTOMATED TEST CASE GENERATION

  • A3Test adapts assertion-generation knowledge to test-case generation by pre-training an assertion model and fine-tuning it on focal-method/test-case pairs.The assertion model uses masked language modeling; fine-tuning treats test generation as translation from a focal method to its developer-written test.Source: §4 · pp. 6–7, §4.1 · p. 7

4.1 Learning Meaningful Assert Statements

  • Assertions assess expected unit-function behaviour; A3Test pre-trains on assertion statements to transfer that knowledge to test-case generation.The source domain is assertion generation and the target domain is test-case generation; pre-training masks 20% of tokens.Source: §4.1 · p. 7

4.2 Learning Meaningful Test Cases

  • The pre-trained assertion model is fine-tuned on Methods2Test, which pairs focal methods with developer-written test methods.The dataset contains 780k pairs, split into 624,022 training pairs and 78,534 validation pairs.Source: §4.2 · p. 7

4.3 Generating Test Cases

  • At inference, beam search tracks top-k candidates and selects one best test case for each focal method.Unlike AthenaTest’s 30 candidates, A3Test generates a single best candidate.Source: §4.3 · p. 8

4.4 Verifying Naming Consistency and Test Signatures

  • Verification detects incomplete parentheses, inconsistent test names, and invalid signatures, correcting the generated tests.It uses push-pop matching for parentheses, adds a missing test prefix, and checks for @Test, public, void, and test in the signature.Source: §4.4 · p. 8

5 EXPERIMENTAL SETUP

The evaluation uses 5,278 public focal methods from five Defects4J projects. The paper reports the model, training setup, and fine-tuning configuration used for the experiments.

Datasets

  • The evaluation covers 5,278 public focal methods from Lang, Chart, Cli, Csv, and Gson in Defects4J.Lang contributes 2,712 methods, Chart 1,328, Cli 645, Csv 373, and Gson 220.Source: §5 · p. 9

Model Training

  • A3Test uses PLBART’s pre-trained tokenizer and model, fine-tuned for test-case generation on Methods2Test.AthenaTest uses the best hyperparameters reported by Tufano et al.Source: §5 · p. 9

Hyper-Parameter Settings for Fine-Tuning

  • Fine-tuning uses PLBART-base with 12 encoder/decoder blocks, 12 attention heads, a 1e-5 learning rate, and AdamW.The learning rate follows a linear schedule.Source: §5 · p. 9

RQ1: How effective is A3Test compared to AthenaTest?

On Defects4J, A3Test produces more correct tests and higher focal-method coverage than AthenaTest, while generating fewer test cases. The authors report consistent improvements across individual projects.

Results

  • A3Test achieves 147% more correct test cases and 15% more focal-method coverage than AthenaTest, with fewer generated test cases.Correct-test-case rates are 40.05% for A3Test and 16.21% for AthenaTest; focal-method coverage is 46.80% and 43.75%, respectively.Source: RQ1: How effective is A3Test compared… · pp. 10–11
  • Across individual Defects4J projects, A3Test performs 112%–387% better in correct-test-case counts and 2%–411% better in focal-method coverage.The reported comparisons are against AthenaTest.Source: RQ1: How effective is A3Test compared… · p. 10
  • A3Test generates 5,278 test cases in one attempt per focal method, versus AthenaTest’s 158,400 tests from 30 attempts each.A3Test’s 2,114 correct tests cover 46.80% of focal methods; AthenaTest’s 25,680 correct tests cover 43.75%.Source: RQ1: How effective is A3Test compared… · p. 10

RQ2: Does A3Test outperform existing pre-trained models?

A3Test outperforms the compared pre-trained models for test-case generation, while results vary substantially among those models. PLBART alone also exceeds AthenaTest in correct-test-case rate.

Results

  • Existing pre-trained models range from 0% for CodeGPT and CodeBERT to 21.5% for PLBART on test-case generation.The authors conclude that pre-trained model performance varies by task and recommend investigating models before adopting them for a downstream task.Source: RQ2: Does A3Test outperform existing… · p. 11
  • PLBART alone improves on AthenaTest from 16.21% to 21.50% correct test cases; the authors distinguish their implicit and explicit pre-training strategies.Both approaches use BART-based architectures, but with different pre-training strategies.Source: RQ2: Does A3Test outperform existing… · p. 11
  • A3Test reaches 40.05% correct test cases, compared with 21.50% for PLBART alone.Table 4 compares correct-test-case percentages across A3Test, AthenaTest, and other pre-trained language models.Source: RQ2: Does A3Test outperform existing… · p. 11, Table 4 · p. 12

RQ3: What is the contribution of the assert pre-training and verification components on the performance of A3Test?

A3Test’s assertion pre-training and verification each improve performance over basic PLBART, and the combined approach performs best. The ablation compares PLBART configurations with neither, either, or both components.

RQ3: What is the contribution of the assert pre-training and verification components on the performance of A3Test?

RQ4: How efficient is A3Test compared to AthenaTest?

A3Test generated tests faster than AthenaTest in the reported comparison, while the discussion examines test quality, line coverage, private-field calls, and deprecated assertions.

RQ4: How efficient is A3Test compared to AthenaTest?

  • For 5,278 total test cases, A3Test generated them in 2.9 hours with one attempt, versus AthenaTest’s 105 hours and 30 attempts, a 97.2% efficiency improvement.Average generation time per test case was 1.98 seconds for A3Test and 2.34 seconds for AthenaTest.Source: RQ4: How efficient is A3Test compared… · p. 13

7 DISCUSSION & FUTURE WORK

  • Early analysis describes generated tests as accurate and human-readable, while noting that more analysis is needed.The tests are described as asserting focal-method behaviour and using appropriate variable and method names.Source: §7 · p. 14

7.2 Generation calls to private fields

  • Some generated tests call private fields using focal context such as class names, constructors, method signatures, and fields; access restrictions can make these tests fail.The authors treat such failures as a way to check access modifiers, while noting that a failing test does not necessarily mean its oracle caught a bug.Source: §7.2 · p. 14

7.3 Deprecated Assert Statements.

  • 6% to 8% of A3Test cases use deprecated Assert.assertThat(); JUnit Jupiter’s Assertions class does not provide this JUnit 4 method.The authors believe the fine-tuning data includes tests using the method and plan further investigation.Source: §7.3 · pp. 14–15

7.1 Line Coverage Analysis

  • For 18 public methods, A3Test generated correct tests with the best line coverage for most focal methods compared with AthenaTest.Table 7 reports absolute and percentage line coverage for each method.Source: §7.1 · p. 14, Table 7 · p. 15

8 THREATS TO VALIDITY

The authors identify threats from replication settings, limited project coverage, and reliance on method coverage as the performance metric.

8 THREATS TO VALIDITY

  • AthenaTest replication settings were not exhaustively optimized; the authors describe A3Test’s reported accuracy as a lower bound that may improve with hyperparameter optimization.They report the settings to support future replication studies.Source: §8 · p. 15
  • Experiments use five Defects4J projects, so further projects would help assess whether the findings generalize.The selected projects differ in input domains, size, and class complexity.Source: §8 · pp. 15–16
  • The comparison uses method coverage; the authors note that other metrics, including mutation score and fault detection, could also be reported.Source: §8 · p. 16

9 CONCLUSION

A3Test augments a pre-trained Transformer with assertion knowledge and verification of naming consistency and test signatures. On 5,278 Defects4J focal methods, it outperformed AthenaTest and other pre-trained models, with higher accuracy and faster generation.

9 CONCLUSION

  • A3Test generated 147% more correct test cases and achieved 15% more method coverage than AthenaTest while using fewer test cases.The evaluation analyzed 5,278 Defects4J focal methods.Source: §9 · p. 16
  • A3Test is a pre-trained Transformer augmented with assertion knowledge and verification of naming consistency and test signatures.Source: §9 · p. 16
  • A3Test outperformed existing pre-trained models and was 97.2% faster than AthenaTest while maintaining higher accuracy.The authors conclude that it generates more accurate test cases.Source: §9 · p. 16
Loading 2302.10352v1…