PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers

Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Neel Sundaresan

arXiv:2009.05634v1cs.SEcs.LG

TL;DR

Generating accurate assert statements is a challenge for automated unit testing, so this paper trains a pretrained transformer to produce them from a partial test and its focal method. It reports 62% exact top-1 accuracy, an 80% relative improvement over the previous RNN approach, and improved coverage when asserts augment existing tests.

  • Problem

    Generating accurate assert statements remains a challenge because asserts in tests can be incomplete or inadequate to check component behavior.

    Source: §I · p. 1

  • Method

    The paper semi-supervisedly pretrains a Transformer on English and source-code corpora, then finetunes it to translate a partial test and focal method into an assert.

    Source: D. Asserts Finetuning · p. 2, §VIII · p. 9

  • Results

    62% top-1 accuracy matching the developer-written assert represents an 80% relative improvement over the previous RNN-based approach [5].

    Source: §IV · p. 5, §VIII · p. 9

  • Takeaways & Limitations

    The approach can augment existing tests, including EvoSuite tests, with generated asserts that lead to improved test coverage.

    Source: §VIII · p. 9

    The comparison is between a 400M-parameter Transformer and a 4M-parameter RNN, so equal-sized models could yield different results.

    Source: §VI · p. 9

Abstract

from arXiv · show

Unit testing represents the foundational basis of the software testing pyramid, beneath integration and end-to-end testing. Automated software testing researchers have proposed a variety of techniques to assist developers in this time-consuming task. In this paper we present an approach to support developers in writing unit test cases by generating accurate and useful assert statements. Our approach is based on a state-of-the-art transformer model initially pretrained on an English textual corpus. This semantically rich model is then trained in a semi-supervised fashion on a large corpus of source code. Finally, we finetune this model on the task of generating assert statements for unit tests. The resulting model is able to generate accurate assert statements for a given method under test. In our empirical evaluation, the model was able to predict the exact assert statements written by developers in 62% of the cases in the first attempt. The results show 80% relative improvement for top-1 accuracy over the previous RNN-based approach in the literature. We also show the substantial impact of the pretraining process on the performances of our model, as well as comparing it with assert auto-completion task. Finally, we demonstrate how our approach can be used to augment EvoSuite test cases, with additional asserts leading to improved test coverage.

I. INTRODUCTION

The introduction frames accurate assert generation as a key challenge in automated unit testing and presents a pretrained transformer approach to address it. The paper reports gains over ATLAS and test-coverage improvements when generated asserts augment EvoSuite tests.

I. INTRODUCTION

  • Assert statements are a central challenge because they can be incomplete or inadequate for testing a software component’s behavior.They let developers check program conditions or states and reason about correctness.Source: §I · p. 1
  • The transformer-based approach predicts the correct assert in 62% of cases on the very first attempt, an 80% relative improvement in top-1 accuracy over previous work [5].Allowing the model to suggest more asserts yields up to 84% correctness.Source: §I · p. 1
  • A pretrained transformer generates asserts from English and source-code pretraining, then is finetuned on the assert-generation task.The approach builds on transfer learning, which pretrains on unlabeled data before downstream-task finetuning.Source: §I · p. 1
  • Pretraining on English and source-code corpora improves performance across BLEU, validation loss, and syntactic-correctness metrics.The paper also evaluates the models on intrinsic and extrinsic measures.Source: §I · p. 1
  • Adding generated asserts to existing tests, including EvoSuite tests, is reported to improve test coverage.The paper presents this as support for automated test-case generation tools.Source: §I · pp. 1–2

II. APPROACH

The approach starts from BART, applies English-text pretraining, and then finetunes models for assert generation. The pretraining objectives reconstruct corrupted text, while the English corpus supplies natural-language material.

II. APPROACH

  • The pipeline starts with BART, applies English and source-code pretraining, and finetunes the models for assert generation.The paper evaluates four variants representing different levels of pretraining.Source: §II · p. 2

A. BART Transformer

  • BART is a denoising autoencoder built on a sequence-to-sequence Transformer architecture, selected to support downstream translation finetuning.Its noising transformations include token masking, deletion, infilling, and sentence permutation.Source: A. BART Transformer · p. 2

B. English Pretraining

  • English pretraining aims to teach semantic and statistical properties of natural language using 160GB of text from books, Wikipedia, and news articles.The corpus is used for 40 epochs.Source: B. English Pretraining · p. 2, 1) Dataset · p. 2

2) Training Strategy:

  • The English-pretraining objective reconstructs text after masking 30% of tokens with Poisson-distributed spans and permuting sentences.The span-length distribution uses λ = 3.Source: 2) Training Strategy · p. 2

C. Code Pretraining

The code-pretraining stage uses Java repositories to learn source-code syntax and properties. Assert finetuning treats each example as a translation pair: a partial test and its focal method form the input, and the developer’s assert is the target.

C. Code Pretraining

  • Java code pretraining aims to teach the model source-code syntax and properties.The corpus contains 25GB from 26,000 public, non-fork GitHub repositories with at least 50 stars, after filtering and file-level deduplication.Source: C. Code Pretraining · p. 2, 1) Dataset · p. 2

2) Training Strategy:

  • Code pretraining corrupts files by independently deleting 20% of tokens and rotating half of the documents, and runs for 10 epochs.Source: 2) Training Strategy · p. 2

D. Asserts Finetuning

  • Finetuning represents assert generation as translation from a partial unit test plus its method under test to the developer-written assert.Source: D. Asserts Finetuning · p. 2

1) Dataset:

  • The finetuning dataset comes from JUnit tests mined across more than 9,000 open-source GitHub projects, selecting test methods with a single assert and identifying their focal methods heuristically.The model uses raw source-code tokens to retain variable and method-name semantics, with an 80/10/10 training, validation, and test split.Source: 1) Dataset · p. 3
  • A Test-Assert Pair combines a test method with its assert removed, the focal method it tests, and the original assert as target.The source concatenates the partial test and focal method; the target is the assert statement.Source: 1) Dataset · p. 3

2) Training Strategy:

The finetuning strategy models the assert as a conditional output from the partial test and focal method. The paper compares four variants that differ in whether they receive English and/or code pretraining.

2) Training Strategy:

  • Training learns the conditional probability of the assert given the partial test and focal method, using cross-entropy loss and Adam with validation-loss early stopping.Encoder and decoder share vocabulary embeddings because both input and output use Java source code.Source: 2) Training Strategy · p. 3

E. Model Variants

  • The four variants are scratch, English-pretrained, code-pretrained, and English-then-code-pretrained models, each finetuned for assert generation.Source: E. Model Variants · p. 3
  • The Test-Assert Pair example depicts the source as the partial test without its assert concatenated with the focal method, and the target as the assert to generate.Source: Figure 2 · p. 3

III. EXPERIMENTAL DESIGN

The study compares assert generation against ATLAS, tests pretraining and focal-method choices, and examines prediction quality and EvoSuite augmentation. Its empirical setup includes a focused Defects4J testbed for the augmentation investigation.

III. EXPERIMENTAL DESIGN

  • Models were compared with ATLAS using the same training and test sets and top-k accuracy for k = 1–50, with beam size capped at 50.Source: §III · pp. 3–4
  • Four BART variants—scratch, English-pretrained, code-pretrained, and English-plus-code-pretrained—were evaluated using top-k accuracy, validation loss, BLEU4, and syntactic correctness.Source: §III · p. 4
  • The focal-method investigation compared the best model with an otherwise similar model trained without the focal method, framing the latter as assert auto-completion from a partial test method.Source: §III · p. 4
  • Generated predictions were manually examined through correct examples and mismatches to assess assert quality.Source: §III · p. 4
  • The EvoSuite investigation inserted generated asserts into automatically generated tests to assess test-coverage changes, using the fixed Lang-1-f version of Apache Commons Lang from Defects4J.Source: §III · p. 4
  • EvoSuite generated tests for 500 seconds; the researchers selected one test for each of 18 unique focal methods, chose one of the top 10 assert predictions, appended it, and recomputed coverage.Source: §III · p. 4

IV. EXPERIMENTAL RESULTS

The results show that the BART models outperform ATLAS across tested k values. BART_English+Code reaches 62% top-1 accuracy, an 80% relative improvement over ATLAS, supporting the prospect of useful recommendations in few suggestions.

IV. EXPERIMENTAL RESULTS

  • Figure 3 plots top-k accuracy against k from 1 to 50; ATLAS is shown at five-step intervals, while BART is plotted at each integer k.Source: §IV · p. 4, Figure 3 · p. 5
  • 62% top-1 accuracy for BART_English+Code is an 80% relative improvement over ATLAS's 27% top-1 accuracy; BART models outperform ATLAS at every k.Source: Figure 3 · p. 5, §IV · p. 5
  • The authors frame the top-1 result as useful for development because developers can receive accurate recommendations in one or a few suggestions rather than sift through many incorrect ones.Source: §IV · p. 5

Extrinsic Metrics:

Pretraining substantially improves assert-generation performance over training from scratch, while adding code pretraining to English pretraining gives smaller but consistent gains. The authors note those gains could be larger on bigger test sets.

Extrinsic Metrics:

  • 22–25% separates BART_Scratch from singly pretrained BART_Code or BART_English, while BART_English+Code leads BART_English by 1.54–2.25%.Source: Extrinsic Metrics · p. 5, Table II · p. 5
  • English pretraining's impact on this source-code task highlights the value of models that understand variable and method-name semantics.Source: Extrinsic Metrics · p. 5
  • The added code-pretraining impact is limited relative to English-only pretraining, though improvements are consistent and could be larger on bigger test sets.Source: Extrinsic Metrics · p. 5

Intrinsic Metrics:

English-plus-code pretraining leads on BLEU4 and validation loss, while Focal Method input improves top-k accuracy. Generated asserts also improve coverage in many EvoSuite test cases, and examples include equivalent and complex predictions.

Intrinsic Metrics:

  • 0.13 versus 0.15 best validation loss and convergence about 2,500 steps earlier distinguish English+Code from English-only pretraining.English+Code also starts at loss 0.21 versus 0.31; the figure tracks validation cross-entropy during training.Source: Intrinsic Metrics · p. 5, Figure 4 · p. 6
  • BART_English+Code obtains the best BLEU4 and validation loss; English+Code also has the best top-1 syntactic correctness, while BART_Code leads at top 25 and 50.The table reports intrinsic metrics for the four model variations.Source: Intrinsic Metrics · p. 5, Table III · p. 6
  • ∼10% more cases are accurately predicted when the Focal Method is included than with the otherwise comparable model without it.The comparison uses top-k accuracy, and some asserts remain uncovered without the Focal Method even across 50 predictions.Source: Intrinsic Metrics · p. 6, Figure 5 · p. 6
  • Simple accuracy misses equivalent valid asserts and plausible asserts where developers wrote none, so these cases require qualitative manual analysis.Equivalent statements are not counted as correct in top-k accuracy.Source: Intrinsic Metrics · p. 7
  • Examples include common and complex asserts, equivalent-but-not-identical checks, and two complete predictions that exactly match the targets.The complex examples involve multiple calls, parameters, attributes, and less common variables.Source: Intrinsic Metrics · pp. 6–7, Figure 7 · p. 7
  • 13 of 18 methods gained line and/or condition coverage, with improvements of 1–3 lines and 1–4 conditions; four correct asserts added no coverage.The experiment reports coverage for the original and augmented EvoSuite tests.Source: Intrinsic Metrics · pp. 7–8, Table IV · p. 8
  • Generated numeric inputs opened different branches from EvoSuite’s null or empty-string tests; the `foo` input added three lines and one condition.The `foo` example concerns the toDouble focal method.Source: Intrinsic Metrics · p. 7
  • Among four correct asserts that added no coverage, three checked EvoSuite return variables without invoking the focal method, while another repeated already-covered behavior.The repeated min call used different values but covered lines and branches already exercised by the original test.Source: Intrinsic Metrics · p. 7

V. DISCUSSION & FUTURE WORK

The authors envision two future deployment scenarios for improving automation in software testing.

V. DISCUSSION & FUTURE WORK

  • The authors identify two possible future deployment scenarios for the model, both aimed at improving automation in software testing.The passage does not specify the scenarios.Source: §V · p. 8

A. Supporting Developers in writing Test Cases

The approach could support developers with assert suggestions during test writing and augment automated test cases with generated assertions. The authors also describe threats involving pretraining overlap, hyperparameter choices, and unequal model sizes.

A. Supporting Developers in writing Test Cases

  • An IDE plugin could suggest asserts while developers write tests, complementing standard code completion.The authors report their approach as more accurate than standard code completion and suggest a hybrid approach.Source: A. Supporting Developers in writing… · p. 8

B. Improving Automated Test Case Generation Tools

VI. THREATS TO VALIDITY

  • Source-code pretraining could expose the model to code similar to finetuning tests; dissimilar organization mitigated this risk, and pretrained-only evaluation found no correct asserts.Within the finetuning dataset, the authors report avoiding data leakage across training, validation, and test sets.Source: §VI · pp. 8–9
  • The authors did not search hyperparameters because training the large models required substantial time, relying instead on literature-suggested configurations.They experimented with different pretraining stages.Source: §VI · p. 9
  • The comparison pits a 400M-parameter BART model against a 4M-parameter RNN, so model capacity differs alongside architecture.The authors note that equal-size comparisons could yield different results; they also report that ATLAS authors found no gains from increasing encoder-decoder layers and units.Source: §VI · p. 9

VII. RELATED WORK

Earlier automated test-generation tools use heuristics, contracts, or mutation-based strategies, while this work learns assert statements from developers’ code. The conclusion reports exact-match performance, benefits from pretraining, and improved coverage when generated asserts augment tests.

VII. RELATED WORK

  • EvoSuite, Randoop, and Agitar use mutation-based strategies, contracts, or other heuristics, whereas this approach learns context-specific asserts from developers’ code.Source: §VII · p. 9
  • Prior work found nearly half of faults undetected by EvoSuite and Randoop could have been detected with a more appropriate assert, [5].Source: §VII · p. 9
  • Against ATLAS [5], the work combines a Transformer architecture, English and source-code pretraining, focal-method context, and qualitative and intrinsic analyses.The authors report an 80% relative improvement in top-1 accuracy over the previous work [5].Source: §VII · p. 9

VIII. CONCLUSION

  • 62% top-1 accuracy matched the developer-written assert exactly, an 80% relative improvement over the previous RNN-based approach [5].Source: §VIII · p. 9
  • Adding generated asserts to existing tests, including EvoSuite tests, led to test coverage improvements.Source: §VIII · p. 9
  • Pretraining on English and source code benefited intrinsic and extrinsic metrics; some syntactically different predictions were nevertheless semantically equivalent and correct.Source: §VIII · p. 9
  • The authors suggest developers could receive accurate, relevant assert suggestions in one or a few attempts to help write complete, robust tests.Source: §VIII · p. 9
Loading 2009.05634v1…