PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Deep Semantics for Test Completion

Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, Milos Gligoric

arXiv:2302.10166v3cs.SEcs.CLcs.LG

TL;DR

Manually writing tests is time-consuming, and test completion offers a way to suggest the next statement using the test context and code under test. TECO combines code semantics with deep learning and test-execution reranking; it outperforms syntax-only baselines on automatic metrics and generates more runnable statements than the best baseline.

  • Problem

    Manually writing tests is tiresome and time-consuming, motivating a way to suggest the next statement in a test method from its context and the code under test.

    Source: §I · p. 1

  • Method

    TECO uses code semantics as model inputs and reranks top predictions by executing generated statements, prioritizing compilable and runnable outputs.

    Source: §I · pp. 1–2, §III · p. 3

  • Results

    29% of TECO’s generated next statements were runnable, versus 18% for the best baseline; TECO also significantly outperformed prior state of the art on test-oracle exact-match accuracy, scoring 16 versus 9.

    Source: §I · p. 2

  • Takeaways & Limitations

    TECO and its corpus provide a starting point for machine learning for code using code semantics and execution data.

    Source: §X · p. 11

    The evaluation focused on JUnit-style tests, so its reported scope does not establish performance on other testing frameworks.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Writing tests is a time-consuming yet essential task during software development. We propose to leverage recent advances in deep learning for text and code generation to assist developers in writing tests. We formalize the novel task of test completion to automatically complete the next statement in a test method based on the context of prior statements and the code under test. We develop TeCo -- a deep learning model using code semantics for test completion. The key insight underlying TeCo is that predicting the next statement in a test method requires reasoning about code execution, which is hard to do with only syntax-level data that existing code completion models use. TeCo extracts and uses six kinds of code semantics data, including the execution result of prior statements and the execution context of the test method. To provide a testbed for this new task, as well as to evaluate TeCo, we collect a corpus of 130,934 test methods from 1,270 open-source Java projects. Our results show that TeCo achieves an exact-match accuracy of 18, which is 29% higher than the best baseline using syntax-level data only. When measuring functional correctness of generated next statement, TeCo can generate runnable code in 29% of the cases compared to 18% obtained by the best baseline. Moreover, TeCo is significantly better than prior work on test oracle generation.

I. INTRODUCTION

TECO frames next-statement test completion as a way to help developers write tests faster, using execution-related code semantics and test execution to guide predictions. On a corpus of Java tests, it outperforms syntax-only baselines on automatic metrics and produces more runnable statements than the best baseline.

I. INTRODUCTION

  • Test completion aims to help developers write tests faster by suggesting the next test-method statement, addressing the time required to write tests manually.Automated test-generation techniques can produce tests with stylistic issues that are hard to maintain, so they are often supplements to manually written tests.Source: §I · pp. 1–2
  • Unlike ordinary code completion, test completion can use the method under test and account for tests’ prepare, execute, and assert sequence.TECO is presented as a deep-learning solution designed for these test-specific characteristics.Source: §I · p. 1
  • TECO feeds execution-related code semantics to its model and reranks top predictions by test execution, prioritizing compilable and runnable statements.Its six semantic signals cover execution results and execution context, including local-variable types, field initialization, setup and teardown, prior calls, and similar statements.Source: §I · pp. 1–2
  • The authors collected and released a corpus of 130,934 Java test methods from 1,270 projects as a test-completion testbed.The paper also describes the corpus as potentially useful for other testing tasks.Source: §I · pp. 1–2
  • 29% of TECO’s generated next statements were runnable, versus 18% for the best baseline; TECO also significantly outperformed syntax-only baselines across automatic metrics and oracle generation.For test oracle generation, TECO achieved exact-match accuracy of 16 versus 9 for prior state of the art.Source: §I · p. 2

II. TASK

Test completion predicts the next statement from the method under test, test signature, and preceding statements. TECO supplements this context with six statically extracted execution-result and execution-context signals.

II. TASK

  • The task is to generate the next statement from the method under test, test signature, and zero or more prior statements.The signature is context rather than the prediction target, and other existing test methods are excluded to prevent copying.Source: §II · p. 2
  • TECO models test bodies as statement sequences and targets projects where developers already know what to test.Fewer than 10% of test methods in the experiments had control flow; the paper focuses on sequential bodies.Source: §II · p. 2

III. EXTRACTION OF CODE SEMANTICS

  • Static analysis avoids runtime configuration and is faster, but may overestimate behavior when values are unknown.For example, it may assume both branches of a conditional can execute; the model may learn to ignore inaccurate results.Source: §III · pp. 2–3
  • Static analysis first collects shared project code elements, then extracts six signals grouped as execution results and execution context.The result signals are local-variable types, absent types, and unset fields; context signals are setup/teardown, the last called method, and a similar statement.Source: §III · p. 3
  • Execution-result signals indicate available or missing inputs and fields that may need initialization in the next statement.Absent types are needed to call the method under test but are not prepared; unset fields are checked through calls up to four jumps deep.Source: §III · p. 3
  • Execution-context signals expose setup and teardown code, the last called method, and a non-test statement with similar prior context.Similar-statement search uses BM25 over two prior statements.Source: §III · p. 3

IV. TECO’S DEEP LEARNING MODEL

TECO uses an encoder-decoder transformer to generate the next test statement from syntax-level context and extracted code semantics. It combines the input pieces into a sequence, with ordering intended to protect important information from truncation.

IV. TECO’S DEEP LEARNING MODEL

  • TECO’s encoder-decoder transformer takes syntax-level data and extracted code semantics as input and generates the next statement.Syntax-level inputs include the method under test, test signature, and prior statements; semantic inputs are S1–S6.Source: §IV · p. 4, A. Encoder-Decoder Transformer Model · p. 4

A. Encoder-Decoder Transformer Model

  • The input includes the method under test, test signature, prior statements, and code semantics extracted from the test and code under test.Input pieces are subtokenized with byte-pair encoding and concatenated using separators.Source: A. Encoder-Decoder Transformer Model · p. 4
  • 6% of cases exceed the 512-subtoken limit, so TECO truncates from the beginning and places syntax-level data at the end.The input length averages 243.88 subtokens; the ordering of semantic data is selected using domain knowledge, and exploring all orderings is left for future work.Source: A. Encoder-Decoder Transformer Model · p. 4
  • The encoder represents the combined input, and the decoder generates the output one subtoken at a time.Source: A. Encoder-Decoder Transformer Model · p. 4

B. Fine-tuning

This section motivates incorporating execution semantics into fine-tuning and describes adapting a pre-trained CodeT5 model to test completion. TECO fine-tunes on test-completion examples using cross-entropy loss.

B. Fine-tuning

  • Pre-training uses syntax-level data for generality across downstream tasks, but prior work indicates pre-trained models may perform poorly on simple code-execution tasks.The authors infer that such models learned little about execution semantics.Source: B. Fine-tuning · p. 4
  • TECO fine-tunes with extracted code semantics so the model can use static-analysis results instead of inferring execution behavior from limited syntax-level context.Source: B. Fine-tuning · pp. 4–5
  • TECO fine-tunes the pre-trained CodeT5 encoder-decoder on a test-completion corpus by minimizing cross-entropy loss.Source: B. Fine-tuning · p. 5

C. Evaluation

At evaluation, TECO generates up to ten candidate statements and reranks them using compilability and execution. It tests candidates in an ad-hoc class that preserves the relevant test context and treats assertion failures as not runnable.

C. Evaluation

  • Beam search keeps the ten highest-probability sequences at each step and stops after ten completed outputs.A repetition-prevention mechanism penalizes generating the same subtoken as the immediately preceding one.Source: C. Evaluation · p. 5

D. Reranking by Execution

  • Reranking prioritizes runnable statements, then compilable statements, and uses model probability to order candidates with the same status.The candidates are first collected as the top ten beam-search predictions.Source: D. Reranking by Execution · p. 5
  • A candidate is tested in an ad-hoc class containing the test signature, prior statements, generated statement, and non-test methods from the original test class.The generated class is compiled with project dependencies and non-test classes; successful compilation is followed by execution.Source: D. Reranking by Execution · p. 5
  • A statement counts as runnable only if execution ends without an exception or assertion failure.JUnit 4 projects use the JUnit runner when available; otherwise, or if the runner fails, the procedure uses an ad-hoc main method.Source: D. Reranking by Execution · p. 5

V. CORPUS

The corpus provides a testbed for Java test completion, assembled from open-source projects and filtered to support analysis of test methods and their associated methods under test. Its scope is bounded by project, control-flow, and string-literal constraints.

V. CORPUS

  • Corpus statistics report project, test-method, and statement counts, plus average token lengths for test methods and methods under test.The reported 130,934 methods include 101,965 deemed runnable under the paper’s procedure; masking can make some originally passing tests count as not runnable.Source: Table I · p. 6, §V · p. 6
  • The collection targets JUnit 4 and JUnit 5 tests, then filters methods by naming and signature and uses a procedure to locate each method under test.Methods without a located method under test were removed; additional size and control-flow filters excluded methods with if statements, loops, or try blocks.Source: §V · p. 6
  • String literals are replaced with a common token, so the corpus supports next-statement completion with masked strings rather than prediction of their contents.The paper leaves generating string-literal contents to future work.Source: §V · p. 6
  • The resulting corpus contains 130,934 test methods and 645,633 statements from 1,270 projects, split at the project level for training, validation, and evaluation.The source projects were selected from CodeSearchNet subject projects based on Maven use, successful compilation, and data-use licensing.Source: §V · pp. 5–6

VI. EXPERIMENTS SETUP

The experiments assess TECO and baseline models on test completion and related evaluations. They measure prediction quality both intrinsically and extrinsically, including runnable statements and test oracle generation.

VI. EXPERIMENTS SETUP

  • The experiments ask how TECO compares with baselines on test completion and how often it predicts a compilable and runnable next statement on the runnable subset.These are the first two research questions.Source: §VI · p. 6
  • The evaluation also asks whether TECO compares favorably with prior work on test oracle generation, a downstream application of test completion.This is the third research question.Source: §VI · p. 6
  • Two further questions examine execution-based reranking and the individual and complementary contributions of the six kinds of code semantics.These are the fourth and fifth research questions.Source: §VI · p. 6
  • Models are trained on training and validation data, then evaluated on test methods or relevant subsets using intrinsic and extrinsic metrics.Experiments are repeated with three random seeds and report averages; model comparisons use bootstrap significance tests at a 95% confidence level.Source: §VI · p. 7

A. TECO Models

The setup compares TECO variants with syntax-only baselines and defines evaluation subsets and metrics for test completion, functional correctness, and test oracle generation.

A. TECO Models

  • TECO uses all six code-semantics kinds with execution reranking; variants omit reranking or use one kind of semantics at a time.The no-reranking model isolates reranking, while six single-semantics variants examine individual semantics types.Source: A. TECO Models · p. 7

B. Baseline Models

  • Syntax-only baselines include fine-tuned CodeT5 and CodeGPT; test oracle generation also uses ATLAS and TOGA, trained on first-assertion data.CodeT5-noFt is included without fine-tuning, and baseline models use author-recommended default configurations.Source: B. Baseline Models · p. 7

C. Subsets of the Evaluation Set

  • The runnable subset contains 25,074 statements whose gold statements are runnable, while the oracle subset contains 4,212 first assertions and its runnable subset contains 3,540.Gold statements may be unexecutable because of runtime-environment setup difficulties, including missing resources or non-Java environments.Source: C. Subsets of the Evaluation Set · p. 7

D. Evaluation Metrics

  • Lexical metrics include exact match, top-10 accuracy, BLEU, CodeBLEU, edit similarity, and ROUGE; %Compile and %Run assess executable predictions.%Run requires a prediction to compile and run without assertion failures or runtime errors, but both functional metrics can overestimate correctness because they do not assess whether code logic is meaningful.Source: D. Evaluation Metrics · pp. 7–8

VII. RESULTS

TECO outperforms syntax-only baselines on test completion, including exact-match accuracy. On the runnable subset, it also produces more compilable and runnable statements than the best baseline.

VII. RESULTS

  • 17.61 exact-match accuracy is 29% higher than CodeT5’s 13.57, and TECO significantly outperforms all baselines on every automatic test-completion metric.The authors attribute the improvement to using code semantics and reranking by execution.Source: §VII · p. 9
  • CodeT5-noFt cannot solve test completion, while CodeGPT performs slightly worse than CodeT5, which the authors relate to test statements’ different style from the method-under-test context.On the runnable subset, neither CodeT5-noFt nor CodeGPT produces compilable or runnable statements; their outputs are broken non-code or invalid Java statements, respectively.Source: §VII · p. 9, B. RQ2: Functional Correctness · p. 9

B. RQ2: Functional Correctness

  • 28.63% of TECO’s predictions are runnable and 76.22% are compilable, compared with the best baseline’s 17.62% and 54.84%, respectively.On the runnable subset, TECO also outperforms all baselines on the other lexical-similarity metrics.Source: B. RQ2: Functional Correctness · p. 9

C. RQ3: Performance on Test Oracle Generation

TECO improves test-oracle generation over prior models, and execution-based reranking contributes to that performance. Results with individual semantic inputs suggest that different kinds of code semantics provide complementary information, while the paper identifies several scope boundaries and future directions.

C. RQ3: Performance on Test Oracle Generation

  • 82% higher exact-match accuracy: TECO scores 16.44 versus TOGA’s 9.01 on test oracle generation.TOGA is strongest among prior models on exact match, but performs worse than CodeT5 on partial-match metrics because its heuristic candidate set may omit the gold statement.Source: C. RQ3: Performance on Test Oracle… · pp. 9–10

D. RQ4: Improvements from Reranking by Execution

  • Reranking by execution adds 2 points in exact-match accuracy and improves oracle-generation exact match by 6–8 points and %Run by 12 points.Execution can expose assertion logic errors, such as an incorrect expected value that causes an assertion to fail.Source: D. RQ4: Improvements from Reranking by… · p. 10

E. RQ5: Comparisons of Code Semantics

  • Single-semantic-input models each beat CodeT5 on at least one metric, with different semantic inputs leading on different metrics and evaluation sets.Setup/teardown and absent types lead on selected full-set metrics, while unset fields and similar statements lead on selected oracle-subset metrics.Source: E. RQ5: Comparisons of Code Semantics · p. 10, Table IV · p. 10

VIII. LIMITATIONS AND FUTURE WORK

  • The work leaves structured code representations and code semantics for large language models to future work, assumes code precedes tests rather than TDD, and focuses on JUnit-style tests.The authors note that structured representations could improve performance and enable quick validity checks; other frameworks such as TestNG are outside the stated focus.Source: §VIII · p. 10

IX. RELATED WORK

Prior test-generation approaches target coverage or test oracles, while TECO completes one test statement at a time as developers write tests. The paper positions code execution as relevant to test completion and concludes that TECO outperforms prior models on test completion and oracle generation.

IX. RELATED WORK

  • Automated test generation typically targets code coverage, but generated tests are often not added to repositories because of low quality and excessive volume.TECO instead targets developer productivity while writing manual tests.Source: §IX · p. 11
  • Prior oracle work targets selected behaviors or assertion generation, whereas TECO completes any part of a test, including oracles.TECO is also compared with deep-learning oracle generators ATLAS and TOGA as a downstream application of test completion.Source: §IX · p. 11
  • Unlike whole-method test generation [67] or test recommendation from existing suites –, TECO completes one statement at a time without requiring existing relevant tests.This lets developers observe and control test-method writing as it proceeds.Source: §IX · p. 11

X. CONCLUSION

  • TECO applies code semantics and test-execution reranking to test completion, and the conclusion reports improvement over state-of-the-art models on code completion and oracle generation.The paper frames execution reasoning as needed for test completion and describes TECO as a starting point for machine learning with code semantics and execution data.Source: §IX · p. 11, §X · p. 11

APPENDIX A COMPARISONS WITH LARGE LANGUAGE MODELS

The appendix evaluates zero-shot Codex for test completion against TECO and other baselines. TECO statistically significantly outperforms Codex on all metrics, although Codex exceeds the smaller baselines on some measures.

APPENDIX A COMPARISONS WITH LARGE LANGUAGE MODELS

  • Codex is evaluated zero-shot with a prompt containing the method under test, test signature, and prior statements, generating only the top-1 next statement.The evaluation used 30,194 statements and took 18 hours.Source: Appendix A · p. 15
  • Codex performs better on test oracle generation than on test completion; the authors suggest the additional prior-statement context may explain the difference.The comparisons cover the full evaluation set, runnable subset, oracle subset, and oracle-runnable subset.Source: Appendix A · pp. 15–16
  • TECO statistically significantly outperforms Codex on all test-completion metrics.Codex performs better than CodeT5 or CodeGPT on some measures and slightly worse on others; the authors hypothesize fine-tuning on the test-completion corpus helped the smaller models.Source: Appendix A · p. 15, Table V · p. 16
Loading 2302.10166v3…