PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Adaptation for Code Model-based Unit Test Case Generation

Jiho Shin, Sepehr Hashtroudi, Hadi Hemmati, Song Wang

arXiv:2308.08033v3cs.SEcs.AI

TL;DR

Code-model test generation faces robustness, evaluation, and project-domain-shift challenges. The paper fine-tunes CodeT5 for test generation and adapts it using project-specific data. The adapted tests improve coverage and mutation score against three baselines, and can complement EvoSuite.

  • Problem

    Code-model test generation faces robustness challenges, time-consuming adequacy evaluation, and domain shift when models are applied to new projects.

    Source: §1 · p. 1

  • Method

    The approach fine-tunes CodeT5 for test generation, then applies project-level domain adaptation using project-specific data.

    Source: §1 · p. 2

  • Results

    Line coverage improves by 18.62%, 19.88%, and 18.02%, and mutation score by 16.45%, 16.01%, and 12.99%, versus CodeT5 without DA, A3Test, and GPT-4, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Alongside EvoSuite, the approach increases overall line coverage and mutation score by averages of 34.42% and 6.8%, respectively.

    Source: §1 · p. 2

    The approach may depend on developer-written tests and is intended to extend existing tests; automatically generated starting tests could hinder targeted benefits such as fault detection and readability.

    Source: §6 · p. 11

Abstract

from arXiv · show

Recently, deep learning-based test case generation approaches have been proposed to automate the generation of unit test cases. In this study, we leverage Transformer-based code models to generate unit tests with the help of Domain Adaptation (DA) at a project level. Specifically, we use CodeT5, a relatively small language model trained on source code data, and fine-tune it on the test generation task. Then, we apply domain adaptation to each target project data to learn project-specific knowledge (project-level DA). We use the Methods2test dataset to fine-tune CodeT5 for the test generation task and the Defects4j dataset for project-level domain adaptation and evaluation. We compare our approach with (a) CodeT5 fine-tuned on the test generation without DA, (b) the A3Test tool, and (c) GPT-4 on five projects from the Defects4j dataset. The results show that tests generated using DA can increase the line coverage by 18.62%, 19.88%, and 18.02% and mutation score by 16.45%, 16.01%, and 12.99% compared to the above (a), (b), and (c) baselines, respectively. The overall results show consistent improvements in metrics such as parse rate, compile rate, BLEU, and CodeBLEU. In addition, we show that our approach can be seen as a complementary solution alongside existing search-based test generation tools such as EvoSuite, to increase the overall coverage and mutation scores with an average of 34.42% and 6.8%, for line coverage and mutation score, respectively.

1 INTRODUCTION

Unit-test generation with code models faces robustness, evaluation, and project-domain-shift challenges. The paper combines task-level fine-tuning and project-level adaptation, reporting higher test adequacy than several baselines and gains when used alongside EvoSuite.

1 INTRODUCTION

  • Unit-test generation is challenging because slight generation errors can break tests, adequacy evaluation requires test execution, and project code distributions cause domain shift.Neural tests can be more readable and maintainable than automatically generated alternatives and target faults differently from search-based tests.Source: §1 · p. 1
  • The approach fine-tunes CodeT5 on test generation, then adapts it with project-specific data to address domain shift and evaluates adequacy as well as textual similarity.It uses Methods2test for task fine-tuning and Defects4j for project-level adaptation and evaluation.Source: §1 · p. 2
  • Line coverage improves by 18.62%, 19.88%, and 18.02%, and mutation score by 16.45%, 16.01%, and 12.99%, versus CodeT5 without DA, A3Test, and GPT-4, respectively.The approach also increases EvoSuite's overall line coverage and mutation score by averages of 34.42% and 6.8%, respectively.Source: §1 · p. 2
  • The framework generates line-level tests that are compilable, similar to human-written tests, and test-adequate, and the study compares it on the Defects4j benchmark.The approach's code, experiment scripts, and raw data are publicly available for replication.Source: §1 · p. 2
  • Project-adapted tests cover lines missed by developer-written tests and a baseline search-based tool, and kill new mutants compared with search-based tools.The paper executes generated tests to measure coverage and mutation score alongside BLEU and CodeBLEU.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

This section reviews search-based testing, domain adaptation, and neural test-generation work. It distinguishes whole-test generation from oracle generation and notes the need to evaluate generated tests by testing adequacy, not only textual similarity.

2.1 Search-based Software Testing

  • Search-based software testing frames test generation as optimizing a test-adequacy criterion, such as code coverage.EvoSuite uses a genetic algorithm to evolve suites toward greater coverage with fewer tests, while prior work reports readability and bug-detection limitations for SBST tools.Source: §2.1 · p. 2

2.2 Domain Adaption

  • Domain adaptation transfers knowledge from a source domain to a related target domain, often where labeled target data are limited.Prior work includes project-level transfer for defect prediction and program repair, along with surveys that categorize adaptation methods.Source: §2.2 · p. 2

2.3 Neural Models for Unit Test Generation

  • Neural test-generation work covers both oracle generation and whole-unit-test generation; an oracle is only one part of a complete test.Oracle generation uses focal context and a test prefix to generate assertions or other oracle statements.Source: §2.3 · p. 3, §2.3.1 · p. 3

2.3.1 Test Oracle Generation.

  • Oracle-generation approaches produce assertions or try-catch clauses, while generating the test prefix remains a separate, non-trivial task.ATLAS [42] was the first deep-neural approach for assertion generation; TOGA [8] generates try-catch clauses and assertions.Source: §2.3.1 · p. 3

2.3.2 Unit Test Case Generation.

  • Whole-test generation studies include AthenaTest [39] and A3Test [4], while the reviewed work extends project-level adaptation and evaluates coverage and mutation adequacy.The related-work discussion says many existing approaches report BLEU-like metrics that do not necessarily reflect test adequacy.Source: §2.3.2 · p. 3

3 TEST CASE GENERATION WITH PROJECT LEVEL DOMAIN ADAPTATION

The framework first fine-tunes CodeT5 for test generation, then adapts it to a target project using developer-written tests. Its pipeline maps covered lines to tests, constructs contextual inputs, and filters generated tests for parsing and compilation.

3 TEST CASE GENERATION WITH PROJECT LEVEL DOMAIN ADAPTATION

  • The framework has two stages: task-level CodeT5 fine-tuning followed by project-level domain adaptation using developer-written tests.The stated aim is to generate test cases with efficacy learned from developer-written tests.Source: §3 · p. 3

3.1 Fine-tuning on Test Case Generation Task

  • The project setup assumes an initial test suite; line-level coverage records which tests cover which source lines and supports a line-to-test mapping.The coverage database could be extended to other code metrics or mutation scores.Source: §3.1 · p. 3
  • CodeT5 is fine-tuned for conditional code generation using Methods2Test tuples of a focal method, its context, and an associated unit test.The dataset mapping connects the source method and context to a test that covers it.Source: §3.1 · pp. 3–4
  • Post-processing replaces end-of-line tokens, removes duplicate test names, repairs some parse errors, and checks generated tests for parsing and compilation.Tests are added to their corresponding project test classes and compiled individually; tests that do not compile are excluded.Source: §3.1 · p. 4

3.2 Domain Adaptation on Project Specific Knowledge

  • Project-adaptation inputs combine a target line with its focal method and context, including method names, method bodies, line numbers, and class information.When multiple tests cover a line, the mapping selects one test, preferring a test from the same class when available.Source: §3.2 · p. 4
  • The model adapts by training for a few epochs on the project dataset, a shorter process than full training; per-project adaptation has a one-time cost.The authors note that SBST avoids this training time, while inference is faster than classic test-generation approaches.Source: §3.2 · p. 5

4 EXPERIMENT SETTINGS

The experiments examine CodeT5 test generation without domain adaptation, project-level adaptation, and whether the adapted approach can augment EvoSuite.

4.1.1 RQ1 (Performance).

  • RQ1 evaluates CodeT5 test generation without domain adaptation, addressing limited prior test-generation studies and their reliance on static BLEU evaluation.Because generated tests may not compile or be effective when executed, the authors replicate prior work with CodeT5 and assess test adequacy; AthenaTest’s model was unavailable.Source: §4.1.1 · p. 5

4.1.2 RQ2 (Effectiveness).

  • RQ2 tests whether project-level domain adaptation improves CodeT5-generated tests and compares it with GPT-4 and A3Test.The authors motivate adaptation by poor RQ1 results and the difficulty of transferring models to new projects with project-specific patterns.Source: §4.1.2 · p. 5

4.1.3 RQ3 (Impact).

  • RQ3 asks whether the proposed test-generation approach can augment search-based test generation, using EvoSuite as its comparator.The authors frame the methods as complementary: EvoSuite can first establish coverage, after which the model can generate additional tests, including after small commits; model inference is contrasted with search-based generation’s training overhead.Source: §4.1.3 · p. 5

4.2 Datasets

Methods2Test fine-tunes the test-generation model, while Defects4J supplies projects for project-level adaptation and evaluation.

4.2 Datasets

  • Methods2Test provides 780,944 instances from 9,410 unique repositories, split into 80% training, 10% validation, and 10% test data.Its Java methods are mapped to tests using a naming-convention heuristic that searches for method names in test cases.Source: §4.2 · p. 6
  • Methods2Test supplies each method’s focal method, focal class name, constructor and public-method signatures, and public fields as model context.The dataset authors report that this combination of context produced the best results; the study uses all provided per-method data for fine-tuning.Source: §4.2 · p. 6
  • Defects4J contains reproducible bugs and supporting infrastructure; the study uses five of its 17 projects for project-level adaptation and evaluation.The initial version contained 357 real bugs from five projects, each with test suites able to expose bugs in project versions.Source: §4.2 · p. 6

4.3 Pre-Trained Code Model

The study selects CodeT5 for project-level fine-tuning and uses its base configuration with the CONCODE task for test generation.

4.3 Pre-Trained Code Model

  • CodeT5 was selected as a model small enough for cost-effective fine-tuning and domain adaptation, while the authors report it beat larger models including GPT-4.The authors also cite its T5 encoder-decoder design for tasks requiring both understanding and generation, and its use of code token-type information.Source: §4.3 · p. 6
  • Initial experiments favored CONCODE over code translation for test generation, so the study fine-tunes CodeT5-base on CONCODE.The authors note that code translation maps code into another programming language, whereas test-generation input and output do not share the same structure and semantics.Source: §4.3 · p. 6
  • CodeT5-small has 60M parameters and CodeT5-base has 220M, with maximum source and target lengths of 512 and 256 tokens.The study uses CodeT5-base with default hyperparameters except for a batch size of four, limited by GPU memory.Source: §4.3 · p. 6

4.4 Baselines

The study compares its approach with GPT-4 and A3Test, selected to represent a large language model and a test-generation method using task-level domain adaptation.

4.4 Baselines

  • GPT-4 serves as a baseline representing a highly advanced large language model.The passage describes it as trained on a vast library of human communication and capable of complex problem solving and reasoning.Source: §4.4 · p. 6
  • A3Test generates unit tests by transferring knowledge from test-oracle generation, its task-level domain-adaptation task.The study uses it as a test-generation baseline.Source: §4.4 · p. 6
  • The baselines are intended to compare project-level adaptation with a state-of-the-art LLM and a test-generation method using task-level adaptation..Source: §4.4 · p. 6

4.5 Evaluation Metrics

The evaluation uses execution-based test adequacy metrics alongside syntax, compilation, and textual-similarity measures. Coverage and mutation scores assess test efficacy, while BLEU and CodeBLEU provide similarity measures.

4.5 Evaluation Metrics

  • Seven criteria cover parse rate, execution rate, line-level coverage, mutation and adapted mutation scores, BLEU, and CodeBLEU.The authors distinguish test adequacy measures from textual-similarity measures.Source: §4.5 · p. 7, §4.5.2 · p. 7

4.5.1 Parse and Compile Rate.

  • Generated tests are assessed for syntax correctness with Tree-sitter and for compilability by compiling them within the project.The parse rate is the ratio of parsable tests; the compile rate is measured among those parsable tests.Source: §4.5.1 · p. 7

4.5.3 BLEU and CodeBLEU.

  • BLEU and CodeBLEU measure similarity to the ground-truth test, but may penalize differing identifiers even when generated tests are functional.CodeBLEU adds weighted token, AST-syntax, and data-flow semantic matching; BLEU was used to select the best model because execution-based adequacy metrics were costly to integrate into training.Source: §4.5.2 · p. 7, §4.5.3 · p. 7

4.5.2 Line-Coverage and Mutation Score.

  • Line coverage is calculated from compilable tests run in the project, while mutation score counts killed mutants among total mutants.For RQ1 and RQ2, coverage excludes test-project lines; RQ3 uses Clover to map covered lines, which was compatible with 5 of 17 Defects4j projects. Adapted mutation score divides killed mutants by covered mutants.Source: §4.5.2 · p. 7

4.6 Configurations and Environment Setup

The experiments use CodeT5-base with its CONCODE configuration, fine-tune on Methods2Test, and perform project-level adaptation on Defects4j.

4.6 Configurations and Environment Setup

  • The study uses CodeT5-base with the CONCODE configuration, fine-tuned on Methods2Test and evaluated on Defects4j projects and EvoSuite-generated tests.The CONCODE configuration’s default model hyperparameters were used.Source: §4.6 · p. 7
  • Project-level domain adaptation fine-tunes all model layers for 20 epochs, with batch size 4, the maximum fitting the experimental GPU memory.The batch size was 4.Source: §4.6 · p. 7
  • Experiments ran on one node with four 32GB V100 GPUs, 10 CPU cores, and 80GB RAM.The authors report that, with minor changes to batch size and training time, experiments can run with 16GB GPU, one CPU core, and 10GB RAM.Source: §4.6 · p. 7

5 RESULTS AND ANALYSIS

Across five Defects4j projects, project-specific domain adaptation improves CodeT5’s evaluation metrics over the no-adaptation model and other baselines. The model also adds coverage beyond EvoSuite, though its mutation comparison has stated constraints.

5.1 RQ1: Effectiveness of CodeT5 without DA

  • The RQ1 evaluation splits Defects4j at the test-case level to avoid identical tests crossing training and evaluation sets; task-only fine-tuning was insufficient for compilable or test-adequate tests.The model was fine-tuned on Methods2Test and directly evaluated on the Defects4j evaluation split.Source: §5.1 · pp. 7–8
  • Without DA, generated tests often failed to compile because of incorrect dependencies or undefined objects, and some were truncated at CodeT5’s 512-token output limit.Their line coverage and mutation scores were less promising than BLEU and CodeBLEU scores.Source: §5.1 · p. 8

5.2 RQ2: Effectiveness of CodeT5 with DA

  • 72.12% in parse rate, 37.34% in compile rate, 20.38% in BLEU, 16.34% in CodeBLEU, 18.62% in line coverage, and 16.45% in mutation score are the average gains of CodeT5 with DA over CodeT5 without DA.Table 1 compares the evaluation scores for CodeT5 with and without DA, GPT-4, and A3Test.Source: Table 1 · p. 8, §5.2 · p. 9
  • CodeT5 with DA outperformed the studied baselines in all reported metrics except GPT-4’s parse rate; GPT-4 outperformed A3Test overall.A3Test had the lowest compile rate, line coverage, BLEU, and CodeBLEU; no-DA CodeT5 had the lowest parse rate and mutation score. GPT-4’s stronger parse rate did not extend to other metrics.Source: §5.2 · p. 9

5.3 RQ3: Augmentation with SBST

  • In all five projects, model-generated tests covered extra lines beyond EvoSuite; the model had higher median and mean coverage than EvoSuite in the reported comparison.EvoSuite’s median and mean coverage were 23% and 35.4%, versus 58% and 57.8% for the model.Source: §5.3 · p. 10, Table 2 · p. 10
  • Model-generated tests had higher mutation scores than EvoSuite in 2 of 5 projects, including 26 new mutants killed in Jacksondatabind.A direct mutation-score comparison is constrained because EvoSuite generates tests for the whole project while the model’s test set covers only 20% of each project.Source: §5.3 · p. 10, Table 3 · p. 10
  • The suggested workflow is to run EvoSuite first, then pass its uncovered lines to the model, which generates each test in about 2 seconds versus EvoSuite’s 153 seconds per test on average.The authors frame the model as an add-on, particularly for new commits, rather than a replacement for EvoSuite.Source: §5.3 · p. 10
  • The study does not systematically evaluate test readability, showing only an example and citing prior user-study evidence for readability benefits.The authors note that their readability claim is based on an example in this paper and a user study in [39].Source: §5.3 · p. 11

6 LIMITATIONS AND THREATS TO VALIDITY

The approach may depend on developer-written tests, and the study leaves readability, LLM prompting, and external validity incompletely assessed. Resource costs also constrained the project set and a comparison with ChatTester.

6 LIMITATIONS AND THREATS TO VALIDITY

  • The approach may depend on developer-written tests; its intended use is to extend existing suites, with LLM-generated tests suggested as a starting point for projects lacking tests.The authors note that automatically generated training tests could hinder the targeted benefits of fault detection and readability.Source: §6 · p. 11
  • The study did not systematically evaluate readability, instead showing an example and relying on a prior user study that reported readability benefits [39].Source: §6 · p. 11
  • The study could not compare with ChatTester because its data-pair collection yielded only 20 pairs for one project and none for the other four.The GPT-4 comparison also omitted advanced prompt engineering, which the authors identify as a subject for future comparison.Source: §6 · p. 11
  • The selected projects may limit external validity; the small project set reflects the resource costs of adequacy metrics and GPT-4 runs.Source: §6 · p. 11

7 CONCLUSION

Project-level domain adaptation improves CodeT5 test generation under domain shift, with results stronger than the stated baselines on most metrics and complementary to search-based testing. The approach is scoped to using existing developer-written tests, and the authors identify other models, datasets, and readability evaluation as future work.

7 CONCLUSION

  • The approach outperforms GPT-4 on all metrics except parse rate and outperforms A3Test on all metrics, while complementing search-based tools to increase line coverage and mutation score.Source: §7 · p. 11
  • Project-level domain adaptation leverages existing developer-written tests to address domain shift, improving generated-test compilability and similarity to human-written, test-adequate tests.Source: §7 · p. 11
  • The approach may depend on developer-written tests, so its stated use case is augmenting existing tests to cover new lines and detect new faults.Source: §7 · p. 11
  • Future work will explore other code models and datasets and conduct a user study of generated-test readability.Source: §7 · p. 11

Related papers

Loading 2308.08033v3…