PaperlayerView exact revision on arXiv

Source-linked AI summary

$\mu$BERT: Mutation Testing using Pre-Trained Language Models

Renzo Degiovanni, Mike Papadakis

arXiv:2203.03289v1cs.SE

TL;DR

Mutation operators can produce code that seems unnatural to developers. µBERT instead uses CodeBERT predictions to generate mutants; on 40 Defects4J faults, it detected 27 versus PiTest’s 26, and its mutants helped improve inferred assertion quality.

  • Problem

    Syntactic mutation operators can produce unnatural code that developers may find unrealistic and that can lead to trivial faults.

    Source: §I · p. 1

  • Method

    µBERT masks tokens in code expressions and uses CodeBERT predictions to generate replacement mutants.

    Source: §I · p. 1, A. Binary Expression Mutation · p. 2

  • Results

    µBERT-guided suites found 27 of 40 Defects4J faults versus PiTest’s 26 of 40, and µBERT was up to 100% more cost-effective; its mutants also helped improve inferred assertion quality.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The authors conclude that natural mutants concern not only readability but also test effectiveness, with µBERT performing comparably to traditional tools and better in some cases.

    Source: §IX · p. 8

    The evaluation covers three Defects4J projects, and results may differ on projects from other domains or against other mutant-generation baselines.

    Source: §VII · p. 6

Abstract

from arXiv · show

We introduce $\mu$BERT, a mutation testing tool that uses a pre-trained language model (CodeBERT) to generate mutants. This is done by masking a token from the expression given as input and using CodeBERT to predict it. Thus, the mutants are generated by replacing the masked tokens with the predicted ones. We evaluate $\mu$BERT on 40 real faults from Defects4J and show that it can detect 27 out of the 40 faults, while the baseline (PiTest) detects 26 of them. We also show that $\mu$BERT can be 2 times more cost-effective than PiTest, when the same number of mutants are analysed. Additionally, we evaluate the impact of $\mu$BERT's mutants when used by program assertion inference techniques, and show that they can help in producing better specifications. Finally, we discuss about the quality and naturalness of some interesting mutants produced by $\mu$BERT during our experimental evaluation.

I. INTRODUCTION

µBERT uses CodeBERT predictions to generate mutants intended to follow programmers’ coding conventions. Its preliminary evaluation reports fault-detection and assertion-inference results alongside examples of mutants that differ from traditional ones.

I. INTRODUCTION

  • The approach targets unnatural mutants, which may seem unrealistic to developers and can produce trivial exceptions, crashes, or infinite loops.Here, naturalness means mutated code follows implicit rules, coding conventions, and patterns representative of competent programmers.Source: §I · p. 1
  • µBERT uses CodeBERT to replace masked tokens with predictions, generating mutants without relying on syntactic mutation operators.The authors say this simplifies mutant creation and limits their number.Source: §I · p. 1
  • µBERT-guided suites found 27 of 40 Defects4J bugs versus PiTest’s 26, with three bugs found only by µBERT and two only by PiTest.µBERT was also up to 100% more cost-effective than PiTest.Source: §I · p. 1
  • µBERT’s mutants can contribute to improving the quality of inferred assertions.The study examines four cases where traditional mutation testing performed poorly.Source: §I · pp. 1–2
  • The paper presents examples of µBERT mutants with interesting properties that differ from traditional mutation.Its preliminary workflow parses Java, masks tokens in expressions, generates replacements from CodeBERT predictions, and discards non-compilable or duplicate mutants.Source: §I · pp. 1–2, Figure 1 · p. 2

II. PRE-TRAINED LANGUAGE MODELS

The paper describes CodeBERT’s masked-token prediction and explains how µBERT applies it to generate Java mutants. The workflow uses surrounding code context, produces candidate replacements, and filters invalid or unchanged programs.

II. PRE-TRAINED LANGUAGE MODELS

  • CodeBERT predicts masked tokens from their surrounding context, supporting masked language modelling across six programming languages, including Java.It accepts sequences of up to 512 tokens with one masked token and predicts the five most likely replacements; µBERT uses all five to create mutations.Source: §II · p. 2
  • Because µBERT treats CodeBERT as a black box, it can benefit from future model improvements and can integrate other language models that support masked language modelling.The paper also notes that generative pre-trained models simplify mutant creation and selection to standard model usage.Source: §II · p. 2

III. µBERT: CODEBERT-BASED MUTANT GENERATION

  • µBERT parses Java classes, extracts expressions, masks a token of interest, and uses CodeBERT predictions to generate mutants.It uses as much surrounding context as possible, up to 512 tokens, creates five mutants per masked expression, and discards non-compiling or unchanged mutants.Source: §III · p. 2
  • The prototype handles unary and binary expressions, assignments, literals, variable names, method calls, and object field accesses, allowing several mutants at one program location.For a + b, it can mask each of the three tokens in turn, potentially producing 15 mutants.Source: §III · p. 2

A. Binary Expression Mutation

µBERT masks operators in binary and unary expressions, uses CodeBERT’s predictions to form replacement mutants, and removes duplicates, unchanged expressions, and non-compilable mutants.

A. Binary Expression Mutation

  • For binary expressions, µBERT masks the operator and uses CodeBERT’s five predicted tokens to create replacement mutants.It discards non-compilable mutants and mutants that are syntactically identical to the original program.Source: A. Binary Expression Mutation · pp. 2–3
  • In the year % 4 example, filtering predictions for duplication, unchanged tokens, and compilability leaves year / 4 and year - 4 mutants.The predictions include spacing variants, which are handled as duplicates or unchanged tokens.Source: A. Binary Expression Mutation · p. 3

B. Unary Expression Mutation

  • For unary expressions, µBERT masks the operator, substitutes predicted tokens, and discards duplicate, unchanged, and non-compilable mutants.The operator may appear before or after the expression; in the --i example, only ++i compiles among the candidate changes.Source: B. Unary Expression Mutation · p. 3

C. Literal and Variable Name Mutation

µBERT masks literals, variable names, and selected expression components, then uses CodeBERT predictions to generate replacement mutants.

C. Literal and Variable Name Mutation

  • For literal mutation, µBERT masks a constant and replaces it with CodeBERT’s predicted tokens.For the literal 4 in year % 4, it discards the unchanged prediction and returns four compilable mutants.Source: C. Literal and Variable Name Mutation · p. 3

D. More Mutation Operators

  • For assignments, µBERT masks the assignment operator; predictions such as -, *, and / can yield alternatives such as avg -= it_result.Source: D. More Mutation Operators · p. 3
  • For method calls, µBERT masks the method name; children.add(c) yields the compilable alternatives children.push(c) and children.remove(c).Source: D. More Mutation Operators · p. 3
  • For object-field access, µBERT masks the field name, with predictions including head, next, tail, last, and first.For array access, it masks the index; arr[mid-1] yields five compilable mutants, including indexes 0, n, mid, 1, and low.Source: D. More Mutation Operators · pp. 3–4
  • For reference-type expressions, µBERT masks the referred class name, with predictions including Math, random, Random, and System.One resulting example is int number = (int)(Random.random() * 10).Source: D. More Mutation Operators · p. 4

IV. RESEARCH QUESTIONS

The paper asks whether µBERT mutants can detect real faults, support assertion selection, and differ usefully from traditional mutation operators.

RQ1

  • RQ1 asks how well tests designed to kill µBERT mutants detect real Defects4J faults compared with PiTest.Source: §IV · p. 4

RQ2

  • RQ2 asks whether µBERT mutants can help retain manually written assertions mistakenly discarded by SpecFuzzer, compared with PiTest.Source: §IV · p. 4

RQ3

  • RQ3 examines whether µBERT generates mutants different from traditional operators, including examples tied to fault detection and assertion preservation.Source: §IV · p. 4

V. EXPERIMENTAL SETUP

The evaluation uses Defects4J faults, manually written assertions, and separate procedures for fault detection, cost-effectiveness, assertion assessment, and qualitative mutant analysis.

A. Faults and Assertions (Ground-truth)

  • The fault-detection study covers 40 bugs from Defects4J across Cli, Collections, and Csv; the assertion study examines four methods and six discarded ground-truth assertions.The project counts are Cli (22), Collections (2), and Csv (16).Source: A. Faults and Assertions (Ground-truth) · p. 4

B. Experimental Procedure

  • For fault detection, the study compares µBERT and PiTest using matched numbers of killed mutants and checks whether selected tests detect the associated real faults.Source: B. Experimental Procedure · p. 4
  • The cost analysis repeatedly selects mutants and tests, counting tests selected plus mutants judged equivalent as effort; it repeats the process 100 times.Source: B. Experimental Procedure · p. 5
  • For assertion assessment, SpecFuzzer is rerun with µBERT and PiTest mutants to test whether the six ground-truth assertions are discarded.Source: B. Experimental Procedure · p. 5
  • The setup also includes qualitative discussion of examples and potential benefits of µBERT mutants for mutation testing and assertion inference.Source: B. Experimental Procedure · p. 5

C. Implementation

The evaluation finds µBERT comparable to PiTest for fault detection, while its cost-effectiveness and assertion-selection results provide additional comparisons. The implementation uses CodeBERT and Spoon.

C. Implementation

  • The implementation uses Spoon to manipulate Java programs and CodeBERT for mutant generation, with scripts for integrating other pre-trained language models.The authors also report making the source code, examples, and preliminary evaluation results publicly available.Source: C. Implementation · p. 5

A. RQ1: Fault Detection Analysis

  • Test suites killing all µBERT mutants detected 27 of 40 faults (67.5%), compared with 26 of 40 (65.0%) for suites killing all PiTest mutants; neither tool detected 11 faults (27.5%).Source: A. RQ1: Fault Detection Analysis · p. 5
  • SpecFuzzer preserved 3 of 6 valid assertions with µBERT mutants and 2 of 6 with PiTest mutants, which could require up to 10 times as many mutants.Source: Table II · p. 5
  • At the same number of analyzed mutants, µBERT was more cost-effective; at maximum effort, mean fault-detection likelihood was 40.0% versus PiTest’s 20.0%.At maximum effort, the reported averages were 46.0% for µBERT and 39.5% for PiTest.Source: A. RQ1: Fault Detection Analysis · p. 5

B. RQ2: Assertion Assessment Analysis

µBERT mutants helped SpecFuzzer preserve valid assertions and supplied examples of mutants that expose faults or violate ground-truth assertions.

B. RQ2: Assertion Assessment Analysis

  • SpecFuzzer preserved 3 of 6 valid assertions with µBERT mutants, compared with 2 of 6 using PiTest, which analyzed up to 10 times more mutants.The study reports the number of generated and killed mutants when assessing assertion selection.Source: B. RQ2: Assertion Assessment Analysis · p. 5

C. RQ3: Qualitative Analysis of µBERT Mutants

  • For Cli 10, Csv 15, and Csv 16, µBERT produced mutants involving required-option handling, special-token values, and control flow, respectively.Examples include replacing remove with add, returning 0 instead of the delimiter, and introducing infinite recursion.Source: C. RQ3: Qualitative Analysis of µBERT… · p. 6
  • In Angle.getTurn and Composite.addChild, µBERT mutants violated ground-truth assertions, so SpecFuzzer did not discard those assertions.For Composite.addChild, examples altered the child’s value or changed which set received an ancestor.Source: C. RQ3: Qualitative Analysis of µBERT… · p. 6

VII. THREATS TO VALIDITY

The authors identify limits in the projects and baseline used, in the mutation implementation and model, and in how testing cost and effectiveness are measured.

VII. THREATS TO VALIDITY

  • The evaluation covers only Cli, Collections, and Csv from Defects4J and uses PiTest as its baseline, so results may differ with other projects or mutant-generation techniques.Source: §VII · p. 6
  • Mutation effectiveness may vary with the selected expressions, projects, programming languages, implementation, or pre-trained model.µBERT uses CodeBERT as a black box, and predictions may change with a different model.Source: §VII · pp. 6, 8
  • Mutant counts and faults found may not reflect actual testing cost or effectiveness, and Defects4J’s test pool may not represent the effort of designing test cases.Source: §VII · p. 8

VIII. RELATED WORK

Prior work has addressed mutant creation through syntax-based operators and mined or crafted fault patterns, while mutant selection targets quality and usefulness. The conclusion presents μBERT as a self-supervised approach and identifies broader evaluation and more targeted mutation locations as future work.

VIII. RELATED WORK

  • Earlier mutant-generation methods use syntax-based operators across programming languages, artefacts, and specialised applications.Other approaches compose fault patterns from historical fixes using syntactic changes, manually crafted patterns, or machine translation.Source: §VIII · p. 8
  • Mutant-selection work targets mutants likely to be killable, couple with real faults, subsume others, or relate to regression changes.Selection addresses the problem that mutants can be too numerous, trivially killed, or redundant.Source: §VIII · p. 8
  • μBERT differs from n-gram naturalness work by generating rather than selecting mutants and using a transformer that captures context before and after the mutation.The paper frames the creation of “good” mutants as an unresolved question despite mutation testing’s long history.Source: §VIII · p. 8

IX. CONCLUSION AND FUTURE WORK

  • μBERT uses self-supervised pre-training on source code rather than training on historical faults or mutation-testing data, which the authors describe as expensive to gather.The authors report performance comparable to traditional mutation tools and better in some cases for fault detection and assertion inference.Source: §IX · p. 8
  • Future work will extend evaluation to the full Defects4J and SpecFuzzer datasets, compare additional tools, and investigate locations likely to yield interesting mutants.The current approach uses CodeBERT as a black box and brute-force masks tokens across program statements.Source: §IX · p. 8

APPENDIX EXTRA EXAMPLES

Table V provides additional μBERT-generated mutant examples for three listed faults.

APPENDIX EXTRA EXAMPLES

  • Table V presents more μBERT-generated mutants for faults Cli 10, Csv 15, and Csv 16.Source: Table V · p. 10
Loading 2203.03289v1…