Source-linked AI summary
Mutation-Guided LLM-based Test Generation at Meta
Christopher Foster, Abhishek Gulati, Mark Harman, Inna Harper, Ke Mao, Jillian Ritchey, Hervé Robert, Shubho Sengupta
TL;DR
Existing LLM-based test generation has offered limited support for targeting specific high-impact fault classes such as privacy, security, and integrity. ACH addresses this gap by generating issue-specific, currently undetected mutants and tests to kill them, then evaluates the approach in Meta’s deployed systems. The deployment produced tests that engineers often accepted, while the authors identify the current workflow as a baseline for future improvements.
Problem
Existing LLM-based test-generation research has offered little work on tests targeting specific high-impact fault classes, including security, integrity, and privacy.
Method
ACH combines mutation-guided fault generation with LLM-based test generation, using relevant, currently unkilled mutants as prompts for tests aimed at specific concerns.
Results
ACH was applied to 10,795 classes across 7 Meta platforms, generating 571 additional unit tests; engineers accepted 73% of automatically generated tests, with 36% judged privacy relevant.
Takeaways & Limitations
ACH demonstrates a deployed approach for transforming vague software concerns into tests that target issue-specific faults, with potential applications beyond privacy hardening.
Takeaways & Limitations
The reported workflow does not yet use more sophisticated prompting, fine-tuning, or language-model ensembles, so the results are presented as a baseline for future developments.
Abstract
from arXiv · showhide
This paper describes Meta's ACH system for mutation-guided LLM-based test generation. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation testing. Instead, it focuses on generating currently undetected faults that are specific to an issue of concern. From these currently uncaught faults, ACH generates tests that can catch them, thereby `killing' the mutants and consequently hardening the platform against regressions. We use privacy concerns to illustrate our approach, but ACH can harden code against {\em any} type of regression. In total, ACH was applied to 10,795 Android Kotlin classes in 7 software platforms deployed by Meta, from which it generated 9,095 mutants and 571 privacy-hardening test cases. ACH also deploys an LLM-based equivalent mutant detection agent that achieves a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing). ACH was used by Messenger and WhatsApp test-a-thons where engineers accepted 73% of its tests, judging 36% to privacy relevant. We conclude that ACH hardens code against specific concerns and that, even when its tests do not directly tackle the specific concern, engineers find them useful for their other benefits.
1 INTRODUCTION
ACH targets specific, currently undetected fault classes by combining mutation-guided test generation with LLMs, using privacy hardening as its deployment focus. Its tests carry verifiable assurances while broader relevance and stylistic fit remain subject to engineering review.
- ACH approach: ACH is an agentic LLM-based tool for generating unit tests that target specific classes of faults, with privacy hardening as its primary reported application.The approach is not confined to privacy and is presented as applicable to other issues.
- Motivation: Recent LLM test-generation work has rarely targeted specific fault classes, despite organizations’ need to test high-impact security, integrity, and privacy concerns.The paper frames issue-specific test generation as an important unmet need.
- ACH approach: ACH constructs relevant, currently unkilled mutants from organizational bug information and uses them as prompts for LLM-based test generation.The workflow generates issue-specific simulated faults and tests that can catch them.
- Assurances: ACH provides buildability, valid-regression, and hardening assurances as unequivocal guarantees, while relevance and fashion-following are assessed through code review.Coverage achieved is also reported when a generated test adds coverage.
- Mutation testing: Generated tests need not extend line coverage because they may expose faults through new data on existing code paths, illustrating mutation testing’s value beyond structural adequacy.This property lets tests find faults without necessarily covering new lines.
- Empirical results: ACH’s deployment covered 7 Meta software platforms and 10,795 classes, producing 4,660 candidate mutants and 571 additional unit tests.The reported tests included 277 that line-coverage-only selection would have discarded.
2 THE ACH SYSTEM
ACH uses issue-specific text and existing development information to generate simulated faults, filter equivalent mutants, and prompt tests aimed at killing currently uncaught faults. Its workflow combines LLM-based agents with automated rule-based components and supports compliance hardening beyond privacy.
- Workflow inputs: ACH starts from an issue summary that can incorporate faults, requirements, constraints, concerns, and regulatory requirements.The issue summary supplies prompts for fault generation and can represent concerns beyond privacy.
- Scope and assurances: ACH is a compliance hardener because it improves regression infrastructure’s ability to detect regressions that could cause non-compliance with the issue of concern.The reported deployment used privacy-hardening concerns, while the workflow is presented as applicable to broader issues.
- Scope and assurances: The reported workflow focuses on mutation-guided agentic testing and does not yet use more sophisticated prompting, fine-tuning, or language-model ensembles.The authors characterize the results as a baseline for future developments.
- Fault generation: The fault-generation agent walks the code base using existing tests, classes under test, and the issue summary to create simulated faults.The generated mutants may target code relevant to the issue being hardened, but they are not guaranteed to change the code.
- Mutant filtering: ACH uses an equivalence-detector agent to judge whether generated mutants differ semantically from the original class.The paper notes that mutant equivalence is undecidable, while reporting that the LLM-as-judge approach was effective for these mutants.
- Test generation: The surviving faults become prompts for a modified TestGen-LLM phase that generates tests targeting mutant coverage rather than uncovered lines.This connects simulated faults to tests intended to kill currently uncaught behaviors.
3 RESULTS FROM META’S ACH DEPLOYMENT
Meta deployed ACH across seven software platforms and 10,795 classes, generating relatively few, issue-specific mutants before producing tests from the non-equivalent candidates. The workflow generated 9,095 buildable mutants, with 4,660 used for test generation, while emphasizing mutation adequacy over structural coverage.
- Deployment: ACH was applied to seven Meta platforms spanning social media, messaging, virtual reality, augmented reality, and cross-app products.The platforms were Aloha, Facebook Feed, Instagram, Messenger, Oculus, Wearables, and WhatsApp.
- Deployment: The workflow terminates after finding a mutant that builds and passes for the current class, so candidate-mutant counts are bounded by classes under test.This termination rule shapes the reported deployment statistics.
- Mutant generation: ACH generates relatively few, highly specific mutants designed to be more relevant to the issue of concern than rule-based mutation approaches.The contrast is intentional: ACH prioritizes issue specificity over the large mutant volumes typical of traditional mutation testing.
- Mutant generation: 9,095 mutants built and passed from 10,795 classes under test, and 25% were trivially syntactically equivalent.The reported equivalent-mutant rate exceeds the typical 10% to 15% rate cited for rule-based approaches.
- Test generation: 4,660 mutants, or 51% of the 9,095 buildable mutants, were deemed non-equivalent and became prompts for test generation.The test-generation prompts targeted mutant coverage rather than uncovered lines.
4 ENGINEERS’ EVALUATION OF ACH
ACH was evaluated through Meta’s normal review process, an initial deployment trial, and WhatsApp and Messenger test-a-thons. Engineers assessed generated tests for usefulness and privacy relevance, with acceptance shaped by coverage, corner cases, and team-specific judgments.
- Initial Trial: 90% of the initial trial’s submitted diffs were accepted, with 27 of 30 tests accepted.The broader reviewed-diff denominator was 27 of 29, or 93%.
- Initial Trial: Engineers valued tests that added coverage, exposed privacy-relevant faults, or improved understanding through concrete fault examples.Reviewers also found non-privacy tests useful when they added other benefits.
- Privacy Test-a-thons: 91 WhatsApp tests and 90 relevance assessments were completed after reviewers requested additional tests beyond the initial allocation.The test-a-thons reviewed usefulness and privacy relevance separately.
- Privacy Test-a-thons: 36% of tests were judged relevant to privacy, which the authors describe as a baseline because little filtering targeted irrelevant tests.The authors suggest static analysis and additional LLM-as-judge agents could improve relevance.
- Privacy Test-a-thons: 73% acceptance corresponded to privacy hardening in about one third of cases, according to the authors’ interpretation of relevance uncertainty.They note that scores 4 and 2 often reflected uncertainty, placing an upper bound near two thirds potentially relevant.
- Privacy Test-a-thons: Acceptance differed sharply by team: WhatsApp accepted 56% of tests versus 89% for Messenger despite higher reported privacy relevance.The authors attribute this difference to potentially different team cultures and subjective landing decisions.
5 THE EQUIVALENT MUTANT PROBLEM
Equivalent mutants are unavoidable in mutation testing and can waste resources, so ACH filters them with an LLM-based detector. The detector’s effectiveness depends strongly on the generated mutants, while simple preprocessing substantially improves reported precision and recall.
- Problem: Equivalent mutants can remain despite existing filtering techniques because program equivalence is undecidable, causing wasted effort on unkillable mutants.This is a general mutation-testing challenge that ACH must manage.
- ACH’s Response: ACH relegates equivalent mutants to an engineering concern because engineers review generated tests rather than mutants, while the workflow still wastes computation trying to kill unkillable mutants.The equivalence detector is therefore used mainly to reduce computational inefficiency.
- Evaluation: The equivalence detector was evaluated through manual analysis of randomly sampled mutants from four platforms against human-assessed equivalence ground truth.The study’s sampling was slightly affected when code changed after mutants were constructed.
- Evaluation: ACH evaluates weak mutation testing, appropriate for unit-test generation, without modeling failed error propagation required for strong mutation testing.Weak mutation focuses on detecting local mutant execution effects rather than requiring propagation to observable outputs.
- Mutant Types: Misleading comments create clearly equivalent mutants because the agent may alter comments without changing executable code, potentially confusing equivalence judgments.The authors identify comment handling as a limitation of text-based language models.
- Detector Results: Precision was 0.79 and recall was 0.47 when unsure cases counted as equivalent; treating unsure as non-equivalent raised precision to 0.97 with recall at 0.44.High precision is prioritized because retaining non-equivalent mutants preserves possible tests, whereas higher recall mainly saves computation.
- Detector Results: Simple comment-removal and lexical preprocessing raised overall precision and recall to 0.95 and 0.96, respectively.Approximately 25% of mutants were syntactically identical, and 61% of equivalent mutants contained only a misleading comment.
- Limitations: The authors caution that the strong detector figures reflect the generated mutants’ obvious semantic changes or obvious equivalence, not general program-equivalence competence.The underlying equivalence problem remains undecidable.
6 THE IMPORTANCE OF MUTATION TESTING
ACH’s industrial results reinforce mutation testing’s value beyond structural coverage: targeting mutants found faults that coverage-only criteria would miss, while also often increasing coverage.
- Mutation testing beyond coverage: 49% of tests that uniquely kill a mutant do not add line coverage.Coverage-only evaluation would wrongly discard tests that catch otherwise undetected faults.
- Mutation testing beyond coverage: ACH kills 15% of mutants versus 2.4% for TestGen-LLM, despite targeting fewer classes.TestGen-LLM generates tests for 32% of classes compared with ACH’s 5.3%, but does not target specific mutants.
- Mutation testing beyond coverage: Targeting mutants can elevate coverage, whereas targeting coverage alone is inadequate for killing mutants.The authors frame this as industrial evidence extending prior laboratory findings.
- Mutation testing beyond coverage: 51% of ACH tests also raise coverage, indicating mutation-guided generation can provide structural-coverage side benefits.The paper suggests mutants could also serve as prompts for coverage-oriented generation, analogous to retrieval augmentation.
7 RELATED WORK
The paper positions ACH as the first large-scale industrial deployment combining LLM-based mutant generation with automatic test generation, extending laboratory mutation-testing work toward targeted faults.
- Assured test generation: ACH uses Assured LLMSE to propose whole compilable test units with assurances about semantics and usefulness, rather than unassured code completions.Its proposals are generated offline through CI and standard code review, creating an audit trail.
- Mutation testing and LLMs: ACH extends realistic LLM-generated mutants by generating faults relevant to a chosen issue of concern.Traditional mutation operators are widely viewed as poorly suited to realistic faults.
- Equivalent mutants: LLM-based equivalence detection is part of ACH’s workflow, addressing the equivalent-mutant problem alongside mutant and test generation.The paper reports prior LLM equivalence-detection work, including a 35% F1 improvement over traditional approaches in one benchmark study.
- Mutation testing and LLMs: Prior LLM mutation-testing research studied benchmark problems and laboratory conditions, while ACH reports deployment at industrial scale.Earlier industrial mutation-testing deployments did not automatically generate tests to kill mutants.
8 OPEN PROBLEMS
The paper identifies open problems around whether ACH’s mutant-equivalence and relevance findings generalize, and around detecting faults already present in the codebase.
- Generalization: The ease of detecting equivalent mutants may be specific to Kotlin, Android, Meta, or simulated privacy faults.The authors state that broader generalization remains an open question.
- Mutant relevance: More research is needed to define and operationalize similarity between generated mutants and original faults.The authors suggest using similarity metrics to guide fine-tuning, prompting, re-prompting, or Chain-of-Thought workflows.
- Existing faults: ACH protects against future regressions but cannot detect existing faults because the current system serves as the regression oracle.Detecting existing faults requires addressing the well-known oracle problem.
- Existing faults: Oracle inference for targeted mutation-guided generation would need higher precision to avoid wasting engineers’ time on false positives.The paper suggests issue-specific fault and test generation may help oracle generation.
9 CONCLUSIONS
ACH combines realistic issue-specific mutant generation, equivalent-mutant filtering, and automatic test generation to deploy mutation-guided testing on Meta’s large-scale industrial systems.
- Conclusions: Mutation testing has remained intellectually important but difficult to deploy in industry.ACH addresses this deployment challenge by combining recent advances in mutant and test generation.
- Conclusions: LLMs help generate realistic faults by translating a concern or issue into simulated faults used to generate tests.This enables ACH to target specific classes of faults rather than relying only on generic mutation operators.
- Conclusions: An additional agent filters equivalent mutants, with simple static-analysis preprocessing strengthening that filtering step.The workflow uses an LLM-based equivalence detector together with preprocessing.
- Conclusions: ACH automatically generates unit tests to kill the generated mutants, combining mutant generation and test generation in one industrial workflow.The paper presents this combined deployment as its central contribution.
- Conclusions: The authors suggest Mutation-as-RAG could improve optimization for structural coverage criteria.This conclusion is based on the observed coverage side benefits of mutation-guided generation.