Source-linked AI summary
Automated Unit Test Improvement using Large Language Models at Meta
Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, Eddy Wang
TL;DR
The paper addresses limited assurance for LLM-generated test improvements by presenting TestGen-LLM, which filters generated cases for buildability, reliable execution, coverage improvement, and non-regression. Deployed at Meta, the tool improved applied test classes and produced recommendations accepted for production, while the authors identify coding-style and deployment-context limitations.
Problem
The paper addresses the need for reliable improvement of existing tests while avoiding unsupported or regressive LLM-generated code.
Method
TestGen-LLM embeds LLMs in an assured software-engineering workflow that filters candidate tests and recommends only measurable, non-regressive improvements.
Results
10% of applied classes improved and 73% of recommendations were accepted by developers and landed in production across Instagram and Facebook.
Takeaways & Limitations
The deployment reports industrial use of LLM-generated test improvements backed by assurances intended to guard against hallucination.
Takeaways & Limitations
The underlying Meta LLMs are commercially sensitive, and the Instagram test-a-thon was not a perfect example of diff-time deployment because test-a-thon focus may have influenced results.
Abstract
from arXiv · showhide
This paper describes Meta's TestGen-LLM tool, which uses LLMs to automatically improve existing human-written tests. TestGen-LLM verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination. We describe the deployment of TestGen-LLM at Meta test-a-thons for the Instagram and Facebook platforms. In an evaluation on Reels and Stories products for Instagram, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage. During Meta's Instagram and Facebook test-a-thons, it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers. We believe this is the first report on industrial scale deployment of LLM-generated code backed by such assurances of code improvement.
1 INTRODUCTION
TestGen-LLM extends human-written Kotlin test classes with LLM-generated cases, filtering recommendations to assure measurable improvement and non-regression. Meta reports evaluation results and industrial deployment experience across Instagram and Facebook.
- TestGen-LLM extends existing human-written Kotlin test classes with additional cases targeting missed corner cases and increased coverage.
- The workflow recommends fully formed software improvements backed by verifiable improvement and non-regression guarantees, discarding cases that cannot meet them.
- The paper reports development, deployment, evolution, lessons learned, open problems, and research challenges from Meta test-a-thons.
- 75% of generated test cases built correctly, 57% passed reliably, and 25% increased coverage in an Instagram Reels and Stories evaluation.
- 10% of applied classes improved, and 73% of TestGen-LLM improvements were accepted by developers and landed in production across Instagram and Facebook.
2 THE TESTGEN-LLM SYSTEM
TestGen-LLM embeds LLM code generation in a filtered software-engineering workflow that retains only buildable, reliable, coverage-improving tests. Its design supports assured regression testing, ensemble-style generation, and human-controlled deployment.
- Progressive semantic filters check buildability, execution reliability, and coverage improvement before accepting generated tests.Non-building, failing, flaky, or non-coverage-improving candidates are discarded.
- Five repeated executions identify flaky tests, retaining only tests that pass every execution for reliable regression testing.
- Candidates clearing all three filters are guaranteed to improve the existing test class and provide reliable regression-test signal.
- TestGen-LLM uses an ensemble approach in which multiple LLMs, prompts, and parameters contribute candidate test cases that compose into test classes.
- The tool supports human engineers rather than replacing them, while automated guarantees address concerns about relying on machine-generated code.
- Providing the class under test improves results, although prompts containing only the existing test class can still find additional tests.
3 TESTGEN-LLM DEPLOYMENT
TestGen-LLM was deployed through Meta test-a-thons and evaluated across models, prompts, and temperatures, with filters retaining tests that build, pass reliably, and increase coverage. Deployment produced engineer-reviewed production diffs, while evaluation and test-a-thon results showed measurable but variable coverage gains.
- 3 TESTGEN-LLM DEPLOYMENT: 16 of 17 TestGen-LLM diffs landed in production during the first Instagram test-a-thon, after one was abandoned for lacking an assertion.The rejected test still extended coverage by executing a previously unexecuted method, but its assertion remained a TODO.
- 3 TESTGEN-LLM DEPLOYMENT: TestGen-LLM ranked sixth among 36 engineers by landed test cases, with each generated test submitted as a separate diff for per-test acceptance or rejection.The ranking used landed test cases rather than diff counts, making the comparison with human engineers fair despite different submission styles.
- 3 TESTGEN-LLM DEPLOYMENT: 75% of test classes had at least one new test case that builds correctly, 57% had one that passes reliably, and 25% had one that increases line coverage.The evaluation covered Instagram Reels and Stories components and used four prompting strategies with two language models.
- 3 TESTGEN-LLM DEPLOYMENT: TestGen-LLM filtered generated tests to retain only cases that build correctly, pass reliably, and increase line coverage.Tests failing any assurance were discarded, preventing recommendations without measurable improvement or non-regression evidence.
- 3 TESTGEN-LLM DEPLOYMENT: The four prompting strategies were extend_coverage, corner_cases, extend_test, and statement_to_complete, with different prompts contributing unique tests for LLM1 and LLM2.LLM1 found 13 tests across 86 classes, while LLM2 found 16; statement_to_complete added no unique test for LLM1 but added two for LLM2.
- 3 TESTGEN-LLM DEPLOYMENT: Zero temperature was selected as the default because generated tests were most likely to build, pass, and extend coverage at that setting.Its effect was only slightly better than temperatures 0.2 and 0.5, with all achieving approximately 4% success over suggested test cases.
- 3 TESTGEN-LLM DEPLOYMENT: In broader deployment, 36 recommendations were accepted, 4 rejected or abandoned, and 2 withdrawn in one reported outcome, while another recorded 144 accepted, 64 rejected or abandoned, 61 unreviewed, and 11 withdrawn.These outcomes reflect engineer review and production-deployment activity across Meta test-a-thon use.
4 QUANTITATIVE RESULTS FROM DEPLOYMENT
Deployment results show TestGen-LLM improved a minority of applied test classes, with most accepted recommendations reaching production. Platform, temperature, and model comparisons require caution because trial counts and deployment conditions differed.
- Overall deployment: 196 of 1,979 applied test classes were improved, or approximately 10%, and 73% of improvements were accepted by developers.These results came from three Instagram and Facebook test-a-thons.
- Platform comparison: Facebook had a slightly higher overall success rate than Instagram, possibly because it had approximately one order of magnitude more human-written Kotlin test code.The paper presents this explanation as a belief rather than a confirmed cause.
- Success criteria: A successful trial required the generated test case to build, pass reliably, and increase coverage over all existing tests, including previous LLM-generated tests.The deployment filtration process used these conditions to define success.
- Configuration comparison: Temperature 0.0 received 30,483 trials because it was the default, while other settings were also available to engineers.The unequal trial allocation complicates direct comparison across temperatures.
- Configuration comparison: Results at temperature 0.4 appeared good, but differing sample sizes and unavoidable deployment confounding factors make interpretation uncertain.Production tests were generated incrementally, so later trials faced a harder coverage baseline.
- Model comparison: LLM2 received far more trials because it was the default, so slight performance differences between the two models require careful interpretation.The model results nevertheless support slightly better overall performance on Facebook than Instagram.
5 QUALITATIVE OBSERVATIONS FROM DEPLOYMENT
Deployment revealed both practical value and recurring quality issues in generated tests. The observations motivate filtering, re-prompting, and richer coverage analysis alongside human review.
- General lesson: The deployment experience underscored opportunities to combine static analysis with language-model inference.The paper places these observations within broader software-engineering practice.
- LLM self-plagiarism: Both LLMs often generated nearly identical tests for the same prompt, sometimes differing only in name.The observed similarity was described as effectively all-or-nothing rather than gradual.
- LLM self-plagiarism: TestGen-LLM later added a filter that removes previously seen test cases by checking syntactic equality of test bodies.The authors suggest semantic clone detection could also help.
- Nuanced coverage reporting: Generated tests sometimes improved coverage mainly in units other than the class under test, indicating either valuable hard-to-reach code coverage or inadequate mocking.The tool already filters flaky tests, but the authors plan more nuanced analysis and reviewer flags.
- Highlighting test need: Some assertion-free tests exposed previously uncovered non-trivial functions and could serve as starting templates for human-written tests.These tests were not deployed, but adding assertions may require less effort than writing tests from scratch.
- Re-prompting: When generated tests covered only part of a previously uncovered method, automatic re-prompting could seek more coverage or provide engineers with a useful template.The proposed workflow preserves human intervention when further automated coverage is unavailable.
6 RELATED WORK
Related work establishes TestGen-LLM as an extension of existing-test improvement rather than test generation from scratch. Its distinctive design combines LLM generation with filtering and measurable assurances.
- Research gap: Previous literature had not addressed extending existing test classes while providing measurable assurances of improvement and absence of regressions.The paper presents this test-extension application and its industrial deployment as its primary novelty and contribution.
- Prior results: Reported coverage and executability vary widely across prior LLM-based test-generation studies and datasets.Examples range from 80% coverage on small HumanEval examples to no more than 2% on EvoSuite SF110.
- Prior results: In this paper, approximately 57% of generated Kotlin test cases were executable and 25% improved coverage, broadly aligning with results on larger open-source systems.The authors compare these figures with 29% executable tests from TeCo and approximately one third from ChatGPT with prompt engineering.
- Model scope: Although the evaluation used two internal Meta models, the Assured LLMSE design is LLM-agnostic and permits arbitrary numbers of models to contribute test cases.The internal model details are not the focus of the paper.
- Approach: TestGen-LLM uses a generate-and-test workflow that filters candidates according to semantic criteria such as reliable passing.It combines traditional software engineering and testing with language models as code-generation engines.
- Positioning: Unlike Genetic Improvement, TestGen-LLM uses an ensemble of language models and configurations rather than genetic programming.The approach can therefore be viewed as a search-based software-engineering strategy using LLMs as a route for candidate generation.
7 FUTURE WORK AND OPEN PROBLEMS
The paper identifies improvement measurement and automated test-oracle design as open problems for scaling assured test improvement. Line coverage is practical but only a proxy, while stronger criteria are computationally demanding.
- Assessing improvement: Line coverage is used as a simple proxy for test improvement, but it is only an expedient measure of improvement.The authors identify improvement measurement as a key Assured LLMSE issue.
- Assessing improvement: In experimental mode, 25% of generated test cases succeeded when success required coverage improvement, while 57% succeeded when requiring only that tests build and pass.The 25% criterion is stricter because line coverage is required.
- Future criteria: Mutation coverage may be a stronger improvement criterion, but its computational demands make deployment at the required scale challenging.The paper cites evidence that strong mutation coverage outperforms other coverage forms.
2. Application–aware probability distribution resolution
TestGen-LLM’s language-model behavior can reproduce application-specific testing styles, while also inheriting deprecated habits that require additional filtering and post-processing.
- Application-aware resolution: Temperature controls variability in repeated language-model outputs by making the generation process more or less exploratory.The paper notes that application-specific methods for converting probability distributions into code require further research.
- Application-aware resolution: LLMs follow existing test-writing styles, often reproducing the conventions prevalent in the surrounding test class and codebase.Engineers viewed this behavior positively because it aligned generated tests with local practices.
- Application-aware resolution: Generated tests replicate bespoke assertion utilities as well as naming, commenting, and overall structural conventions.These styles would be difficult to reproduce with rule-based algorithms, whereas LLMs acquire them from examples.
- Application-aware resolution: Fashion following can also reproduce deprecated coding habits that remain common in the model’s training data.The paper proposes prompt engineering, linter filters, and static-analysis post-processing as mitigations.
8 CONCLUSIONS
TestGen-LLM progressed from proof of concept to a deployed Meta test-support tool that landed test cases in production. Its central safeguard is submitting for human review only tests guaranteed to improve the existing code base.
- Conclusions: TestGen-LLM evolved from proof of concept through minimum viable product to a deployed test-support tool at Meta.The paper describes this development and deployment trajectory as part of its contribution.
- Conclusions: TestGen-LLM submits for human review only test cases that it can guarantee improve the existing code base.This assurance mechanism is presented as protection against LLM hallucination.
- Conclusions: The paper presents this deployment as its first reported industry-scale use of Assured Large Language Model Software Engineering.The reported production use occurred at Meta.