Source-linked AI summary
TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
Elena Bruches, Vadim Alperovich, Dari Baturova, Roman Derunets, Daniil Grebenkin, Georgy Mkrtchyan, Oleg Sedukhin, Mikhail Klementev, Ivan Bondarenko, Nikolay Bushkov, Stanislav Moiseev
TL;DR
LLM-based unit-testing research has largely focused on isolated generation rather than broader suite maintenance, motivating a realistic evaluation framework. TAM-Eval benchmarks creation, repair, and updating at test-file level across 1,539 multilingual scenarios, and finds limited first-attempt capability with only marginal effectiveness gains.
Problem
Existing LLM-based testing work largely targets isolated test generation or oracle prediction, leaving comprehensive unit-test suite maintenance underexplored.
Method
TAM-Eval evaluates test creation, repair, and updating at test-file level using unified prompts, iterative execution feedback, and reference-free pass-rate, coverage, and mutation metrics.
Results
State-of-the-art models show limited first-attempt test-maintenance capability without fail feedback; GPT-5 reaches a PassRate of 30.7% at Attempt@1.
Takeaways & Limitations
TAM-Eval provides an open, modular benchmark for comparing raw LLMs and agentic workflows in automated unit-test maintenance.
Abstract
from arXiv · showhide
While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval (Test Automated Maintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of both raw LLMs and agentic workflows, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are publicly available at https://github.com/trndcenter/TAM-Eval.
I. INTRODUCTION
TAM-Eval addresses the underexplored problem of maintaining unit-test suites through creation, repair, and updating. It evaluates these tasks at test-file granularity using repository context, execution feedback, and multiple effectiveness metrics.
- Motivation: Existing LLM-based testing tools largely generate isolated tests or assertions rather than maintaining suites as code evolves.The paper frames maintenance as continuous test generation, repair, and updating needed to avoid outdated suites, broken pipelines, and undetected regressions.
- Technical Challenges: Automated test-maintenance evaluation requires reasoning over production code and existing tests in dynamic environments that check correctness, coverage, and mutation score.
- Contributions: TAM-Eval covers test creation, repair, and updating at entire test-file granularity, reflecting realistic developer workflows.
- Contributions: 1,539 validated scenarios from actively maintained open-source repositories form TAM-Eval’s curated benchmark dataset.
- Contributions: The framework uses unified prompts, iterative execution feedback, and reference-free evaluation through pass rate, coverage, and mutation testing.
- Empirical Analysis: Mutation-coverage improvements rarely exceed 12 percentage points, while leading models show substantial variability across tasks and languages.
II. BACKGROUND
Unit-testing research has progressed from traditional coverage-oriented automation to LLM-based generation, repair, and improvement. Existing approaches increasingly use repository history and specialized pipelines, but repair and maintenance require distinct capabilities.
- Background: Traditional automated unit-testing systems focused on code coverage, while LLM methods expanded test generation through prompting and self-improvement techniques.
- B. Unit tests repairing: Test repair can use rule-based, LLM-based, or combined pipelines, including syntax analysis, import transfer, and entity mutations.
- B. Unit tests repairing: Test repair requires a different set of LLM skills from test generation despite sometimes appearing within the same pipeline.
- Background: Test maintenance also includes periodically updating tests that lack functionality coverage or are unstable under mutations.
- Background: Commit-based benchmarks simulate maintenance by reverting repository states or pairing pre-fix and post-fix implementations with generated tests.
D. Unit-testing related benchmarks
Unit-testing benchmarks assess varied dimensions and scenarios, but prior resources differ in task coverage, repository-level realism, language scope, and expansion cost. TAM-Eval introduces an automated benchmark focused on creation, repair, and updating.
- D. Unit-testing related benchmarks: Existing unit-testing benchmarks evaluate different task dimensions rather than a single standardized capability.
- D. Unit-testing related benchmarks: TestBench evaluates Java examples across syntactic correctness, compilation, validity, coverage, and defect detection.
- D. Unit-testing related benchmarks: ProjectTests spans Python, Java, and JavaScript but its costly manual expansion may hinder long-term utility.
- D. Unit-testing related benchmarks: TESTEVAL measures overall, targeted line and branch, and path coverage, but lacks repository-level data.
- D. Unit-testing related benchmarks: TAM-Eval provides a fully automated pipeline for unit-test creation, repair, and updating, with 1,539 filtered and annotated samples across Java, Python, and Go.
Stage 2. Execution-based Filtering
TAM-Eval’s execution-based filtering retains buildable, runnable, stable, and meaningfully test-covered project pairs. Additional content checks and balanced sampling further constrain the benchmark toward substantive and diverse examples.
- Stage 2. Execution-based Filtering: Only projects buildable through predefined commands without manual intervention are retained for execution-based evaluation.
- Stage 2. Execution-based Filtering: Tests must run within 30 seconds and pass consistently across two executions, while language-specific import constraints help identify suitable focal-test pairs.
- Stage 2. Execution-based Filtering: At least 40% original line coverage of the corresponding focal function is required, providing a meaningful baseline for coverage and mutation-score improvements.
- Stage 2. Execution-based Filtering: Repositories are retained only when they contain at least five focal-test pairs satisfying the filtering conditions.
- Content-Based Checks: Content checks exclude test files with fewer than two cases, trivial focal functions, extreme file sizes, comment-heavy code, and generated files.
- Sampling: Balanced sampling limits repositories to up to 10 focal-test file pairs, and 45% of the data remaining after the third stage was filtered out.
Stage 4. Test Maintenance Tasks Creation
TAM-Eval derives non-overlapping creation, repair, and updating tasks from executable focal–test file pairs, using structured modifications and realistic defects to model maintenance needs.
- Task construction: Three task families cover test creation, repair, and updating as core subproblems of automated test maintenance.The final-stage samples are assigned without overlap.
- Test creation: Creation scenarios include clearing entire test suites, extending partially covered tests, and recovering removed tests.The recovery setup guarantees that new test generation is feasible, unlike the lower-coverage extension setting.
- Test repair: Repair scenarios inject syntactic, execution, coverage, and efficiency defects to test correction and restoration of meaningful behavioral checks.Execution-targeted faults account for 47.37% of the repair challenge, while coverage-targeted cases account for 17.77%.
- Test updating: Updating reverts a test file to an earlier executable-code revision while retaining the current focal file, requiring recovery when maintenance metrics degrade.Samples are retained after at least a 5% coverage drop or incorrect execution with the current focal file.
IV. EVALUATION FRAMEWORK
The evaluation framework uses interpretable, reference-free metrics to assess whether modified test suites execute correctly and improve line coverage and mutation-based fault detection.
- Framework: The pipeline evaluates test creation, repair, and updating without requiring reference test suites.Its metrics target coverage improvement, fault detection, and correctness.
- Metrics: PassRate measures the ratio of successful test executions to total executed tests and filters for basic utility.Failing tests cannot reliably validate correctness.
- Metrics: ∆TestCov measures the change in line-level coverage between the initial and final test suites.Coverage is computed from executed lines relative to executable lines in the focal file.
- Metrics: Mutation testing measures whether tests distinguish the original implementation from faulty mutants, with ∆MutCov capturing the resulting change.An ideal suite passes on the original file and fails on as many mutants as possible.
B. Evaluation Pipeline
Each benchmark sample undergoes a reproducible automated pipeline that filters trivial outputs, validates syntax, executes tests, measures coverage, and performs fixed-set mutation testing before aggregation.
- Evaluation pipeline: The pipeline begins with sanity checks that discard empty outputs and exact duplicates of the input test file.
- Evaluation pipeline: Language-specific parsers validate that generated test code is syntactically valid and compilable.
- Evaluation pipeline: Test execution collects line-level coverage against the corresponding focal file to compute ∆TestCov.
- Evaluation pipeline: Mutation testing generates fixed mutant sets and counts how many mutants the test suite fails to compute ∆MutCov.For each run, the mutant set is fixed rather than sampled.
- Evaluation pipeline: Individual metric deltas are aggregated across the benchmark into average performance indicators.
C. Infrastructure
The infrastructure isolates evaluations in Docker, supports language-specific analysis tools, and provides a modular interface for model inputs, retries, and unified task prompting.
- Infrastructure: Sandboxed Docker environments provide consistency, scalability, and isolation across evaluations.Build and test commands are detected automatically using language and project-structure heuristics.
- Infrastructure: Coverage analysis uses coverage.py for Python, JaCoCo for Java, and cover for Go, while mutation testing uses mutpy, PIT, and go-mutesting.
- Infrastructure: The infrastructure is modular and language-extensible, allowing future additions of languages or testing tools.
- Model interaction: Models receive focal and corresponding test files and are prompted to rewrite the entire test file to improve coverage and effectiveness.The default context is streamlined, but full-repository context is supported.
- Model interaction: Models may make up to k attempts, with subsequent attempts receiving automatically extracted syntax, compiler, or stack-trace feedback.All maintenance tasks share one unified prompt with minimal task-specific guidance.
B. Inference Configuration
The evaluation benchmarks diverse LLMs on automated unit test maintenance using reproducible inference settings and up to three recovery attempts. Performance generally improves with iterative attempts, but first-attempt capability remains limited for most models.
- The evaluation queried a diverse set of open-source and proprietary LLMs through the OpenRouter API.
- Sampling used a fixed random seed and temperature of 0.25, with default system prompts unless otherwise specified.
- GPT-5 achieved the highest Attempt@3 PassRate at 42.3%, followed by GPT-OSS-120B at 32.8%.
- All three metrics generally improved with additional attempts, with GPT-5 and GPT-OSS-120B showing the most consistent gains.The metrics are Pass Rate, mean ∆Test Coverage, and mean ∆Mutation Coverage.
- Most models performed poorly on the first attempt without failure feedback, except GPT-5, which reached a 30.7% PassRate at Attempt@1.
B. Performance by Language
Performance varies substantially across programming languages, models, and task types. GPT-5 leads overall, while coverage gains, failure modes, and task difficulty reveal limits beyond merely producing executable tests.
- Performance by Language: GPT-5 consistently leads across languages, with ∆TestCov of 18.7 and ∆MutCov of 10.2, while GPT-OSS-120B leads Python coverage metrics.
- Performance by Language: Go generally appears most suitable for LLMs, whereas Java can achieve a 29.3% GPT-5 Pass Rate without comparable coverage gains.
- Performance by Language: Qwen3 Coder 480B A35B produces the shortest test files, while Gemini 2.5 Flash produces the longest across Python, Go, and Java.For Python, lengths range from 3,698 to 15,641 characters; for Go, 4,410 to 13,463; and for Java, 3,017 to 12,711.
- Performance by Language: Java has the highest assert density despite shorter files, with models averaging 15 to 66 asserts per file.
- Performance by Task: Create and Repair generally outperform Update, whose reduced ∆MutCov reflects the difficulty of precise context-aware edits.GPT-OSS-120B remains competitive with GPT-5 on Update Pass Rate and ∆TestCov.
- Performance by Task: Add New Tests and Recover Tests show relatively high Pass Rates but negligible coverage improvements, indicating limited semantic depth.
- Failure Analysis: Execution_error accounts for over 60% of invalid outputs for most models, reaching up to 80% for Devstral-Small and Gemini 2.5 Flash.Syntax error rates remain low, making correct executable test-suite setup the principal bottleneck.
- Failure Analysis: Models share similar weaknesses, with Go tests frequently containing unused imports, undefined names, and function references.
VII. CONCLUSION
TAM-Eval is an open benchmark and framework for evaluating LLM-based unit test maintenance across three languages and multiple effectiveness dimensions. Its baseline results show substantial shortcomings, while iterative verifier feedback improves performance and motivates better context handling.
- TAM-Eval evaluates automated unit test maintenance across Python, Java, and Go using 1,539 curated task instances.Its modular design supports extensions to additional languages and repositories.
- The framework uses reference-free test pass rates, test coverage gains, and mutation coverage gains to assess test creation, repair, and updating.
- Baseline analysis reveals significant shortcomings in LLM performance for unit test maintenance.
- Performance improves with multiple attempts, underscoring the importance of iterative feedback from automated verifier systems such as compilers.
- The findings identify enhanced context handling and support for higher-order testing paradigms as areas for improvement.The benchmark has also been used as a validation set for training reward models.