Source-linked AI summary
RM -RF: Reward Model for Run-Free Unit Test Evaluation
Elena Bruches, Daniil Grebenkin, Mikhail Klementev, Vadim Alperovich, Roman Derunets, Dari Baturova, Georgy Mkrtchyan, Oleg Sedukhin, Ivan Bondarenko, Nikolay Bushkov, Stanislav Moiseev
TL;DR
Automatically generated unit tests require informative feedback, but repeatedly compiling, executing, and measuring candidates is costly. RM-RF predicts three execution-derived test signals from source and test code using multilingual execution-labeled data, achieving an average F1 of 0.69 across the targets while reducing evaluation overhead.
Problem
Repeatedly building, running, and measuring generated tests imposes substantial computational overhead, limiting efficient feedback for code-generation workflows.
Method
RM-RF is a lightweight reward model trained on Java, Python, and Go examples to predict suite execution success, coverage improvement, and mutation-kill improvement from code without compilation or execution.
Results
0.69 average F1 is achieved across the three targets, with fine-tuning improving predictive quality over zero-shot baselines.
Takeaways & Limitations
RM-RF provides lower-latency, lower-cost feedback that remains closely aligned with execution-derived outcomes for generated-test evaluation.
Takeaways & Limitations
The reward model was not tested inside an RL pipeline, and the dataset and models cover only Java, Python, and Go.
Abstract
from arXiv · showhide
We present RM-RF, a lightweight reward model for run-free evaluation of automatically generated unit tests. Instead of repeatedly compiling and executing candidate tests, RM-RF predicts - from source and test code alone - three execution-derived signals: (1) whether the augmented test suite compiles and runs successfully, (2) whether the generated test cases increase code coverage, and (3) whether the generated test cases improve the mutation kill rate. To train and evaluate RM-RF we assemble a multilingual dataset (Java, Python, Go) of focal files, test files, and candidate test additions labeled by an execution-based pipeline, and we release an associated dataset and methodology for comparative evaluation. We tested multiple model families and tuning regimes (zero-shot, full fine-tuning, and PEFT via LoRA), achieving an average F1 of 0.69 across the three targets. Compared to conventional compile-and-run instruments, RM-RF provides substantially lower latency and infrastructure cost while delivering competitive predictive fidelity, enabling fast, scalable feedback for large-scale test generation and RL-based code optimization.
I. INTRODUCTION
RM-RF addresses the cost of execution-based feedback for generated unit tests by predicting test-suite correctness, coverage improvement, and mutation-kill improvement directly from code. It is trained on multilingual test examples and evaluated across model and tuning configurations, reaching an average F1 of 0.69 across three targets.
- Execution-based validation delays training and inference because builds, test runs, and metric computation impose substantial computational overhead.
- RM-RF predicts whether an added test preserves successful suite compilation and execution, increases code coverage, and improves mutation kill rate.
- RM-RF eliminates repeated compilation and execution during test evaluation, reducing latency and resource costs for generated-test assessment.
- The dataset spans Java, Python, and Go and contains focal files, test files, and candidate test additions labeled through coverage and mutation-analysis tools.
- 0.69 average F1 is achieved across the three prediction targets after comparing zero-shot, full-parameter fine-tuning, and PEFT with LoRA.
- Execution-derived feedback can indicate whether a generated test is useful during code generation or model training.
II. BACKGROUND
Prior work evaluates generated tests through coverage, mutation outcomes, execution feedback, and reinforcement learning, but existing tools often operate at narrower scopes or incur execution costs. RM-RF extends run-free behavioral prediction to multilingual candidate-test evaluation.
- B. Evaluation of the Test Quality: Existing test benchmarks assess dimensions including syntactic correctness, compilation success, validity, coverage, and defect detection.
- B. Evaluation of the Test Quality: Prior toolkits commonly target individual functions and may lack complete-suite assessment at file or repository level, while requiring manual annotation or static oracles.
- B. Evaluation of the Test Quality: Coverage alone may not reflect bug-finding ability, whereas mutation testing measures the percentage of injected mutants caught by a test suite.
- D. Learning and Testing via Adversarial RL: Process-supervised reinforcement learning uses compiler- or execution-derived labels to provide denser feedback than outcome-only rewards.
- D. Learning and Testing via Adversarial RL: Regression language models demonstrate that quantitative code behavior can be predicted from source text without execution.
D. Learning and Testing via Adversarial RL
Unit-test generation research increasingly uses reinforcement learning, co-evolution, and execution feedback to improve test and code quality. RM-RF supplies a run-free reward-modeling approach for the same feedback-oriented setting, while its dataset construction uses execution-derived labels.
- D. Learning and Testing via Adversarial RL: UTRL co-trains a test generator to catch code-generator bugs and a code generator to satisfy the resulting tests.
- D. Learning and Testing via Adversarial RL: CURE jointly trains coder and tester models under reinforcement learning without ground-truth code, allowing each to learn from the other’s feedback.
- D. Learning and Testing via Adversarial RL: Iterative critique, reinforcement learning, and coder–tester co-evolution are presented as promising directions for more reliable code and test generation.
- D. Learning and Testing via Adversarial RL: RM-RF predicts execution-derived test signals without compiling or running, targeting compilation success, coverage improvement, and mutation-kill improvement.
A. Data evaluation
The evaluation targets test correctness, line-coverage improvement, and mutation-coverage improvement, using execution-based metrics and language-specific tooling. These measures assess whether added tests are valid and improve functional coverage and fault detection.
- Evaluation targets: Correctness labels tests without runtime or syntax errors during execution.The label is estimated from test-running logs.
- Evaluation targets: Line coverage measures executed test lines relative to total executable focal-file lines.∆TestCov compares initial coverage from existing tests with final coverage after adding the new test.
- Tooling: Coverage values were computed with coverage.py for Python, JaCoCo for Java, and cover for Go.Mutation analysis used mutpy for Python, PIT for Java, and go-mutesting for Go.
- Evaluation targets: Mutation analysis introduces systematic program changes and measures whether the test suite kills detectable mutants.The mutation-coverage delta is estimated similarly to the line-coverage delta.
- Evaluation targets: A correct test with positive mutation- or line-coverage delta is classified as well-written and useful.This criterion combines correctness with improvement on at least one coverage measure.
B. Data collection
The datasets were collected from open-source GitHub repositories selected for test availability, licensing, recency, adoption, and contributor activity. The collection pipeline combined repository selection and downloading with execution- and content-based filtering.
- Collection pipeline: The collection pipeline selected and downloaded GitHub projects before execution- and file-content-based filtering.These stages were applied to both the Holdout and training datasets.
- Project selection: Repositories required at least five focal-test file pairs.
- Project selection: Holdout projects and source files were required to have recent updates, while training projects needed updates after January 1, 2023.The stated source-file cutoff was January 1, 2025, or January 1, 2024 for Java.
- Project selection: Projects had to provide data under licenses such as MIT or Apache-2.0.
- Project selection: Holdout repositories required at least 40 stars, training projects more than five stars, and contributions from at least two developers.
2) Project execution-based filtering:
Execution-based filtering removes projects and samples that undermine reproducibility, runtime practicality, or training suitability. The resulting dataset includes human-written, generated, correct, broken, and no-existing-test subsets.
- Project execution-based filtering: Projects requiring external dependencies, installation steps, or extra downloads were excluded to preserve scalability and reproducibility.Tests running longer than 30 seconds were also removed, and tests were executed twice for stability and runtime estimation.
- Project execution-based filtering: Filtering removed samples with fewer than two tests, insufficient executable functions, unsuitable file lengths, generated files, or focal files exceeding 70% comments.
- Dataset structure: The final dataset contains 22,285 samples split into five training subsets and one validation subset without repository overlap.The validation subset contains 1,192 samples, or 5.34% of the total.
- Dataset structure: Human-correct samples comprise 3,303 instances, while human-broken samples comprise 4,349 instances.Broken examples include syntax errors, runtime errors, redundant logic, and missing verification logic.
- Dataset structure: Generated-correct samples comprise 11,543 instances, compared with 1,486 generated-not-correct samples.Generated tests were produced with Qwen2.5-Coder-32B-Instruct and evaluated individually after isolating test cases.
- Dataset structure: The no-existing-test subset contains 396 samples and is intended to reduce hallucination when only one new test is available.
A. Training configuration
RM-RF receives source, existing-test, and new-test code and predicts binary or float-formulated evaluation targets. Training used full fine-tuning for smaller models and LoRA for larger models, with target correlations found to be weakly positive.
- Training configuration: 7B models used full supervised fine-tuning, whereas 20B models used LoRA for parameter-efficient fine-tuning.Swift supported fine-tuning and vLLM supported inference.
- Training configuration: The model prompt contains the focal file, existing test file, and new test case in diff-style format.The instruction asks whether the new test is valid and increases source-code coverage.
- Training targets: Binary targets represent correctness, coverage increase, and mutation increase.Correctness means successful compilation and execution; coverage and mutation targets indicate improvement in the corresponding measures.
- Training targets: Float targets retain correctness while expressing coverage and mutation as ∆TestCov and ∆MutCov.Reversed binary targets were explored initially but omitted from final experiments because their behavior was similar.
- Training targets: The target correlations were weakly positive: 0.21 for Correctness–MutCov, 0.19 for Correctness–TestCov, and 0.35 for TestCov–MutCov.
V. EVALUATION
RM-RF evaluation compares target formulations, tuning strategies, execution alignment, and efficiency. Fine-tuned models achieve the strongest validation results, while RM-RF closely tracks execution-derived outcomes and substantially reduces evaluation time and compute.
- Evaluation setup: Weighted F1 evaluates binary and float targets, with continuous gains binarized as True when greater than 0.Class-frequency weights are used for the final weighted F1-score.
- Validation results: Fine-tuning consistently improves predictive quality over zero-shot baselines across settings.Both fine-tuned and zero-shot models were evaluated on the validation dataset.
- Validation results: 0.69 average F1 is achieved by Qwen2.5-Coder-7B-Instruct after SFT on binary targets, including 0.76 on ∆TestCov.This is the best reported average F1 in the validation comparison.
- Holdout results: Float targets generalize better on the Holdout dataset, while binary targets produce the strongest average validation results and stable cross-language performance.The reported languages are Go, Java, and Python; ∆MutCov improvements are most pronounced for Java, reaching 0.71 on binary targets.
- Tuning comparison: 0.69 average F1 for 7B SFT exceeds 0.63 for 14B LoRA on binary targets in this experimental regime.LoRA remains competitive for some metrics and is attractive when memory is constrained, but SFT provides the most reliable gains here.
- Error analysis: Missed Dependencies and Duplicated Entity errors are particularly difficult for the model to detect in intentionally incorrect generated tests.Invalid Constructor, Undefined Entity, Runtime Errors, and Invalid Call errors are relatively easier to identify.
- Execution alignment: 0.74 overall Spearman correlation indicates that RM-RF preserves relative test-case quality rankings against execution-derived metrics.The coefficients are 0.4 for Go and 0.6 for both Java and Python.
- Efficiency: Less than three hours are required for the 22B model to predict all samples, compared with several days for complete building and execution.RM-RF removes project building and source-code execution from the evaluation loop, reducing latency and computational resources.
VII. LIMITATIONS
The study identifies language coverage, evaluation design, and deployment validation as key limitations of RM-RF. Its dataset covers only three languages, omits language-specific training experiments, and does not test the reward model inside an RL pipeline; publicly available code may also permit partial pretraining overlap.
- The dataset and models are limited to Java, Python, and Go, leaving language coverage to be increased.
- The study does not evaluate models trained exclusively for one programming language, leaving their performance and practical maintainability unresolved.
- RM-RF was not tested within an RL pipeline, although the authors propose incorporating it to improve generated-test quality and training speed.
- The publicly available open-source dataset may introduce partial data leakage if models encountered its code during pre-training.
VIII. CONCLUSION
The conclusion presents RM-RF as a lightweight alternative to compile-and-run evaluation, predicting execution-related test outcomes from source and test code. Across Java, Python, and Go experiments, it achieved an average F1 score of 0.69 while reducing latency and resource consumption.
- RM-RF evaluates automatically generated unit tests without requiring compilation or execution.
- It predicts successful compilation, increased code coverage, and improved mutation kill rates directly from source and test code.
- 0.69 average F1 was achieved across Java, Python, and Go experiments and multiple model configurations.
- The authors report that RM-RF can provide rapid, scalable, and accurate feedback for automated test generation and reinforcement-learning-driven code optimization.