Source-linked AI summary
Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping
Yiwen Zhang, Xiaodong Yan, Zhenyu Huang, Deng Zhao, Liang Jiang, Qing Cui, Zujie Wen, Zhiqiang Zhang, Jun Zhou
TL;DR
RLVR for code generation is limited by incomplete and noisy test cases that can distort training feedback. RobustTests synthesizes tests from near-correct faulty programs and applies validation, behavioral filtering, and dense rewards; Qwen3-32B training with it achieved an absolute 3% performance gain on LiveCodeBench over baselines.
Problem
RLVR coding performance is limited by incomplete, poorly characterized, and hallucination-prone test cases that can produce misleading reward feedback.
Method
RobustTests drives test synthesis with near-correct faulty programs, validates and behaviorally filters tests, and uses stepwise dense rewards based on pass rates.
Results
RobustTests improved augmented-test diagnostic coverage and yielded an absolute 3% performance gain on LiveCodeBench for Qwen3-32B versus baseline methods.
Takeaways & Limitations
The findings support RobustTests as an effective framework for improving code-generation proficiency through higher-quality test cases and robust RL feedback.
Takeaways & Limitations
The approach primarily requires programming tasks with available ground-truth solutions, limiting utility when reference implementations are absent.
Abstract
from arXiv · showhide
Reinforcement learning from verifiable rewards (RLVR) has emerged as a pivotal technique for enhancing the code generation capabilities of Large Language Models (LLMs). However, the efficacy of RLVR in coding implementations is fundamentally limited by the comprehensiveness of test cases, because insufficient test coverage in code validation often causes false positives, further leading to reward hacking and policy degradation. To mitigate the reward bias stemming from the suboptimal quality of current automated generation methods, we propose the RobustTests framework, which introduces a faulty-code-driven test case synthesis strategy that leverages "near correct" faulty codes to guide the model in precisely capturing latent logical discrepancies and further integrates validator agents with behavioral feature clustering to facilitate the granular filtering of invalid and redundant test cases. To address false negatives caused by inherent hallucination noise in synthetic test cases, RobustTests also incorporates a stepwise dense reward function based on pass rates, bolstering training robustness through fine-grained feedback. By employing this pipeline, we construct a high-quality dataset that augmented the test cases in CodeContests, encompassing a broader spectrum of faulty code scenarios and significantly enhances diagnostic utility. Experimental results demonstrate that, by leveraging a moderately challenging subset of problems from CodeContests for training, RL fine-tuning of Qwen3-32B via RobustTests achieves an absolute 3% performance gain on the LiveCodeBench benchmark compared to baseline methods, confirming the effectiveness of the RobustTests framework in advancing the code generation proficiency of LLMs.
1 Introduction
RobustTests addresses test-coverage and reward-bias limitations in RLVR through faulty-code-driven synthesis and stepwise dense rewards. It augments CodeContests+ with broader diagnostic coverage and improves Qwen3-32B performance on LiveCodeBench.
- Motivation: RLVR effectiveness is limited by insufficient test coverage, which permits faulty code to pass sparse tests and causes reward hacking and policy degradation.LLM-generated tests may also miss boundary conditions or violate problem constraints, producing biased verification signals.
- RobustTests framework: RobustTests synthesizes tests from near-correct faulty programs refined with original tests to target specific logical defects.The framework also uses validator agents and behavioral feature clustering to filter invalid and redundant tests.
- RobustTests framework: RobustTests uses a stepwise dense reward function based on pass rates to support learning from partially correct signals despite hallucinated test-case noise.This mechanism targets false negatives caused by unavoidable errors in synthetic test suites.
- Dataset and evaluation: 7% improvement in TSP relative to the original CodeContests+ indicates broader coverage of failure modes in the augmented dataset.The dataset contains more diverse test cases and strengthens diagnostic utility across faulty-code scenarios.
- Dataset and evaluation: 3% absolute performance gain on LiveCodeBench was achieved by RL fine-tuning Qwen3-32B with RobustTests compared to baseline methods.The experiments used moderately difficult CodeContests+ problems for training and reported a correlation between TSP and model performance.
2 Related work
Related work contrasts highly accurate but costly human-curated test cases with scalable LLM-based synthesis methods, while coding-oriented RL uses executable feedback, compiler signals, or unit tests for optimization.
- Test case synthesis: Human experts currently produce the most accurate test cases, but manual curation is prohibitively expensive and unsuitable for massive training corpora.This approach underpins benchmarks including MBPP, HumanEval, and LiveCodeBench.
- Test case synthesis: LLM-based methods improve scalability through stochastic input perturbations or seed-input synthesis guided by reference implementations.Examples include CodeContests+, EvalPlus, KodCode, and AceCoder.
- Reinforcement learning: Coding-oriented reinforcement learning leverages code executability to provide objective feedback for model optimization.Early systems such as CodeRL and AlphaCode used compiler feedback or unit tests as reward signals with algorithms including PPO.
3 Method
RobustTests addresses RLVR reward bias through faulty-code-driven test synthesis, validation and diversity-based selection, then uses pass-rate-based dense rewards to improve robustness to residual test-case noise.
- Automated Test case Generation: RobustTests generates tests by first sampling faulty implementations and retaining nearly correct code that exposes logical defects.Faulty candidates are evaluated on the base suite, while fully correct and completely incorrect implementations are filtered out.
- Automated Test case Generation: Deduplicating faulty implementations by execution vector preserves one representative per failure mode and reduces semantic redundancy.The resulting pool is designed to cover diverse logical discrepancies.
- Automated Test case Generation: Faulty-code-driven prompting generates tests that remain correct on the reference solution while inducing failures in at least one faulty implementation.Generated tests and base-suite tests form the candidate pool for subsequent filtering.
- Test case Validation and Selection: A three-stage pipeline validates inputs, checks instruction compliance, and selects diverse tests using failure-profile clustering to maximize diagnostic coverage and parsimony.K-means partitions test cases by binary execution vectors, followed by round-robin medoid selection.
- Dense Reward: The reward mechanism uses synthesized-test pass rates to provide granular feedback despite residual validator loopholes and reference-solution flaws.This supports resilience to test-case noise and stable curriculum learning.
4 Experiment
Experiments evaluate RobustTests against alternative test-case augmentation strategies on LiveCodeBench and CodeForces. RobustTests improves performance through diverse, diagnostically stronger tests and stepwise dense rewards that mitigate false positives and noisy test cases.
- Performance on LiveCodeBench and CodeForces: RobustTests outperforms HardTests and Naive LLM Generation by approximately absolute 3% on both LiveCodeBench and CodeForces.The gains are attributed to higher-quality tests that detect logical discrepancies and dense rewards that provide intermediate feedback despite hallucinated tests.
- Diversity and TSP: Higher TSP values correspond to stronger LiveCodeBench performance, with RobustTests achieving the highest TSP among compared test suites.Lower-TSP methods such as HardTests and CodeContests+ produce weaker downstream performance.
- Ablation Study: Test Case Synthesis and Diversity-Driven Selection each yield approximately 1.5% gains on LiveCodeBench when applied independently, while their integration performs best for Qwen3-32B.The experiments fix the test-case budget at approximately 40 and use binary sparse rewards.
- Dense Reward Analysis: Dense rewards consistently improve LiveCodeBench performance with RobustTests or CodeContests+, with larger gains under no-validator RobustTests* settings across LiveCodeBench and Codeforces metrics.Setting the reward scale to zero causes a substantial performance drop, whereas scales from 0.05 to 0.20 maintain stable performance.
- Validator Filtering: Validator filtering rejects approximately 30% of raw generated test cases, but about 10% of accepted cases remain invalid and cause false-negative judgments for roughly 10% of all test cases.No reference-execution failures are observed, indicating that residual invalidity remains despite filtering.
5 Conclusion
RobustTests establishes a robust code-generation RL framework by combining faulty-code-driven test synthesis with stepwise dense rewards. It expands diagnostic coverage, reduces false positives, and mitigates false negatives through partially correct signals and curriculum learning.
- 5 Conclusion: RobustTests combines faulty-code-driven test case synthesis with a stepwise dense reward mechanism for robust reinforcement learning in code generation.The framework is designed to improve the diagnostic utility of the augmented CodeContests+ dataset.
- 5 Conclusion: Near-correct faulty codes expand diagnostic coverage and effectively minimize false positives.
- 5 Conclusion: Stepwise dense rewards mitigate false negatives by letting models learn from partially correct signals through curriculum learning.
Limitations
RobustTests remains limited by its reliance on ground-truth solutions and by the need to establish broader generalization beyond competitive programming benchmarks.
- Solution dependence: RobustTests primarily applies to programming tasks with available ground-truth solutions, limiting utility where reference implementations are absent.This dependency constrains applicability in real-world scenarios lacking reference solutions.
- Domain generalization: The framework’s generalizability beyond competitive programming benchmarks such as LiveCodeBench and CodeForces remains to be fully explored.Broader software development tasks are not yet comprehensively evaluated.
A Training Settings
Experiments use Qwen3-32B as the base model on 32 NVIDIA H200 GPUs, with detailed hyperparameters summarized in Table 4.
- A Training Settings: Experiments run on 32 NVIDIA H200 GPUs using Qwen3-32B as the base model.Detailed experimental parameters are summarized in Table 4.
B Evaluation Settings … D.2 Original Test cases
The paper evaluates its test-synthesis pipeline under standardized long-context settings, compares several baseline generation strategies, and illustrates the synthesis process through a CodeContests+ case study. The case study specifies a tree-reconstruction problem whose original base suite contains 177 test cases.
- B Evaluation Settings: Evaluations on LiveCodeBench (2024.08–2025.01) and CodeForces use a 128K context window.Qwen3-32B uses YaRN for context extension and follows the official open-weight hyperparameters.
- C Introduction of Baselines: Naive LLM Generation directly synthesizes test-case suites, then validates and revises them against the reference solution.This baseline is described as conforming generated suites to the problem specifications before validation.
- C Introduction of Baselines: HardTests prompts an LLM to produce generator programs that synthesize inputs, with outputs obtained from the reference solution.The approach uses dedicated generator programs rather than directly generating a test-case suite.
- C Introduction of Baselines: CodeContests+ uses a generator-validator multi-agent framework to produce and validate inputs before obtaining outputs from the reference solution.Both the generator program and input validator are generated by the LLM.
- C Introduction of Baselines: CodeContests-O closes the synthesis loop by using execution feedback from correct and incorrect solutions to refine test cases for fidelity and discriminability.This baseline is characterized as a Feedback-Driven Iterative Framework.
- D Case Study: The case study uses a representative CodeContests+ problem to explain synthesis mechanics, component necessity, and causes of invalid test cases.The selected problem is p03520.
- D.2 Original Test cases: The original base test suite for this problem contains 177 test cases, denoted as |Tbase| = 177.The section also provides an illustrative example of the suite.
D.3 Faulty Codes and Generated Test cases … D.6 Diversity-Driven Selection
The framework generates faulty-code-guided tests, validates and refines them, then selects a diverse final subset. This pipeline produces a calibrated, constraint-compliant, discriminative test suite for training and evaluation.
- D.3 Faulty Codes and Generated Test cases: |Fbase| = 12 qualified faulty implementations are generated to guide synthesis of discriminative test cases.The faulty implementations are intended to expose latent logical discrepancies between correct and faulty solutions.
- D.3 Faulty Codes and Generated Test cases: 75 test cases are synthesized to discriminate between correct and faulty implementations.The generated cases are subsequently merged with the base test suite.
- D.3 Faulty Codes and Generated Test cases: |Tcand| = 252 after synthesized test cases are merged with the base suite, with outputs requiring reference-solution calibration.The candidate set combines synthesized and existing tests before refinement.
- D.4 Input Validator: The input validator checks bounds, self-loops, duplicate edges, cycles, connectedness, and the s_i vector format.It enforces valid tree structure and input constraints before test cases enter refinement.
- D.5 LLM Instruction-compliance Validation: |Tfinal| = 248 after outputs are calibrated with the reference solution and non-discriminative cases are pruned.Two test cases passed by faulty implementations and were filtered out during two-stage refinement.
- D.6 Diversity-Driven Selection: |Tsynth.| = 40 representative test cases are selected for the experimental test suite.The selection follows the formal diversity-driven procedure described in Appendix E.
D.7 Limitations of the Synthesized Test cases · E Algorithm definition
The synthesized tests can hallucinate invalid cases and reject correct solutions, creating false positives and false negatives. RobustTests addresses these issues with stepwise dense rewards and clustering-based test-case filtering.
- D.7 Limitations of the Synthesized Test cases: Missing validation of valid integer side lengths causes some synthesized test cases to be falsely accepted by the input validator.The issue arises for arrays s, with hallucinated test cases serving as an example.
- D.7 Limitations of the Synthesized Test cases: Hallucinated test cases illustrate how inadequate validation can admit invalid inputs into the synthesized test set.The passage specifically attributes this failure to the absence of a valid-integer-side-length check.
- D.7 Limitations of the Synthesized Test cases: Test cases that reject correct Qwen3-32B answers produce false negatives during evaluation.This failure reflects imperfect synthesized tests rather than incorrect model answers.
- D.7 Limitations of the Synthesized Test cases: A stepwise dense reward function is used to improve model robustness against imperfect test cases.The reward design is introduced specifically to address false negatives caused by test-case rejection of correct answers.
- E Algorithm definition: Clustering techniques filter test cases to enhance diversity in the Section 3.2 pipeline.The corresponding procedure is presented as pseudo-code in Algorithm 1.
- E Algorithm definition: Algorithm 1 provides pseudo-code for the clustering-based test-case filtering procedure.The procedure operationalizes the Section 3.2 strategy for improving test-case diversity.
F Implementation Details … F.3 Input Validator Generation
The implementation uses Kimi-k2-based prompt templates across faulty-code generation, faulty-code-driven test synthesis, and input validation. Faulty implementations are sampled for diversity, while validators enforce specification compliance and the accompanying selection procedure refines the test suite.
- F Implementation Details: Kimi-k2 underlies prompt templates for faulty-code generation, test-case synthesis, and input-validator generation.These templates correspond to the three stages described in Section 3.
- F.1 Faulty Code Generation: Three distinct faulty-code prompts are each sampled three times, producing nine model invocations per problem.The multi-prompt design targets high intra-class diversity among generated faulty implementations.
- F.2 Faulty-Code-Driven Test Case Generation: The faulty-code-driven test-generation prompt produces discriminative cases that distinguish correct from faulty implementations.The prompt is provided in Figure 11.
- F.3 Input Validator Generation: Input validators are constructed to ensure synthesized test inputs strictly follow the problem specifications.The corresponding prompt is provided in Figure 12.
- F.3 Input Validator Generation: Diversity-driven test-case selection refines an initial suite Tfinal into a synthesized suite Tsynth under cluster-count and budget constraints.Algorithm 1 specifies Tfinal, K, and M as inputs and Tsynth as the refined output.
- F.3 Input Validator Generation: The selection procedure represents tests with binary execution vectors and partitions them into behavior-based clusters.The fragments describe constructing V^t and partitioning Tfinal into clusters {C1, C2, . . . , CK′}.
- F.3 Input Validator Generation: Within each cluster, candidates are ordered by ascending centroid distance, and selection continues while the synthesized suite remains below its budget and the initial suite size.The procedure iterates over clusters and applies the stated stopping conditions.