PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning

Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, Mengdi Wang

arXiv:2506.03136v2cs.CL

TL;DR

The paper asks whether coding and unit-test generation can co-evolve without ground-truth code solutions. CURE trains them through code-test interactions, and ReasonFlux-Coder models improve coding metrics after optimization.

  • Problem

    Can unit-test and code generators co-evolve to improve coding ability without access to ground-truth code solutions?

    Source: §1 · p. 3

  • Method

    CURE co-evolves a self-play model as code and unit-test generator using pairwise rewards from their interactions, without ground-truth code supervision.

    Source: §1 · p. 3

  • Results

    ReasonFlux-Coder 7B and 14B models improve code generation accuracy by 5.3% and Best of N accuracy by 9.0% after optimization on Qwen2.5-Instruct models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The trained unit-test generator can serve as a reward model for reinforcement learning, improving coding performance without human-labeled or ground-truth unit-test supervision.

    Source: §1 · p. 3

    The reward assumption can break down when generated unit tests have low accuracy, problem formulations are ambiguous, or tasks have binary outputs.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

We propose CURE, a novel reinforcement learning framework with a dedicated reward design that co-evolves coding and unit test generation capabilities based on their interaction outcomes, without any ground-truth code as supervision. This approach enables flexible and scalable training and allows the unit tester to learn directly from the coder's mistakes. Our derived ReasonFlux-Coder-7B and 14B models improve code generation accuracy by 5.3% and Best-of-N accuracy by 9.0% after optimization on Qwen2.5-Instruct models, outperforming similarly sized Qwen-Coder, DeepSeek-Coder, and Seed-Coder. They naturally extend to downstream tasks such as test-time scaling and agentic coding-achieving a 8.1% improvement over the base model. For the long-CoT model, our ReasonFlux-Coder-4B consistently outperforms Qwen3-4B while achieving 64.8% inference efficiency in unit test generation. Notably, we also find that our model can serve as an effective reward model for reinforcement learning on base models. Project: https://github.com/Gen-Verse/CURE

1 Introduction

CURE co-evolves coding and unit-test generation without ground-truth code, using interactions between generated code and tests for mutual supervision. The resulting models improve coding and related capabilities, including test-time scaling and agentic coding.

1 Introduction

  • Unit tests align directly with code correctness, can be reused across candidate solutions, and need not require a complete solution to construct.The paper presents generated tests as rewards for reinforcement learning and test-time scaling or agentic coding, and as a basis for self-check and self-correction.Source: §1 · p. 2
  • Training unit-test generators from ground-truth code is costly and labor-intensive, limiting the scale and diversity of usable data.CURE instead leverages the code generator to supervise the unit-test generator while improving the coder.Source: §1 · p. 3
  • CURE uses a single self-play agent as both coder and unit-test generator, with a pairwise reward matrix from their interactions enabling mutual supervision.The coder's incorrect solutions reveal failure modes that help the unit tester distinguish good code from bad code.Source: §1 · p. 3
  • 5.3% code-generation accuracy and 9.0% Best of N accuracy improvements were reported for ReasonFlux-Coder 7B and 14B after optimization on Qwen2.5-Instruct models.The models outperformed similarly sized Qwen-Coder, DeepSeek-Coder, and Seed-Coder.Source: §1 · p. 3
  • The trained unit-test generator can serve as a reward model for reinforcement learning on base models without human-labeled or ground-truth unit-test supervision.For long-CoT models, the paper introduces a response-length-guided reward transformation to improve test-time efficiency.Source: §1 · p. 3
  • Figure 3 depicts generating code and tests alongside ground-truth tests, building an execution table, extracting rewards, and iteratively optimizing both components.For long-CoT models, the pipeline transforms the reward to ensure efficiency.Source: Figure 3 · p. 4

2 Related Work

Prior work trains unit-test generators with code-derived supervision, applies generated tests to coding workflows, and uses reinforcement learning to improve LLMs. The paper positions CURE as compatible with existing coding-focused RL algorithms.

2.1 Unit Test Generation

  • Ground-truth-code datasets have been used to fine-tune unit-test generators, but relying on those solutions limits flexibility and scalability.O1-Coder and UTGEN are examples; UTGEN also uses incorrect tests from perturbed code.Source: §2.1 · p. 4

2.2 Application of Unit Tests

  • Generated unit tests support candidate selection, self-revision, iterative debugging, and pairwise discrimination in coding workflows.The cited approaches include selecting samples by execution results against generated tests and combining public and generated tests for revision.Source: §2.2 · p. 4

2.3 Reinforcement Learning for LLM Improvement

  • PPO, DPO variants, and GRPO offer different RL optimization approaches, and coding-focused RL methods can be integrated into CURE to optimize its coding component.The paper states that competing with existing coding-RL algorithms is not its aim.Source: §2.3 · p. 4

3 Method

The method develops a reward-precision objective for generated tests and uses execution outcomes to jointly optimize a coder and unit tester. Its analysis gives conditions for reward precision to improve as more tests are generated.

3 Method

  • The method section derives individual test rewards from reward precision, presents CURE's co-evolving RL framework, and adds a response-length-guided transformation for long-CoT efficiency.The sections cover the overall objective, theoretical analysis, joint optimization, and the long-CoT reward transformation.Source: §3 · pp. 4–5

3.1 Motivation: Using Unit Tests for Inference

  • For BoN coding, the method executes candidate solutions against generated tests to form a binary evaluation matrix used to select a solution.The reward precision objective is proposed because generated tests may be low-accuracy, problem formulations may be ambiguous, or tasks may have binary outputs.Source: §3.1 · p. 5

3.2 Analysis on Reward Precision

  • The reward-precision analysis models solution and test correctness as Bernoulli variables and shows convergence depends on conditions involving p_u, p_00, and p_01.The method uses μ as the unit-test optimization objective, estimating each test's individual value from the execution matrix.Source: §3.2 · pp. 5–6
  • As the number of generated tests m grows, the aggregated reward ranks correct solutions above incorrect ones with probability approaching 1 if and only if μ > 0.Under this condition, larger μ means fewer tests are needed for a reliable reward signal.Source: §3.2 · p. 5

3.3 Co-evolving Coder and Unit Tester with RL

  • CURE executes generated solutions against generated and ground-truth tests in one matrix, then uses it to estimate rewards for both the coder and tester.Coder rewards increase with the number of ground-truth tests passed; each generated test receives an estimated μ-based reward.Source: §3.3 · p. 6
  • The test reward favors tests that reject incorrect solutions while passing correct ones, avoiding incentives for trivial or overly permissive tests.A correct solution passes all ground-truth tests, while an incorrect solution fails at least one.Source: §3.3 · p. 6

Algorithm 1 CURE

The algorithm repeatedly generates code solutions and unit tests for each task, evaluates their interactions, assigns rewards, and fine-tunes its policy. For long-CoT unit testing, it adjusts rewards using response lengths and truncates responses beyond 8K tokens.

Algorithm 1 CURE

  • For each task, the policy generates code solutions and unit tests whose executions form a binary evaluation matrix, then uses their rewards to compute advantages and fine-tune the policy.The algorithm takes the task set, policy, iteration count, numbers of generated solutions and tests, learning rate, and KL coefficient as inputs.Source: Algorithm 1 CURE · p. 7

3.4 Improve Efficiency of Long-CoT Unit Tester

  • For long-CoT unit testing, response-length-guided reward transformation penalizes overly long responses while aiming to keep originally positive and negative rewards on their respective sides of zero.The method standardizes rewards, uses response lengths in the transformation, then balances and normalizes the transformed rewards.Source: §3.4 · pp. 7–8
  • Responses longer than 8K tokens are truncated to their first 8K tokens during training, alongside the reward adjustment intended to penalize excessive length while retaining some original reward information.Source: §3.4 · p. 8

4 Experiments

CURE is evaluated across coding benchmarks, model scales, and test-time and agentic settings; its optimized models improve coding and unit-test performance, with further gains in long-CoT, API-based, and label-free RL settings. Ablations favor CURE’s co-evolution and reward design over the tested alternatives.

4.1.1 Datasets; 4.1.2 Models and Optimization

  • The evaluation uses five coding datasets and derives 7B and 14B models from Qwen2.5-Instruct and the long-CoT 4B model from Qwen3-4B.CodeContests tasks at difficulty ≤2 are split into 4.5k training and 200 evaluation examples; LiveCodeBench uses version 2 with 511 problems.Source: §4.1.1 · p. 8, §4.1.2 · p. 8

4.1.3 Test-time Scaling and Agentic Coding; 4.1.4 Agentic Unit Test Generation

  • BoN selects among generated code candidates using generated unit tests, while agentic coding pipelines also include iterative refinement or debugging.For agentic unit-test generation, tests are generated first and iteratively refined based on execution results on the corresponding code.Source: §4.1.3 · pp. 8–9, §4.1.4 · p. 9

4.2.1 CURE Significantly Improves the Overall Coding Ability

  • 37.8% higher unit-test accuracy, 5.3% higher one-shot code accuracy, and 9.0% higher BoN accuracy than the base models were achieved by ReasonFlux-Coder-7B and 14B.The models also consistently outperform Qwen2.5-Coder-Instruct across all three metrics and improve BoN performance across sampling settings.Source: §4.2.1 · p. 9

4.2.2 Robust for Long-CoT Models and Achieves Inference Efficiency

  • ReasonFlux-Coder-4B consistently outperforms Qwen3-4B in unit-test, code, and BoN accuracy, while unit-test generation response length falls to 64.8% of its original length.The authors report that standard base models gain more than long-CoT models, which already capture much of the benefit from chain-of-thought reasoning.Source: §4.2.2 · pp. 9–10

4.2.3 Boosts API-inference Models’ Performance and Cost-efficiency

  • ReasonFlux-Coder-4B as unit tester improves GPT-4o-mini and GPT-4.1-mini BoN accuracy by averages of 5.5% and 1.8%, respectively.With GPT-4o-mini as coder, it yields a 7.0% improvement over GPT-4o one-shot performance while reducing cost; the table reports cost and accuracy by benchmark.Source: Table 2 · p. 9, §4.2.3 · pp. 10–11

4.2.4 Facilitates Label-free RL

  • Using ReasonFlux-Coder-4B-generated tests to supervise Qwen2.5-14B-Instruct RL produces improvements comparable to ground-truth labeled tests across code, unit-test, and BoN accuracy.The comparison trains both models for 150 steps and uses 16 code and 16 unit-test samples in BoN.Source: §4.2.4 · p. 11, Figure 7 · p. 12

4.2.5 Broad Application to Test-time Scaling and Agentic Coding Methods

  • Across MPSC, AlphaCodium, and S*, ReasonFlux-Coder-14B averages an 8.1% improvement over Qwen2.5-14B-Instruct; agentic unit-test accuracy improves by 25.1%.The agentic test-generation setting refines tests using execution results; Figure 6 evaluates its panels on LiveBench with Qwen2.5-14B-Instruct as base.Source: §4.2.5 · p. 11, Figure 6 · p. 11

4.2.6 Ablation Study on Optimization Methods and Reward Designs

  • CURE outperforms coder-only optimization, positive-reward supervised fine-tuning, and the simplified unit-test reward across the reported ablations.The theoretically derived reward constrains p01 and p00 to 36.5% and 9.1%, versus 42.2% and 14.7% under the simple reward, improving selection precision and solution ranking.Source: §4.2.6 · pp. 11–12

5 Discussions

CURE co-evolves coding and unit-test generation without ground-truth code, using a theoretically derived reward for the unit tester. The discussion reports gains across coding, testing, downstream applications, and reward-model use.

5 Discussions

  • ReasonFlux-Coders improve test-time scaling by 8.1% and agentic unit test generation by 25.1%, and are also effective as reward models for reinforcement learning.The discussion also reports significant improvements in code and unit-test generation across five benchmarks.Source: §5 · p. 12
  • CURE co-evolves coding and unit-test generation without ground-truth code, using a theoretically derived reward for the unit tester.The discussion characterizes this approach as enhancing training flexibility and scalability.Source: §5 · p. 12
  • ReasonFlux-Coder-4B consistently outperforms Qwen-4B while achieving significantly higher efficiency in unit-test generation.The discussion identifies scaling CURE through self-supervision without labeled data as a promising future direction.Source: §5 · p. 12

A Proofs of Theoretical Results

The proof models each test as a random comparison between a correct and an incorrect solution, then studies the sign of their cumulative reward difference. The expected difference determines the asymptotic ranking outcome, with an exponential finite-sample guarantee when it is positive.

A Proofs of Theoretical Results

  • Each test contributes +1 when the correct solution wins, −1 when the incorrect solution wins, and 0 for a tie; independent tests make these contributions i.i.d.The expected difference is μ = p_u(1 − p_01) − (1 − p_u)p_00.Source: A Proofs of Theoretical Results · p. 16
  • When the expected per-test reward difference μ is positive, the cumulative difference is eventually positive almost surely, so the probability of the correct solution ranking higher tends to 1.When μ is negative, the cumulative difference is eventually negative almost surely, and that probability tends to 0.Source: A Proofs of Theoretical Results · pp. 16–17
  • For positive μ, the proof applies Hoeffding’s inequality to obtain an exponential guarantee at finite test counts.The result is stated for the case μ > 0.Source: A Proofs of Theoretical Results · p. 17

B Additional Experimental Results

The additional experimental tables describe error-analysis and response-length reporting, but the supplied captions do not include outcome values. They specify the reported measures and evaluation-run counts.

B Additional Experimental Results

  • Table 3 reports accuracy improvement over the base model with standard errors, but its caption provides no cell values.It is described as the error-analysis table corresponding to Table 1.Source: Table 3 · p. 19
  • Unit-test and code accuracies in Table 3 are evaluated over 16 independent runs, while BoN accuracy uses one run per benchmark.The caption notes that BoN scaling is computationally intensive.Source: Table 3 · p. 19
  • Table 4 reports response length in tokens for Qwen3-4B and ReasonFlux-Coder-4B on unit-test generation.The caption links it to Figure 2(e) but supplies no response-length values.Source: Table 4 · p. 19

C Details of Experiments

The experiment prompts ask the coder to produce a Python solution and the unit tester to create a task-derived test example. The tester prompt emphasizes correctness, discriminative power, and a specified output format.

C.1 Prompt Design

  • The code-generation prompt instructs the model to think first, then write a Python script using input() and print().The problem statement is inserted into the prompt.Source: §C.1 · p. 20
  • The unit-test prompt asks for a new test example with input, output, and explanation, designed to distinguish correct code from incorrect code.It instructs the model to derive and check the example carefully before presenting it in the prescribed format.Source: §C.1 · p. 20

C.2 Preprocess Data

The experiments standardize inputs and outputs in stdio format, convert selected functional-format tasks to that format, and use ground-truth code from benchmark sources or filtered model generations for evaluation.

C.2 Preprocess Data

  • The experiments convert functional input/output formats to stdio by placing each variable on a separate line and flattening lists into space-separated values.This conversion covers some LiveBench and LiveCodeBench tasks and all MBPP tasks.Source: §C.2 · p. 20
  • Evaluation uses provided ground-truth code for CodeContests and MBPP, and QwQ-32B-generated code that passes all ground-truth tests for Codeforces, LiveCode, and LiveCodeBench.The QwQ-32B code is selected using BoN with a maximum of 3 samples.Source: §C.2 · p. 21

C.3 Test-time Scaling and Agentic Coding

This section describes the application of MPSC, AlphaCodium, and S* to test-time scaling and agentic coding, with method-specific sampling and refinement procedures.

C.3 Test-time Scaling and Agentic Coding

  • The section covers applying MPSC, AlphaCodium, and S* in test-time scaling and agentic coding.Source: §C.3 · p. 21

C.3.1 MPSC

  • For MPSC, each task receives 8 code, unit-test, and specification samples, which are used to derive consistency scores through iterative optimization.A specification is a pre-condition/post-condition function pair defining valid inputs and expected program behavior.Source: §C.3.1 · p. 21

C.3.2 AlphaCodium

  • For AlphaCodium, 8 code solutions are reasoned over public tests and paired with unit tests, then refined in two public-test and two generated-test execution-feedback iterations.A separate procedure uses 4 rounds of self-debugging on public tests before pairwise selection using generated tests.Source: §C.3.2 · p. 21, §C.3.3 · p. 21

C.4 Agentic Unit Test Generation Methods

The agentic unit-test procedure iteratively refines tests using execution feedback, and the appendices illustrate generated tests for the 14B and long-CoT 4B models.

C.4 Agentic Unit Test Generation Methods

  • The agentic unit-test pipeline follows iterative refinement and generation, prompting the policy model to revise a test after execution failure, for up to 3 iterations.For this function-specific task, the input includes both the task description and ground-truth code.Source: §C.4 · p. 21

C.5.2 Examples for ReasonFlux-Coder-4B (long-CoT)

  • The 4B course-selection example initially reports 10, then recognizes that two nonoverlapping same-day courses yield 11.Source: §C.5.2 · pp. 27–28
Loading 2506.03136v2…