PaperlayerView exact revision on arXiv

Source-linked AI summary

ReVeal: Self-Evolving Code Agents via Reliable Self-Verification

Yiyang Jin, Kunzhao Xu, Hang Li, Xueting Han, Yanmin Zhou, Cheng Li, Jing Bai

arXiv:2506.11442v2cs.SEcs.LG

TL;DR

Outcome-only RLVR does not explicitly optimize verification, contributing to unreliable self-verification and limited test-time scaling. ReVeal trains code generation and self-verification together in iterative, tool-evaluated turns; on LiveCodeBench it supports refinement beyond its training horizon and improves Pass@k over the base model.

  • Problem

    RLVR relies on outcome rewards without explicitly optimizing verification, while prior verification approaches provide limited or non-generalizable signals for multi-turn refinement.

    Source: §1 · p. 2

  • Method

    ReVeal uses multi-turn reinforcement learning to co-train code generation and self-verification with constructed tests, tool feedback, and turn-level credit assignment.

    Source: §1 · p. 2, §2.2.2 · p. 5

  • Results

    20+ inference turns after three training turns on LiveCodeBench; ReVeal also significantly outperforms the base model in Pass@k.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support ReVeal as a framework for long-horizon reasoning in tasks with verification-generation asymmetry.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has advanced the reasoning capabilities of large language models. However, existing methods rely solely on outcome rewards, without explicitly optimizing verification or leveraging reliable signals from realistic environments, leading to unreliable self-verification and limited test-time scaling. To address this, we widen the verification-generation asymmetry by explicitly optimizing self-verification, making it a reliable driver of deeper test-time scaling. We introduce ReVeal, a multi-turn reinforcement learning framework that evolves code generation through self-verification and tool-based evaluation. ReVeal structures long-horizon reasoning as iterative generation-verification turns and incorporates TAPO for turn-level credit assignment, fostering the co-evolution of code and test generation. At inference, this strengthened self-verification enables the model to use self-constructed tests and tool feedback to continuously evolve code for 20+ turns on LiveCodeBench despite training on only three. It also significantly improves Pass@k, indicating stronger exploration that expands the reasoning boundaries of the base model. These findings highlight the promise of ReVeal as a scalable paradigm for RL training and test-time scaling, paving the way for more robust and autonomous AI agents.

1 INTRODUCTION

Although RLVR has strengthened LLM reasoning, outcome-only training leaves self-verification unreliable and limits test-time scaling. ReVeal explicitly trains verification alongside generation, enabling code agents to refine solutions over substantially more inference turns than training turns.

1 INTRODUCTION

  • Existing RLVR relies on outcome rewards without explicitly optimizing verification, while earlier studies report performance plateaus beyond the training horizon.The paper links this gap to unreliable self-verification, including uninformative reflection or guessing on hard problems.Source: §1 · pp. 1–2
  • Prior multi-turn approaches use separate critics without tool feedback or depend on public tests that are rarely available in real-world scenarios.The paper characterizes these approaches as providing limited, non-generalizable verification.Source: §1 · p. 2
  • ReVeal explicitly optimizes self-verification to widen the verification-generation gap, which Figure 2 presents as enabling deeper test-time scaling.The framework is intended to let verification drive sustained improvements in generation.Source: §1 · p. 2, Figure 2 · p. 2
  • ReVeal alternates code generation with self-verification using constructed tests and external tools, then applies turn-level rewards and TAPO to assign credit.TAPO is designed to prevent reward gaming, such as generating trivial code to exploit verification rewards.Source: §1 · p. 2
  • 20+ inference turns after only three training turns on LiveCodeBench; ReVeal also significantly outperforms the base model in Pass@k.The paper reports that verification signals and tool feedback guide exploration beyond the base model’s reasoning boundaries.Source: §1 · p. 2

2 METHODS

ReVeal trains code generation and self-verification through tool-supported iterative turns, then uses joint rewards and turn-aware credit assignment to guide their co-evolution. Its design aims to make verification a reliable signal for multi-turn refinement.

2.1.1 ITERATIVE GENERATION-VERIFICATION LOOP

  • ReVeal uses a shared policy to alternate candidate-code generation and test-based verification, appending tool feedback to guide subsequent turns without external critics or predefined tests.The loop ends when a valid solution is found or the turn budget is reached.Source: §2.1.1 · p. 3

2.1.2 TOOL-AUGMENTED VERIFICATION

  • Tool execution supplies fine-grained feedback for refining code and verification strategies, while training filters generated tests through golden solutions and test-time verification runs autonomously.The paper reports consistently higher Pass@k than the base model with tool interaction.Source: §2.1.2 · pp. 3–4

2.2 TURN-AWARE RL FOR THE GENERATION-VERIFICATION PARADIGM

  • ReVeal explicitly optimizes verification because outcome-only rewards give imprecise credit to intermediate verification and can lead to blind reflection.The authors propose that more reliable verification signals can guide generation across multiple test-time turns.Source: §2.2 · p. 4

2.2.1 JOINT VERIFIABLE REWARDS

  • ReVeal combines outcome, generation, and verification rewards to supervise final code quality, improvement across generation turns, and the quality of tests executed on golden code.The outcome reward includes a format reward and a pass-rate reward; the generation reward is configured to encourage accuracy improvements across turns.Source: §2.2.1 · p. 4

2.2.2 TURN-AWARE POLICY OPTIMIZATION

  • TAPO replaces PPO’s GAE-based advantages with a turn-aware return that bootstraps from token- and turn-level returns, assigning generation rewards to the generation turn and preceding verification turn.Verification rewards remain confined to their own turns; the authors state this prevents reward gaming and supports stable co-evolution of code and tests.Source: §2.2.2 · p. 5

3 EXPERIMENTS

ReVeal is evaluated on code-generation benchmarks and compared with outcome-only RL, critic-based methods, and prompt-only tool use. Results cover benchmark performance, longer inference horizons, and the joint development of code and verification capabilities.

3.1 SETTINGS

  • The training set contains 11,151 problems and the test set 509, after filtering unsupported formats and checking golden solutions.The data come from TACO; training uses DAPO-Qwen-32B with at most three turns, and evaluation covers LiveCodeBench and CodeContests.Source: §3.1 · pp. 5–6
  • Evaluation includes Pass@1, correction rate ∆↑, degradation rate ∆↓, and Pass@k up to k = 128; the baselines include a separate-critic method and outcome-only RL.A prompt-only ReVeal variant using the same prompt and execution tool does not sustain effective deep multi-turn refinement.Source: §3.1 · p. 6, §3.2 · p. 7

3.2 MAIN RESULTS

  • On LCB V6, ReVeal reaches 34.8% Pass@1 at turn 1 and surpasses single-turn RL; versus critic-based CTRL, it achieves larger correction rates with near-zero degradation.Table 1 compares performance on LiveCodeBench V6 and CodeContests.Source: §3.2 · pp. 6–7, Table 1 · p. 7

3.3 ANALYSIS

  • TAPO with joint verifiable rewards improves Pass@1 and ∆↑ while suppressing ∆↓ versus outcome-only training at the same turn budget.The analysis reports larger gains with longer sequences and harder verification, where outcome-only signals provide imprecise credit to intermediate steps.Source: §3.2 · p. 7, §3.3 · pp. 8–9

3.2 MAIN RESULTS

  • On Qwen2.5-32B-Instruct, ReVeal outperforms single-turn RL by 4.1%; the stronger DAPO-Qwen-32B backbone gains more from deeper inference.The authors repeated the experiment eight times and reported mean ± standard deviation in the appendix.Source: §3.2 · p. 7

3.3 ANALYSIS

  • Pass@1 rises from 34.8% at turn 1 to 36.7% at turn 3 and 38.7% at turn 25 on LiveCodeBench, although training used at most three turns.The extended-turn evaluation tests whether refinement continues beyond the training horizon.Source: §3.3 · p. 7
  • ReVeal outperforms both the base model and single-turn RL across Pass@k values from 1 to 128 on LiveCodeBench.The paper attributes this to verification-driven exploration guided by tool-assisted, execution-based feedback.Source: §3.3 · p. 8
  • After format learning, test-case accuracy rises from about 50% to nearly 88%, and correctness judgments exceed 85% on correctly generated test cases.Final-turn code accuracy improves over first-turn accuracy, with the gap widening during training.Source: §3.3 · p. 8, Figure 4 · p. 8

4 RELATED WORK

Prior tool-augmented and self-verification methods provide context for ReVeal’s focus on explicitly optimizing verification. ReVeal instead generates and executes its own tests, avoiding dependence on pre-existing test suites.

4.1 TOOL-AUGMENTED REASONING

  • Prior tool-augmented RL methods use search or code execution in multi-turn reasoning, but typically rely on outcome rewards without explicit verification optimization or turn-level credit.The paper describes ReVeal and prior tool-augmented approaches as complementary and potentially combinable.Source: §4.1 · p. 9

4.2 SELF-VERIFICATION OF LLMS

  • Separate critic models add the cost and complexity of coordinating two models, while RLEF depends on publicly available tests.The cited examples are CTRL and RLEF (Gehring et al., 2025).Source: §4.2 · p. 9
  • ReVeal has the model generate and execute its own tests, eliminating dependence on pre-existing test suites and improving applicability to real-world software systems.Source: §4.2 · p. 9

5 CONCLUSION

The conclusion presents ReVeal as a multi-turn RL framework that jointly optimizes generation and verification, with tool feedback supporting refinement beyond the training horizon. It also identifies conditions under which the approach may apply beyond code and discloses the authors’ limited use of an LLM.

5 CONCLUSION

  • ReVeal uses TAPO with joint verifiable rewards to support code self-evolution during training and iterative refinement for 20+ inference turns.The authors state that newly discovered solutions can be distilled back into the code LLM through continued training.Source: §5 · p. 10
  • The framework’s generation-verification design may apply beyond code where both capabilities have verifiable rewards and the domain exhibits verification asymmetry.Source: §5 · p. 10

A THE USE OF LARGE LANGUAGE MODELS

B.1 HYPERPARAMETERS

The appendix identifies tables listing training and evaluation hyperparameters for the two model variants.

B.1 HYPERPARAMETERS

B.2 CASE STUDY OF REVEAL

The appendix presents a ReVeal multi-turn case study and details its generation-verification prompt and tool-feedback templates.

B.3 PROMPT TEMPLATES

  • The Generation-Verification Prompt alternates generation and verification until the correct answer is obtained, with code enclosed in Python blocks for reward extraction.Source: §B.3 · p. 14

B.2 CASE STUDY OF REVEAL

  • The case study illustrates ReVeal performing iterative generation and verification.Table 4 is identified as a case study of iterative generation and verification.Source: §B.2 · p. 14, Table 4 · p. 15

B.4 TEMPLATES USED FOR TOOL FEEDBACK

  • Tool feedback distinguishes passed and failed tests, invalid or wrong tests, and formatting errors with corresponding signals or guidance.Source: §B.4 · p. 16

C THE EFFECTIVENESS OF SHORT-TERM MEMORY

ReVeal uses short-term memory to retain recent generation-verification interactions; adding memory did not reduce Pass@1 at turn 15.

C THE EFFECTIVENESS OF SHORT-TERM MEMORY

D MORE EXPERIMENTS ON QWEN2.5-32B-INSTRUCT

Experiments on Qwen2.5-32B-Instruct report improved Pass@1 over single-turn RL, gains from joint verifiable rewards, and concurrent improvements in code and test-case accuracy.

Main Results

Training Curves

  • Figure 5 shows concurrent improvements in code accuracy and test-case accuracy with TAPO and joint rewards.The passage links explicit self-verification optimization with more reliable verification and stronger multi-turn refinement.Source: D MORE EXPERIMENTS ON… · p. 18

Significance

E COMPARE WITH PROMPT-ONLY REVEAL

The prompt-only ReVeal variant uses the same prompt and code-execution tool without RL training, but does not sustain deep multi-turn refinement. RL-trained verification is associated with higher verification reliability and test-case accuracy.

E COMPARE WITH PROMPT-ONLY REVEAL

Loading 2506.11442v2…