Source-linked AI summary
REVES: REvision and VErification--Augmented Training for Test-Time Scaling
Yuanxin Liu, Ruida Zhou, Xinyan Zhao, Amr Sharaf, Hongzhou Lin, Arijit Biswas, Mohammad Ghavamzadeh, Zhaoran Wang, Mingyi Hong
TL;DR
Test-time sequential revision is not directly optimized by standard single-shot objectives, motivating a training method aligned with multi-step reasoning. REVES uses continual mistake-driven augmentation for revision and verification, improving performance across coding, math, puzzles, and circle packing, including +6.5 points over single-shot RL on LiveCodeBench.
Problem
Standard single-shot objectives do not directly optimize the multi-step contexts and sequential revision capability generated by test-time scaling algorithms.
Method
REVES alternates policy optimization with continual augmentation that converts successful trajectories’ near-miss answers into decoupled revision and verification prompts.
Results
+6.5 points over single-shot RL on LiveCodeBench, with consistent gains across math, coding, out-of-distribution puzzles, circle packing, and revision-using test-time algorithms.
Takeaways & Limitations
Improving sequential revision transfers to other revision-using test-time algorithms and supports strong performance across diverse verified reasoning tasks.
Takeaways & Limitations
REVES relies on a verifier signal to provide correctness feedback.
Abstract
from arXiv · showhide
Test-time scaling via sequential revision has emerged as a powerful paradigm for enhancing Large Language Model (LLM) reasoning. However, standard post-training methods primarily optimize single-shot objectives, creating a fundamental misalignment with multi-step inference dynamics. While recent work treats this as multi-turn reinforcement learning (RL), conventional approaches optimize over the multi-step trajectories directly, failing to further exploit the high-quality mistakes in intermediate steps that model can learn from correcting them. We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization. By converting the intermediate steps (``near-miss'' answers) in the successful recovery trajectories into decoupled revision and verification prompts, our approach concentrates training on both effective answer transformation and error identification. This approach enables efficient off-policy data generation and reduces the computational overhead of long-horizon sampling compared to standard multi-turn RL. On LiveCodeBench, using publicly available test cases as feedback, we observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training. Beyond coding, our approach matches the previously reported SOTA result on circle packing while using the smallest base model (4B) and far fewer rollouts than the much larger evolutionary search systems. Math results under ground-truth verification further confirm improved correction ability. It also generalizes to out-of-distribution constraint-satisfaction puzzles such as n\_queens and mini\_sudoku, where correctness is defined entirely by problem constraints. Code is available at https://github.com/yxliu02/REVES.git.
1 Introduction
REVES addresses the mismatch between single-shot post-training and multi-step sequential revision by decomposing the objective into per-state recovery signals and training decoupled revision and verification prompts. It improves test-time performance across coding, math, circle packing, and constraint-satisfaction puzzles while reducing reliance on long-horizon sampling.
- Motivation: Sequential revision is central to test-time scaling because revision-using methods eventually call the policy on revise-from-prior-attempt inputs.The paper focuses on sequential revision because improvements transfer to other revision-using test-time scaling algorithms.
- Problem: Standard single-shot post-training is misaligned with multi-step deployment, while multi-turn RL suffers from path-dependent trajectory-level credit assignment.A wrong, wrong, correct rollout can broadcast credit across all turns despite only the recovery step being successful.
- Method: REVES exactly decomposes the sequential-revision objective into weighted per-state one-step recovery probabilities, eliminating horizon-level credit assignment.The framework runs rollouts under the current policy, retains successful recovery trajectories, and uses intermediate near-miss answers for training.
- Empirical Results: +6.5 points over the single-shot RL baseline and +4.0 points over standard multi-turn training are achieved on LiveCodeBench with public test cases as feedback.These results quantify REVES’s coding gains against both single-shot and multi-turn training baselines.
- Empirical Results: REVES delivers consistent gains across math, coding, out-of-distribution puzzles, and circle packing, matching prior best circle-packing results with a Qwen3-4B base.The circle-packing comparison is against larger evolutionary search systems built on Gemini-2.0 Pro/Flash and Qwen3-8B.
2 Related Work
Related work aligns post-training with test-time inference, including parallel sampling and sequential revision. REVES differs by targeting revision-using inference objectives rather than single-shot pass@1.
- Test-time-aware post-training: Test-time-aware post-training optimizes objectives aligned with deployment strategies such as best-of-N, majority voting, or learned verification.The cited work includes pass@k optimization for parallel sampling families.
- Training sequential revision capability: Prior work trains sequential revision through supervised, preference-based, RL-based, and multi-turn RL approaches with natural-language or numerical critiques.These methods share the broader goal of strengthening revision capability.
- Training sequential revision capability: REVES primarily differs by aligning training with the test-time objective of a revision-using inference strategy.Its framing contrasts with prior methods that explicitly train policies to revise.
- Weakness-driven data synthesis and guided exploration: SwS synthesizes weakness-targeted problems with strong external teachers, while POPE uses privileged oracle hints to guide on-policy exploration on hard problems.These approaches are described as orthogonal to REVES.
- Weakness-driven data synthesis and guided exploration: SwS and POPE target single-shot pass@1, whereas REVES targets a fundamentally different test-time objective.The contrast is framed as orthogonal to REVES’s revision-oriented objective.
3 Test-Time Scaling as a Meta-RL Problem
The paper formulates test-time scaling as a meta-RL problem: policies must perform well across multi-step contexts induced by a scaling algorithm, rather than optimize only single-shot answers. It adopts the sequential-revision objective JϕSR because sequential revision is strong and its recovery gains transfer to revision-using test-time scaling methods.
- Induced decision process: A test-time scaling algorithm induces a decision process in which the policy generates responses across prompts formed from prior history, with execution stopping at the first correct response or step K.Sequential revision uses the latest response, whereas tree search may branch from earlier responses, making the transition process potentially history-dependent.
- Objective mismatch: Optimizing Jϕ is a meta-RL problem because the policy must adapt across multi-step contexts generated by ϕ, unlike the single-shot objective optimized by RLHF, RLVR, and GRPO.The goal is to improve Jϕ for every TTS algorithm ϕ ∈ΦR.
- Transfer guarantee: Theorem 3.1 establishes that improving sequential-revision recovery transfers to every revision-using TTS algorithm that covers the relevant inputs.The transfer result enables training against one algorithm while improving deployment-time choices across ΦR.
- Training target: The paper adopts JϕSR as its training target because sequential revision is empirically strong, simple to implement, and offers transfer guarantees to other revision-using algorithms.The stated takeaway is to use the sequential-revision objective JϕSR rather than single-shot pass@1.
- Objective mismatch: Theorem 3.2 shows that single-shot performance does not determine sequential-revision performance: policies can have identical JOneShot values yet differ substantially under JϕSR.Thus, standard single-shot RLVR training does not directly optimize the sequential-revision objective.
4 REVES: From Path-Level Credit to Recovery-Level Supervision
REVES decomposes sequential-revision success into per-state one-step recovery probabilities, then trains local recovery behavior using intermediate mistakes from successful trajectories. Its two-stage augmentation and single-turn RL design adds verification, reuses offline data, filters for successful trajectories, and supports efficient test-time stopping.
- Recovery-level objective: REVES expresses sequential-revision success as a visit-weighted sum of per-state one-step recovery probabilities, eliminating horizon-level credit assignment.Fresh policy samples at each visited state estimate and optimize the local recovery factor.
- Comparison with multi-turn RL: Multi-turn RL weights wrong–wrong–correct trajectories with the same terminal advantage across all turns, producing a high-variance, coarse signal for local revision behavior.The estimator remains unbiased for terminal trajectory success but is indirect for improving intermediate responses.
- Recovery-level supervision: Intermediate wrong responses become revision states rather than imitation targets, converting successful long-horizon trajectories into several short-horizon recovery problems.This provides a sharper local signal while using the current policy to discover useful intermediate states.
- Two-stage realization: Stage I retains trajectories that succeed within budget K and creates revision and verification prompts from their intermediate responses; Stage II applies single-turn RL to the augmented prompt union.The updated policy regenerates augmentation in the next epoch.
- Efficiency: Offline augmentation generates visited states once per epoch for reuse, while asynchronous parallel generation and single-turn gradient steps avoid interrupting the main RL loop.These choices reduce the serial overhead associated with fresh multi-turn rollouts at every gradient step.
- Test-time stopping rule: At test time, REVES replaces unavailable oracle stopping with domain-specific signals, selecting the most confident response across K revisions for mathematical reasoning or using public-test-case rewards for coding.For coding, the process stops at the first response passing all public tests, otherwise output is yK.
5 Experimental Results
REVES consistently improves sequential revision across coding and mathematical reasoning tasks, matches the best reported circle-packing result with a smaller model, and generalizes to unseen puzzle domains. Its learned revision capability also transfers across test-time algorithms, while continual and larger-budget augmentation strengthen training outcomes.
- Experimental setup: REVES is evaluated against single-response RL, multi-turn training without explicit self-verification, and PAG with self-verification.The evaluation covers MATH500, AIME24/25, LiveCodeBench, CodeContest, n_queens, and mini_sudoku.
- Coding results: REVES consistently improves coding performance across test-time budgets under execution-based feedback and surrogate stopping criteria.Publicly available test cases determine stopping and provide feedback for subsequent revisions.
- Mathematical reasoning: REVES substantially improves sequential revision across all evaluated math benchmarks under oracle and self-confidence stopping.Oracle stopping isolates the ability to revise an incorrect solution into a correct one.
- Circle packing: 2.635983 is the optimal sum of radii reached by REVES on Qwen3-4B for circle packing, matching the best reported result with the smallest base model and fewer rollouts.The comparison is against larger evolutionary search systems built on Gemini-2.0 Pro/Flash and Qwen3-8B.
- Generalization and transfer: REVES generalizes to n_queens and mini_sudoku despite using checkpoints trained only on math and code, and its revision-trained policies improve MCTS, AB-MCTS variants, and Mind Evolution.These results indicate transferable sequential-revision capability across out-of-distribution tasks and diverse test-time inference strategies.
- Ablations: Continual data augmentation outperforms augmenting only at the initial epoch, while increasing augmentation budget consistently improves performance at higher computational cost.Continual regeneration keeps supervision aligned with the model’s current failure modes; REVES also yields sharper confidence separation than RevisionOnly.
6 Conclusion · A Sequential Revision Protocol · A.1 Example Math Task
REVES aligns training with sequential revision by combining reinforcement learning with continual, mistake-driven augmentation that teaches models to revise incorrect solutions and identify errors. Its protocol repeatedly revises responses using feedback with early stopping, illustrated by correcting a complex-number rotation answer after feedback that the initial response was incorrect.
- 6 Conclusion: REVES combines RL with continual, mistake-driven data augmentation to learn both solution revision and error identification.The conclusion states that static data becomes uninformative once mistakes disappear, motivating continual augmentation.
- 6 Conclusion: REVES improves test-time sequential-revision performance across math, coding, and out-of-distribution tasks over single-shot RL and multi-turn training.
- A Sequential Revision Protocol: The protocol takes a revision budget K, question x, model π, and feedback function as inputs before producing the selected answer ẏ.The final answer is the first response satisfying r⋆(x, yi) = 1, or yK when early stopping does not occur.
- A Sequential Revision Protocol: The sequential revision protocol samples an initial response, repeatedly generates revisions conditioned on the previous response and feedback, and stops early when the revised answer is correct.If no revision is correct within budget K, the protocol returns the final response yK.
- A.1 Example Math Task: In the math example, feedback consists only of “The response is incorrect.” without diagnostic information.
- A.1 Example Math Task: For the 90° clockwise rotation of 3 around −4 −5i, the initial solution computes −9 + 2i and receives incorrectness feedback.The example asks for the resulting complex number and presents −9 + 2i as the previous attempt’s final answer.
- A.1 Example Math Task: After feedback, the corrected solution translates 3 relative to −4 −5i, multiplies by −i for the clockwise rotation, and obtains 1 −12i.The corrected derivation uses z − a = 7 + 5i, rotates it to 5 − 7i, then adds a.
A.2 Example Coding Task
The coding-task evaluation iteratively executes a generated program on public tests and feeds failures back for revision. Final correctness requires passing both public and private test cases, while an example illustrates permutation-sorting code generation and runtime-error diagnosis.
- Evaluation protocol: At test time, the model generates a candidate program, executes it on public tests, and revises it using appended error messages or failing-test feedback.This forms an iterative execution-based evaluation protocol.
- Evaluation protocol: Passing public tests is insufficient: a solution is correct only when it passes all public and private test cases.The public tests provide feedback during generation, whereas correctness is assessed against the complete test suite.
- Example task: The example asks for a Python program that transforms a permutation into the identity permutation using the specified operation.The task requires reading standard input, writing standard output, and returning only the final Python code.
- Example task: code_v1 produces a runtime error because index-based updates and repeated linear searches with P.index cause inconsistent state updates and excessive runtime overhead.The diagnosis identifies both correctness and efficiency problems in the attempted solution.
B Limitation and Future Direction · C Proofs
REVES is directly applicable when training tasks provide deterministic ground-truth verifiers, while extending it to unverifiable tasks depends on the policy’s self-verification calibration. The proofs establish theoretical results on objective mismatch, transfer from sequential-revision recovery, and hazard decomposition.
- B Limitation and Future Direction: REVES requires a ground-truth verifier during training and applies directly to math, coding, puzzle, and circle-packing benchmarks with deterministic verifiers.The stated limitation is task-dependent rather than benchmark-wide.
- B Limitation and Future Direction: For free-form QA, creative writing, and RLHF-style preference learning, REVES does not directly extend because such tasks lack the required verifier.Verification prompts can instead support self-verification at test time, but reliability depends on policy calibration.
- B Limitation and Future Direction: REVES is especially suited to scientific tasks with clear automated rewards, including PDEs, symbolic regression, theorem proving, and discovery-style optimization.Examples use residual norms or analytic checks, fit and complexity scores, proof checkers, or optimization verification.
- C Proofs: The proofs cover the objective-mismatch result, showing that optimizing single-shot pass@1 does not optimize the sequential-revision objective JϕSR.This result is stated as Theorem 3.2.
- C Proofs: The transfer guarantee establishes that improving SR recovery improves every revision-using TTS algorithm that covers SR’s revision inputs.This result is stated as Theorem 3.1.
- C Proofs: The hazard decomposition exposes the structure underlying the corresponding theoretical analysis.The passage identifies this result as Lemma 4.1, but the supplied text truncates its full statement.
C.1 Proof of Theorem 3.2 … D Training Prompt Template
The appendices establish the objective-mismatch result, derive the recovery-based off-policy training signal, and justify its identities through first-success stopping and Markov-policy arguments. The training templates use structured supervision without appending execution-error information, including revision prompts with binary correctness feedback.
- C.1 Proof of Theorem 3.2: For every revision length K ≥2, there exist policies π1, π2 and a problem distribution X with a positive objective gap ∆(K) > 0.This is the stated objective-mismatch theorem.
- C.1 Proof of Theorem 3.2: Under sequential revision, π1 succeeds with probability JϕSR(π1) = 1/N, whereas π2 succeeds with probability JϕSR(π2) = 1 −(1 −1/N)^K.The construction uses one problem, N ≥2 candidate answers, uniformly drawn ground truth, binary correctness feedback, and independent uniform draws for π2.
- C.2 Proof of Theorem 3.1: The recovery decomposition yields an off-policy fixed-reference form that holds the visit measure at baseline π0, avoiding contamination from policy-induced shifts in ρϕSR.The on-policy objective is JϕSR(π) = ∑z ρϕSR π (z) Vπ(z), while the fixed-reference form uses ∑z ρϕSR π0 (·)(Vπ1 −Vπ0).
- C.2 Proof of Theorem 3.1: REVES optimizes this off-policy signal by freezing rollout states under π0 in Stage I and training Vπ on those fixed states with single-turn RL in Stage II.Each epoch refreshes rollouts and resets the baseline, keeping the ℓ1 shift in ρϕSR small.
- C.2 Proof of Theorem 3.1: The bound is conditional on revision-input coverage, improved one-step recovery, and stable visit measures under the policy update.These conditions correspond to small Cϕ, a policy update that raises one-step recovery, and small ℓ1 shifts between ρϕ and ρϕSR.
- C.3 Proof of Lemma 4.1: Lemma 4.1 derives its first identity from SR’s first-success stopping event and its third identity by substituting Vπ(z) and exchanging the visit-count sum with the outer expectation.The success event at step t is {τ ≥t, yt ∈Gx}, with Gx := {y : r⋆(x, y) = 1}.
- D Training Prompt Template: Training prompts omit execution-error information from the context and instead rely on structured supervision, including revision templates with binary correctness feedback.The appendix illustrates these templates for math revision tasks and shows concrete problems with partial incorrect answers.
E Implementation Details … F.3 Sensitivity to the Confidence Threshold
The paper standardizes implementation and evaluation across baselines and REVES, while comparing revision-specific training, efficiency, and confidence-threshold effects. REVES uses iterative data augmentation, avoids naive multi-turn credit-assignment problems, reduces wall-clock time, and exhibits a cost-quality threshold trade-off.
- E Implementation Details: All experiments run on 8 × NVIDIA H200 GPUs, using a unified framework that serializes outputs and evaluates them with task-specific external verifiers.The framework applies each task’s prescribed test-time protocol uniformly across methods.
- E Implementation Details: The RL baseline uses official Skywork-OR1 scripts for single-shot reinforcement learning, without revision or multi-turn generation during training.Multi-Turn and PAG instead use Policy-As-GenVerifier with an 8,092-token maximum response length and adapted dataset reward computation.
- E Implementation Details: REVES initializes data offline, then augments it at each epoch by adding revision trajectories generated by the current policy.This augmentation procedure is the key implementation difference from the shared baseline training framework.
- F.1 Comparison with Naive Multi-Turn RL: Naive multi-turn schemes misassign credit because sequential revision consists of fresh attempts on the same problem, not longer-horizon actions with intermediate sub-rewards.Trajectory-level broadcast gives every turn identical credit, while pooled per-response fails to capture the structure of intermediate responses.
- F.2 Wall-Clock Efficiency: REVES performs the length-K serial cost once offline and uses parallelizable single-turn sampling during online reinforcement learning, unlike length-M serial sampling at every gradient step.This design targets the computational burden of online multi-turn training while retaining iterative data generation.
- F.2 Wall-Clock Efficiency: Three REVES iterations took roughly 30 hours on 8× NVIDIA H200 GPUs, versus over 48 hours for the equivalent online multi-turn baseline on identical hardware.The paper notes that concurrent Stage I generation and Stage II training could reduce time further with optimized infrastructure.
- F.3 Sensitivity to the Confidence Threshold: At budget K = 16, increasing the TailConfidence threshold monotonically increases accuracy across three math benchmarks while also increasing average revision rounds.Higher thresholds enforce stricter stopping, revealing a cost-quality trade-off.
G Additional Experiments and Ablations
Additional experiments with DeepSeek-R1-Distill-7B show that REVES improves test-case-based performance over both the original base model and an RL baseline, despite the higher cost of multi-turn training for this model.
- DeepSeek-R1-Distill-7B Evaluation: DeepSeek-R1-Distill-7B produces approximately 9k-token responses on average, compared with ∼2k tokens for Qwen-7B, making multi-turn training significantly more expensive.This higher response length limits the scale of the experiments.
- DeepSeek-R1-Distill-7B Evaluation: REVES consistently improves test-case-based performance over both the base model and the RL baseline.The authors expect larger gains with increased data scale and longer training.
H Circle Packing Implementation Details
REVES applies sequential revision to circle packing by optimizing verifier-checked improvements, with reward shaping that amplifies tiny objective differences while penalizing infeasible or worsening candidates. It then constructs prompts from successful intermediate revisions and trains Qwen3-4B with GRPO.
- Circle packing implementation: REVES optimizes circle packing through sequential revisions, where each state is a partially improved configuration and the verifier rewards feasible candidates that improve the running best.The benchmark maximizes the sum of radii of non-overlapping circles inscribed in the unit square, using non-overlap, inscription, and objective-value feedback.
- Reward shaping: Reward shaping converts the compressed objective and constraint signal into scores that penalize violations, flag infeasible high-sum candidates, and amplify small improvements.For n = 26, feasible objectives cluster near g ≈2.63; the transformation R(s) := (s −2.5)2 makes a ∆s = 10−3 gain near s = 2.6 produce ∆R ≈2 · 10−4.
- Stage I: search rollouts and prompt-set construction: Stage I runs 4 sequential-revision rollouts with budget K = 100 and retains intermediate states whose subsequent revisions yield verified improvements for the augmented prompt set.Each refinement proposes a new configuration from the previous candidate and receives the shaped reward U(y | x).
- Stage II: RL training: Stage II trains Qwen3-4B with GRPO, sampling 8 prompts per gradient step and 64 rollouts per prompt for the group advantage estimate.The training prompt set is constructed in Stage I from successful intermediate revisions.
- A representative solution: A representative policy-generated solution initializes a 26-circle grid placement and refines it with SLSQP under non-overlap and inscription constraints.The configuration shown for the n = 26 instance achieves sum-of-radii = 2.635983.