PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

Difan Jiao, Qianfeng Wen, Blair Yang, Zhenwei Tang, Ashton Anderson

arXiv:2604.01591v2cs.AI

TL;DR

LLMs can make correctable reasoning errors, while prompt-only refinement is brittle and many training-based methods require external supervision. ThinkTwice jointly trains solving and self-refinement in two GRPO phases using the same binary correctness reward, improving both capabilities across five mathematical benchmarks and two model families. Its training dynamics reveal a rectify-then-fortify curriculum, while the paper supports extensions to other outcome-rewarded tasks and multiple refinement turns.

  • Problem

    Existing self-refinement approaches are brittle or depend on process supervision, critique annotations, or other external signals, limiting reusable refinement training.

  • Method

    ThinkTwice alternates reasoning and self-refinement optimization on the same problems with GRPO, using a task-agnostic instruction and the same binary correctness reward without external information.

  • Results

    Across five mathematical reasoning benchmarks and two model families, ThinkTwice consistently outperforms competitive online policy optimization baselines on direct reasoning and self-refinement.

  • Takeaways & Limitations

    Training jointly for reasoning and self-refinement yields a rectify-then-fortify curriculum and a more rectified reward signal with minimal overhead compared to GRPO.

  • Takeaways & Limitations

    The evaluation focuses on mathematical reasoning with verifiable answers; applying ThinkTwice to other outcome-rewarded tasks and multiple refinement turns remains future work.

Abstract

from arXiv · show

We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.

1 Introduction

ThinkTwice addresses brittle and supervision-heavy approaches to LLM self-refinement by jointly training a shared model to solve problems and revise its own solutions. It improves both reasoning and refinement, while its training dynamics shift from correcting errors to fortifying correct answers.

  • RLVR improves LLM reasoning, but strong reasoners still produce correctable errors such as incomplete derivations, algebraic mistakes, and unproductive paths.
  • Prompt-only refinement can be brittle, while training-based methods often rely on process supervision, critique annotations, or other external signals.
  • ThinkTwice alternates solving and refining the same problems with a shared backbone, using the same binary correctness reward without correctness signals, critique annotations, or external teacher models.
  • 5 percentage points before refinement and 11.5 points after one self-refinement step: ThinkTwice outperforms GRPO on Qwen3-4B AIME problems by pass@4.
  • Early refinement mainly corrects failed solutions, then shifts toward preserving and polishing correct ones as the model improves, forming an implicit rectify-then-fortify curriculum.

2 Related Work

Related work places ThinkTwice within RLVR, second-pass reasoning, and self-refinement research. Existing self-refinement methods add inference-time computation or feedback but generally do not learn a reusable refinement policy.

  • RLVR optimizes reasoning models with automatically checkable outcome signals such as exact answers, including the critic-free GRPO algorithm.
  • ThinkTwice is presented as a sequential two-phase design with reasoning optimization followed by refinement optimization.
  • Training-free self-refinement methods use inference-time self-feedback, verification, tools, or self-consistency, but do not learn a reusable refinement policy.

3 Methodology

ThinkTwice builds self-refinement into GRPO by first generating and rewarding base solutions, then conditioning a second optimization phase on selected base solutions. The refinement instruction is task-agnostic, and binary correctness rewards support an emergent correction-to-polishing curriculum.

  • 3.1 Preliminaries: ThinkTwice uses GRPO, which computes advantages from grouped samples and avoids a separate critic model.
  • 3.1 Preliminaries: GRPO samples a group of G responses from the old policy for each input and optimizes a clipped surrogate objective.
  • 3.1 Preliminaries: For verifiable mathematics, each response receives a binary reward of 1 when its extracted answer equals the ground-truth answer.
  • 3.2 Self-Refinement Formulation: The refinement prompt combines the problem, a base solution, and a task-agnostic instruction asking the model to review and improve its previous solution.
  • 3.3 ThinkTwice: Each training pair first optimizes solving, then optimizes refining the model’s own solutions to the same problems under complementary objectives.
  • 3.3 ThinkTwice: The same binary correctness reward evaluates refined solutions, encouraging error correction for wrong bases and preservation or polishing for correct bases.
  • 3.3 ThinkTwice: ThinkTwice randomly selects one base solution per problem from GRPO rollouts before constructing refinement prompts.
  • 3.3 ThinkTwice: The sampling strategy creates an emergent curriculum that shifts from correcting frequent early errors toward polishing correct solutions as reasoning accuracy improves.

4 Results

ThinkTwice improves direct reasoning and self-refinement across five mathematical benchmarks and two model families, while cross-model evaluation isolates refinement capability from base-solution quality.

  • 4.2 Reasoning: ThinkTwice achieves the highest average direct-reasoning score for both Qwen3-4B and OLMo3-7B before self-refinement.The evaluation uses pass@4 across five mathematical reasoning benchmarks.
  • 4.2 Reasoning: 5.05 percentage points separate ThinkTwice from GRPO on Qwen3-4B AIME direct reasoning, with scores of 44.11% and 39.06%.
  • 4.3 Refinement: ThinkTwice reaches 71.88% average self-refinement pass@4 on Qwen3-4B, exceeding DAPO by 2.9 points and GRPO by 4.5 points.On AIME, ThinkTwice reaches 60.43%, more than 11 points above GRPO.
  • 4.3 Refinement: On OLMo3-7B, ThinkTwice has the highest average self-refinement score and outperforms all baselines on every benchmark.
  • 4.3 Refinement: Cross-model evaluation generates one greedy base solution and four refinements per problem, attributing resulting accuracy differences to refinement models.Results are averaged across all five benchmarks.
  • 4 Results: ThinkTwice improves both reasoning and self-refinement within one framework across two model types and sizes.

5 Discussion

ThinkTwice develops a rectify-then-fortify refinement curriculum: it increasingly corrects failed solutions early, then preserves and shortens correct solutions as training advances.

  • 5.1 Training Dynamics: ThinkTwice maintains a higher fix-wrong rate than GRPO throughout training, peaking around the middle of training.Fix-wrong measures incorrect base solutions that become correct after refinement.
  • 5.1 Training Dynamics: ThinkTwice’s damage-correct rate drops near zero in the second half of training as refinement shifts toward preserving correct solutions.The baseline’s damage-correct rate remains more than five times higher than ThinkTwice’s best-checkpoint rate.
  • 5.1 Training Dynamics: Route switching, solution completion, and late-stage fortification are recurring qualitative refinement behaviors aligned with the correction-to-preservation transition.
  • 5.1 Training Dynamics: The refinement phase provides useful gradient information on problems where the base attempt alone provides limited learning signal, yielding a more rectified reward signal.
  • 5.1 Training Dynamics: ThinkTwice’s mean reward remains comparably stable to GRPO, while refinement responses become progressively shorter during training.Base response lengths stay in the same range as GRPO.
  • 5.2 Training Cost: 3% slower total wall-clock time than GRPO is observed through step 300, while ThinkTwice reaches its best checkpoint in 7.2 hours versus 8.6 hours.ThinkTwice converges in 220 steps versus GRPO’s 280 and maintains higher benchmark accuracy for most of training.

6 Conclusion

ThinkTwice jointly optimizes reasoning and self-refinement with the same binary correctness reward, consistently outperforming online policy-optimization baselines while adding minimal overhead. Its training dynamics form a rectify-then-fortify curriculum, and multi-turn refinement remains a proposed extension.

  • ThinkTwice jointly optimizes direct reasoning and self-refinement with the same binary correctness reward, without correctness signals, critique annotations, or external verifiers.
  • Its analyses identify an implicit rectify-then-fortify curriculum that produces a more rectified reward signal during training.
  • ThinkTwice adds only minimal training overhead compared with GRPO.
  • The framework is evaluated on mathematical reasoning with publicly available datasets and models, without human subjects, private data, or additional dual-use concerns beyond general-purpose language-model research.

Ethics Statement

The work focuses on improving mathematical reasoning in language models through RLVR using publicly available datasets and models. It involves no human subjects or private data.

  • The study uses publicly available datasets and models to improve mathematical reasoning through reinforcement learning with verifiable rewards.
  • The work does not involve human subjects, private data, or dual-use concerns beyond those inherent to general-purpose language-model research.

Reproducibility Statement

The paper provides implementation resources and positions ThinkTwice against related self-refinement approaches. Its comparison emphasizes reinforcement learning, supervision requirements, verifier dependence, and the shared policy with sparse correctness rewards.

  • Reproducibility Statement: Appendix B.1 provides the full training workflow, refinement instruction, hyperparameters, dataset sources, and code-level illustrations.
  • Reproducibility Statement: Appendix B.2 documents the implementation of all training-free baselines, and an anonymous ThinkTwice codebase is linked in the abstract.
  • Related Work Comparison: Table 3 compares related methods by reinforcement-learning use, supervision beyond final-answer correctness, and reliance on an additional verifier, teacher, or critique channel.
  • Related Work Comparison: ThinkTwice is closest to RL-based self-refinement but uses a shared policy and the same sparse final-answer correctness reward in solve and revise phases.
  • Related Work: The related literature includes second-pass reasoning, reflection, reconsideration, and training-free self-refinement methods across multiple application settings.

B.1.1 Datasets

The experiments train on MATH and evaluate exact boxed answers across five mathematical benchmarks, while comparing base reasoning and several fixed-prompt refinement procedures. The implementation alternates reasoning and refinement phases, samples multiple initial branches, and scores only final branch outputs.

  • Datasets: Training uses 7,500 MATH problems spanning difficulty levels 1–5 across seven subjects.
  • Datasets: Evaluation covers 1,526 problems from MATH500, OlympiadBench, Minerva Math, AIME, and AMC.
  • Evaluation: Each task requires a boxed final answer, and accuracy is measured by exact matching with Huggingface Math-Verify.
  • Training Procedure: ThinkTwice alternates reasoning and refinement steps, applying both objectives to the same batch within each training iteration.
  • Refinement Procedure: Refinement prompts include the original problem, the Phase 1 solution, and a task-agnostic instruction to check errors, explain corrections, or verify correct reasoning before producing a refined answer.
  • Baselines: Evaluation compares base reasoning, one-step refinement, SELF-REFINE, and REFLEXION under a fixed refinement instruction and common random seed and evaluation set.
  • Evaluation: Each problem uses 32 sampled initial branches, while second-stage generations are greedy and only each branch’s final answer contributes to pass@k.

C Additional Results

Across five benchmarks and both model families, ThinkTwice improves reasoning and self-refinement across the pass@k spectrum. Qualitative cases show refinement correcting errors or unfinished derivations early, then compressing and polishing correct solutions later.

  • ThinkTwice’s reasoning and self-refinement performance are evaluated across k ∈{1, 2, 4, 8, 16, 32} on five benchmarks and two model families.
  • Quantitative results: ThinkTwice outperforms other baselines across nearly all reasoning pass@k values, with its advantage widening on AIME as k increases.
  • Quantitative results: Nearly 77% pass@32 on Qwen3-4B versus roughly 62% for the next-best baseline marks ThinkTwice’s self-refinement advantage on AIME.
  • Qualitative mechanisms: The qualitative mechanisms include route switching, solution completion, and late-stage exploitation that removes exploratory clutter from already-correct solutions.

D.3 Case C: sample 194 — concrete error repair early, concise proof late

Case C illustrates two refinement behaviors on the same optimization problem: repairing a concrete derivative mistake early and removing exploratory detours once the base proof is correct.

  • Early checkpoint: At step 30, the base trace reaches the correct one-variable reduction but makes a concrete derivative arithmetic error.
  • Early checkpoint: The refinement explicitly identifies the incorrect d4 = 1 calculation and corrects it to d4 = 16.
  • Early checkpoint: This early refinement demonstrates bug-fixing behavior rather than merely rewriting an already-correct solution.
  • Late checkpoint: At step 220, the base solution is correct but exploratory, testing and rejecting an inequality route before returning to calculus.
  • Late checkpoint: The refinement starts near the final proof and strips away the abandoned routes, behaving like an exploitation layer once the base policy has learned the answer.

D.4 Case D: sample 425 — vector recomputation as a backup geometry case

Case D shows ThinkTwice shifting from repairing unresolved vector algebra early in training to concise invariant-based proofs later. Compared with pure GRPO, the late refinement is shorter and more controlled despite reaching the same correct result.

  • The geometry target is AF^2 + BF^2 + CF^2 = 3R^2, with the ground truth given as 3R^2.
  • At step 100, the reasoning trace reaches an unresolved mixed term because it never substitutes the identity needed to eliminate S·O.
  • The step-100 refinement recomputes the vector expansion term by term, replaces the mixed term using the missing identity, and concludes with 3R^2.
  • By step 220, ThinkTwice uses a circumcenter-at-origin derivation and compresses the refinement into a short invariant argument that cancels cross terms.
  • Across examples, early refinement changes the model’s route by discovering missing relations, closing derivations, or repairing algebra, while later refinement preserves correctness and removes detours.
Loading 2604.01591v2…