Rectifying LLM Thought from Lens of Optimization

• RLVR lets LLMs explore reasoning steps from terminal rewards, supporting scalable inference-time self-improvement.Source: §1 · p. 1
• Long-CoT reasoning can produce protracted paths that contribute little to solving a problem and incur substantial computational costs.Even simple queries can elicit chains exceeding thousands of tokens, increasing latency and resource demands.Source: §1 · pp. 1–2
• The paper models each reasoning step as a task-specific gradient-descent update, with suboptimal reasoning characterized as oscillation around saddle points or local optima.Source: §1 · p. 2
• REPRO monitors CoT optimization with a surrogate objective and combines intensity and stability scores into a process-level reward for post-training.The method is described as plug-and-play with prevalent reinforcement learning algorithms.Source: §1 · p. 2
• Experiments across mathematics, science, and coding benchmarks show significant reasoning-performance improvements with REPRO across reinforcement learning algorithms and diverse LLMs.The evaluated models include base, supervised-finetuned, and native long-CoT variants.Source: §1 · p. 2

• PPO uses an actor to optimize a reward and a critic to estimate state values, with a clipped surrogate objective that constrains policy updates for training stability.Source: §2 · p. 2
• PPO's advantage estimate uses generalized advantage estimation, derived from temporal-difference errors accumulated over time steps.Source: §2 · p. 2
• Critic-free algorithms address PPO's value-model computational cost by replacing the value estimate with a reward baseline; examples include ReMax, RLOO, GRPO, and REINFORCE++.Source: §2 · pp. 2–3
• The REPRO framework treats reasoning trajectories as optimization of LLM internal states, then scores that process and uses the score as a reward in RLVR.Source: Figure 1 · p. 3
• For critic-free algorithms, the objective uses groups of trajectories and normalized reward-baseline advantages, with a KL penalty weighted by β.Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• REPRO frames each reasoning step as an update to the model’s internal states that iteratively increases the likelihood of the correct answer.The authors note that the actual optimization process is complex and nontrivial.Source: §3.1 · p. 3

3.2 OBJECTIVE FUNCTION DEFINITION

• The proxy objective ˜J uses the probability of generating the ground-truth answer to track changes in the underlying objective indirectly.As the partial reasoning trajectory updates internal states, the ground-truth answer probability is expected to increase.Source: §3.2 · p. 4

3.3 QUANTIFYING OPTIMIZATION PROCESS

• The composite score S combines magnitude, which measures net objective improvement, and stability, which captures oscillation across reasoning steps.The magnitude score is normalized relative to a question-specific baseline; stability uses the association between objective values and step indices.Source: §3.3 · pp. 4–5

3.4 LEARNING WITH RECTIFYING PROCESS-LEVEL REWARD

• To limit the overhead and noise of token-level scoring, REPRO selects the top-k reasoning segments by the entropy of each segment’s first token.The authors motivate selection by proposing that uncertainty near segment endings signals a greater likelihood of suboptimal optimization.Source: §3.4 · p. 5
• For each selected segment, REPRO appends the ground-truth answer and rewards the change in its score relative to the preceding selected segment.The resulting process-level reward is normalized separately from the outcome reward to prevent noise signals from interfering with correctness reward.Source: §3.4 · p. 6
• In an example, low-reward steps involve self-doubt or redundant re-examination, while high-reward steps correspond to computations or preliminary conclusions that advance reasoning.The authors propose penalizing low-reward steps and incentivizing high-reward ones.Source: §3.4 · p. 6

4.1 SETUP

• Experiments cover mathematical, scientific, and code reasoning benchmarks and several prominent LLMs, using a training corpus of approximately 40,000 mathematical samples.The evaluation includes AIME24, AIME25, MATH500, LiveMathBench, GPQA-Diamond, MBPP, and LiveCodeBench.Source: §4.1 · p. 7

4.2 EFFECTIVENESS AND GENERALIZATION OF REPRO

• REPRO improves performance across PPO, REINFORCE++, REINFORCE++ Baseline, and GRPO, generalizes to science and coding benchmarks, and benefits LLMs from different families and sizes.For PPO on DeepSeek-R1-Distill-Qwen-1.5B, AIME24 accuracy rises from 34.8 to 36.3 and MATH500 from 86.9 to 87.7.Source: §4.2 · pp. 7–8

4.3 ABLATION STUDY

• Across tested w values, REPRO outperforms the baseline; slightly higher performance at lower w suggests the Magnitude Score may matter more than the Stability Score.The scores assess optimization intensity and stability, respectively.Source: §4.3 · p. 8
• REPRO maintains relatively stable performance across tested α values, although the coefficient is introduced to balance outcome advantage and REPRO advantage.The paper sets α to 0.1.Source: §4.3 · pp. 8–9
• Increasing the number of selected segments k yields only marginal performance gains, making the trade-off between training cost and performance important.The paper identifies k as a key factor in that trade-off.Source: §4.3 · p. 9

4.4 FURTHER ANALYSIS OF THE CHANGING OF THINKING BEHAVIORS

• REPRO reduces reasoning-token costs as training progresses and reduces inference token costs across all benchmarks.The reported training dynamics are for DeepSeek-R1-Distill-Qwen-1.5B.Source: Figure 3 · p. 8, §4.4 · p. 9, Figure 6 · p. 9
• As REPRO training progresses, the prevalence of backtracking—a pattern associated with ineffective and excessive reasoning—decreases compared with vanilla GRPO.The pattern is tracked on AIME24 every 10 training steps.Source: Figure 5 · p. 8, §4.4 · p. 9

Demystifying Reasoning Trajectories of LLMs

• Prior studies establish Transformers’ expressive capabilities, and more recent work finds they can meta-learn optimization algorithms such as gradient descent within forward trajectories.The paper identifies work treating LLM reasoning trajectories as optimization processes for model parameters and internal states as especially related.Source: §5 · p. 10

Promoting and Improving LLM Reasoning

• RL methods improve LLM reasoning, while long-CoT models can incur computational inefficiency through redundant tokens or unnecessary reasoning steps.The reviewed approaches include training-time length or information rewards and adaptive policies that alter reasoning patterns.Source: §5 · p. 10

6 CONCLUSION

• REPRO conceptualizes CoT reasoning as an optimization process and evaluates its intensity and stability.The scores are integrated as a process-level reward into RLVR training.Source: §6 · p. 10
• Experiments across diverse reasoning benchmarks demonstrate REPRO’s effectiveness.Source: §6 · p. 10
• REPRO is reported to improve LLM reasoning behavior and efficiency.Source: §6 · p. 10

ETHICS STATEMENT

• The study focuses on general research tasks and reports no risks to health, safety, personal security, or privacy.Source: ETHICS STATEMENT · p. 11
• The work involves no human participants and releases no new datasets.Source: ETHICS STATEMENT · p. 11
• The authors anticipate no ethical risks or conflicts of interest and state a commitment to scientific integrity and ethical guidelines.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• The REPRO framework is described in Section 3.Source: REPRODUCIBILITY STATEMENT · p. 11
• Datasets, model configurations, and other implementation specifics are detailed in Appendix B and Section 4.1.Source: REPRODUCIBILITY STATEMENT · p. 11
• Key code implementations are in the supplementary materials, with complete code to be released publicly upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 11

A DISCUSSIONS

• REPRO adds computation for entropy-based selection and reward calculation, while prefix overlap enables inference engines’ KV caching to accelerate forward computation.Source: A DISCUSSIONS · p. 20
• Against Vanilla GRPO, REPRO gains 3.1% on AIME24 while cutting tokens from 12,367 to 6,158; it also beats AdaThink by 4.3% with 28% fewer tokens.At matched token efficiency, L1-Max scores 26.7% versus REPRO’s 36.0% on AIME24.Source: Table 3 · p. 20, A DISCUSSIONS · p. 20

B.1 IMPLEMENTATION OF NORMALIZATION RECTIFYING PROCESS REWARD IN CRITIC-FREE RL

• GRPO normalizes rectifying process rewards across segments within each question’s trajectory group, and the same approach applies to GRPO variants.The named variants include Dr.GRPO, VAPO, DAPO, and GSPO.Source: §B.1 · p. 21
• REINFORCE++ normalizes each segment’s reward across all segments in all trajectories in the batch.Source: §B.1 · p. 21
• RLOO normalizes within each question’s trajectory group, whereas ReMax normalizes against the mean rectifying process reward from a greedy-decoded trajectory.Source: §B.1 · p. 21

B.2 TRAINING DATA

• Training uses DeepScaleR-Preview-Dataset, comprising about 40,000 mathematics problem-answer pairs from AIME, AMC, Omni-MATH, and Still.Source: §B.2 · p. 21

B.3 EVALUATION BENCHMARKS

• Mathematical evaluation uses AIME24, AIME25, MATH500, and LiveMathBench, including the 202505 hard split of 100 high-quality English questions.AIME24 and AIME25 each contain 30 challenging problems; MATH500 uses Level 5 questions.Source: §B.3 · p. 21
• Scientific reasoning is evaluated on GPQA-Diamond, a 198-question subset of a multiple-choice benchmark designed to resist web search.Source: §B.3 · p. 22
• Code reasoning is evaluated with MBPP elementary Python problems and LiveCodeBench problems drawn from LeetCode, AtCoder, and Codeforces.Source: §B.3 · p. 22

B.4 TRAINING PARAMETERS

• Training sets w = 0.5, α = 0.1, and the top 10 segments per trajectory; veRL and vLLM support training, with parameters tabulated for PPO, GRPO, and REINFORCE++ variants.Source: §B.4 · p. 22, Table 4 · p. 22, Table 5 · p. 22

B.5 PROMPT FOR THINKING PATTERN RECOGNITION

• Thinking-pattern recognition uses Qwen3-235B-A22B-Instruct-2507 to assess reasoning chains for beneficial behaviors.Source: §B.5 · p. 24
• The prompt highlights backtracking, verification, subgoal setting, and enumeration, requests explicit examples, and allows other behaviors such as creative analogies or generalizations.It directs the evaluator to return an empty list if no beneficial behaviors are observed.Source: §B.5 · p. 24

C.1 REPRO ON LLMS OF DIVERSE FAMILIES

• REPRO improves performance across Qwen3-1.7B, Hunyuan-1.8B-Instruct, and MobileLLM-R1-950M, including PPO+REPRO’s AIME24 increase from 45.2 to 49.0 on Qwen3-1.7B.The reported gains span mathematics, science, and coding benchmarks.Source: §C.1 · pp. 24–25, Table 7 · p. 25
• On Qwen3-1.7B and Hunyuan-1.8B-Instruct, PPO+REPRO uses fewer inference tokens than PPO while achieving higher performance.The comparisons are shown in Figures 7 and 8.Source: §C.1 · p. 25, Figure 7 · p. 25, Figure 8 · p. 26

C.2 REPRO ON LARGER LLMS

• On Qwen3-8B, GRPO+RePro improves AIME24 from 75.6 to 76.1, AIME25 from 67.9 to 68.5, and results on science and code benchmarks.GPQA-Diamond rises from 59.5 to 60.4, MBPP from 68.8 to 72.1, and LiveCodeBench from 52.2 to 53.4.Source: §C.2 · p. 26, Table 8 · p. 27
• On Qwen3-8B, GRPO+REPRO uses fewer inference tokens than standalone GRPO while achieving higher performance.The comparison is shown in Figure 9.Source: §C.2 · p. 26, Figure 9 · p. 26

C.3 REPRO FOR ZERO-RLVR

• On reasoning benchmarks evaluated with Qwen3-4B-Base, REPRO and GRPO achieve comparable in- and out-of-domain performance; GRPO slightly outperforms REPRO on AIME24, while REPRO leads on AIME25.They perform comparably on MATH500, with REPRO marginally ahead; they perform closely on MBPP and LiveCodeBench, where REPRO slightly leads on LiveCodeBench.Source: §C.3 · p. 27, Table 9 · p. 27
• For zero-RLVR, RePro’s token-cost growth is described as relatively modest compared with baselines.The passage presents Figure 10 as illustrating the comparison.Source: §C.3 · p. 27, Figure 10 · p. 27

C.4 ABLATION OF THE NUMBER OF SELECTED SEGMENTS

• AIME24 rises from 35.7 at k = 5 to 36.9 at k = 30, while MATH500 rises from 86.3 to 87.8; gains become marginal beyond k = 20.Source: §C.4 · p. 27, Table 10 · p. 28
• The authors use k = 10 as a balance between performance and training efficiency, since larger k adds computational overhead for increasingly marginal gains.Source: §C.4 · p. 27

C.5 ENTROPY DYNAMICS OF REPRO

• Vanilla GRPO entropy falls from about 0.275 to 0.255, while RePro stays around 0.28 for 100 steps and finishes above 0.27.Figure 11 compares the entropy dynamics of GRPO with RePro and standard GRPO.Source: §C.5 · p. 28, Figure 11 · p. 28

Case D.1: Case in Math Reasoning

• At least two matches comprise 90 draws with exactly two matches, 24 with exactly three, and one with all four.The counts use C(4,2)C(6,2), C(4,3)C(6,1), and C(4,4)C(6,0), respectively.Source: D CASES · p. 29
• The conditional probability of winning the grand prize given a prize is 1/115, so m + n = 116.There are 115 draws with at least two matches and one draw with all four matches.Source: D CASES · pp. 30–31

Case D.2: Case in Science Reasoning

• The science-reasoning case asks which of four described exoplanets has the highest density.The choices include an Earth-mass, Earth-radius planet; a 2-Earth-mass planet with density approximately 5.5 g/cm3; and planets with Earth-like composition at five times or half Earth’s mass.Source: D CASES · p. 33

• The reasoning repeatedly treats same-composition planets as having Earth’s density and concludes every option is equal, then struggles to reconcile that tie with a single-choice question.It cycles through mass, volume, and density explanations without resolving the contradiction before its stated answer.Source: D CASES · pp. 33, 37
• An earlier attempt instead chooses option c (C), speculating that its greater mass causes compression and higher density while acknowledging uncertainty.It presents this as an assumption rather than a confident calculation.Source: D CASES · p. 37
• The final answer selects option b (A) at 5.5 g/cm3 because it is the only option explicitly assigned that density, despite saying all four options tie.The answer states that options c and d are implicitly equal to Earth’s density because they have the same composition.Source: D CASES · p. 37
• The exoplanet trace selects C after assigning c 27.5 g/cm3 and d 2.75 g/cm3, but later says all four options are 5.5 g/cm3 while still concluding C.The later account says c has the same composition and density as Earth, yet calls it the highest-density option.Source: D CASES · pp. 38–39

Case in Code Reasoning

• The product test also covers zero, floating-point inputs, and large integers, according to the explanation.The explanation says zero yields False and gives examples of floats and large integers.Source: D CASES · p. 40
• The code solution returns `a * b < 0` and reports that all provided test cases pass.A zero product returns False, so zero is not treated as having an opposite sign.Source: D CASES · pp. 40–41

E LIMITATIONS

• REPRO’s scalability beyond models with 8B parameters remains an open question because larger-scale experiments were not feasible.The authors cite computational resource and time constraints.Source: E LIMITATIONS · p. 42
• REPRO’s efficacy on subjective or difficult-to-verify problems is unexplored; reported results cover verifiable objective benchmarks in mathematics, science, and coding.Source: E LIMITATIONS · p. 42

F LLM USAGE

• LLMs were used only to refine and proofread the manuscript, not to develop the methodology, design experiments, or analyze and interpret results.Source: F LLM USAGE · p. 42