PaperlayerView exact revision on arXiv

Source-linked AI summary

Inducing Process Supervision from Outcome-Only Reinforcement Learning

Shengda Fan, Xin Cong, Zhong Zhang, Haotian Chen, Yankai Lin

arXiv:2609.36641v1cs.LGcs.AI

TL;DR

Strong process reward models are costly to train because human step labels are difficult to scale and Monte Carlo methods require extensive rollouts. TIPS uses outcome-only reinforcement learning to train a generative model to produce step judgments, and its Qwen3-4B variant reaches 85.2 F1 on ProcessBench with 3.2K outcome-labeled examples.

  • Problem

    Training strong process reward models remains costly because human step annotation is hard to scale and Monte Carlo methods require massive rollouts while defining step quality through correct continuations.

    Source: §1 · p. 2

  • Method

    TIPS trains a generative reward model to produce a chain of thought, step labels, and an outcome label, using outcome correctness alone for reward.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    With 3.2K outcome-labeled examples, TIPS-Qwen3-4B-Thinking-2507 achieves 85.2 F1 on ProcessBench and improves AgentProcessBench FirstErrAcc across tested backbones.

    Source: §5 · p. 10

  • Takeaways & Limitations

    TIPS induces process supervision from outcome-only RL, with CoT and the backbone’s CoT prior identified as key carriers of the induced signal.

    Source: §1 · p. 2, §5 · p. 10

    Transfer can weaken when input shortcuts predict outcomes, outcome optimization fails, step correctness is loosely coupled to success, or the backbone cannot express faithful verification.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Process reward models (PRMs) have become a key component for LLMs, as their step-level feedback supports both post-training and test-time reasoning. However, training strong PRMs remains costly: human step annotation is difficult to scale, while Monte Carlo estimation is computationally expensive and can drift from the intrinsic correctness of steps. To get effective PRMs at low cost, we introduce TIPS (Thinking-Induced Process Supervision), an outcome-only reinforcement learning (RL) framework for training generative PRMs. In TIPS, the model generates a chain-of-thought (CoT) followed by step-level labels and an outcome label. The reward depends solely on whether the predicted outcome matches the ground truth, and the resulting group-relative advantage is used to optimize the entire generated response. Intuitively, when checking intermediate steps helps determine the outcome, more accurate checks can lead to better outcome judgments and higher rewards. Outcome-only RL can therefore reinforce step-level verification without explicit process supervision. We validate the effectiveness of TIPS across math and agent benchmarks and four backbone families. Notably, TIPS-Qwen3-4B-Thinking-2507 reaches 85.2 F1 on ProcessBench with only 3.2K outcome-labeled trajectories, surpassing all evaluated trained PRMs and strong prompt-only judges such as GPT-5.4-Instruct and Claude-4.7-Opus, while still trailing o1-mini. Code and data are available at https://github.com/RUCBM/TIPS.

1 INTRODUCTION

Strong process reward models are costly to build because human step annotation is difficult to scale and Monte Carlo labels can diverge from intrinsic step correctness. TIPS asks whether outcome-only reinforcement learning can strengthen process verification through generated thoughts.

1 INTRODUCTION

  • Human step annotation is expensive and hard to scale, while Monte Carlo methods require massive rollouts and may treat a correct continuation as evidence of step quality.A correct answer reachable from a step does not necessarily mean that step itself is correct.Source: §1 · p. 2
  • The motivating question is whether improving outcome verification can, in turn, strengthen process verification.The authors observe that an ORM may inspect intermediate steps in its thought before judging the final answer.Source: §1 · p. 2
  • TIPS uses outcome-only reinforcement learning to strengthen step-level verification as a byproduct, without direct process supervision.The paper also provides an information-theoretic lower bound on information about step correctness encoded in the model’s thought.Source: §1 · p. 2
  • TIPS is validated on mathematical reasoning and agent tasks across multiple backbone families.The supplied introduction lists Qwen, DeepSeek, LLaMa, and SmolLM families.Source: §1 · p. 2

2 RELATED WORK

Prior process reward models use costly human labels or automatically constructed labels, while outcome-supervised approaches can struggle to localize errors. TIPS instead uses outcome-only RL to induce step-level verification in a generative model.

2 RELATED WORK

  • Prior PRMs commonly learn from human annotations, Monte Carlo labels, or stronger-model distillation, but these approaches retain supervision or rollout costs.Discriminative PRMs often use an encoder with a scalar scoring head rather than the model’s generative reasoning capability.Source: §2 · p. 3
  • Outcome-supervised ImplicitPRM and AutoPSV are close supervision precedents, but their induced signals often fail to localize step-level errors.Recent generative PRMs produce reasoning before step judgments but still rely on explicit process supervision.Source: §2 · p. 3
  • Unlike RL work focused on improving outcome reward models, TIPS uses outcome-only RL to induce a process reward model as a by-product.The cited RL applications include mathematical reasoning, tool-using agents, retrieval, and code generation.Source: §2 · p. 3

3 THINKING-INDUCED PROCESS SUPERVISION

TIPS trains a generative reward model to produce thoughts and both step-level and outcome judgments, while using only outcome correctness for RL reward. Its analysis explains when outcome feedback can carry process information and when that transfer weakens.

3 THINKING-INDUCED PROCESS SUPERVISION

  • The section frames TIPS as a framework that moves from motivation to training design and then an information-theoretic account.The account is conditioned on nontrivial verification, where the pre-reasoning representation does not fully determine the outcome.Source: §3 · p. 3, §3.3 · p. 4

3.1 MOTIVATION

  • TIPS is motivated by the observation that an ORM’s thought can inspect intermediate steps before producing an outcome verdict.If more faithful step checks help predict the trajectory outcome, RL can reinforce those thought patterns without step annotations.Source: §3.1 · p. 3

3.2 METHODOLOGY

  • TIPS prompts a model to generate a thought, then process labels for each step and an outcome label for the trajectory.The process labels indicate step validity given preceding steps; the outcome label predicts whether the full trajectory is correct.Source: §3.2 · p. 3, Figure 3 · p. 4
  • Only outcome correctness determines reward, and a group-relative advantage reinforces the generated response; process labels are used as step-level labels at inference.Rollouts with correct outcome verdicts receive higher relative advantage; the model requires outcome-labeled trajectories for training.Source: §3.2 · p. 4

3.3 ANALYSIS

  • By Fano’s inequality, low outcome error requires the thought to explain some uncertainty about the outcome that remains after observing the pre-reasoning representation.This is the outcome-information component of the analysis before relating outcome information to process information.Source: §3.3 · p. 4
  • The analysis relates information in the thought about step labels to outcome information, less residual outcome uncertainty after step labels are known.It defines the pre-reasoning representation R, reasoning trace Z, ground-truth step labels Y, and outcome D for the analysis.Source: §3.3 · pp. 4–5
  • When outcome prediction requires tracking step validity, outcome-only RL can favor thoughts that encode process-relevant information despite never observing step labels.Correct outcome judgments receive higher relative advantage across sampled rollouts.Source: §3.3 · p. 5
  • The transfer can weaken when input shortcuts predict the outcome, outcome optimization is ineffective, step validity loosely determines task success, or the backbone cannot express faithful verification.These conditions respectively reduce uncertainty in the outcome beyond the representation, keep outcome error high, leave outcome uncertainty after step labels, or limit faithful step-by-step reasoning.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate TIPS on mathematical reasoning and agent tasks, across multiple model families, using process-verification and response-selection benchmarks. Results show strong math performance, gains on agent tasks, and analyses of when outcome supervision transfers to process verification.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans Qwen, DeepSeek, Llama, and SmolLM backbones; math measures Best-of-8 selection accuracy and ProcessBench F1, while agent tasks use FirstErrAcc.Comparisons include prompt-only judges, published discriminative and generative PRMs, and outcome-supervised implicit PRMs.Source: §4.1 · pp. 5–6

4.2 RESULTS ON MATH

  • 85.2 average F1 on ProcessBench: TIPS-Qwen3-4B-Thinking-2507 surpasses evaluated trained PRMs and GPT-5.4-Instruct and Claude-4.7-Opus, while trailing o1-mini.The result uses 3.2K outcome-labeled trajectories.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 RESULTS ON AGENT

  • TIPS improves average AgentProcessBench FirstErrAcc for all three backbones, with gains of +5.4 and +4.5 for Qwen3-4B-Thinking-2507 and Qwen3-8B-Thinking, versus +2.5 for Qwen3-4B-Instruct-2507.AgentProcessBench gains are smaller and more heterogeneous than those on ProcessBench; agent failures may not correspond to a clearly invalid observed step.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 DISCUSSIONS

  • Using 3.2K outcome-only trajectories, TIPS-Qwen3-4B-Thinking-2507 reaches 73.3 Avg, 4.5 points above 197K-scale ImplicitPRM, but 4.8 below the Pass@8 oracle.For TIPS, Best-of-8 reranking filters trajectories predicted to have correct outcomes, then selects the retained trajectory with the highest mean step-validity score.Source: §4.4 · pp. 7–8, Table 3 · p. 8
  • TIPS improves ProcessBench F1 across DeepSeek-R1-distilled, Llama-based, SmolLM, Qwen2.5-7B-Instruct, and Qwen3-8B models, extending beyond one backbone family.Source: Figure 4 · p. 8, §4.4 · p. 8
  • ORM accuracy and process metrics generally improve together, with Pearson correlations from 0.78 to 1.00; math has larger gain ratios than agent tasks.The paper attributes weaker agent transfer to a more indirect relation between local step correctness and final task success.Source: §4.4 · p. 8, Figure 5 · p. 9
  • Outcome-to-process transfer depends on model priors: Qwen2.5-7B-Instruct's ProcessBench F1 peaks early and deteriorates despite rapidly improving ORM accuracy, unlike the distilled model's coupled gains.The authors suggest using strong CoT-prior backbones and propose stopping when ORM accuracy plateaus and response length declines.Source: Figure 6 · p. 9, §4.4 · p. 9
  • Replacing CoT with label-only prompting yields much lower base ProcessBench F1 and little improvement from outcome-supervised RL, indicating that reasoning traces carry the induced signal.The label-only variant also has near-zero entropy, shorter outputs, and smaller KL loss.Source: Table 4 · p. 9, §4.4 · p. 9

5 CONCLUSION

TIPS induces process supervision through a generative reward model’s chain of thought, using outcome-only RL. The conclusion reports strong benchmark results with 3.2K outcome-labeled examples, alongside factors that support or limit transfer.

  • 85.2 F1 on ProcessBench with 3.2K outcome-labeled examples, while AgentProcessBench FirstErrAcc improves across all tested backbones.The reported model is TIPS-Qwen3-4B-Thinking-2507.Source: §5 · p. 10
  • TIPS induces process supervision through the chain of thought of a generative reward model using outcome-only reinforcement learning.Source: §5 · p. 10
  • CoT and the backbone’s CoT prior carry the induced signal, while weak outcome–process coupling and outcome shortcuts limit transfer.The authors identify light Monte Carlo priors and CoT distillation as possible future additions.Source: §5 · p. 10

A LIMITATIONS

The limitations concern the evaluated settings and the conditions under which outcome rewards induce reliable process judgments. The broader-impact discussion also flags potential misuse and calls for validation and oversight.

A LIMITATIONS

  • Evaluation covers pure-text reasoning and agent trajectories, not multimodal settings requiring verification grounded in non-textual observations.Source: A LIMITATIONS · p. 14
  • Induced process supervision weakens when success is only indirectly tied to local correctness or the backbone cannot express verification in generated thoughts.For weaker backbones or label-only prompting, outcome RL may improve outcome prediction without reliably producing faithful step-level judgments.Source: A LIMITATIONS · p. 14

B BROADER IMPACTS

  • TIPS labels may be biased, miscalibrated, or unreliable where final success is weakly tied to local step correctness, so they are not ground truth for high-stakes use.Source: B BROADER IMPACTS · p. 14
  • The authors call for domain-specific validation, human oversight, and separate evaluation of outcome- and step-level behavior, while noting stronger reward models may raise harmful-agent capabilities.Source: B BROADER IMPACTS · p. 14

C GRPO OBJECTIVE

The GRPO training objective samples rollout groups, assigns binary rewards, and computes group-relative advantages before optimizing a clipped surrogate objective. The implementation uses a non-normalized advantage variant.

  • For each input, GRPO samples G rollouts from the old policy, assigns each a binary reward, and computes a group-relative advantage.Source: C GRPO OBJECTIVE · p. 14
  • The method uses a non-normalized GRPO variant that does not divide the advantage by the group standard deviation.Source: C GRPO OBJECTIVE · p. 14
  • The model is updated by maximizing a clipped surrogate objective.Source: C GRPO OBJECTIVE · p. 14
  • |o(i)| denotes the i-th rollout’s length, and ϵ is the clipping threshold.Source: C GRPO OBJECTIVE · p. 14

D DERIVATIONS FOR THINKING-INDUCED PROCESS SUPERVISION

The appendix provides proofs of two theorems using conditional mutual information and entropy bounds. One relates process information to a binary prediction error; the other bounds outcome information using process information and residual uncertainty.

E IMPLEMENTATION DETAILS.

The section specifies TIPS’s math and agent training setup and the prompt format for producing step-level and final judgments.

E IMPLEMENTATION DETAILS.

  • Math training uses 3,200 SCAN-Pro1 trajectories, while agent training uses 2,905 trajectories from four datasets; training runs for one epoch with batch size 32 and learning rate 1 × 10−6.Experiments use eight NVIDIA A800 GPUs, verl, a 16K-token maximum rollout, and KL penalty coefficient 0.Source: E IMPLEMENTATION DETAILS · p. 15

F PROMPT DESIGN

  • The math prompt asks the model to inspect a paragraph-indexed solution, identify the first invalid reasoning step, and return step_labels and a final_label as JSON.The prompt requires one 0/1 step label per paragraph, with later steps also labeled incorrect when they depend on an earlier error.Source: F PROMPT DESIGN · p. 16
  • For each paragraph, a label of 1 means it is correct and logically valid given previous paragraphs; 0 marks an incorrect, unsupported, or dependent step.Source: F PROMPT DESIGN · p. 16

G HUMAN EVALUATION OF PROCESS-LEVEL FEEDBACK

In a 60-trajectory human evaluation, annotators assessed whether TIPS localized the first error and explained it accurately.

H DOES TIPS SIMPLY IMPROVE PROBLEM-SOLVING ABILITY?

TIPS’s process-verification gains are substantially larger than its changes on two solver benchmarks, and evaluated models with stronger solver scores do not necessarily score higher on ProcessBench.

H DOES TIPS SIMPLY IMPROVE PROBLEM-SOLVING ABILITY?

  • The analysis compares mean@4 accuracy on AIME 2026 and HMMT February 2026 with ProcessBench F1.Source: H DOES TIPS SIMPLY IMPROVE… · p. 16, Table 5 · p. 17
  • 24.9 and 13.9 F1-point gains on ProcessBench for Qwen3-4B-Thinking-2507 and Qwen3-4B-Instruct-2507, respectively, accompany much smaller solver-benchmark changes.The thinking model improves modestly on both solver benchmarks; the Instruct model is unchanged on AIME 2026 and declines on HMMT February 2026.Source: H DOES TIPS SIMPLY IMPROVE… · p. 16, Table 5 · p. 17
  • The solver-format RLVR baseline scores higher than TIPS-Qwen3-4B-Instruct-2507 on both solver benchmarks but trails by 14.9 F1 points on ProcessBench.Source: H DOES TIPS SIMPLY IMPROVE… · p. 17
  • GooseReason-4B-Instruct and Qwen3-30B-A3B-Thinking-2507 score higher on solver benchmarks but trail their TIPS counterparts by 5.0 and 4.9 ProcessBench F1 points, respectively.Source: H DOES TIPS SIMPLY IMPROVE… · p. 17
Loading 2609.36641v1…