Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models

1 Introduction

• HRM combines fine-grained single-step and coarse-grained consecutive multi-step supervision, allowing later steps to correct earlier reasoning errors during evaluation.At inference, HRM remains step-wise but incorporates multi-step coherence learned during training.Source: §1 · p. 2, Figure 1 · p. 3
• HRM provides more stable and reliable reward evaluations than PRM and ORM in Best-of-N search on PRM800K, with accuracy stabilizing at 80% as N increases.PRM- and ORM-based policy performance fluctuates and degrades as N grows.Source: §1 · p. 2
• HNC merges two consecutive reasoning nodes into one, expanding MCTS training data with controlled noise and minimal computational overhead.The approach is intended to increase training-data diversity and label robustness.Source: §1 · p. 2
• Across GSM8K and MATH500, HRM shows stronger reasoning consistency, generalization, and robustness than PRM.The paper reports consistent outperformance of PRM across these cross-domain evaluations.Source: §1 · p. 2
• The paper additionally fine-tunes the policy model on high-quality MCTS-filtered reasoning trajectories to improve reasoning performance.This self-training step uses trajectories selected from the reward-model training data.Source: §1 · p. 2

2.1 RLHF

• RLHF uses a reward model to distinguish response quality and optimize an LLM with PPO.The framework incorporates human feedback into language-model optimization.Source: §2.1 · p. 3
• ORM evaluates complete outputs, whereas PRM evaluates intermediate reasoning steps for finer-grained supervision.These approaches represent the two main reward-design strategies discussed in the related work.Source: §2.1 · p. 3

2.2 ORM

• ORM suffers from delayed feedback and credit-assignment problems, and can reinforce spurious reasoning when correct answers arise from flawed intermediate steps.Rule-based ORM is cited as a separate approach that can guide long-chain reasoning and self-reflection.Source: §2.2 · p. 3

2.3 PRM

• PRM is vulnerable to reward hacking, in which agents exploit reward-function flaws without genuinely completing the intended task.The related work also identifies PRM’s human-annotation requirements as prohibitively expensive.Source: §2.3 · p. 3
• MCTS automates reasoning-trajectory annotation, but constraints on tree height and width reduce the diversity of generated reasoning data as search cost grows.Figure 2 depicts reasoning steps as tree nodes and their assigned scores.Source: §2.3 · p. 3, Figure 2 · p. 4

3.1 Hierarchical Reward Model

• HRM trains on both individual steps and consecutive multi-step segments, linking local correctness with global coherence while allowing later steps to repair earlier errors.Although training uses merged segments, inference remains step-wise and assigns rewards from the current step, as in PRM.Source: §3.1 · p. 4
• Two-step merging is used as a minimal hierarchical-supervision design that preserves step-level evaluation while smoothing annotation noise from ambiguous boundaries.The merged training data forms a superset of PRM supervision and maintains computational efficiency.Source: §3.1 · pp. 4–5

3.2 Hierarchical Node Compression in MCTS

• MCTS annotation requires deep and wide search trees for reliable intermediate-step scores, creating exponential computational growth that hinders large-scale implementation.Each node score reflects the proportion of correct trajectories in its subtree.Source: §3.2 · p. 5
• HNC merges two consecutive MCTS nodes into one, creating augmented branches with minimal computational overhead and controlled noise.The method assumes each node has sufficiently many children and randomly merges consecutive nodes.Source: §3.2 · p. 5
• HNC preserves the expected parent score while increasing variance from σ^2/N to σ^2/(N−1), with the change remaining moderate when N is sufficiently large.This variance shift provides low-cost data augmentation while maintaining the expected score.Source: §3.2 · p. 5

3.3 Self-Training

• Self-training filters high-quality MCTS reasoning with MC-Score and uses supervised fine-tuning rather than PPO or GRPO under computational constraints.The objective combines causal language-modeling loss with KL divergence regularization against a reference model.Source: §3.3 · p. 5
• The self-training objective applies logarithmic KL scaling because KL loss can reach 20000 while causal LM loss remains within 0 to 12, causing severe loss imbalance.The scaling stabilizes optimization while balancing general-capability retention against reasoning improvement.Source: §3.3 · pp. 5–6
• Table 2 evaluates ORM, PRM, and HRM on PRM800K using best-of-N selection with Qwen2.5-72B-Math-Instruct fine-tuned on manually labeled data.The supplied passage identifies the comparison scope but does not provide the table’s numerical outcomes.Source: Table 2 · p. 6

4.1 HRM

• HRM training extends PRM by labeling consecutive reasoning steps, enabling evaluation of self-reflection and sequential coherence from manual PRM800K annotations.ORM uses complete trajectories, PRM individual steps, and HRM multiple consecutive steps.Source: §4.1 · p. 6
• HRM maintains stable performance as best-of-N grows, converging to 80% accuracy while ORM and PRM fluctuate and decline.The evaluation uses Qwen2.5-72B-Math-Instruct, with PRM and HRM scoring intermediate steps and ORM selecting among complete trajectories.Source: §4.1 · p. 6

4.2 HNC

• HNC adds roughly 30% more training samples, raising training time from about 84 to 120 A100 GPU hours, while MCTS generation remains the dominant cost at over 2,400 GPU hours.The augmentation itself takes around 30 minutes on CPU, compared with approximately 2,457 A100 GPU-hours for full MCTS simulation.Source: §4.2 · p. 7
• HRM trained with HNC-generated data consistently shows greater stability and robustness than PRM across policy models on PRM800K.HNC augments MCTS-generated data; the experiments use multiple policy models and report results in Table 3.Source: §4.2 · pp. 6–7, Table 3 · p. 7
• Qwen2.5-72B-Math-Instruct performance is constrained by MCTS templates requiring at most six steps and explicit “# Step X” formatting, causing some outputs to be unretrievable by regex post-processing.This formatting mismatch affects measured performance despite the policy model's strong reasoning capability.Source: §4.2 · pp. 7–8

4.3 Self-Training

• When λ = 0.001, the policy rapidly loses generalization within a few iterations and KL loss rises to approximately 20,000, while λ = 10.0 slows convergence.For λ = 0.001, causal LM loss remains between 0 and 12; λ = 10.0 prioritizes adherence to the reference distribution and constrains reasoning improvements.Source: §4.3 · p. 8
• Fine-tuning on high-MC-score reasoning data yields greater robustness for HRM than PRM, using Qwen2.5-7B-Math-Instruct and λ = 0.5.The high-quality data are selected with MC-score > 0.9 from the respective reward-model training sets.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 HRM Generalization Across Different Domains

• When trained only on PRM800K, HRM generalizes more robustly than PRM to MATH500 and GSM8K, particularly on complex mathematical reasoning in MATH500.The HRM–PRM difference is marginal on GSM8K, where problems are usually solvable within three steps, although HRM retains a slight edge.Source: §4.4 · pp. 8–9, Table 5 · p. 8

• HRM combines fine-grained and coarse-grained assessments to improve reasoning coherence and self-reflection.Source: §5 · p. 9
• HNC optimizes MCTS-based autonomous annotation by expanding training data and enhancing label diversity with minimal computational cost.Source: §5 · p. 9
• Experiments on PRM800K show HRM is more robust than PRM and generalizes strongly across GSM8K and MATH500.Source: §5 · p. 9
• MCTS-generated auto-labeled data supports policy-model fine-tuning that further improves reasoning performance.Source: §5 · p. 9

6.1 Merged Steps

• HNC merges only two consecutive steps because larger merges create rapidly increasing label combinations and potential conflicts.Source: §6.1 · p. 9
• Merging four steps can produce combinations such as one positive, two negative, and one neutral label, complicating unified labeling rules.Source: §6.1 · p. 9
• The two-step design preserves straightforward and consistent labeling rules.Source: §6.1 · p. 9

A.1 Labeling Rules for HRM Training Data

• Table 6 defines labeling rules for merging consecutive reasoning steps in the manually labeled PRM800K dataset.Source: §A.1 · p. 11
• Consecutive neutral steps are labeled negative to penalize non-progressive and inefficient reasoning.Source: §A.1 · p. 11
• The labeling design encourages concise and goal-directed reasoning.Source: §A.1 · p. 11

A.2 HRM Training Details

• A reasoning step ends at a formula boundary marked by '\]\n\n', '\)\n\n', or '# END!'.Source: §A.2 · p. 11
• The prompt requires step-by-step reasoning followed by an answer beginning with '# Answer ' and ending with '# END!'.Source: §A.2 · p. 11
• Example reasoning includes sequential arithmetic steps and terminates with '# END!'.Source: §A.2 · p. 11

A.3 HNC Setting Details

• MCTS-generated reasoning is segmented at explicit # Step X delimiters because constrained tree width and height prevent treating a complete formula as one step.The same separators are used during inference with Best-of-N path selection.Source: §A.3 · p. 11
• The prompt requires each reasoning step to begin with # Step X, followed by a repeated answer and the terminating # END! marker.The expected format also specifies an answer-only final response.Source: §A.3 · p. 11
• HRM training data extends PRM800K's Positive, Negative, and Neutral manual labels with multi-step reasoning labels.This labeling strategy distinguishes HRM from PRM's original single-step supervision.Source: Table 6 · p. 12

A.4 Self-Training

• Without KL loss, the policy model rapidly loses generalization ability even as evaluation loss continuously decreases.KL regularization is introduced to constrain training relative to a reference model.Source: §A.4 · p. 12
• Logarithmic scaling and weighting factor λ stabilize KL loss because its unscaled range can greatly exceed the language-modeling loss range.The cited ranges are 0 to 20000 for KL loss and 0 to 12 for language-modeling loss.Source: §A.4 · p. 12
• λ = 0.5 balances KL and language-modeling losses, preventing excessive divergence from the reference model during stable training.The result is illustrated in Fig. 4.Source: §A.4 · p. 12