Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

• Current alignment methods rely on human demonstrations or judgments, limiting supervision when tasks exceed human expertise; easy-to-hard generalization asks whether easy-task supervision can enable hard-task success.The setting requires no human supervision on the harder tasks.Source: §1 · pp. 1–2
• Because SFT or in-context learning on easy tasks often fails to generalize to hard tasks, the authors propose using evaluators, based on the intuition that evaluation is easier than generation.An evaluator trained on easy tasks can supervise a generator on hard tasks.Source: §1 · p. 2
• The proposed approach trains outcome- and process-supervised reward models on easy tasks, then uses their evaluations to guide hard-task generation through re-ranking or reinforcement learning.The training setup pairs labeled easy questions and high-quality solutions with unlabeled hard questions.Source: §1 · p. 2
• Training with human supervision only on Level 1–3 problems and answers can outperform SFT and Final-Answer RL trained on the full Level 1–5 dataset.The paper uses this finding to argue that easy-to-hard evaluation can improve generators across varying task complexity.Source: §1 · p. 3

2.1 Scalable Oversight

• Unlike approaches that use AI to evaluate other AIs [1] [38] [60] [9], this framework enables models to excel at hard tasks where human supervision may be unavailable, unlike weak-to-strong generalization [11], where supervision may be unreliable.It aligns with the “sandwiching” concept, in which domain experts evaluate outputs of AI-assisted non-experts [9] [57].Source: §2.1 · p. 3

2.2 Compositional Generalization

• Least-to-most prompting [95] [20] achieves perfect easy-to-hard generalization on SCAN [37] and CFQ [35], and has also succeeded on mathematical reasoning in GSM8K [16] and DROP [21].The cited work connects this success to language models’ capacity to break complex tasks into simpler sub-tasks (Perez et al. [51]).Source: §2.2 · p. 4

2.3 Easy-to-Hard Generalization

• Prior work trains easy-to-hard generators on easy tasks using SFT or ICL, while this paper evaluates on challenging MATH [32] and reports gains beyond those approaches.Prior findings vary: BERT performed poorly on common-sense reasoning trained only on easy data [71], and ICL showed similar limits on GSM8K [24]; concurrent work found ICL or SFT strong on multiple-choice ARC [15] and MMLU [30]. Iterative machine teaching [43] theoretically supports training classifiers from easy to hard examples.Source: §2.3 · p. 4

3 Methodology

• The study treats MATH levels 1–3, with demonstrations and answers, as easy tasks and levels 4–5, represented only by questions, as hard tasks.MATH has 12,500 problems; MATH500 is the test set, 500 problems are sampled for validation, and the remaining 11,500 problems form the training set. SFT uses filtered PRM800K and MetaMATH data, with process labels from human annotations or Math-Shepherd [75].Source: §3 · p. 4, §3.1 · p. 5

3.2 Generators

• Generator baselines include full- and hard-level ICL, easy-to-hard ICL and SFT, and full SFT, evaluated with greedy decoding or self-consistency [76].Full SFT is treated as a task-specific ceiling [11] [29], while easy-to-hard SFT may generalize to hard tasks with performance degeneration [71].Source: §3.2 · p. 5

3.3 Evaluators

• OPRM combines process-level judgments on intermediate steps with outcome-level assessment of the final answer, alongside ORM and PRM evaluator baselines.The setup considers evaluators trained on easy tasks or the full dataset; unlike final-answer rewards, reward models trained on easy tasks can score hard-task solutions.Source: §3.3 · pp. 5–6

3.4 Optimizing Generators Against Evaluators

• Generators are optimized against evaluators by best-of-n reranking, reward-weighted voting, or ReST, DPO, and PPO reinforcement learning.Best-of-n selects the highest-scoring sampled solution; weighted voting weights solutions by reward-model scores.Source: §3.4 · p. 6

3.5 Evaluation Metrics

• Evaluation focuses on generators’ absolute and relative MATH500 performance, rather than weak-to-strong or easy-to-hard supervision-gap recovery metrics [11, 29, 40].The authors note that models trained only on simpler tasks, particularly with RL, can sometimes outperform models trained across all difficulty levels.Source: §3.5 · p. 6

3.6 Implementation Details

• Figure 4 reports that easy-trained PRMs improve weighted-voting reranking on hard tasks; experiments use 7b and 34b Llemma models.Models are fully fine-tuned with input-output embeddings and normalization layers frozen; reward models add a scalar head, and PPO’s value model is initialized from the reward model.Source: §3.6 · pp. 6–7, Figure 4 · p. 7

4.1 Easy-to-Hard Generalization of Generators

• 35.4 for full MetaMath SFT exceeds 32.2 for its easy-to-hard counterpart by 3.2; only MetaMath-trained models show clear easy-to-hard gaps, unlike PRM800K.SFT also outperforms ICL; the authors hypothesize that lower-quality SFT data may teach task format, while high-quality imitation data can teach solving principles.Source: §4.1 · p. 7

4.2 Easy-to-Hard Generalization of Evaluators

• A 34b evaluator significantly improves a 7b generator, while a 7b evaluator also improves a 34b generator.Both generators and evaluators are trained on easy data in this comparison.Source: Figure 5 · p. 8, §4.2.1 · p. 8

4.2.1 Re-ranking

• OPRMs outperform PRMs and ORMs, and weighted voting beats Best-of-n and majority voting, with its advantage growing on harder tasks.On MetaMath/Math-Shepherd, adding ORM training examples yields no significant improvement; Best-of-n remains competitive with majority voting when producing one full solution.Source: §4.2.1 · p. 8

4.2.2 Reinforcement Learning (RL)

• Top performance of 34.0 with easy-task human PRM rewards exceeds the 33.0 prior state of the art; PPO leads on PRM800K, while PPO and DPO lead on MetaMATH.DPO is on par with PPO when initialized from high-quality SFT, whereas PPO is less dependent on SFT quality.Source: Table 2 · p. 9, §4.2.2 · p. 9

4.3 Easy-to-Hard Generalization on the Coding Domain

• An ORM trained on outcome supervision from easy Introductory APPS data significantly improves model performance on hard Interview and Competition problems, including Best-of-N reranking performance.The APPS evaluation subsampled 500 test questions: 100 Introductory, 300 Interview, and 100 Competition.Source: §4.3 · p. 10, Table 3 · p. 10

5 Conclusion

• Easy-to-hard evaluators enabled policy models to generalize to difficult tasks through reranking and reinforcement learning, offering a scalable alignment direction.The authors frame the findings as a promising path for AI systems to advance in complex domains without direct human supervision on harder tasks.Source: §5 · p. 10
• The study is limited by its focus on specific model sizes, datasets, and reasoning domains, with long-term implications and robustness left for future research.Source: §5 · p. 10

A.1 Rationale-Augmented (Mathematical) Reasoning

• The paper evaluates easy-to-hard generalization in mathematical reasoning, a task with clear difficulty levels, asking whether models can generalize to more challenging problems when trained only on easy mathematical data.Source: §A.1 · p. 18

A.2 Outcome Reward Models & Process Reward Models

• Unlike prior PRM studies of IID settings, this work studies PRMs as easy-to-hard evaluators and reports that they can enable policy models to generalize to harder tasks.Prior work reported PRM advantages over ORMs in reranking, with similar performance in RL via expert iteration in one study [74] and significant reranking gains in another [40].Source: §A.2 · p. 18

B Reinforcement Learning Algorithms

• ReST alternates policy sampling with reward-model-weighted supervised fine-tuning, while DPO trains on paired positive and negative samples and PPO samples from the policy during fine-tuning.The passage identifies expert iteration [4] and rejection sampling fine-tuning [72] [87] as ReST variants, and cites PPO use in RLHF [7] [50] and RLAIF [69].Source: B Reinforcement Learning Algorithms · p. 18

C.1 Supervised Fine-Tuning & Reward Modeling

• SFT uses correct-labeled PRM800K steps, while MetaMath training filters to at most three solutions per question per epoch to reduce overfitting.The PRMs are trained on their corresponding released datasets; ORM solution generation samples 32 solutions per question with top-K 20 and temperature 0.7, balancing positive-to-negative samples between 1:3 and 3:1.Source: §C.1 · p. 19

C.2 Re-Ranking

• For reranking, the ORM scores a solution at its final token, whereas the PRM scores intermediate reasoning steps and the final token.Sampling uses top-K 20 and temperature 0.7 for majority voting, weighted voting, and best-of-n.Source: §C.2 · p. 19

C.3 Reinforcement Learning

• RL implementation details include two ReST-EM epochs, with performance degeneration observed from epoch three, and PPO training for 500 steps on non-MATH500 training and validation questions.For iterative DPO, preference pairs require a score difference greater than 1.0; the PPO setup initializes the value model from the reward model and clips gradient norm at 1.Source: §C.3 · pp. 19–20

D Re-ranking Results on MetaMath

• PRMs are more effective on harder tasks when trained on MetaMath/Math-Shepherd [86] [75].Figure 6 evaluates easy-to-hard generalization with both SFTs and reward models trained on easy data.Source: D Re-ranking Results on MetaMath · p. 20, Figure 6 · p. 21
• Table 6 reports Pass@N scores, the upper bound of Best-of-N, on APPS coding problems.Source: Table 6 · p. 21
• Figure 6 presents easy-to-hard evaluator generalization with both SFTs and reward models trained on easy data.Source: Figure 6 · p. 21

E Qualitative Comparison

• Table 7 compares MetaMath-7b-SFT with MetaMath-7b-PPO and shows the first error Math-Shepherd detected in SFT greedy decoding.The examples are cherry-picked for illustration.Source: Table 7 · p. 22

F Performance of DPO Iterations

• Figure 7 plots test performance as a function of DPO iterations.Source: Figure 7 · p. 22
• The authors suggest that a large pre/post-PPO distribution shift may explain PPO’s exceptional performance on PRM800K from a weak SFT initialization.Table 8 reports the shift for a Llemma-7B model trained on PRM800K.Source: Table 8 · p. 23

G Re-ranking Performance Analysis of PRMs, ORMs & OPRMs

• OPRMs significantly outperform PRMs and ORMs, which perform similarly, with PRMs slightly ahead on hard tasks in the Llemma + MATH setting.Figures 8 and 9 compare 7b and 34b models; both SFTs and reward models are trained on easy data.Source: Figure 8 · p. 24, G Re-ranking Performance Analysis of… · p. 24, Figure 9 · p. 25

H Re-ranking Results on MetaMath

• PRMs trained on MetaMath/Math-Shepherd are more effective on harder tasks, extending the reported easy-to-hard reward-model result to this dataset.The comparison includes PRMs, ORMs, and OPRMs for 7b models trained on MetaMath, with both SFTs and reward models trained on easy data.Source: H Re-ranking Results on MetaMath · p. 26, Figure 10 · p. 26

I More Comparisons

• Figures 11–14 compare weighted voting and Best-of-N across 7b and 34b PRM800K settings and 7b MetaMath settings, with SFTs and reward models trained on easy data.The comparisons also include a 34b SFT paired with a 7b reward model on PRM800K.Source: Figure 11 · p. 27, Figure 12 · p. 27, Figure 13 · p. 28

J Hard-to-Easy Generalization

• Full-data SFT consistently performs best overall, while performance deteriorates when training transfers between easy and hard tasks.On hard test questions, Full SFT trails Easy-to-Hard and Hard-to-Easy SFT, but it outperforms both across all test questions.Source: Table 9 · p. 29, J Hard-to-Easy Generalization · p. 29

I More Comparisons

• Figure 15 compares agreement between a Llemma-7b reward model trained on ORM or PRM data and agreement with ground-truth final-answer labels.Source: Figure 15 · p. 29

J Hard-to-Easy Generalization

• Hard-task-only reinforcement learning underperforms training on easy tasks or all tasks, especially for PPO on PRM800K.This motivates asking whether training only on hard tasks generalizes to easy tasks.Source: J Hard-to-Easy Generalization · p. 29
• Training on all data consistently beats easy-only or hard-only Llemma-7b SFT, while transferring from either task difficulty to the other reduces generator performance.Source: J Hard-to-Easy Generalization · p. 29, Table 9 · p. 29
• Full SFT trails Easy-to-Hard and Hard-to-Easy SFT on hard test questions but surpasses both across all test questions.The authors attribute the full-distribution advantage to exposure to a wider variety of unique questions and difficulties.Source: J Hard-to-Easy Generalization · p. 29, Table 9 · p. 29

K On ORM’s Approximation of PRM Labels

• ORMs had higher agreement with PRMs on PRM800K and MetaMath despite being trained on final-answer rewards.The evaluation addresses whether ORM’s approximation of PRM labels also holds on the more challenging MATH dataset variants.Source: K On ORM’s Approximation of PRM Labels · p. 29
• The results indicate that ORMs tend to predict whether the full trace is correct, rather than only whether its final answer is correct.The paper attributes this tendency to ORMs approximating PRM labels [74]; Uesato et al. [74] suggest recognizing correct steps can be simpler than internally computing the final answer.Source: K On ORM’s Approximation of PRM Labels · pp. 29–30
• Easy-to-hard generalization also applies to value-estimation ORMs, not only reward models trained on explicit step-wise annotations.The paper describes these ORMs as practically evaluating each solution step.Source: K On ORM’s Approximation of PRM Labels · p. 30
• In DPO training on a MetaMath-initialized Llemma-7b model, ORM re-ranking gave similar performance to PRM re-ranking (29.2 vs. 30.4; 31.2 vs. 34.0).The passage does not specify the metric labels for these paired values.Source: K On ORM’s Approximation of PRM Labels · p. 30

L Analysis of Aggregation Functions in PRMs & OPRMs

• Lightman et al. [40] report comparable performance for minimum and product aggregation, and PRM800K training treats neutral steps as positive labels.Source: L Analysis of Aggregation Functions in… · p. 30
• Product works universally well for weighted voting, while minimum works well for best-of-n across PRM800K and Math-Shepherd analyses, so the paper adopts both strategies in its main experiments.The analyses cover PRMs and OPRMs, with weighted voting and best-of-n decoding.Source: L Analysis of Aggregation Functions in… · p. 30, Figure 16 · p. 31, Figure 17 · p. 32
• The last-step strategy performs poorly for reward models trained on human-annotated process rewards but works much better for OPRMs and pseudo-PRMs, potentially explaining why OPRMs do not further improve Math-Shepherd performance.Source: L Analysis of Aggregation Functions in… · p. 30

M Societal Impact

• Easy-to-hard generalization could enable AI systems to tackle increasingly complex problems in scientific discovery, healthcare, and education, potentially benefiting society.Source: M Societal Impact · p. 33
• Systems operating beyond human supervision raise concerns about transparency, accountability, and potential misuse.Source: M Societal Impact · p. 33
• The paper calls for robust safeguards, ethical consideration, ongoing research, and public discourse to mitigate risks and support responsible development and deployment.Source: M Societal Impact · p. 33

N Fine-Grained Analysis of OPRMs’ Re-ranking strategies

• Best-of-N and Weighted Voting perform strongly across difficulty levels, including unseen level 4–5 problems, although both SFTs and OPRMs were trained on level 1–3 data.The results indicate that OPRM-based re-ranking can generalize from easier to harder tasks.Source: Figure 19 · p. 34, N Fine-Grained Analysis of OPRMs’… · p. 34
• OPRMs generalize best on Algebra, Intermediate Algebra, and Precalculus, and worst on Counting & Probability problems.The comparison covers Algebra, Counting & Probability, Geometry, Intermediate Algebra, Number Theory, Prealgebra, and Precalculus.Source: N Fine-Grained Analysis of OPRMs’… · p. 35, Figure 20 · p. 35

O Pass@N Analysis for Different Math Questions

• Level 1 problems nearly saturate with more samples, while many Level 5 instances remain unsolved even at larger N.Not all problems reach a 100% Pass@N rate, even with a larger N.Source: Figure 21 · p. 36, O Pass@N Analysis for Different Math… · p. 36
• 34b models perform significantly better than 7b models on Pass@1, but their advantage diminishes on Pass@1024.Source: O Pass@N Analysis for Different Math… · p. 36
• Pass@N varies across math-question categories, reflecting differences in category difficulty for large language models.Source: O Pass@N Analysis for Different Math… · p. 36

P Case Studies for OPRM

• OPRM assigns low confidence to majority-voted solutions that mishandle units or misuse a polygon-angle theorem.The examples concern a problem mixing 2.5 hours and 90 seconds and an incorrect use of the exterior-angle sum theorem.Source: P Case Studies for OPRM · p. 37, Figure 22 · p. 37, Figure 23 · p. 37

Q Step and Outcome ROC Curve

• PRM and OPRM discriminate reasoning steps better than ORM, while OPRM discriminates final outcomes better than both ORM and PRM.The step judgments use balanced samples from PRM800K-test; outcome curves use balanced labeled generations from the 7B policy model on MATH500, and stronger step discrimination does not necessarily make an evaluator more helpful for generation.Source: Q Step and Outcome ROC Curve · p. 38, Table 10 · p. 38, Figure 24 · p. 38

R Few-Shot Prompt in In-Context Learning

• The cited text identifies 4-shot examples for PRM800K Hard ICL, PRM800K Easy-to-Hard ICL, and MetaMath Full ICL.Source: R Few-Shot Prompt in In-Context Learning · pp. 40, 41, 43
• The MetaMath Hard ICL examples include worked algebra, triangle-area, and rectangle-to-square area-ratio problems.The examples demonstrate elimination for a system of equations and geometric or proportional reasoning.Source: R Few-Shot Prompt in In-Context Learning · pp. 43–44
• Additional MetaMath Hard ICL examples solve inverse-variation and divisor-remainder problems with worked steps.The inverse-variation example concludes z = 1/512, while the divisor-remainder example sums the eligible two-digit divisors to 71.Source: R Few-Shot Prompt in In-Context Learning · p. 44
• The MetaMath Easy-to-Hard ICL examples include worked counting, factorial-logarithm, and modular-arithmetic problems.The counting example uses complementary counting; the other examples solve for k and compute a remainder modulo 50.Source: R Few-Shot Prompt in In-Context Learning · p. 45