Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards

1 Introduction

• Pre-training scaling supports broad linguistic and knowledge representations but leaves gaps in alignment, task adaptation, and deep reasoning, motivating post-training and test-time scaling.The introduction relates these limitations to models falling short of robust and general AI.Source: §1 · p. 1
• Learning from rewards guides LLM behavior across post-training and test-time scaling, including preference optimization, reasoning, reward-guided decoding, and post-hoc correction.Examples include RLHF, RLAIF, DPO, GRPO, Best-of-N, reward-guided decoding, and post-hoc correction.Source: §1 · pp. 1–2, Figure 1 · p. 1
• Reward signals support learning from dynamic feedback rather than only static data, which the introduction connects to aligned preferences and deep reasoning and planning abilities.The survey also names applications in mathematics, code, multimodality, agents, and embodied AI.Source: §1 · p. 2
• The survey organizes learning from rewards through reward models and strategies, reviews methods across training, inference, and post-inference, and covers benchmarks, applications, challenges, and future directions.Figure 2 depicts reward models evaluating generated outputs and strategies using rewards to update models or refine outputs.Source: §1 · p. 2, Figure 2 · p. 2

2 A Taxonomy of Learning from Rewards for LLMs

• The taxonomy distinguishes reward source, reward model, learning stage, and learning strategy as dimensions for describing how reward signals are acquired, represented, and used.These dimensions are presented as a unified way to understand learning-from-rewards systems.Source: §2 · p. 2

2.1 A Unified Conceptual Framework

• The unified framework links input-conditioned language-model outputs, reward-model evaluation, and strategies that use rewards to update models or adjust outputs.The framework covers both model-based and model-free reward models and both training-based and training-free strategies.Source: §2.1 · pp. 2–3

2.2 Reward Source

• Reward sources include human preferences and automated feedback from self-judging models, trained judges, rules, knowledge, and tools.Human preference data can train reward models or directly fine-tune language models, but collecting it is resource-intensive and may not scale easily across domains or tasks.Source: §2.2 · p. 3

2.3 Reward Model

• Reward models vary in format (scalar, critique, or implicit), scoring pattern (pointwise or pairwise), and granularity (outcome or process).Scalar rewards are simple and common but sparse and less interpretable; critiques are more expressive but can require processing, while implicit rewards shift more inference work to the learning strategy.Source: §2.3 · pp. 4–5
• Reward-model architectures may be model-based or model-free; model-based approaches offer reusable evaluation but require preference data, while model-free approaches are simpler but typically task-specific.The survey also describes scalar, generative, and semi-scalar model-based reward models, alongside model-free examples such as DPO and GRPO.Source: §2.3 · p. 4

2.4 Learning Stage

• Reward use spans training, inference, and post-inference: training converts rewards into optimization signals, while inference guides decoding and post-inference refines generated outputs.Inference and post-inference strategies can support test-time scaling without modifying model parameters.Source: §2.4 · p. 5

2.5 Learning Strategy

• Training-based strategies convert rewards into gradient-based updates, including reinforcement learning and supervised fine-tuning methods such as PPO, DPO, GRPO, and RSFT.The taxonomy describes these as strategies that optimize the language model.Source: §2.5 · p. 5
• Training-free strategies guide or refine outputs without updating model parameters, using methods such as Best-of-N, reward-guided decoding, and post-inference correction.They are described as relatively lightweight and useful when fine-tuning is infeasible or computationally expensive.Source: §2.5 · p. 5

3 Training with Rewards

• The survey groups reward-based training into scalar, critique, implicit, rule-based, and process rewards, used with reinforcement learning or supervised fine-tuning.Rule-based and process rewards are treated separately because of their emerging importance, despite conceptual overlap with other categories.Source: §3 · p. 5, Figure 4 · p. 6

3.1 Training with Scalar Rewards

• Human-feedback scalar rewards train a reward model on preference comparisons, then use it with PPO to align language models with preferences such as helpfulness and harmlessness.Automated feedback, including LLM judgments and tool outcomes, offers an alternative to costly human annotations; RLAIF has achieved comparable performance with RLHF.Source: §3.1 · pp. 6–7

3.2 Training with Critique Rewards

• Critique rewards use generative reward models to provide pointwise or pairwise judgments with explanations and refinement suggestions, supporting flexible, interpretable feedback.Recent generalist models extend this approach across tasks and scoring formats, including vision-language tasks and diverse NLP scoring formats.Source: §3.2 · p. 8

3.3 Training with Hybrid Rewards

• Hybrid rewards combine scalar scores and critiques, including models with separate scalar-prediction and critique-generation heads and multimodal reward models.Critic-RM (Yu et al., 2024b) uses a multi-task objective; MM-RLHF (Zhang et al., 2025g) trains on image and video understanding and safety.Source: §3.3 · p. 8

3.4 Training with Implicit Rewards

• Implicit rewards encode preferences in curated training data rather than supplying an explicit reward during fine-tuning; DPO uses preference log-likelihood differences and reduces RLHF to supervised fine-tuning.Training data can be constructed from human or automated feedback, including model judgments, self-consistency, external knowledge, and tool execution.Source: §3.4 · pp. 8–10

3.5 Training with Rule-based Rewards

• Rule-based rewards verify outputs against criteria such as correctness and format; DeepSeek-R1 (DeepSeek-AI et al., 2025) uses them with GRPO to elicit long-CoT reasoning, while random and incorrect rewards can work for some models such as QWen2.5-Math.Shao et al. (2025) report the latter finding; DeepSeek-R1’s accuracy rewards include correct math solutions or code passing unit tests, while format rewards enforce structural constraints.Source: §3.5 · p. 10

3.6 Training with Process Rewards

• Process rewards evaluate intermediate reasoning steps for finer-grained supervision, especially in complex tasks; automated methods use LLM-generated labels, estimated step scores, or outcome supervision to train process reward models.Automated feedback addresses the labor-intensive step-level human annotations required by early process reward models; PRIME (Cui et al., 2025) estimates token-level process rewards from outcome-level supervision without step-level annotations.Source: §3.6 · p. 11
• The reward model samples high-quality responses to supervise subsequent fine-tuning iterations.Source: §3.6 · p. 12

4.1 Generate-then-Rank

• Best-of-N samples candidate responses, scores them with a reward model, and selects the top-ranked response or uses voting for the final output.Outcome rewards score whole responses, while process rewards score intermediate steps and aggregate them to rank candidates.Source: §4.1 · pp. 12–13
• Best-of-N becomes computationally expensive as candidate counts rise; Fast Best-of-N uses reward-based early rejection to avoid fully generating every candidate.The reward model is queried during generation and unpromising candidates are terminated early.Source: §4.1 · p. 13
• Process-reward ranking can favor solutions built from reliable intermediate steps, rather than selecting solely for a correct final answer.Methods aggregate step rewards by multiplication or minimum, or use weighted voting.Source: §4.1 · p. 13

4.2 Reward-Guided Decoding

• Reward-guided decoding integrates rewards into generation rather than separating candidate generation from evaluation, using token- or step-level guidance with search algorithms.The section identifies greedy search, beam search, and Monte Carlo Tree Search as examples.Source: §4.2 · pp. 13–14
• Token-level methods combine token likelihoods with reward signals, while step-level search selects intermediate steps and can recover from earlier errors.Token-level methods may be inefficient because they query the reward model for every candidate token at every decoding step.Source: §4.2 · p. 14

5.1 Self-Correction

• Self-correction uses the language model to evaluate and revise its outputs, with feedback serving as reward signals for iterative refinement.Approaches include natural-language feedback, reflection stored in memory, and verification questions about the model’s own output.Source: §5.1 · pp. 15–16

5.2 Correction with External Feedback

• External feedback is grouped by source into trained models, external knowledge, and external tools, addressing difficulties models can have correcting errors unaided.The passage notes particular difficulty for less capable small-scale models.Source: §5.2 · p. 16
• Trained critics provide quality assessments, correction suggestions, or step-level critiques; external knowledge supplies evidence, while tools provide execution or verification feedback.Examples include compiler and unit-test feedback for code, retrieved evidence for factual correction, and symbolic reasoners or proof checkers.Source: §5.2 · pp. 16–17

6.1 Benchmarking Outcome Reward Models

• Outcome-reward benchmarks evaluate whole-output quality, with datasets covering chat, reasoning, safety, subtle content changes, style bias, math, and real-world scenarios.Other benchmarks assess whether generated critiques identify errors and can guide correction.Source: §6.1 · pp. 17–18

6.2 Benchmarking Process Reward Models

• Process-reward benchmarks test step correctness, including locating the first error in math or logical reasoning and evaluating error types across multiple dimensions.Coverage ranges from math to seven domains, including physics, medicine, and code.Source: §6.2 · p. 18

6.3 Benchmarking Multimodal Reward Models

• Multimodal benchmarks assess outcome and process reward models across image-text alignment, safety, hallucination detection, reasoning, and other capabilities.Some evaluate step-level relevance, logic, or attribute correctness; others combine reasoning steps, outcomes, and critiques.Source: §6.3 · pp. 18–19

6.4 Other Benchmarks

• Specialized benchmarks target reward modeling for retrieval-augmented generation and multilingual settings, while PPE evaluates models indirectly through downstream RLHF performance.Source: §6.4 · p. 19

7.1 Preference Alignment

• Preference alignment rewards factuality, harmlessness, and helpfulness while penalizing toxicity, bias, and hallucination.The surveyed approaches span stages for factuality and hallucination reduction, while safety methods primarily target training and inference.Source: §7.1 · p. 19

7.2 Mathematical Reasoning

• Math-reasoning methods train with reward models or preference data, and use inference-time candidate ranking or reward-guided search such as MCTS.Process reward models provide step-level signals for math solutions.Source: §7.2 · pp. 19–20

7.3 Code Generation

• Code-generation methods use reward models, compiler and test feedback, or code analysis for fine-tuning, inference-time ranking and decoding, and correction.Correction may use the language model itself or external feedback.Source: §7.3 · p. 20

7.4 Multimodal Tasks

• Multimodal applications use reinforcement learning and reward-guided decoding for image understanding, visual reasoning, video reasoning, counting, detection, segmentation, and generation.Source: §7.4 · p. 20

7.5 Agents

• Agent applications use rewards to train agents or guide choices of actions and trajectories across navigation, planning, tool use, research, and GUI tasks.Methods include process-reward training, reward-guided MCTS, reinforcement learning, and multi-turn reinforcement learning.Source: §7.5 · p. 20

7.6 Other Applications

• Other applications include embodied reasoning, retrieval, tool use, recommendation, software engineering, text-to-SQL, and medical or financial tasks.The survey describes reinforcement learning for embodied systems and retrieval, as well as reward designs tailored to specific tasks and domains.Source: §7.6 · pp. 20–21

8.1 Interpretability of Reward Models

• Reward models are often opaque scalars or critiques; multi-objective scores expose predefined dimensions, while generative rationales still need consistency, reliability, and faithfulness checks.ArmoRM's interpretability is limited to predefined objectives.Source: §8.1 · p. 21

8.2 Generalist Reward Models

• Narrow-domain models generalize weakly and emit static rewards; generalist models target varied inputs, domains, and inference scaling, as DeepSeek-GRM's critiques support voting and meta-reward filtering.The survey describes question answering, math reasoning, and code generation as example domains for generalist reward models.Source: §8.2 · p. 22

8.3 Reward Hacking

• Reward hacking exploits proxy-reward shortcuts, including fabricated answers, test tampering, and self-judge loops; proposed mitigations include tamper-resistant rewards, anomaly detection, decoupled feedback, and dataset audits.The section relates reward hacking to difficulty specifying a reward function that perfectly captures true objectives.Source: §8.3 · p. 22

8.4 Grounded Rewards from Real-World Interactions

• Dependence on human preferences or curated feedback limits surpassing human knowledge and adapting to complex environments; grounded outcome rewards require agent-environment interaction but bring noisy, delayed, partial-feedback challenges.Examples include health assistants using physiological signals and scientific agents using experimental data.Source: §8.4 · p. 22

8.5 Continual Learning from Rewards

• Fixed datasets, predefined rewards, and short episodes constrain adaptation; continual reward learning targets changing signals and distribution shifts but must address forgetting, stability-plasticity, and dynamic reward modeling.The section presents continual learning from rewards as a foundation for lifelong competent and aligned agents.Source: §8.5 · pp. 22–23

• The survey categorizes learning from rewards into training, inference, and post-inference stages and reviews representative studies through reward models and learning strategies.Source: §9 · p. 23
• The survey also summarizes recent progress in reward-model benchmarking and applications.Source: §9 · p. 23
• It identifies core challenges and outlines promising future directions for learning from rewards in LLMs.Source: §9 · p. 23