Source-linked AI summary
Progress Reward Modeling for Robotic Learning: A Comprehensive Survey
Jianshu Zhang, Keliang Wu, Haoran Lu, Anbang Liu, Ce Zhang, Weijie Yin, Chengxuan Qian, Xiyuan Yang, Zhenyu Pan, Guo Ye, Han Liu
TL;DR
Terminal success signals provide little guidance about intermediate behavior, while progress-reward research lacks a shared framework for comparison and validation. This survey unifies the field through model interfaces, reward-construction mechanisms, and data and benchmark analysis, showing that progress quality requires task grounding and uncertainty evaluation alongside downstream policy performance. It also identifies fine-grained sensitivity, adaptive temporal modeling, efficient inference, and long-horizon memory as limitations for practical deployment.
Problem
Terminal success signals reveal task completion but provide little information about intermediate progress, stagnation, or regression, especially in long-horizon robotic tasks.
Method
The survey organizes progress reward modeling by model interfaces, reward-construction mechanisms, and the data and benchmarks used for supervision and evaluation.
Results
The survey shows that progress evaluation must assess task grounding, answerability, uncertainty, and downstream utility rather than relying solely on policy performance.
Takeaways & Limitations
Progress rewards are best understood as task-conditioned behavior evaluators whose quality depends on faithful progress estimation, not merely improved policy outcomes.
Takeaways & Limitations
Current progress models often lack fine-grained sensitivity, adaptive temporal modeling, efficient online inference, and reliable long-horizon memory.
Abstract
from arXiv · showhide
Robotic learning takes place in dynamic environments with large behavior spaces. A terminal success signal only tells the robot whether the task is completed. It does not explain whether the current behavior is making progress, remaining unchanged, or undoing earlier progress. For this reason, recent studies have increasingly explored progress rewards that provide feedback during task execution. However, the current literature lacks a shared framework. Existing methods use different observations, goal specifications, output signals, supervision sources, and evaluation protocols. This makes it difficult to compare them and understand what their results actually validate. In this survey, we provide a unified view of progress reward modeling for robotic learning. We organize the field in three connected steps. We first study the interface of a progress model. This defines the problem from the outside by asking what information the model receives and what form of progress signal it produces. We then move inside the model and study the methods used to construct this signal. This reveals the different assumptions and mechanisms behind progress estimation and reward generation. Finally, we examine the data and benchmarks that support these methods. This shows how progress supervision is obtained and what different evaluations actually measure. Together, these three perspectives connect what a progress model is, how it is built, and how its quality is validated. We further summarize the main limitations of current approaches and discuss future research directions.
1 Introduction
Progress reward models address the limitations of terminal success signals by providing task-conditioned feedback about intermediate execution. This survey unifies the fragmented literature through model interfaces, reward-construction mechanisms, and the data and evaluations used to validate them.
- Motivation: Terminal success signals reveal only whether a task was completed, not whether intermediate actions advance, stagnate, or undo progress.The limitation is especially severe for long-horizon tasks with rare successful outcomes and many decisions.
- Motivation: Progress rewards provide dense temporal feedback that improves credit assignment and supports monitoring, reranking, filtering, planning, action selection, and early failure detection.They help distinguish useful behavior from irrelevant motion and identify stagnation or regression before final task completion.
- Challenges: Progress estimation is harder than success detection because it must infer completed subtasks, current activity, and remaining distance to a task-conditioned goal.Unlike binary final-goal prediction, progress estimation places the current observation within an evolving longer-horizon execution.
- Challenges: The literature remains fragmented across observations, goal specifications, output forms, supervision sources, reward uses, and heterogeneous evaluation protocols.Existing methods may predict success probability, estimate video progress, compare trajectories, or generate executable reward programs.
- Survey Scope and Contributions: The survey organizes progress reward modeling into interfaces, construction mechanisms, and data and evaluation evidence.It distinguishes progress-fidelity benchmarks from robustness, generalization, and downstream-utility evaluations, noting that policy improvement does not guarantee faithful progress representation or closed-loop suitability.
2 Interface: Input and Output of Progress Models
The progress-model interface specifies what evidence the model receives and how it represents task goals, then determines the form in which progress is produced. Inputs range from observations and temporal or comparative context to explicit state access, while outputs include scores, deltas, rankings, and executable reward functions.
- Interface overview: The interface is organized around task-state representation, goal specification, and output form.These three questions define what evidence a model can access and what kind of progress or reward signal it provides.
- State representation: Input interfaces use single observations, temporal context, comparisons between reference states, or explicit state access to estimate progress.Temporal windows help infer prior events, comparisons make progress relational, and state-access methods rely on environment code, simulator APIs, training statistics, or proprioceptive features.
- Goal specification: Goals are commonly specified through language, visual references, or structured programs, each trading descriptive flexibility against access and robustness requirements.Visual goals capture physical details but require curated references and can be sensitive to viewpoint, embodiment, appearance, and layout; structured goals support precise, dense, compositional rewards but work best in instrumented environments.
- Output form: Output interfaces encode progress as state-wise scalar scores, transition-level deltas, rankings or preferences, or executable reward functions.These forms respectively represent state desirability, local change, comparative advancement, or an explicit computation over available state variables, features, or observations.
3 Methods: How Are Progress Rewards Constructed?
This section organizes progress-reward construction by how progress signals are obtained and converted into usable rewards. It identifies four paradigms: frozen foundation-model scoring, temporal or relative supervision, instruction-tuned prediction, and programmatic construction.
- Methods: How Are Progress Rewards Constructed?: The survey categorizes progress-reward methods by the mechanism used to obtain and convert progress signals into usable rewards.This organization provides the section’s framework for comparing construction approaches.
- Methods: How Are Progress Rewards Constructed?: Four paradigms are identified: frozen foundation-model scoring, learning from temporal or relative supervision, instruction-tuned progress prediction, and programmatic reward construction.These paradigms span model-based scoring, supervised prediction, and explicitly constructed rewards.
2. Temporal / Relative Supervision
Temporal or relative supervision learns reward geometry from supervision, including pairwise preferences across time-indexed observations.
- Temporal / Relative Supervision: Temporal or relative supervision can learn reward geometry from supervision.
- Temporal / Relative Supervision: Pairwise preference provides a form of relative supervision.
- Temporal / Relative Supervision: Temporal comparisons can involve observations at t-k, t-1, and t.
Methods: How Are Progress Rewards Constructed? · 3. Instruction-Tuned Progress Models · 4. Programmatic Reward Construction
Progress rewards are constructed through four paradigms: frozen foundation-model scoring, learned rewards from temporal or relative supervision, instruction-tuned progress prediction, and executable reward programs. These methods differ in their supervision, output formulation, and reliance on calibration or program execution.
- Methods: How Are Progress Rewards Constructed?: Existing methods fall into four paradigms based on where the progress signal originates and how it becomes a usable reward.The paradigms are summarized as frozen foundation-model scoring, temporal or relative supervision, instruction-following models, and programmatic reward construction.
- 3.1 Frozen Foundation Models as Semantic Reward Scorers: Frozen foundation models provide zero-shot semantic reward signals by measuring image-text alignment or prompting a frozen VLM about task completion.CLIP-style methods use image-text similarity, while TOPReward operates in token space.
- 3.1 Frozen Foundation Models as Semantic Reward Scorers: Because frozen-model scores are semantic priors rather than calibrated progress rewards, they often require normalization, thresholding, differencing, or prompt engineering.This paradigm does not require task-specific reward labels or fine-tuning.
- 3.2 Learning Rewards from Temporal and Relative Supervision: Supervised reward methods infer goal proximity or behavioral progress from temporal order in successful demonstrations, often treating later states as closer to completion.Goal-proximity imitation learning uses demonstration time as a progress proxy and converts changes in goal proximity into rewards.
- 3.2 Learning Rewards from Temporal and Relative Supervision: Preference-based methods compare states, clips, or trajectories to learn which behavior is better, rather than assigning an absolute progress value.RL-VLM-F derives pairwise preferences with a frozen VLM and trains a scalar reward model, while PEARL transfers preferences across tasks and Rank2Reward learns shaped rewards from rankings.
- 3. Instruction-Tuned Progress Models: Instruction-tuned progress models explicitly define progress judgments in prompts and fine-tune VLMs or video-language models to follow those instructions.Targets may concern absolute progress, task success, relative improvement, preferences, or reasoning-grounded progress scores.
- 3. Instruction-Tuned Progress Models: These models learn progress estimation as a dedicated instruction-following capability rather than as an implicit by-product of visual representation learning.Training pairs visual observations or trajectories with progress-related instructions and target responses.
- 4. Programmatic Reward Construction: Programmatic reward methods translate progress estimation into executable code, predicates, feature functions, or structured reward logic evaluated during policy learning.Text2Reward and Language2Rewards convert natural-language task descriptions into executable reward functions, while Eureka refines LLM-generated reward programs using policy feedback.
4 Data and Benchmarks
This section organizes progress-data construction by human involvement and benchmarks by the evaluation goals of progress fidelity, robustness and generalization, and downstream utility. Together, these perspectives connect how supervision is obtained with what evaluation results validate.
- Progress-data construction: Progress supervision is constructed through human-driven, human-in-the-loop, or fully automated pipelines using trajectories and labels for completion, progress change, or relative behavior quality.Human-driven methods capture semantic task information but are costly and subjective; human-in-the-loop methods combine sparse semantic anchors with automated processing; automated methods scale supervision through temporal structure, environments, models, and augmentation.
- Progress-data construction: Human supervision provides stronger semantic grounding, whereas automation increases scale but can inherit model errors or create physically implausible synthetic failures.Human annotations capture properties such as grasp stability and acceptable contact, while automated labels may reflect annotator-model bias or exploitable artifacts from unrealistic failures.
- Progress fidelity: Progress-fidelity benchmarks test scalar calibration, temporal consistency, relative ordering, task grounding, and uncertainty rather than treating fidelity as a single property.Calibration compares predictions with explicit references; temporal consistency tests chronological ordering; relative benchmarks support preference learning; grounding varies the specified goal; uncertainty tests answerability and abstention.
- Progress fidelity: Task-grounding evaluations verify that progress judgments depend on the specified goal rather than generic motion, scene change, or temporal position.Counterfactual instructions, mismatched video–language pairs, and contrastive prompts test whether one observation receives different judgments under different goals.
- Downstream utility: Online policy-learning benchmarks evaluate whether progress rewards support interactive reinforcement learning using success rate, sample efficiency, and learning stability, while results also depend on exploration and environment design.These outcomes are influenced by exploration, policy architecture, reset design, and environment factors, so they measure downstream utility in addition to reward quality.
5 Limitations and Future Directions
Current progress reward models remain limited by coarse-grained estimation, fixed-rate temporal assumptions, high inference latency, and insufficient long-horizon memory. Addressing these issues is necessary for real-time robotic learning deployment.
- Limitations: Current progress models often lack fine-grained sensitivity, adaptive temporal modeling, efficient online inference, and reliable long-horizon memory.These practical limitations hinder wider real-time use of progress rewards in robotic learning.
- Limitations: Coarse representations may miss precise alignment, slight object motion, unstable grasping, and early contact failure.Existing methods commonly rely on sparse frames, short clips, discrete progress bins, or broad task stages.
- Limitations: Temporal-order and timestamp supervision can incorrectly encourage progress to increase smoothly, despite uneven progress with sudden gains and extended plateaus.Waiting, searching, and careful alignment may produce plateaus, while major subgoals can cause sudden increases.
- Limitations: Large Vision-Language Models can be too slow for reward production at every control step during online deployment.They may need to process multiple frames, interpret instructions, and reason about execution before producing a reward.
- Limitations: Long-horizon tasks require explicit memory because similar observations can represent different stages or progress levels depending on execution history.Current observations may not reveal completed subgoals, their order, or whether earlier progress was preserved or undone.