PaperlayerView exact revision on arXiv

Source-linked AI summary

Progress Reward Model for Reinforcement Learning via Large Language Models

Xiuhui Zhang, Ning Gao, Xingyu Jiang, Yihui Chen, Yuheng Pan, Mohan Zhang, Yue Deng

arXiv:2610.22056v1cs.AI

TL;DR

Long-term, sparse-reward RL tasks are difficult, and LLM-based planners and rewarders each leave challenges in low-level learning or complex reward construction. PRM4RL combines subtask planning with progress-based dense rewards; experiments report improved performance and efficiency on robotics tasks.

  • Problem

    Long-term tasks with sparse rewards challenge RL, while LLM planners may not guide low-level learning and LLM rewarders face the complexity of whole-task reward construction.

    Source: §1 · pp. 1–2

  • Method

    PRM4RL decomposes tasks into subtasks with fine-grained progress functions and uses progress as a potential to construct a dense reward model with optimality and convergence guarantees.

    Source: §1 · p. 2, §4.2 · p. 5, §4.3 · p. 6

  • Results

    PRM4RL improves convergence speed and performance on complex multi-stage tasks, and is reported to improve success rate while reducing LLM calls compared with existing methods.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The paper concludes that combining high-level planning with low-level reward shaping is effective and efficient for complex, multi-stage RL tasks.

    Source: §6 · p. 9

    The approach may be affected by LLM hallucinations, and its applicability beyond robotics remains unexplored.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at https://github.com/deng-ai-lab/PRM4RL.

1 Introduction

Long-term tasks with sparse rewards challenge RL, while existing approaches rely on expert data or address planning and reward shaping separately. PRM4RL combines high-level task decomposition with low-level dense rewards guided by fine-grained progress.

1 Introduction

  • Traditional RL approaches use task decomposition or reward shaping, but both rely heavily on expert priors or supervision that can be difficult to acquire at scale.Hierarchical RL decomposes tasks, while inverse RL and intrinsic-reward mechanisms provide dense signals.Source: §1 · p. 1
  • LLM-based methods act either as planners that decompose tasks into short-term subtasks or as rewarders that generate dense guidance for the policy.Both paradigms draw on priors encoded in LLMs.Source: §1 · p. 2
  • Planner-only methods lack dense reward guidance for effective subtask learning, whereas rewarder-only methods struggle to construct rewards for whole complex tasks and may require extensive search or expert assistance.Prior approaches also use pretrained policy libraries or traditional motion planning, limiting flexibility and efficiency through non-end-to-end operation.Source: §1 · p. 2
  • PRM4RL decomposes complex tasks into subtasks with termination conditions and fine-grained progress evaluations, then uses progress to construct dense rewards with optimality and convergence guarantees.The framework integrates long-term planning and low-level reward generation.Source: §1 · p. 2

2 Related Works

Prior LLM-augmented RL work uses language models for planning or reward generation. The paper contrasts these approaches and motivates decomposing tasks into subtasks while using progress-based rewards.

2.1 LLM for Planning

  • LLM planning methods use pretrained policies or motion planning to connect language-generated subtasks to low-level control, while the paper positions PRM as a dense reward bridge to policy training.ELLM computes reward from similarity between executed actions and a plan, but the passage describes it as computationally expensive and unsuitable for robotics.Source: §2.1 · p. 3

2.2 LLM for Reward Generation

  • LLM reward-generation methods produce executable rewards with human feedback or evolutionary search, but treat complex tasks as a whole rather than accounting for stage-wise features.The paper motivates short-term subtasks as a way to provide more specific guidance for each stage.Source: §2.2 · p. 3

3 Preliminary

The preliminary section models standard RL tasks as Markov decision processes and states the objective as maximizing expected cumulative return.

3 Preliminary

  • A standard RL task is modeled as an MDP comprising state and action spaces, transition and reward functions, a discount factor, and an initial-state distribution.The passage also defines trajectories, policies, and the state-action value function as expected cumulative return after an action.Source: §3 · p. 3
  • The RL objective is to learn a policy that maximizes expected cumulative return.Source: §3 · p. 3

4 Method

PRM4RL combines LLM-generated subtask planning with a progress-based dense reward to guide RL. It uses state-based subtask tracking and potential-based reward shaping, then augments policy learning with both subtask priors and PRM rewards.

4 Method

  • PRM4RL combines subtask priors with progress-based rewards to augment RL, with the shaping approach presented as having optimality and convergence guarantees.The framework decomposes long-term tasks into subtasks, derives progress from them, and uses that progress to construct the reward.Source: §4 · pp. 3–4, §4.2 · p. 6

4.1 Subtask Decomposition

  • The LLM decomposes the task into subtasks and generates verb-plus-noun priors, while a state-based determination function Ψ(s) tracks the current subtask without repeated LLM calls.The priors use a fixed pattern, and Ψ(s) identifies the current subtask from the state and subtask completion conditions.Source: §4.1 · pp. 4–5

4.2 Potential-Based Progress Reward Model

  • PRM uses fine-grained subtask progress as a potential to form a dense reward, and potential-based shaping preserves optimal policies and RL convergence.The reward combines a sparse success reward with a potential-based shaping term; the paper also describes a convergence-efficiency result through equivalence to Q-value initialization.Source: §4.2 · pp. 5–6

4.3 Reinforcement Learning Augmentation

  • The framework embeds the current subtask prior into the state and uses PRM as the reward, yielding an augmented MDP for policy optimization.The paper reports using SAC and PPO to optimize policies under the augmented MDP.Source: §4.3 · p. 6

5 Experiment

PRM4RL is evaluated on long-term sparse-reward robotics tasks, where it combines high-level planning with low-level training guidance. It outperforms comparison methods across nearly all environments, and ablations examine the roles of its reward and subtask prior.

5 Experiment

  • The evaluation covers a diverse suite of long-term, sparse-reward reinforcement-learning tasks, testing high-level planning and low-level training guidance.Source: §5 · p. 6

5.1 Experiment Settings

  • Experiments use the same RL algorithm and training parameters across methods, GPT-4o for LLM requirements, and five random seeds.The baselines include ELLM, T2R, and the environment-provided expert-written dense reward called Oracle.Source: §5.1 · pp. 6–7

5.2.1 Outperform previous LLM approaches

  • PRM4RL outperforms comparison methods across nearly all environments, with nearly 100% success on MW-pick-out-of-hole and MS-LiftPegUpright where baselines achieve less than 15%.The authors also report faster convergence than ELLM and stronger performance on intricate tasks than T2R.Source: Figure 3 · p. 7, §5.2.1 · p. 7, Figure 4 · p. 8

5.2.2 Ablation Study

  • Replacing PRM reward with Oracle reward significantly reduces performance, and the no-PRM variant is often similar to Oracle.The authors conclude that the subtask prior alone does not provide an effective, efficient reward signal.Source: §5.2.2 · p. 8, Figure 5 · p. 8
  • Removing the subtask prior slows convergence, although the no-prior variant still outperforms T2R; the authors report that the prior and PRM reward work synergistically.Source: §5.2.2 · p. 9
  • Using progress directly as reward significantly reduces performance; its episode reward can rise while success rate declines during training.The authors contrast this with stable convergence under potential-based shaping.Source: Figure 6 · p. 8, §5.2.2 · p. 9

5.2.3 Generalization Results

  • On unseen MetaWorld tasks, PRM4RL experiences the least performance loss among compared methods, while removing the subtask prior causes a significant drop.The authors attribute the prior's transfer value to its designed “verb + noun” pattern.Source: Figure 7 · p. 9, §5.2.3 · p. 9

6 Conclusion

PRM4RL combines high-level task decomposition with low-level reward shaping; the paper reports faster convergence and improved performance on complex, multi-stage tasks.

  • PRM4RL combines high-level decomposition into manageable subtasks with a low-level Progress Reward Model that has optimality and convergence guarantees.Source: §6 · p. 9
  • Experiments report improved convergence speed and an increased ability to solve complex, multi-stage tasks.Source: §6 · p. 9
  • LLM hallucinations during reasoning could affect the approach, as with other methods that rely on LLM planning or coding.Source: §6 · p. 9
  • Applicability beyond robotics remains unexplored, and further framework adaptation may be needed for diverse environments and tasks.Source: §6 · p. 10

A Theoretical Proof

The appendix analyzes potential-based reward shaping in sparse-reward MDPs, establishing policy invariance and an update-level equivalence related to convergence efficiency.

  • The policy-invariance proof considers an MDP with sparse success reward and a potential-based reward-shaping function.Source: A Theoretical Proof · p. 13
  • Adding the shaping function preserves the set of optimal policies: each optimal policy in either MDP is optimal in the other.Source: A Theoretical Proof · p. 13
  • For the convergence-efficiency argument, induction maintains a Q-function offset invariant across updates, establishing equivalence at every step.Source: A Theoretical Proof · pp. 13–14

B.1 Additional Generalization Results

The appendix specifies additional MetaWorld generalization evaluations and describes how the ELLM and Text2Reward baselines were configured.

B.1 Additional Generalization Results

  • Figure 8 evaluates transfer from Metaworld-pick-place to four unseen MetaWorld tasks, with results averaged over five seeds and 100 tests per seed.Source: §B.1 · p. 14, Figure 8 · p. 14

B.2 Baseline Settings

  • For ELLM, the authors re-implemented the baseline and supplied their subtask prior, subtask success reward, and Ψ for tracking the current subtask.Source: §B.2 · p. 14
  • For Text2Reward, the authors reused its few-shot reward functions where available and generated functions following its code for other tasks.Source: §B.2 · p. 15

C Hyper-Parameter Details

The appendix reports the progress-function generation settings, RL implementations, hardware, and approximate per-task training times.

D Tasks

The appendix lists tasks for MetaWorld and ManiSkill, describes their robot-control setups, and gives the prompt structure and a generated progress-function example.

D Tasks

D.1 Metaworld

  • MetaWorld uses a fixed-base 7 DoF Sawyer arm with end-effector delta-position control and variable object and robot initial and target positions.Observations combine end-effector position, gripper openness, object pose, prior measurements, and goal position.Source: §D.1 · p. 16, Figure 9 · p. 16, Table 3 · p. 16

D.2 Maniskill

  • ManiSkill uses a 7 DoF Franka Panda with end-effector delta-pose control; initial and target positions, robot joints, and physical parameters vary.Observations include robot proprioception and task-specific information.Source: §D.2 · pp. 16–17, Figure 10 · p. 17, Table 4 · p. 17

E Prompt Details

  • The Pythonic prompt uses expert-authored class-like environment descriptions and instructs the LLM to generate subtasks, completion conditions, and progress functions.Subtask progress ranges from 0 to 1; main progress combines the current subtask index and progress, treating completed subtasks as complete.Source: E Prompt Details · pp. 17–19

F Example of Generated Progress Function

  • The generated pick-place example computes progress from distance-to-object, object lift, and distance-to-goal, then sets progress to complete at release.For the first three subtasks, main progress is the subtask index plus subprogress; the function returns main progress and the current subtask index.Source: F Example of Generated Progress Function · p. 20

Related papers

Loading 2610.22056v1…