PaperlayerView exact revision on arXiv

Source-linked AI summary

Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning

Qi Wang, Mian Wu, Yuyang Zhang, Mingqi Yuan, Wenyao Zhang, Haoxiang You, Yunbo Wang, Xin Jin, Xiaokang Yang, Wenjun Zeng

arXiv:2512.00961v2cs.LG

TL;DR

RL reward design is difficult and often requires task-specific engineering that may not generalize. GenReward uses pretrained video diffusion models to provide video-level and frame-level goal-driven rewards, and it significantly outperforms existing methods across the reported benchmarks. The approach incurs additional computational overhead during training.

  • Problem

    Designing appropriate RL rewards requires substantial domain expertise and human labor, limiting scalable application across complex tasks.

    Source: §1 · p. 1

  • Method

    GenReward generates goal-conditioned videos, compares video latents for video-level rewards, and uses CLIP-selected goal frames with forward-backward representations for frame-level rewards.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    GenReward significantly outperforms existing methods in episode return on Meta-World and Distracting Control Suite.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pretrained video diffusion models can provide informative goal-driven rewards without handcrafted reward design for the evaluated RL tasks.

    Source: §1 · p. 2, §6 · p. 8

    Computing video-level and forward-backward rewards during training introduces additional computational overhead.

    Source: §6 · p. 8

Abstract

from arXiv · show

Reinforcement Learning (RL) has achieved remarkable success in various domains, yet it often relies on carefully designed programmatic reward functions to guide agent behavior. Designing such reward functions can be challenging and may not generalize well across different tasks. To address this limitation, we leverage the rich world knowledge contained in pretrained video diffusion models to provide goal-driven reward signals for RL agents without ad-hoc design of reward. Our key idea is to exploit off-the-shelf video diffusion models pretrained on large-scale video datasets as informative reward functions in terms of video-level and frame-level goals. For video-level rewards, we first finetune a pretrained video diffusion model on domain-specific datasets and then employ its video encoder to evaluate the alignment between the latent representations of agent's trajectories and the generated goal videos. To enable more fine-grained goal-achievement, we derive a frame-level goal by identifying the most relevant frame from the generated video using CLIP, which serves as the goal state. We then employ a learned forward-backward representation that represents the probability of visiting the goal state from a given state-action pair as frame-level reward, promoting more coherent and goal-driven trajectories. Experiments on Meta-World and Distracting Control Suite demonstrate the effectiveness of our approach.

1. Introduction

GenReward addresses the difficulty of designing scalable RL rewards by using pretrained video diffusion models to transfer world knowledge through video-level and frame-level goals.

1. Introduction

  • Generated goal-conditioned videos transfer world knowledge to downstream agents and support performance on unseen tasks.Figure 1 presents this world-knowledge-transfer role in the proposed framework.Source: Figure 1 · p. 1
  • GenReward uses pretrained video diffusion models as goal-driven reward models, avoiding handcrafted reward design for RL agents.The framework is motivated by the difficulty and labor involved in designing task-specific reward functions.Source: §1 · p. 1
  • The framework computes video-level rewards from correlations between agent-observation latents and generated goal-video latents.A finetuned video diffusion model generates goal-conditioned videos, whose encoded representations are compared with agent observations.Source: §1 · p. 2
  • For fine-grained goal achievement, CLIP selects a goal frame and a forward-backward representation estimates goal-reaching probability from state-action pairs.This action-aware representation supplies the frame-level reward.Source: §1 · p. 2
  • GenReward significantly outperforms existing methods in episode return on Meta-World and Distracting Control Suite.The paper contrasts its generative, action-aware reward framework with competitive reward models that may use expert demonstrations or omit action information.Source: §1 · p. 2, Table 1 · p. 2

2. Problem Setup

The paper formulates online visual RL as a POMDP and seeks to improve downstream performance by using pretrained video diffusion models to provide intrinsic rewards.

2. Problem Setup

  • Online visual RL is modeled as a POMDP in which observations, actions, transitions, and reward feedback evolve over time.At timestep t, the agent receives observation o_t, selects action a_t according to π(a_t|o_t), and receives the next observation and reward.Source: §2 · p. 2
  • The objective is to improve online downstream-task performance by leveraging a pretrained video diffusion model’s world knowledge for intrinsic rewards.The setup assumes access to a pretrained video diffusion model containing rich prior knowledge.Source: §2 · p. 2
  • Existing reward models differ in whether they use expert demonstrations, rely on generative models, and incorporate action information.These distinctions define the compared learning paradigms in the problem setup.Source: §2 · p. 2

3. Method

GenReward combines adapted video diffusion priors with video-level latent alignment and frame-level forward-backward rewards to guide RL without handcrafted reward design.

3. Method

  • GenReward adapts a pretrained video diffusion model, then combines video-level and frame-level goal rewards with the environment reward.The method has three stages: diffusion-model adaptation, video-level reward construction, and frame-level reward construction.Source: §3 · p. 2, §3.3 · p. 5

3.1. Preliminary: VDM Adaptation

  • CogVideoX is finetuned with task-specific text and image conditions to generate videos aligned with desired goals.The model uses a 3D VAE and latent-space diffusion, with task-specific conditioning from text and image embeddings.Source: §3.1 · pp. 2, 4

3.2. Video-Level Goal as Reward

  • Video-level reward is the cosine similarity between latent representations of sampled agent observations and a generated goal video.The method samples 16 frames from each sequence and encodes them with the finetuned CogVideoX 3D causal VAE.Source: §3.2 · p. 4
  • Video-level reward encourages the agent to mimic the generated goal video temporally, while text-to-video generation can operate without finetuning datasets.The paper also notes that expert video datasets are used for finetuning, but are not required when using text-to-video generation.Source: §3.2 · p. 4

3.3. Frame-Level Goal as Reward

  • CLIP selects the most relevant generated-video frame as a goal image by comparing each frame with the task description.OpenCLIP uses separate text and image encoders to score the goal-video frames.Source: §3.3 · p. 4
  • Forward-backward representations estimate long-term goal occupancy, with F(s, a, z)^T B(s′) approximating the probability of reaching s′ from a state-action pair.The forward representation takes state, action, and goal representation inputs, while the backward representation maps states into the shared representation space.Source: §3.3 · p. 4
  • The frame-level reward uses the goal embedding as both the forward-network goal and reward target, encouraging actions likely to reach the selected goal state.The goal image is encoded with DINOv3, while observations are encoded into model states using DreamerV3 before forward-backward learning.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

4. Experiments

GenReward is evaluated on Meta-World and Distracting Control Suite using generated-video goals, video-level alignment rewards, and frame-level goals. It outperforms key baselines on Meta-World tasks, while analyses show benefits from both reward components and sensitivity to their weighting.

4.1. Experimental Setups

  • The evaluation covers five medium- or hard-level Meta-World manipulation tasks and Distracting Control Suite, using generated videos from RT-1, Bridge, and RLBench-related sources.Episodes are limited to 256 steps, and models are evaluated under both dense and sparse environment rewards; DCS results are provided in supplementary material.Source: §4.1 · p. 5
  • GenReward compares against Dense Reward, RoboCLIP, Diffusion Reward, and TADPoLe using DreamerV3 as the underlying model-based RL algorithm.The comparison includes video-, diffusion-, and text-conditioned reward models.Source: §4.1 · pp. 5–6

4.2. Main Comparison

  • GenReward outperforms DreamerV3 with raw dense reward on Pick Out of Hole (193 →582), Bin Picking (398 →822), and Shelf Place (154 →814).The study reports competitive episodic returns across all five Meta-World tasks and a significant advantage over RoboCLIP and Diffusion Reward.Source: §4.2 · p. 6
  • CLIP selects the generated-video frame with highest task similarity as a more accurate goal image for forward-backward representation learning.The selected frame provides the frame-level goal used in the Meta-World Pick Place analysis.Source: §4.2 · p. 7, Figure 7 · p. 7

4.3. Model Analyses

  • Removing video-level reward degrades performance, whereas adding frame-level forward-backward reward improves performance on Bridge Pick Video →Pick Place.The ablation isolates the contributions of the video-level and frame-level goals.Source: §4.3 · p. 7, Figure 8 · p. 7
  • Too-small or too-large video-level and forward-backward reward weights respectively weaken behavior imitation or exploration.The coefficient α balances video-level imitation, while β controls reliance on generated frame-level goals.Source: §4.3 · p. 7, Figure 8 · p. 7
  • Alternative generated videos from RT-1 and RLBench consistently benefit GenReward over DreamerV3 without a generative prior.The result supports transfer of world knowledge from generated videos to the downstream agent.Source: §4.3 · pp. 7–8, Figure 9 · p. 8

5. Related Work

Related work has used diffusion models, vision-language models, and pretrained videos to support RL reward design and decision making. GenReward differs by using pretrained video diffusion-model representations to align agent trajectories with generated goal videos.

Reinforcement learning with diffusion model

  • Diffusion-based RL methods use diffusion models for world modeling, trajectory generation, or reward estimation, whereas GenReward uses video-diffusion priors for goal-driven rewards.The cited methods include DIAMOND, PolyGRAD, TADPoLe, and Diffusion Reward.Source: §5 · p. 8

VLMs as reward function

  • VLM-based reward methods compute similarity between observations and text or demonstration videos, while GenReward measures alignment with generated goal videos using a video encoder.RoboCLIP, VLM-RMs, and FuRL represent the comparison set.Source: §5 · p. 8

Decision making with videos

  • Video-based decision-making methods learn representations, world models, value functions, rewards, or policies from pretrained or generated videos.GenReward specifically adopts generated videos as video-level goals for goal-conditioned behavior learning.Source: §5 · p. 8

6. Conclusions and Limitations

The paper presents GenReward as a video-diffusion-based framework for video-level and frame-level goal rewards. It reports effectiveness on Meta-World, while noting additional computational overhead during training.

Conclusions

  • GenReward uses pretrained video diffusion models to provide video-level and frame-level goal-driven rewards without explicit reward engineering.The framework exploits generative-model world knowledge for informative RL reward signals.Source: §6 · p. 8

Limitations

  • Computing video-level and forward-backward rewards during training introduces additional computational overhead.This is identified as a limitation of GenReward.Source: §6 · p. 8

Supplementary Material

The supplementary material includes additional performance comparisons for Meta-World Bin Picking and across various environments.

Supplementary Results

A.1. Results with Sparse Rewards

GenReward maintains consistent improvements over other baselines for Meta-World tasks under sparse rewards, where feedback arrives only every 64 steps.

  • GenReward achieves consistent improvements over other baselines under sparse Meta-World rewards, with feedback provided every 64 steps.Rewards are set to 0 before each 64-step feedback interval.Source: §A.1 · p. 11

A.2. Evaluation on Challenging Benchmark

Across DCS, Adroit, and Meta-World evaluations using different reward settings, GenReward consistently outperforms the baselines, including under limited-step sparse rewards.

  • GenReward consistently outperforms baselines across DCS, Adroit, and Meta-World benchmarks under dense and sparse reward settings.The evaluation uses DrQv2, with dense rewards on DCS and Adroit and limited-step 0/1 sparse rewards on Meta-World.Source: §A.2 · p. 11
  • The hallucination evaluation reports Pick Place returns of 796 ± 73 for GenReward with hallucination, compared with 454 ± 30 for dense reward.Source: §A.2 · p. 11
  • The component-sensitivity and variant evaluations are represented by Tables C and D, covering individual component replacements and GenReward variants.Source: Table C · p. 11, Table D · p. 11

A.3. Results on Video Hallucinations

Additional experiments show that GenReward remains effective with hallucinated videos, depends on its original component choices and action-aware reward, trains efficiently, and can still benefit from imperfect frame selection.

A.3. Results on Video Hallucinations

  • GenReward benefits consistently from videos containing hallucinations such as object teleportation, rather than explicitly addressing hallucinations.Source: §A.3 · p. 11

A.4. Component Sensitivity

  • The original CLIP-based goal selection and DINOv3 design achieve the best performance among the tested component variants.Source: §A.4 · p. 11

A.5. Comparison of GenReward Variants

  • Removing action information from the forward-backward reward decreases performance, while GenReward remains competitive with its real-video variant.Source: §A.5 · p. 11

A.6. VDM Adaption Details.

  • The video diffusion model adaptation uses 400 finetuning samples and reports an FVD score of 21.6, indicating high fidelity.Source: §A.6 · p. 11

A.7. Computational Complexity

B. Forward-Backward Network Details

C. Effect of Frame-Level Goal Selection

D. Hyperparameters

  • The supplementary hyperparameter tables specify GenReward’s model dimensions, optimization settings, world-model parameters, and imagination controls.Source: D. Hyperparameters · p. 12

Related papers

Loading 2512.00961v2…