Source-linked AI summary
ViVa: A Video-Generative Value Model for Robot Reinforcement Learning
Jindi Lv, Hao Li, Jie Li, Fankun Kong, Yang Wang, Pengfei Yi, Yifei Nie, Xiaofeng Wang, Zheng Zhu, Chaojun Ni, Qiuping Deng, Hengtao Li, Jiancheng Lv, Guan Huang
TL;DR
Existing VLM-based value models struggle to represent temporal dynamics and physical interactions needed for reliable long-horizon value estimation. ViVa repurposes a pretrained video generator to jointly predict future proprioception and scalar value, achieving strong results across three tasks and an average RECAP success rate of 80%.
Problem
VLM-based value models lack explicit modeling of temporal dynamics and physical interactions, limiting reliable value estimation for long-horizon robot tasks.
Method
ViVa repurposes a pretrained video generator to jointly predict future proprioception and a scalar value from current observations and robot proprioception.
Results
ViVa achieves state-of-the-art metric-based results across three tasks, reliably tracking task progress and detecting execution errors; integrated into RECAP, it achieves an average success rate of 80%.
Takeaways & Limitations
Grounding value estimation in predicted embodiment dynamics couples value with foresight beyond static snapshots for robotic reinforcement learning.
Takeaways & Limitations
Jointly predicting future visual latents degraded value-estimation accuracy, likely because high-dimensional visual reconstruction interfered with the simpler value objective.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models have advanced robot manipulation through large-scale pretraining, but real-world deployment remains challenging due to partial observability and delayed feedback. Reinforcement learning addresses this via value functions, which assess task progress and guide policy improvement. However, existing value models built on vision-language models (VLMs) struggle to capture temporal dynamics and physical interactions, undermining reliable value estimation in long-horizon tasks. In this paper, we propose ViVa, a video-generative value model that repurposes a pretrained video generator to jointly predict future proprioception and a scalar value. By grounding value estimation in anticipated embodiment dynamics, ViVa leverages spatiotemporal priors to intrinsically couple value with foresight beyond static snapshots. ViVa achieves state-of-the-art results in metric-based evaluation across three tasks, producing reliable value signals that accurately track task progress and detect execution errors. Integrated into RECAP, it achieves an average success rate of 80%, highlighting the promise of video-generative models for value estimation.
1. Introduction
Real-world robot manipulation requires value estimates that connect current behavior with delayed future outcomes. ViVa addresses this by using video-based future prediction to produce temporally grounded value signals for reinforcement learning.
- Partial observability and delayed feedback make connecting present robot behavior with future task outcomes a central challenge.
- Value functions estimate expected future outcomes, distinguish beneficial from undesirable behavior, and provide learning signals for policy improvement.
- VLM-based value models use static image–text representations that do not explicitly model how scenes evolve over time.
- ViVa repurposes a pretrained video generator to jointly predict future proprioception and a scalar value from current observations and robot proprioception.
- Across three long-horizon manipulation tasks, ViVa shows sensitivity to fine-grained execution errors, generalizes to novel objects, and improves RECAP real-robot performance.
2. Related Works
Prior robotic value-estimation methods primarily use static-image VLMs, while video generation models provide temporal dynamics useful for anticipating future states. ViVa applies this video-generative capability directly to scalar value estimation rather than action generation or guidance.
- Robotic reinforcement-learning value methods span offline Q-learning, autonomous real-world interaction, and end-to-end approaches.
- Prior VLM-based approaches capture per-frame semantics but lack explicit modeling of temporal dynamics and physical interactions.
- Video generation models are optimized to capture temporal evolution and are therefore suited to anticipating visual dynamics.
- In robotics, video generation has mainly supported planning, policy learning, action extraction, goal conditioning, or joint video-action generation.
- ViVa instead repurposes a pretrained video generator to predict scalar values grounded in anticipated embodiment dynamics.
3. Method
ViVa formulates robotic value estimation as future prediction, using a pretrained video generator to jointly predict future proprioception and current value from multimodal observations. Its latent-frame diffusion design and outcome-aware returns ground value estimation in temporal and embodied dynamics.
- 3.1. Problem Formulation: ViVa represents the joint observation as multi-view images and proprioception, then predicts future proprioception alongside the current value.The value function estimates expected future return, while ViVa grounds that estimate in anticipated physical dynamics.
- 3.2. Overall Architecture: The architecture embeds images, proprioception, and scalar values as latent frames, using repeat-padding for proprioception and broadcast for values.All modalities are mapped into the pretrained video generator’s latent space.
- 3.2. Overall Architecture: During training, clean conditioning frames prefix noisy targets for future proprioception and value, which a diffusion Transformer denoises jointly.The conditioning prefix contains a blank token, current proprioception, and current multi-view images.
- 3.2. Overall Architecture: At inference, reverse diffusion generates future proprioception and value from the current observation prefix, then decodes the value by averaging latent elements.The predicted value is rescaled from [-1, 1] to [0, 1], while future proprioception is reconstructed by inverting repeat-padding.
- 3.2. Overall Architecture: Jointly predicting future visual latents degraded value accuracy, so the method focuses its targets on value and future proprioception.The paper attributes this observation to interference between high-dimensional visual reconstruction and simpler value prediction.
- 3.3. Reward Definition and Value Training: The return provides outcome-aware supervision: successful episodes occupy [0, 1), while failed episodes occupy [1, 2), preserving a margin of 1.0.This return supervises the value latent and is jointly learned with future proprioception to capture task integrity and embodied dynamics.
4. Experiments
Experiments evaluate ViVa across three long-horizon manipulation tasks using held-out episodes and annotated failure modes. The dominant errors differ by task, concentrating on flattening textiles, sequential box assembly, and precise paper handling.
- Evaluation Tasks: Evaluation covers shirt folding, box packaging, and paper roll organization as three long-horizon manipulation tasks.Each task involves multi-stage manipulation with task-specific success conditions and failure modes.
- Evaluation Dataset: 50 held-out episodes per task are evaluated with manually annotated milestone and error frames.Milestone frames appear in every episode, whereas error frames are annotated only when the corresponding failure occurs.
- Error Analysis: 45.5% of shirt-folding failures are Failed to spread flat, followed by Poor spreading quality at 27.3%.Failures concentrate during flattening, reflecting the difficulty of manipulating deformable textiles without wrinkles or misalignment.
- Error Analysis: 44.5% of box-packaging failures involve Side panel operation, while Box operation accounts for 26.7%.The task requires precise sequential coordination across multiple assembly stages.
- Error Analysis: 50.0% of paper-roll failures involve Label and tear operation, while Roll retrieval deviation accounts for 35.7%.Small deviations in tearing angle or retrieval trajectory can damage the flexible-paper product.
4.2. Evaluation Metrics
The evaluation metrics measure whether predicted values respond directionally to annotated milestones and execution errors. They summarize both response strength and event-detection accuracy.
- Event Response Score: Event Response Score measures directional value changes around annotated key frames labeled as milestones or execution errors.Values are averaged over windows immediately before and after each annotated frame.
- Sensitivity Metrics: Milestone Sensitivity and Error Sensitivity average Event Response Score over milestone and error frames, respectively.Positive values indicate correct directional responses, with larger magnitudes reflecting stronger sensitivity.
- Event Detection Rate: Event Detection Rate classifies an annotated frame as detected when ERS_k > 0.This criterion converts directional response into a binary event-detection decision.
- Event Detection Rate: Milestone Detection Rate and Error Detection Rate are proportions of correctly detected frames within their respective event types.The rates use positive-ERS milestone and error counts divided by the corresponding total frame counts.
4.3. Value Model Evaluation
ViVa provides stronger value estimates than sparse-sampled VLM methods, especially for fine-grained task events. It tracks milestones and execution errors more precisely across the three evaluation tasks.
- ViVa establishes new state-of-the-art value-model performance on two of three tasks, while dense-supervised methods outperform sparse-sampled approaches.
- ViVa achieves positive milestone and error sensitivity across tasks, whereas π*0.6 records negative MS (−0.266) and ES (−0.299).
- ViVa attains MDR of 0.990 and ErrDR of 1.000, compared with π*0.6’s 0.407 and 0.500, respectively.
- ViVa produces clear value increases at milestones and decreases at execution errors, while GVL and TopReward generate noisy, flat trajectories.
4.4. Object Generalization
On an out-of-domain pants-folding task, ViVa generalizes better than π*0.6 and remains responsive to milestone events. Its largest advantages appear on fine-grained event metrics.
- The evaluation uses 50 held-out pants-folding episodes annotated with milestone frames, while error detection on novel objects remains challenging without large-scale pretraining.
- ViVa outperforms π*0.6 across all four metrics on the out-of-domain pants-folding task.
- MS more than doubles from 0.016 to 0.037, while MDR improves from 0.397 to 0.643 for ViVa versus π*0.6.
- ViVa consistently responds to milestone events on the unseen task, whereas π*0.6 captures the overall trend but misses key milestones.
4.5. Real-Robot Experiments
In real-robot experiments, replacing a VLM-based value function with ViVa improves RECAP performance across three long-horizon manipulation tasks. ViVa also improves faster across rollout rounds.
- RECAP with ViVa outperforms imitation-learning baselines and RECAP with a VLM value function on all three tasks.
- ViVa’s shirt-folding success rate rises from 50% to 80% between R0 and R1, while the VLM variant stagnates at 60%.
- On paper-roll organization, ViVa improves from 20% to 80% between R0 and R1, compared with the VLM variant’s increase from 20% to 50%.
- From R1 to R2, the VLM variant plateaus while ViVa reaches 90% on shirt folding and extends its lead on box packaging.
4.6. Ablation Studies
Ablations show that video-generator backbones, future proprioception prediction, and pretrained video weights each contribute to ViVa’s value estimation and error detection.
- Replacing the VLM backbone with a video generator improves value metrics, including shirt-folding MS from −0.266 to 0.137 and MDR from 0.407 to 1.000.
- Adding input state and future-state prediction raises box-packaging ErrDR from 0.449 to 0.612 and changes shirt-folding ES from −0.026 to 0.012.
- Removing pretrained video weights drops box-packaging MS from 0.034 to 0.009 and shirt-folding ES from 0.012 to −0.524.
- On paper-roll organization, removing pretrained weights reduces MDR from 0.762 to 0.489 and ErrDR from 0.778 to 0.667.
4.7. Hyperparameter Analysis
Hyperparameter studies identify λprop = 0.5 and K = 50 as the settings that best balance value with future proprioception prediction and provide reliable error awareness. ViVa also offers a favorable computational-cost balance relative to the VLM baseline.
- Loss weight analysis: λprop = 0.5 yields positive error sensitivity across all three tasks, whereas insufficient proprioception weighting weakens error awareness.At λprop = 0.1, shirt folding shows negative ES and zero ErrDR.
- Loss weight analysis: The λprop = 0.5 setting achieves the best trade-off between value and future proprioception prediction and is used throughout the experiments.
- Prediction horizon analysis: K = 50 provides sufficient temporal context, producing positive ES across all tasks and ErrDR = 1.0 on shirt folding.At K = 10, shirt folding reaches ErrDR = 0.000 and ES = −0.002.
- Prediction horizon analysis: The K = 50 setting achieves the best trade-off across tasks and is adopted for all experiments.
5. Conclusion
ViVa grounds value estimation in predicted future dynamics by jointly predicting future proprioception and value. It achieves strong results across three real-robot tasks, while large-scale pretraining remains future work because of resource constraints.
- ViVa jointly predicts future proprioception alongside value to learn temporally grounded representations that track task progress and detect execution errors.
- ViVa achieves strong results across all three real-robot tasks, supporting spatiotemporal priors and embodiment-aware prediction for value learning.
- Resource constraints prevented pretraining the value model at scale, leaving potential cross-task generalization for future work.