PaperlayerView exact revision on arXiv

Source-linked AI summary

iVideoGPT: Interactive VideoGPTs are Scalable World Models

Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, Mingsheng Long

arXiv:2405.15223v3cs.CVcs.LGcs.RO

TL;DR

Existing world models must reconcile step-level interactivity with the scalability of modern video generation, which otherwise lacks sufficiently granular action control. iVideoGPT uses an autoregressive transformer with compressive tokenization, achieving competitive results across video prediction, visual planning, and visual model-based reinforcement learning after pre-training on millions of manipulation trajectories.

  • Problem

    The central gap is developing visual world models that combine scalable video generation with interactive, step-by-step agent control.

  • Method

    iVideoGPT autoregressively models visual observations, actions, and rewards as token sequences, using compressive tokenization to reduce sequence length.

  • Results

    iVideoGPT achieves competitive performance against state-of-the-art methods across video prediction, visual planning, and visual model-based RL.

  • Takeaways & Limitations

    The pre-trained model serves as a versatile foundation for accurate video prediction and simplified model-based planning or reinforcement learning.

  • Takeaways & Limitations

    Public robotic data have limited diversity, and the compressive tokenization assumes initial frames provide sufficient context for future frames.

Abstract

from arXiv · show

World models empower model-based agents to interactively explore, reason, and plan within imagined environments for real-world decision-making. However, the high demand for interactivity poses challenges in harnessing recent advancements in video generative models for developing world models at scale. This work introduces Interactive VideoGPT (iVideoGPT), a scalable autoregressive transformer framework that integrates multimodal signals--visual observations, actions, and rewards--into a sequence of tokens, facilitating an interactive experience of agents via next-token prediction. iVideoGPT features a novel compressive tokenization technique that efficiently discretizes high-dimensional visual observations. Leveraging its scalable architecture, we are able to pre-train iVideoGPT on millions of human and robotic manipulation trajectories, establishing a versatile foundation that is adaptable to serve as interactive world models for a wide range of downstream tasks. These include action-conditioned video prediction, visual planning, and model-based reinforcement learning, where iVideoGPT achieves competitive performance compared with state-of-the-art methods. Our work advances the development of interactive general world models, bridging the gap between generative video models and practical model-based reinforcement learning applications. Code and pre-trained models are available at https://thuml.github.io/iVideoGPT.

1 Introduction

iVideoGPT addresses the tension between scalable video generation and step-level interactive world modeling with an autoregressive transformer and compressive tokenization. Pre-training on millions of manipulation trajectories yields a versatile foundation adaptable to video prediction, planning, and model-based reinforcement learning.

  • Applications: The framework is presented as a single versatile foundation for interactive world models across a wide range of downstream applications.Figure 1 highlights its scalability and adaptation to practical robotic manipulation tasks.
  • Motivation: Scalable video generators offer realistic trajectories but lack the step-by-step action intervention needed for precise skill learning.Recurrent world models provide step-level interactivity but have limited ability to model complex, in-the-wild data at scale.
  • Approach: iVideoGPT integrates visual observations, actions, and rewards into an interactively autoregressive sequence of tokens.Its compressive tokenization conditions each observation on contextual observations, achieving an asymptotic 16× reduction in token sequence length.
  • Approach: Pre-training spans over one million robotic and human manipulation trajectories, followed by domain-specific adaptation for downstream tasks.The pre-trained model supports action-conditioned video prediction, visual planning, and visual model-based RL.
  • Evidence: Experiments cover video prediction, visual planning, and visual model-based RL, with competitive performance against state-of-the-art methods.The authors report that iVideoGPT can simulate accurate and realistic experiences across these applications.

2 Problem Formulation

The paper formulates visual-control world modeling as interactive video prediction in a partially observable environment. The central challenge is enabling causal, action-conditioned step-level interaction without sacrificing the scalability of modern video generation models.

  • World-model formulation: A world model is an internal model learned by an agent to simulate its environment.The environment is represented as a partially observable Markov decision process.
  • Interactive prediction: Given visual history and actions, the world model approximates and samples the next observation and reward transition.At each step, the agent acts from previous observations and imagined observations, receiving predicted feedback from the model.
  • Interactive prediction: Video is used as a task-agnostic, widely available modality for self-supervised visual-control world modeling.The formulation treats video frames as the observation space and casts control as interactive video prediction.
  • Challenge: Non-causal temporal modules in many video generators prevent causal intermediate action control during generation.Recurrent MBRL world models support interaction but lack scalability, creating the problem addressed by iVideoGPT.

3 Interactive VideoGPT

iVideoGPT combines compressive visual tokenization with an autoregressive transformer to model interactive video and multimodal signals. It is pretrained on diverse manipulation videos and adapted for goal-conditioned prediction, action conditioning, reward prediction, and downstream tasks.

  • Architecture: iVideoGPT uses a compressive tokenizer and autoregressive transformer to predict video tokens while integrating visual observations, actions, and rewards.The model can also omit actions and rewards during action-free video pre-training.
  • Compressive tokenization: Conditional VQGAN encoding represents context frames with N tokens but future frames with n ≪ N tokens, reducing redundancy while preserving essential dynamics.Future-frame tokenization is conditioned on context through dual encoders and decoders.
  • Compressive tokenization: 16× asymptotic token-sequence reduction supports faster rollouts for model-based planning and reinforcement learning while improving temporal consistency.The reduction uses N = 16 × 16 and n = 4 × 4 tokens.
  • Interactive prediction: The transformer flattens tokenized frames into a sequence with slot tokens marking frame boundaries and generates predictions frame by frame through next-token prediction.The sequence length is L = (N + 1)T0 + (n + 1)(T − T0) − 1.
  • Pre-training: Action-free pre-training predicts subsequent video tokens with cross-entropy and excludes context-frame generation so capacity focuses on dynamics.The first predicted-frame token begins at index (N + 1)T0 + 1.
  • Adaptation and applications: The model is pretrained on 1.4 million trajectories and adapted for goal-conditioned prediction, action conditioning, reward prediction, and domain-specific applications.The architecture supports sequence rearrangement for goal-conditioned prediction and multimodal conditioning for interactive world models.

4 Experiments

Experiments evaluate iVideoGPT for video prediction, visual MPC, and visual model-based RL, alongside analyses of transfer, scaling, tokenization, and context-dynamics separation. Across these settings, iVideoGPT provides competitive prediction and control performance while supporting efficient, adaptable world-model use.

  • Experimental scope: Experiments cover video prediction, visual planning, and visual model-based reinforcement learning, with additional analyses of data efficiency, model scaling, and computational efficiency.The evaluation compares iVideoGPT with prior methods across multiple control-relevant settings.
  • Video prediction: iVideoGPT achieves competitive video-prediction performance against MAGVIT on BAIR and FitVid on RoboNet while enabling action conditioning and scalable architecture.Action conditioning improves BAIR FVD by almost 20%, and the model extends from 64×64 to 256×256 resolution on RoboNet.
  • Visual planning: iVideoGPT outperforms all baselines on two RoboDesk tasks with a large margin and reaches comparable average performance to SVG′ on the VP2 benchmark.Its suboptimal open-slide performance is attributed to discretization limitations and the benchmark’s imperfect built-in reward design.
  • Visual model-based reinforcement learning: The model-based RL method improves sample efficiency over DrQ-v2 and matches or exceeds DreamerV3 across six Meta-World tasks.The method uses synthetic rollouts to augment the replay buffer and is presented as the first successful application of MBPO to visual continuous control tasks.
  • Model analysis: Large-scale pre-training transfers to unseen robot motion and becomes especially beneficial when downstream data are scarce.With an adapted tokenizer, the transformer transfers pre-trained knowledge without transformer fine-tuning; gains are significant with 100 or 1,000 BAIR trajectories.
  • Model analysis: Increasing iVideoGPT size accelerates validation-loss reduction, while compressive tokenization lowers token counts, saves time and memory, and enables larger models at lower cost.The tokenization analysis also finds slightly lower reconstruction quality than standard 16 × 16 tokenization but more consistent contextual information for video prediction.
  • Model analysis: Removing decoder cross-attention to context frames preserves movement trajectories while discarding most contextual information, supporting a separation of context and dynamics.Goal-conditioned prediction also produces more accurate paths toward specified goals than action-free prediction.

5 Related Work

Related work spans world models for visual control and video generation or prediction. iVideoGPT addresses limitations in scalability and stepwise interactivity through compressive, context-aware tokenization within an autoregressive framework.

  • World models for visual control: Visual-control world models often use action-conditioned video prediction, while latent-imagination methods improve rollouts but tightly couple model and policy learning.The paper positions its approach as reducing this design complexity.
  • Video generation and prediction: iVideoGPT combines compressive tokenization with context-aware representations and cross-attention to provide a more generic form of interactive video tokenization.Its autoregressive transformer also integrates with established language-modeling infrastructure and handles diverse conditions without specialized adapter modules.

6 Discussion

iVideoGPT is presented as a scalable, generic world-model foundation pretrained on human and robotic manipulation trajectories and adapted across downstream tasks. The authors identify limited robotic-data diversity, missing modalities, long-video camera-motion challenges, and the need for more complex real-robot evaluation as remaining issues.

  • iVideoGPT integrates multimodal signals with autoregressive next-token prediction and supports video prediction, planning, and model-based reinforcement learning.
  • Public robotic data have limited diversity, motivating more extensive pretraining data and stronger knowledge transfer between human videos and robots.
  • The current formulation omits modalities such as multi-view observations, proprioceptive states, and actions beyond action-free video prediction.
  • Compressive tokenization may falter for long videos with substantial camera motion, while more complex real-robot tasks remain an important future direction.

A Implementation and Experimental Details

The implementation-details section points readers to Tables 2, 3, and 5 for the experiment’s main hyperparameters and provides a comprehensive explanation of experimental details.

  • Tables 2, 3, and 5 contain the experiment’s main hyperparameters, alongside a comprehensive explanation of the experimental details.

A.1 Architecture

The architecture uses compressive VQGAN tokenization followed by an autoregressive transformer. Context and future frames are encoded separately, with cross-attention transferring contextual information to dynamic future-frame representations.

  • Tokenizer: Compressive tokenization uses context and prediction encoder-decoder pairs, with the prediction pathway employing a tighter bottleneck for dynamic information.
  • Tokenizer: Multi-scale cross-attention filters contextual features into dynamics information and retrieves context during decoding for gradual scene reconstruction.
  • Tokenizer: Cross-attention is restricted to smaller feature maps to reduce memory usage, with thresholds of 16 × 16 for 64 × 64 inputs and 32 × 32 for 256 × 256 inputs.
  • Tokenizer: Each context frame is independently encoded and decoded, then its feature maps are concatenated for cross-attention in the prediction encoder and decoder.
  • Transformer: The transformer flattens video into tokens, separates context and future-frame token IDs, and uses dedicated slot tokens before each frame group.

A.2 Action-free Video Pre-training

The experimental setup combines filtered and weighted robotic and human-action datasets with standardized training, evaluation, and model-based reinforcement-learning procedures. The study uses 35 datasets, task-specific reward handling, and matched baselines including DreamerV3-related APV.

  • Data mixture: The pretraining mixture contains 35 datasets from Open X-Embodiment and Something-Something-v2, filtered for image availability, robot type, repetition, and resolution.
  • Training details: Training samples use uniformly sampled video segments with dataset-specific step sizes to account for differing collection frequencies.
  • Visual Model-based RL: Meta-world experiments add a task-success bonus of r_bonus = 10.0 because original rewards do not consistently correlate with success rates.
  • Visual Model-based RL: The model-based RL implementation embeds iVideoGPT within MBPO and uses DrQ-v2 as the actor-critic algorithm.
  • Baselines: The DreamerV3 comparison uses APV modified with DreamerV3 features and pretrained on the same dataset as iVideoGPT.

B.1 Qualitative Evaluation

The supplementary evaluations cover qualitative video prediction, human preferences, visual model-based RL, efficiency, and comparisons with related tokenization approaches. Across these analyses, iVideoGPT is evaluated on diverse datasets and tasks, with reported advantages in human preference, task performance, and computational efficiency.

  • Qualitative video prediction: iVideoGPT was qualitatively evaluated on Open X-Embodiment, BAIR, RoboNet, VP2, and Meta-world across action-free, goal-conditioned, action-conditioned, and reward-prediction settings.The supplementary figures include zero-shot predictions and high-resolution RoboNet results.
  • Human study: 386 annotations from 9 participants found iVideoGPT preferred over VideoGPT and MCVD based on physical naturalness and feasible robot-object interactions.The study compared videos generated on the action-free BAIR dataset.
  • Visual model-based RL: MBPO with iVideoGPT outperformed FitVid on 5 of 6 Meta-world tasks and performed comparably on the remaining task.FitVid’s imagined trajectories were also qualitatively blurrier, which may hinder accurate environment simulation and model-based RL performance.
  • Efficiency: Compressive tokenization provides significant training-memory savings and faster generation rollouts without making the tokenizer the generation-time bottleneck.Generation time remains primarily determined by the autoregressive transformer’s number of forward passes.
  • Tokenization comparison: Compared with VideoGPT, iVideoGPT’s cross-attention tokenizer handles temporal redundancy more efficiently and represents videos with fewer tokens.The reported configuration uses L = 511 tokens with N = 256, n = 16, T = 16, and T0 = 1.

C.3 Failure Case Analysis for Visual Planning

iVideoGPT underperforms on RoboDesk open slide because low-resolution observations and two-stage tokenization hinder modeling subtle gripper–handle contact. Mixed VP2 results also reflect a learned reward classifier that can favor unlikely predicted trajectories.

  • iVideoGPT performs sub-optimally on RoboDesk open slide, with analysis attributing the issue to limitations in both the model and benchmark.
  • Inaccurate model prediction: At 64 × 64 resolution, the model misses subtle observation changes needed to determine whether the gripper contacts the slide handle.Despite strong validation mean square error and perceptual loss, the model predicts incorrect outcomes on some trajectories.
  • Inaccurate model prediction: Two-stage tokenization and prediction may worsen uncertainty because discrete tokenization loses observation information, while end-to-end models perform significantly better in this comparison.
  • Inaccurate model prediction: Training and evaluating at 256 × 256 resolution could mitigate these issues and improve control, but experiments use lower resolution for fair comparison.
  • Imperfect built-in reward design: VP2’s learned success classifier can be fooled by out-of-distribution inputs, assigning high rewards to low-quality or unlikely-to-succeed predicted trajectories.This imperfect reward design likely contributes to mixed benchmark results, and no current VP2 model consistently outperforms others across all tasks.
Loading 2405.15223v3…