Source-linked AI summary
VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, Xiaojie Jin
TL;DR
Language cannot fully capture real-world knowledge, while prior video-based approaches have largely relied on language instructions and simple commands. VideoWorld trains an autoregressive video model on unlabeled task videos; it learns across Go and robotics, including 5-dan Go performance with 300 million parameters.
Problem
Language alone cannot capture all real-world knowledge, and existing video approaches are limited to simple commands and rely heavily on language instructions.
Source: §1 · p. 1, §2.1 · p. 3Method
VideoWorld trains an autoregressive model on unlabeled videos and predicts video frames alongside latent codes that represent multi-step future visual changes.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 5Results
Video-only training learns Go rules and fundamental robotic operations; VideoWorld reaches a 5-dan professional level with 300 million parameters and approaches oracle performance in robotics.
Source: §1 · p. 2Takeaways & Limitations
The findings indicate that video-only learning can acquire complex task knowledge and that compact visual-change representations improve knowledge learning.
Source: §1 · p. 2Applying the approach to real-world scenarios still faces challenges including high-quality video generation and generalization.
Source: §6 · p. 10
Abstract
from arXiv · showhide
This work explores whether a deep generative model can learn complex knowledge solely from visual input, in contrast to the prevalent focus on text-based models like large language models (LLMs). We develop VideoWorld, an auto-regressive video generation model trained on unlabeled video data, and test its knowledge acquisition abilities in video-based Go and robotic control tasks. Our experiments reveal two key findings: (1) video-only training provides sufficient information for learning knowledge, including rules, reasoning and planning capabilities, and (2) the representation of visual change is crucial for knowledge acquisition. To improve both the efficiency and efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional level in the Video-GoBench with just a 300-million-parameter model, without relying on search algorithms or reward mechanisms typical in reinforcement learning. In robotic tasks, VideoWorld effectively learns diverse control operations and generalizes across environments, approaching the performance of oracle models in CALVIN and RLBench. This study opens new avenues for knowledge acquisition from visual data, with all code, data, and models open-sourced for further research.
1. Introduction
VideoWorld investigates whether unlabeled videos alone can teach task rules, reasoning, and planning, addressing limits of language and the relative scarcity of work on pure visual learning. Its Go and robotics experiments find that video-only learning can acquire complex task knowledge, while compact representations of visual change improve learning.
1. Introduction
- The study asks whether AI can learn knowledge solely from visual input, testing Go rules, reasoning, and planning alongside robotic manipulation in CALVIN and RLBench.Go provides well-defined rules and requires strategic reasoning and forward planning.Source: §1 · pp. 1–2
- VideoWorld trains an autoregressive transformer on discrete tokens from task-execution videos and infers task operations from generated frames.The videos are the sole training data and source of knowledge; Go moves and robotic actions are inferred from newly generated frames.Source: §1 · p. 2
- The Latent Dynamics Model compresses future visual changes into latent codes, improving video-learning efficiency and effectiveness while providing a way to probe learned knowledge.The model predicts both video frames and latent codes to capture multi-step visual context, including object interactions and scene dynamics.Source: §1 · p. 2, Figure 2 · p. 2
- VideoWorld reaches a 5-dan professional level against KataGO with only 300 million parameters, using visual observation without search or reward learning.The paper describes 5-dan as representing a highly trained human player.Source: §1 · p. 2
- In CALVIN and RLBench, VideoWorld learns diverse robotic controls, generalizes across environments, and approaches oracle-model performance.The oracle models are trained with ground-truth action labels.Source: §1 · p. 2
- The authors introduce Video-GoBench for video-based Go training and evaluation, while describing knowledge learning from unlabeled videos as early-stage.They identify improved visual representations and large-scale pretraining as likely ways to enhance learning.Source: §1 · pp. 2–3
2. Related Work
Prior work explores video generation and knowledge learning through language, labels, or latent actions, but this work investigates complex reasoning and planning learned solely from visual signals. It distinguishes its approach by studying multi-step latent representations of future visual changes for long-horizon tasks.
2.1. Video Generation
- Video-generation research includes planners and text-conditioned decision systems, but prior approaches are limited to simple commands and rely heavily on language instructions.Video models have also been studied as agents and environment simulators, while standard generation work uses diffusion or autoregressive methods in compressed latent spaces.Source: §2.1 · p. 3
2.2. Learning Knowledge
- Whereas knowledge-learning work often uses language, labels, or annotated game data, this study examines long, complex tasks learned from visual signals without labels.Prior examples include language-model reasoning and planning, plus chess models trained on annotated games.Source: §2.2 · p. 3
2.3. Latent Actions
- LAPO, Genie, and LAPA learn latent actions from video, whereas this work studies multi-step future-change representations for complex reasoning and long-horizon planning.The paper frames its focus as going beyond action learning to examine how future visual changes support knowledge acquisition.Source: §2.3 · p. 3
3. VideoWorld
VideoWorld predicts video frames autoregressively, then maps generated content to task actions. Its Latent Dynamics Model compresses multi-frame visual changes into codes that the transformer uses alongside frame tokens.
3.1. Basic Video Generation Framework
- The basic framework trains a VQ-VAE and autoregressive transformer to predict video tokens, then decodes generated tokens into frames for mapping to task actions.The generator models the next frame from observation history; a task-relevant policy converts generated frames into actions.Source: §3.1 · p. 3
3.2. Learning with Latent Dynamic Model
- Raw video uses many more tokens than state sequences to represent decision-relevant changes, motivating compact codes for visual dynamics.The LDM uses query embeddings to represent changes across multiple frames, including temporally correlated Go moves and robotic actions.Source: §3.2 · p. 4
- The LDM causally encodes unquantized features from each frame and its next H frames, quantizes learned change representations, and decodes them to predict those future frames.Its training objective minimizes the ℓ2 distance between predicted and target future frames.Source: §3.2 · pp. 4–5
- The transformer autoregressively predicts LDM codes alongside discretized video frames, using separate codebooks whose vocabularies are combined for prediction.During inference, the generated codes and predicted frame are passed to an independently trained Inverse Dynamics Model, which uses a small amount of video-action labels to produce actions.Source: §3.2 · pp. 5–6
4. Video-GoBench
Video-GoBench is a visual Go benchmark designed to evaluate rule learning, reasoning, and planning through gameplay. It uses a large-scale video dataset and evaluates models with legality, tournament strength, and oracle-based move metrics.
4. Video-GoBench
- Video-GoBench uses simplified visuals and online gameplay to assess Go rule learning, reasoning, and planning while reducing texture and dynamics distractions.The simulated environment supports converting large-scale state-based gameplay data into video sequences.Source: §4 · p. 6
4.1. Dataset Generation
- The training corpus combines 3.2 million KataGo self-play examples with 7.8 million human-match games, totaling about 400 million unique board states.Its test set contains 1,000 matches and 56,000 unique board states after opening moves are removed; human-game moves are reannotated with KataGo.Source: §4.1 · p. 6
4.2. Evaluation
- Evaluation measures legal-move rate, Elo from an eight-agent tournament, action accuracy against KataGo’s best move, and action-value relative to KataGo-9d.Figure 5 visualizes play against KataGo and predicted latent codes as a probe for signs of forward planning.Source: §4.2 · p. 6, Figure 5 · p. 6
5. Experiment
Experiments in Go and robotic simulation test whether video-only training can acquire task knowledge. Results emphasize the value of compact visual-change representations, with LDM-based VideoWorld learning Go strategy and robotic control across tasks and environments.
5.1. Implementation Details; 5.2. Benchmarks
- The experiments use Go, CALVIN, and RLBench; training uses 4×4 frame compression for Go and CALVIN, with six-frame Go and ten-frame CALVIN sequences.The CALVIN evaluation uses the ABCD→D split; RLBench training uses 20,000 trajectories.Source: §5.1 · p. 6, §5.2 · p. 7
5.3. Key Findings with The Basic Framework
- The basic framework learns Go rules with nearly 100% legality and nearly 50% best-move accuracy; compact state sequences improve Go metrics, while robotic action labels boost task completion.State sequences converge faster, although they omit some local stone patterns; raw video includes additional visual detail and requires substantially more tokens.Source: §5.3 · p. 7, Table 4 · p. 8
5.4. Results with LDM
- VideoWorld-300M reaches Elo 2317 versus KataGo-5d’s 2253, while CALVIN performance approaches action-label-supervised models and joint CALVIN–RLBench training approaches oracle performance across environments.VideoWorld-300M also scores Elo 2317 versus 1998 for its counterpart without LDM.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 7
5.5. Understanding Learned Knowledge with LDM
- LDM latent codes capture temporal dependencies and Go move patterns; during inference, Go codes group by output position and imagined opponent moves achieve 71.2% average action-value and 74.3% action accuracy.In robotics, predicted codes group by task type, with later prediction steps more distinctly separated by task.Source: §5.5 · pp. 8–9, Figure 6 · p. 9
5.6. More Ablations
- Go performs best with five-step compression, but longer horizons fail to converge; simpler CALVIN dynamics tolerate longer compression, while robotic tasks need larger codebooks than Go.Codebooks that are too large hinder convergence in both scenarios.Source: §5.6 · p. 9
- Human Go data from players of varying skill levels affects performance, and predicting next frames alongside latent codes further improves performance over code-only training in Go and robotics.The authors hypothesize that next-frame prediction improves environmental understanding and code accuracy.Source: Table 5 · p. 8, §5.6 · pp. 9–10
5.7. Visualizations
- VideoWorld’s Go matches against KataGo-5d illustrate squeeze and self-sacrifice tactics, including a deliberate stone sacrifice to create an opportunity for greater capture.The paper also shows robotic-arm control results on manipulation tasks.Source: §5.7 · p. 10, Figure 7 · p. 10, Figure 8 · p. 10
6. Conclusion
The conclusion presents video-based next-token prediction as an initial approach to knowledge learning from raw visual data. It identifies real-world video generation quality and generalization as remaining challenges.
6. Conclusion
- The paper’s experiments find that video-trained models can master Go rules and learn fundamental robotic operations, while visual-change representation is crucial to knowledge learning.The authors propose LDM to improve the efficiency and effectiveness of knowledge acquisition.Source: §6 · p. 10
- The work is an initial exploration of raw-video knowledge learning using next-token prediction in custom Go and robotic simulation environments..Source: §6 · p. 10
- Real-world application still faces challenges including high-quality video generation and generalization.The authors see video generation models as having potential to become general knowledge learners.Source: §6 · p. 10
A. Implementation Details
The implementation uses an LDM to encode visual changes and reconstruct video, while a transformer predicts future frames and an inverse dynamics model converts predictions into robotic actions. The appendix describes training configurations and evaluation setups for CALVIN and RLBench.
Latent dynamics model
- The LDM causally encodes image features, extracts change information with learnable embeddings, and reconstructs frames from the first-frame features and embeddings.Its encoder, embedding, and decoder functions form the three-part LDM described in the appendix.Source: A. Implementation Details · p. 14
Training details
- Detailed training configurations for the LDM and auto-regressive transformer are presented in Table A.1.The supplied table caption identifies the two model configurations but does not state their values.Source: A. Implementation Details · p. 14, Table A.1 · p. 14
Robotics action execution
- For robotic control, the model predicts a next frame and latent codes, then an inverse dynamics model infers actions that are executed open-loop with new observations fed into the next prediction cycle.The action vector contains arm displacement, Euler angles, and gripper state.Source: A. Implementation Details · p. 14
Inverse dynamics model
- The inverse dynamics model combines predicted-frame and latent-code features to produce seven control signals, and is trained separately for each environment.It uses a ResNet-18 and MLPs, with AdamW and mean squared error for training.Source: A. Implementation Details · p. 14
RLBench evaluation setup
- VideoWorld is evaluated in CALVIN and RLBench; both the LDM and transformer train on their combined datasets, while the inverse dynamics model is environment-specific.RLBench training data consists of 20,000 generated trajectories, and its environment appearance and task settings differ from CALVIN.Source: A. Implementation Details · p. 14
Latent dynamics model
- During the LDM training stage, z is quantized using FSQ, and the model is trained with mean squared error to reconstruct video from z and first-frame features.Source: A. Implementation Details · p. 15
B. Details on Video-GoBench
Video-GoBench contains many unique board states, and repeated training states become scarce as games progress against reinforcement-learning agents. The measured overlap reaches zero by move 30, supporting the claim that high performance cannot rely on memory alone.
Board state count
- Approximately 400 million unique board states populate Video-GoBench, concentrated mostly in the first 100 moves, with substantially fewer at later moves.Figure A.1a presents the distribution by move number.Source: B. Details on Video-GoBench · p. 15
Board state repetition rate
- By move 30, none of the 400 evaluated games against KataGo-9d share board states with the training set.Figure A.1b presents repetition rate across move numbers.Source: B. Details on Video-GoBench · p. 15
- The decline in repeated states means strong performance cannot be explained by memorizing training scenarios alone, highlighting generalization.The benchmark’s combinatorial growth makes repeated states less common as games progress.Source: B. Details on Video-GoBench · p. 15