Source-linked AI summary
Learning Universal Policies via Text-Guided Video Generation
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, Pieter Abbeel
TL;DR
The paper asks whether general-purpose agents can be built despite diverse environments, task specifications, and control spaces. It casts decision making as text-conditioned video generation, then extracts actions from generated plans. UniPi demonstrates combinatorial and multitask generalization, real-world transfer, and realistic robot-video synthesis, while remaining limited by slow generation and assumptions about observability.
Problem
Constructing agents that solve diverse tasks is difficult because environments lack a universal state interface and reward specifications vary across tasks.
Method
UniPi uses a diffusion video planner conditioned on an image and textual task description, followed by a task-specific action generator that extracts controls from generated frames.
Results
UniPi achieves combinatorial generalization, multitask learning, and real-world transfer; pretrained models reach 77.1% success on generated-video task evaluation.
Takeaways & Limitations
Text-conditioned video provides a shared representation for planning and generalizing across tasks, environments, and robot manipulation settings.
Takeaways & Limitations
Video diffusion can take about a minute to generate photorealistic videos, and partially observable environments may produce physically unfaithful hallucinations.
Abstract
from arXiv · showhide
A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress in text-guided image synthesis has yielded models with an impressive ability to generate complex novel images, exhibiting combinatorial generalization across domains. Motivated by this success, we investigate whether such tools can be used to construct more general-purpose agents. Specifically, we cast the sequential decision making problem as a text-conditioned video generation problem, where, given a text-encoded specification of a desired goal, a planner synthesizes a set of future frames depicting its planned actions in the future, after which control actions are extracted from the generated video. By leveraging text as the underlying goal specification, we are able to naturally and combinatorially generalize to novel goals. The proposed policy-as-video formulation can further represent environments with different state and action spaces in a unified space of images, which, for example, enables learning and generalization across a variety of robot manipulation tasks. Finally, by leveraging pretrained language embeddings and widely available videos from the internet, the approach enables knowledge transfer through predicting highly realistic video plans for real robots.
1 Introduction
The paper proposes text-conditioned video generation as a universal planning strategy for agents operating across diverse environments and tasks. UniPi uses language to specify goals, synthesizes future image trajectories, and extracts actions to support generalization, multitask learning, and internet-scale transfer.
- Different environments have distinct state and action spaces, hindering knowledge sharing, learning, and generalization across tasks.
- The planner conditions on a current image and textual goal, generates an image-sequence trajectory, and regresses actions from synthesized frames.The instantiated planner uses video diffusion followed by an inverse dynamics model to extract control actions.
- Combinatorial Generalization: Language enables combinatorial generalization to unseen combinations of geometric relations and novel behaviors.The paper reports rearranging objects into new unseen combinations of geometric relations.
- Multi-task Learning: Video-based action prediction supports multitask learning and generalization to new language-conditioned tasks without test-time finetuning.
- Action Planning: Hierarchical, steerable, and human-interpretable video plans support long-horizon action planning, test-time constraint incorporation, and plan verification.Plans can be generated sparsely and refined, biased by new constraints, and inspected in video space.
- Internet-Scale Knowledge Transfer: Internet-scale text-video pretraining transfers knowledge by enabling realistic robot-motion video synthesis from language instructions.
- UniPi formulates policy generation as text-conditioned video planning, synthesizing diverse behaviors instead of directly predicting subsequent actions.The approach uses video as a universal interface for observations and actions, with text specifying task goals.
2 Problem Formulation
The paper introduces UPDP as a video-based alternative to MDPs for unified decision making across diverse environments. It uses images as a shared interface, text as task specification, and a separated planner-control design.
- 2.1 Markov Decision Process: MDP-based agents struggle to generalize across environments because state spaces, reward functions, and dynamics models differ.These differences complicate shared learning and transfer across tasks.
- 2.2 Unified Predictive Decision Process: UPDP uses images as a universal interface, text descriptions instead of reward design, and a task-agnostic planner separated from environment-specific control.This abstraction is intended to support knowledge sharing and generalization.
- 2.2 Unified Predictive Decision Process: A UPDP is defined by an image observation space, textual task-description space, finite horizon H, and conditional video generator ρ(·|x0, c).The generator produces H-step image trajectories conditioned on the initial image and task description.
- 2.2 Unified Predictive Decision Process: The trajectory-conditioned policy π maps generated image sequences and task descriptions to action sequences, learned using offline experience data.The policy aligns actions with planned trajectories through an offline RL setting.
- 2.2 Unified Predictive Decision Process: Unlike MDPs, UPDP directly models video trajectories, bypasses explicit reward specification while separating environment-agnostic planning from action selection.This separation allows the planner to be reused across agents and environments.
- 2.2 Unified Predictive Decision Process: UPDP implicitly plans over an MDP and directly outputs trajectories from instructions, but requires video and text data and introduces non-Markovian modeling difficulties.Its suitability depends on the available training data; pretrained text-video models may alleviate some difficulties.
- 2.3 Diffusion Models for UPDP: Diffusion models are used to represent the conditional distribution ρ(τ|x0, c) over image-sequence trajectories.The formulation is also compatible with variational autoencoders, energy-based models, and generative adversarial networks.
- 2.3 Diffusion Models for UPDP: The diffusion generator reverses a noise-adding process with a conditional denoiser guided by the text instruction and initial image.Classifier-free guidance controls the strength of these conditioning signals during sampling.
3 Decision Making with Videos
UniPi plans behavior by generating future image frames conditioned on an observation and text instruction, then converts those frames into actions. Its design addresses constrained video planning, temporal consistency, hierarchical planning, and test-time behavioral modulation.
- 3 Decision Making with Videos: UniPi combines a diffusion video planner conditioned on the first frame and task description with a task-specific inverse-dynamics action generator.The action generator infers action sequences from generated videos.
- 3 Decision Making with Videos: Figure 2 depicts planning image sequences from an input observation and text instruction, followed by conversion of images into actions through inverse dynamics.The figure summarizes the planner-to-control pipeline.
- 3.1 Universal Video-Based Planner: Unlike unconstrained text-to-video generation, the planner must generate an executable video beginning from the observed initial configuration.Fixing the first frame during sampling is one approach to enforce this initial-state constraint.
- 3.1 Universal Video-Based Planner: Trajectory consistency is enforced by providing the observed image as additional context while denoising each intermediate noisy frame.This gives the model a signal to maintain the underlying environment state over time.
- 3.1 Universal Video-Based Planner: Hierarchical planning first generates coarse, low-dimensional plans and then refines them into plans in the underlying state and action spaces.The hierarchy addresses the exponential search-space growth of long-horizon, high-dimensional planning.
- 3.1 Universal Video-Based Planner: UniPi supports test-time plan modulation by composing a prior over trajectories, including learned image classifiers or a target-image constraint.These priors can guide generated plans toward desired states or task properties.
- 3.2 Inverse Dynamics: A small inverse-dynamics model can be trained independently of the planner on a separate, smaller simulator-generated dataset.The dataset may be suboptimal because inverse-dynamics training is decoupled from planner training.
- 3.3 Action Execution: Generated image sequences are converted into actions and executed either with closed-loop model predictive control or open-loop action execution.Closed-loop control regenerates actions after each execution step, whereas open-loop control follows the initially inferred sequence.
4 Experimental Evaluation
The experiments evaluate UniPi’s combinatorial generalization, multitask transfer, adaptable planning, and real-world video synthesis. Across these settings, UniPi generalizes to unseen language combinations and tasks, while internet-scale pretraining improves real-robot plan quality and generalization.
- Combinatorial Generalization: Each UniPi component improves task-completion accuracy on the seen Place and Relation tasks.The ablation comparison reports performance for the model’s individual components and their combinations.
- Combinatorial Generalization: UniPi generalizes to seen and novel combinations of language prompts in combinatorial Place and Relation tasks.The evaluation uses task-completion accuracy across language instructions and randomized environment instances.
- Adaptable Planning: Test-time sampling can adapt plans to color and move a particular block toward a specified geometric relation.The adaptation experiment guides sampling toward an intermediate image to impose a new constraint.
- Real-World Transfer: Pretraining on non-robot internet videos improves UniPi’s FID and FVD and marginally improves CLIP score for real-robot plan generation.Without pretraining, generated plans often fail to complete the task despite this failure being poorly reflected in CLIP score.
- Real-World Transfer: Internet-scale pretraining enables generalization to novel task commands and scenes, while task-specific robot-data training fails to generalize in the reported test split.The pretrained model is also relatively robust to black cropping and photoshopped background objects.
5 Related Work
Prior work includes world models, diffusion-based decision making, and generalist or text-conditioned agents, but UniPi uses text and images as universal interfaces for cross-environment policy learning.
- World-model approaches separate environment modeling from planning and policy learning, whereas UniPi jointly learns a world model and conducts hierarchical planning.
- Diffusion decision-making methods generate trajectories or goal images, while UniPi uses video diffusion to learn universal policies directly.
- Existing generalist agents are restricted to shared state-action spaces or require customized tokens, limiting direct use of pretrained vision-language knowledge.
- Existing text-conditioned control policies typically operate in a specific robot’s action space, leaving cross-morphology, multi-environment generalist learning unresolved.
6 Conclusion
The paper reports that text-conditioned video generation supports combinatorial generalization, multi-task learning, and real-world transfer, while identifying speed and partial observability as limitations.
- UniPi demonstrates effective combinatorial generalization, multi-task learning, and real-world transfer through text-conditioned video generation.
- Video diffusion can take a minute to generate highly photorealistic videos, although diffusion distillation initially produced a 16x speed-up.
- The evaluated environments are generally fully observed; in partially observable settings, video diffusion might hallucinate physically unfaithful objects or movements.
A.1 Video Diffusion Training Details
The appendix describes UniPi’s video-diffusion architecture, first-frame conditioning, text encoder, training resolutions, inverse dynamics model, and baseline implementations.
- UniPi uses a Video U-Net with three residual blocks, channel multiplier [2] [4], attention resolutions [6] [12] [24], and 1024-dimensional conditioning embeddings.
- Training uses T5-XXL prompts and first-frame-conditioned video diffusion models with 1.7B parameters at 10x48x64 and 20x48x64 temporal resolutions.
- The inverse dynamics model predicts the simulated robot arm’s 7-dimensional controls from image observations using convolutional layers, mean pooling, and an MLP.
- Baseline training details follow the inverse dynamics setup, while Transformer TT predicts eight future controls instead of Transformer BC’s immediate next control.
- The state-based diffusion baseline applies UniPi’s U-Net structure to future controls replicated across pixel locations rather than generated image frames.
A.4 Details of the Combinatorial Planning Task
The combinatorial planning tasks require a robot to color blocks by placing them in bowls, then move them to specified geometric relations in a placement box.
- Blocks begin white, acquire color in bowls, and must then be placed according to desired geometric relations in the final placement box.
- The simulated robot’s action space contains six joint values and a discrete contact action controlling interaction with the nearest block.
- CLIPort uses the same action space and executes actions through PyBullet’s built-in joint controller.
B.1 Additional Results on Combinatorial Generalization
Figure 10 presents additional results for UniPi’s generated videos on unseen language goals at test time, illustrating combinatorial video generation.
- Figure 10 shows UniPi-generated videos for unseen language goals at test time.
- The figure provides additional results on UniPi’s generated videos.
- These results concern combinatorial video generation.
B.2 Additional Results on Multi-Environment Transfer
Figure 11 presents additional results for UniPi’s generated video plans on different new tasks in the multitask setting.
- Figure 11 shows UniPi-generated video plans for different new tasks.
- The results are presented in a multitask setting.
- The figure provides additional results on UniPi’s generated video plans.
B.3 Additional Results on Real-World Transfer
Figure 12 presents additional results for UniPi’s high-resolution video plans across different language prompts.
- Figure 12 shows UniPi’s high-resolution video plans across different language prompts.
- The plans are evaluated across different language prompts.
- The figure provides additional results on high-fidelity plan generation.