PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Interactive Real-World Simulators

Sherry Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans, Pieter Abbeel

arXiv:2310.06114v3cs.AI

TL;DR

Real-world simulators must unify datasets containing different objects, actions, and movements to model interactive experience. UniSim orchestrates diverse data into an action-conditioned video simulator, enabling action-rich and long-horizon interactions, real-robot policy deployment, and improved video captioning from simulated data.

  • Problem

    Learning a real-world simulator is difficult because available datasets cover different information, actions, and environments that must be brought together.

  • Method

    UniSim unifies heterogeneous actions and videos, then trains an action-conditioned video generation model to predict observations through video diffusion.

  • Results

    UniSim simulates action-rich and eight-step long-horizon interactions, supports effective real-robot vision-language actions, and improves video captioning using generated data.

  • Takeaways & Limitations

    A unified visual action interface can support interactive simulation, simulation-trained embodied policies, and generated-data training for other vision-language tasks.

  • Takeaways & Limitations

    UniSim can hallucinate unrealistic outcomes when given actions that are impossible for the scene or robot.

Abstract

from arXiv · show

Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative models is to simulate realistic experience in response to actions taken by humans, robots, and other interactive agents. Applications of a real-world simulator range from controllable content creation in games and movies, to training embodied agents purely in simulation that can be directly deployed in the real world. We explore the possibility of learning a universal simulator (UniSim) of real-world interaction through generative modeling. We first make the important observation that natural datasets available for learning a real-world simulator are often rich along different dimensions (e.g., abundant objects in image data, densely sampled actions in robotics data, and diverse movements in navigation data). With careful orchestration of diverse datasets, each providing a different aspect of the overall experience, we can simulate the visual outcome of both high-level instructions such as "open the drawer" and low-level controls from otherwise static scenes and objects. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies, each of which can be deployed in the real world in zero shot after training purely in simulation. We also show that other types of intelligence such as video captioning models can benefit from training with simulated experience, opening up even wider applications. Video demos can be found at https://universal-simulator.github.io.

1 INTRODUCTION

UniSim aims to learn a universal simulator that generates visual consequences of diverse real-world actions, enabling interactive experiences for humans and robots. It unifies broad datasets through an action-and-video interface while demonstrating several applications.

  • Motivation: UniSim targets simulation of visual effects from diverse actions, enabling human interaction with scenes and objects and risk-free robot learning in simulation.The motivating examples include driving cars and preparing furniture or meals.
  • Scope: “Universal” means simulating through a unified interface of actions and videos, rather than simulating everything.Sound is explicitly not simulated.
  • Data and modeling: The simulator learns from broad data covering objects, scenes, human activities, navigation and manipulation motions, panorama scans, simulations, and renderings.It uses an autoregressive prediction model to support consistent simulation across video-generation boundaries.
  • Applications: The paper demonstrates three specific use cases for the real-world simulator.The supplied passage introduces these applications but does not specify them further.

2 LEARNING AN INTERACTIVE REAL-WORLD SIMULATOR

This section defines UniSim as a universal simulator that predicts video consequences of actions from observed history, despite heterogeneous action formats, environments, and frame rates. It unifies these inputs through continuous action embeddings and trains a video diffusion model for next-observation prediction.

  • Simulator formulation: UniSim predicts the visual consequence of an action as a video, given a world state such as an image frame.The simulator is framed as an observation prediction model that maps history and action to the next observation.
  • Data unification: Different action modalities and video frame rates make learning difficult, so UniSim processes them into common representations.Inputs can include language instructions, robot controls, and camera movements, while videos are converted into a shared temporal format.
  • Simulator formulation: UniSim formulates interaction as learning p(o_t|h_{t−1}, a_{t−1}) to predict the next video frames from history and a temporally extended action.Actions may resolve into sequences of low-level robot commands, and the predicted observation comprises the next set of captured frames.
  • Diffusion model: The model generates observations from initial noise through iterative denoising, using classifier-free guidance to condition predictions on actions.In practice, previous video frames serve as history conditioning, while action conditioning is controlled through the guidance formulation.
  • Architecture and training: UniSim uses a video U-Net with interleaved temporal and spatial attention and convolution layers across downsampling and upsampling passes.Conditioning frames are replicated at all future frame indices and concatenated with the model inputs.

3 SIMULATING REAL-WORLD INTERACTIONS

UniSim simulates action-rich manipulation and navigation from natural-language instructions, supports temporally consistent long-horizon interactions, and produces diverse stochastic environment transitions. Ablations show that conditioning on multiple recent frames improves simulation, while language-plus-dataset identifiers enable action completion in low-data domains.

  • Action-Rich Simulation: UniSim simulates kitchen manipulation, switch pressing, and navigation from natural-language actions starting from the same initial observation.The model was trained on generic internet data without action-rich manipulation data such as EPIC-KITCHENS.
  • Long-Horizon Simulations: UniSim autoregressively simulates 8 interactions while maintaining temporal consistency in object identities and locations.Examples preserve a can on the counter and an orange in a drawer across sequential frames.
  • Simulating Low-Data Domains: With only 700 Habitat HM3D training examples, prefixing language actions with a dataset identifier produces video samples that complete the specified action.This demonstrates simulation in a low-data domain during joint training.
  • Diversity and Stochasticity in the Simulator: UniSim generates diverse and stochastic transitions, including varied revealed objects, object colors and locations, and camera angles.Diffusion-model flexibility supports stochastic environments whose outcomes cannot be fully controlled by actions.
  • Frame Conditioning Ablations: Conditioning on 4 frames outperforms a single frame, whereas exponentially distant history can hurt performance and additional frames provide no further Ego4D improvement.Distant history may still help applications requiring long-term memory, such as navigation for retrieval.

4 APPLICATIONS OF UNISIM

UniSim supports training vision-language policies, reinforcement-learning agents, and vision-language models by simulating realistic experiences. These applications include long-horizon action learning, zero-shot real-robot deployment, and video captioning from generated data.

  • Vision-language policies: UniSim generates training data for vision-language policies, addressing the large language-action labeling burden that grows with task horizon and complexity.The simulator creates long-horizon trajectories by combining multiple scripted-language rollouts and hindsight-style relabeling.
  • Vision-language policies: A VLM trained only on simulated data produces effective high-level language actions from initial and goal images in the real Language Table domain.The simulator generates video trajectories from real initial observations for training and evaluation.
  • Reinforcement learning: UniSim enables RL training by providing a realistic simulator that can be accessed in parallel, overcoming a major barrier to real-world RL applications.The method treats simulator rollouts as on-policy real-environment rollouts and uses a learned reward model to predict rewards.
  • Reinforcement learning: An RL policy trained in simulation can execute zero-shot on the real Language Table task and successfully move a blue cube to a green circle.UniSim supports low-level endpoint controls, including horizontal, vertical, and diagonal movements.
  • Vision-language models: UniSim also supplies purely generated training data for vision-language models, providing significant performance benefits in video captioning when natural data is rare or difficult to collect.The proof of concept fine-tunes PaLI-X on data generated by UniSim.

5 RELATED WORK

Prior generative models excel at language and media synthesis but generally do not support physical tasks requiring perception, control, and multi-turn interaction. UniSim addresses limited real-world training access by learning a universal simulator that provides realistic, unlimited environments for agent training and benefits video captioning.

  • Video Captioning: PaLI-X finetuned only on simulated UniSim data achieves a large CIDEr advantage over no finetuning and transfers better than finetuning on true ActivityNet Captions video data.
  • Internet-Scale Generative Models: Internet-scale language and media models primarily support text and generative-media tasks rather than physical tasks requiring perception and control.
  • Internet-Scale Generative Models: Existing video-generation policies face a major bottleneck from limited access to real-world environments for policy learning.
  • Internet-Scale Generative Models: UniSim learns a universal real-world simulator to provide realistic and unlimited environment access for training sophisticated agents.

6 LIMITATIONS AND CONCLUSION … A.1 ADDITIONAL LONG-HORIZON INTERACTION

UniSim demonstrates learned real-world simulation from text instructions to robot controls, enabling visually realistic interaction and autonomous-agent training. The paper identifies hallucination as a limitation and presents additional long-horizon interaction results for humans and robots.

  • 6 LIMITATIONS AND CONCLUSION: UniSim learns a real-world simulator that responds to action inputs ranging from text instructions to robot controls.
  • 6 LIMITATIONS AND CONCLUSION: The simulator generates visually realistic experiences for human interaction and autonomous-agent training.
  • 6 LIMITATIONS AND CONCLUSION: Hallucination is identified as a limitation requiring future work when actions are unrealistic given the scene.
  • Appendix: The appendix reports additional qualitative results on long-horizon simulation of human and robot interactions.
  • A ADDITIONAL RESULTS: The appendix also covers long-horizon vision-language policies, low-level reinforcement-learning policies, UniSim’s dataset, architecture, training, and application setups.
  • A.1 ADDITIONAL LONG-HORIZON INTERACTION: UniSim generates consistent video rollouts across 3-4 high-level language actions in additional long-horizon human and robot interactions.

A.2 ADDITIONAL REAL-ROBOT RESULTS FOR LONG-HORIZON LANGUAGE POLICY · A.3 ADDITIONAL RESULTS ON LEARNING RL POLICY IN UNISIM

UniSim supports vision-language policies for long-horizon block rearrangement and enables reinforcement-learning policies trained in simulation to execute in the real world zero-shot. The additional results show both simulated rollouts and real-world execution, including reward-model progress over a 70-step run.

  • A.2 ADDITIONAL REAL-ROBOT RESULTS FOR LONG-HORIZON LANGUAGE POLICY: A vision-language policy finetuned with hindsight-labeled data generates long-horizon instructions for rearranging blocks.The policy is evaluated on tasks involving two or three blocks.
  • A.2 ADDITIONAL REAL-ROBOT RESULTS FOR LONG-HORIZON LANGUAGE POLICY: The policy successfully moves two or three blocks to match their locations in a goal image.This result extends the long-horizon language-policy evaluation to multi-block spatial goals.
  • A.3 ADDITIONAL RESULTS ON LEARNING RL POLICY IN UNISIM: The RL policy trained in UniSim is visualized through its first real observations and last simulated observations during rollout.Figure 11 compares observations across the transition from real inputs to simulated rollout outcomes.
  • A.3 ADDITIONAL RESULTS ON LEARNING RL POLICY IN UNISIM: The UniSim-trained RL policy is executed in the real world zero-shot, with first and last real observations documenting the run.The policy is trained from UniSim rather than from real-world interaction for this evaluation.
  • A.3 ADDITIONAL RESULTS ON LEARNING RL POLICY IN UNISIM: The real-world execution includes a learned reward model that outputs steps-to-completion throughout policy execution.The middle plot reports the reward-model output alongside the initial and final observations.
  • A.3 ADDITIONAL RESULTS ON LEARNING RL POLICY IN UNISIM: Step 0 corresponds to the initial observation, while step 70 corresponds to the final observation in the real-world rollout.These endpoints define the temporal span used to present reward-model progress during execution.

B DATASETS

This section describes the datasets used to train UniSim and the metadata reported for each dataset. It also notes preprocessing choices, including dataset splitting and T5 language-model embeddings for available text.

  • Dataset composition: Miscellaneous data comprise collections of datasets that have not been published.
  • Preprocessing: Some datasets are processed into training and validation splits, so their training-example counts may differ from original dataset sizes.
  • Preprocessing: When original datasets include text, UniSim preprocessing uses embeddings from the T5 language model.The passage attributes this embedding method to Raffel et al. (2020).
  • Dataset composition: The UniSim training data are documented by dataset name, approximate training-example count, and mixture weight.These fields are provided for each dataset in the accompanying table.

C ARCHITECTURE AND TRAINING · D DETAILS OF EXPERIMENTAL SETUPS · D.1 DETAILS OF LEARNING LONG-HORIZON POLICY

UniSim uses a history-conditioned 3D U-Net video predictor with two spatial super-resolution models, while long-horizon policy learning draws on Language Table trajectories containing instructions, controls, and image frames. The Language Table data combines 160k simulated and 440k real trajectories, whose original tasks have short horizons.

  • C ARCHITECTURE AND TRAINING: UniSim’s video model is parametrized by a 3D U-Net architecture.The architecture uses spatial downsampling and upsampling passes with skip connections, interleaved 3D convolution and attention layers.
  • C ARCHITECTURE AND TRAINING: The video system includes one history-conditioned base video prediction model and two additional spatial super-resolution models.The super-resolution models are described as similar to Ho et al. (2022).
  • D.1 DETAILS OF LEARNING LONG-HORIZON POLICY: Language Table contains 160k simulated trajectories and 440k real trajectories.Each trajectory includes a language instruction, visuomotor controls, and image frames corresponding to task execution.
  • D.1 DETAILS OF LEARNING LONG-HORIZON POLICY: Each Language Table trajectory pairs a language instruction with visuomotor controls and image frames.Example instructions include “move blue cube to the right.”
  • D.1 DETAILS OF LEARNING LONG-HORIZON POLICY: The original Language Table trajectories have short horizons, such as moving one block.The dataset description motivates learning policies capable of longer-horizon behavior beyond these short tasks.
  • D.1 DETAILS OF LEARNING LONG-HORIZON POLICY: The Language Table environment records visuomotor control sequences alongside the corresponding visual execution frames.This provides synchronized action and observation sequences for policy learning.

D.2 DETAILS OF RL POLICY TRAINING · D.3 DETAILS OF VIDEO CAPTIONING · E ADDITIONAL ABLATIONS

The paper details a two-stage RL policy-training pipeline built around PaLI 3B and a video-generation environment, and describes text-only UniSim conditioning for generating ActivityNet Captions data to train vision-language models.

  • D.2 DETAILS OF RL POLICY TRAINING: PaLI 3B uses a ViT G/14 image encoder and UL2 encoder-decoder to process observations, instructions, and tokenized controls.The model can decode language, control actions, or other values of interest.
  • D.2 DETAILS OF RL POLICY TRAINING: Behavioral cloning discretizes continuous Language-Table actions into token sequences such as “+1 -5” for supervised prediction.The action is represented using extra tokens from PaLI’s vocabulary.
  • D.2 DETAILS OF RL POLICY TRAINING: Supervised finetuning runs for 300k gradient steps with 1k warmup steps, learning rate 0.001, dropout 0.1, and batch size 128.Training uses both simulated and real Language-Table data, following a setup similar to RT-2.
  • D.2 DETAILS OF RL POLICY TRAINING: The RL environment exposes video-generation inference through RPC wrapped by the DM Env API, with randomly sampled demonstration goals at episode reset.The RL policy is initialized from the Stage 1 checkpoint.
  • D.2 DETAILS OF RL POLICY TRAINING: Actors are rate limited to keep generated trajectories from becoming substantially off-policy, and RL-training hyperparameters are reported in Table 7.The cited passage identifies the trajectory-control mechanism but does not provide Table 7’s values.
  • D.3 DETAILS OF VIDEO CAPTIONING: UniSim supports text-only conditioning by supplying placeholder white frames and increasing classifier-free guidance strength on text.This approach generated videos purely from ActivityNet Captions and used 30,740 training-split text-video examples for VLM data generation.

E.1 ABLATIONS OF DATASETS · E.2 ABLATIONS OF MODEL SIZE

Dataset ablations show that combining internet, human-activity, and robot data yields the strongest UniSim performance, while removing internet data substantially worsens FVD. Model-size ablations show larger models improve video modeling and achieve the best scores at the largest size, but FVD gains plateau as models grow.

  • E.1 ABLATIONS OF DATASETS: UniSim dataset ablations evaluate FVD and CLIP scores on 1024 held-out test samples.The evaluation uses the test split and computes both metrics over 1024 samples.
  • E.1 ABLATIONS OF DATASETS: Including internet data and diverse human-activity and robot data produces the best FVD and CLIP scores.The ablation results identify the combined dataset mixture as the strongest configuration.
  • E.1 ABLATIONS OF DATASETS: Removing internet data significantly worsens FVD, highlighting internet data’s importance to UniSim.This finding comes from comparing dataset variants in the ablation study.
  • E.2 ABLATIONS OF MODEL SIZE: Model-size ablations evaluate FVD and CLIP scores on 1024 held-out test samples.The study measures video-modeling performance on the held-out test split.
  • E.2 ABLATIONS OF MODEL SIZE: FVD improvement plateaus as model size increases, limiting the gains from scaling despite overall performance improvements.The paper characterizes this plateau as slightly disappointing from a scaling perspective.
  • E.2 ABLATIONS OF MODEL SIZE: Increasing model size improves video modeling performance, with the largest model achieving the best FVD and CLIP scores.The table-level result favors the largest evaluated model on both reported metrics.

F FAILED SIMULATIONS WITHOUT JOINT TRAINING

Joint training across broad internet, manipulation, and other datasets is important for reliable long-horizon environment simulation. Without this data integration, simulations fail on actions such as “uncover bottle” and can fail half the time.

  • Failed simulations without joint training: Training only on SSV2 produces failed environment simulations for the action “uncover bottle.”These failures are shown in the top two videos of Figure 13.
  • Failed simulations without joint training: Training only on generic internet data, excluding SSV2, EpicKitchen, Ego4D, and various robotics datasets, also produces failed “uncover bottle” simulations.These failures are shown in the bottom two videos of Figure 13.
  • Failed simulations without joint training: Half of long-horizon interaction simulations fail when UniSim’s text-to-video model is trained only on Brohan et al. (2022) data instead of broad and manipulation datasets.The failures are marked in red text in Figure 14.
Loading 2310.06114v3…