Source-linked AI summary
VirtualHome: Simulating Household Activities via Programs
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, Antonio Torralba
TL;DR
Household robots need executable representations of complex activities, but no database provides programs describing multistep home tasks. The paper crowdsources such programs, builds the VirtualHome simulator, and learns to generate programs from language or video. VirtualHome then supports synthetic activity videos with rich ground truth and agent execution, while the simulator currently supports only the 12 most frequent actions.
Problem
Household activities require executable multistep programs, but a database describing such activities for robots was missing.
Method
The paper crowdsources household descriptions and programs, implements frequent atomic interactions in a Unity3D simulator, and trains program-generation models from language or video.
Results
VirtualHome creates a large activity-video dataset with rich ground truth and supports agents driven by programs inferred from textual descriptions or video demonstrations.
Takeaways & Limitations
VirtualHome provides a shared playground for vision and robotics in which agents can use language or visual demonstrations to execute novel household activities.
Takeaways & Limitations
The current simulator supports only the 12 most frequent actions, rather than the full variability of household actions and objects.
Abstract
from arXiv · showhide
In this paper, we are interested in modeling complex activities that occur in a typical household. We propose to use programs, i.e., sequences of atomic actions and interactions, as a high level representation of complex tasks. Programs are interesting because they provide a non-ambiguous representation of a task, and allow agents to execute them. However, nowadays, there is no database providing this type of information. Towards this goal, we first crowd-source programs for a variety of activities that happen in people's homes, via a game-like interface used for teaching kids how to code. Using the collected dataset, we show how we can learn to extract programs directly from natural language descriptions or from videos. We then implement the most common atomic (inter)actions in the Unity3D game engine, and use our programs to "drive" an artificial agent to execute tasks in a simulated household environment. Our VirtualHome simulator allows us to create a large activity video dataset with rich ground-truth, enabling training and testing of video understanding models. We further showcase examples of our agent performing tasks in our VirtualHome based on language descriptions.
1. Introduction
VirtualHome represents household activities as executable programs and builds the data, simulator, and generation methods needed to connect language or video demonstrations to agent execution.
- Motivation: Programs represent complex household tasks as sequences of atomic actions and object interactions that can drive agents toward goals.Examples include actions such as “sit” and interactions such as picking up an object.
- Program generation: The paper proposes automatic program generation from both natural-language descriptions and video demonstrations, enabling language- or video-driven agent execution.This is intended to let nonexpert users teach robots a wide variety of novel tasks.
- Data and representation: The project crowdsources household instructions and translates them into programs through a Scratch-inspired interface.The programs include steps required for task completion even when those steps are omitted from natural-language descriptions.
- Simulation: VirtualHome implements frequent atomic interactions in Unity3D and uses physics, navigation, and kinematic models to execute programs in a simulated household.The simulator supports an artificial agent performing tasks defined by the collected programs.
- Dataset and contribution: VirtualHome creates a large activity-video dataset with rich ground truth for training and testing video-understanding models.Programs drive the agent in a synthetic world to produce the videos.
2. Related Work
Related work represents activities as action plans, generates code from language, teaches robots through instructions, and uses simulation for autonomous systems. VirtualHome differs by targeting broad household activities with object interactions and human poses.
- Actions as programs: Prior work generated action plans from cooking videos or recipes and represented activities as sequences of atomic steps.These approaches included probabilistic grammars, action-plan treebanks, and Markov random fields.
- Actions as programs: Crowdsourced household scripts provide useful natural-language descriptions but are difficult to convert into executable robot programs.The scripts often contain only one or two sentences describing short action sequences.
- Code generation: Code-generation research has produced programs from natural-language queries, conditions, game rules, and visual questions.Examples include retrieving code snippets, generating If-This-Then-That code, and inferring programs for visual question answering.
- Robotics: Most instruction-following robotics work translates relatively simple navigational instructions into small sequences of robotic actions.VirtualHome additionally addresses object interactions and more complex everyday activities.
- Simulation: Existing simulations support autonomous driving, flying, navigation, question answering, or limited object interaction, whereas VirtualHome simulates broad household actions and human poses.The paper states that it is not aware of another simulator with comparable household-scale coverage of objects and actions.
3. KB of Household Activities for Robots
The paper builds a robot-oriented repository of household activities as complete symbolic programs, collected from natural-language descriptions and organized into executable action sequences. The resulting dataset contains diverse activities, objects, actions, and simulated video-ground truth, while completeness remains imperfect.
- Motivation: Programs specify all steps needed for robots to perform household activities, including commonsense actions omitted from human-directed descriptions.They provide a clear, non-ambiguous representation of complex tasks as sequences of simpler actions.
- Dataset Analysis: The simulator-generated activity videos include randomized household configurations and rich ground truth such as atomic-action timestamps, poses, segmentation, depth, and optical flow.Programs are animated after varying homes, agents, cameras, object placements, initial agent locations, action speeds, and interaction objects.
- Data Collection: Workers first described activities from eight household scenes, then translated descriptions into programs using a Scratch-based graphical language.The interface used predefined action blocks and required annotators to compose programs as sequences of steps.
- Program Representation: Each program step combines an action with object arguments and instance identifiers, allowing different objects of the same class to be disambiguated.For example, repeated references to the television can point to the same object instance across multiple steps.
- Dataset Analysis: The ActivityPrograms dataset contains 2821 programs covering 75 atomic actions, 308 objects, and 2709 unique steps.Descriptions average 3.2 sentences and 21.9 words, while programs average 11.6 steps.
- Dataset Analysis: 64% of sampled programs were complete, 28% missed minor steps, and 8% missed crucial steps.Completeness was assessed by five workers rating 100 collected programs.
4. VirtualHome: Simulator of Household Tasks
VirtualHome uses programs to drive agents through simulated household tasks, while generating diverse environments and richly annotated activity videos. The simulator supports a subset of frequent actions and prepares scenes by assigning program-referenced objects to plausible locations.
- Simulator purpose: VirtualHome uses programs to drive characters in a simulated 3D household environment, providing a playground for teaching artificial agents tasks.The simulator focuses on building the environment and leaves learning inside it to future work.
- Generated data: The simulator generates dense ground truth including semantic segmentation, depth, pose, and related video annotations.Automatically generated annotations also include step timestamps, object segmentation, optical flow, and camera parameters.
- Program execution: Programs are executed by mapping their referenced objects to simulator instances and computing feasible interaction positions and animation information.The mapping accounts for all program steps so related interactions use coherent object assignments, such as typing beside the selected computer.
- Supported actions: The current simulator implements the 12 most frequent atomic actions, including navigation, object manipulation, switching, opening, placing, looking, sitting, standing, and touching.Action animations depend on the object involved, and navigation uses Unity’s NavMesh framework to avoid obstacles.
- Scene preparation: Missing objects are added to a household before execution using a knowledge base of plausible supporting locations.Annotators specify other objects, floors, or walls that can support each object class.
- VirtualHome Activity dataset: 5,193 synthesized programs were described by human annotators, animated in the simulator, and used to automatically generate training and evaluation ground truth.The resulting VirtualHome Activity dataset uses programs restricted to the 12 currently supported actions.
5. From Videos and Descriptions to Programs
The paper treats program generation from language descriptions and videos as a translation task using an encoder–decoder model. It combines sequence similarity with simulator executability during reinforcement-learning training.
- Task and model: The authors introduce program generation from either natural-language descriptions or video demonstrations as a translation task.They adapt a sequence-to-sequence model and train it with reinforcement learning using simulator-derived rewards.
- Task and model: An RNN encoder–decoder represents the input sequence and generates one program step at a time.The encoder uses 100-dimensional LSTM hidden states, while the decoder predicts instruction probabilities from action and object embeddings.
- Learning and inference: Training uses two phases: cross-entropy pretraining followed by policy-gradient reinforcement learning.During pretraining, the ground-truth step is fed to the next decoder time instance.
- Learning and inference: The reinforcement-learning reward combines normalized longest-common-subsequence similarity with a binary simulator-executability reward: r(ws, g) = rLCS(ws, g) + 0.1 · rsim(ws).The first term encourages semantic agreement with the ground-truth program, while the second rewards executable programs.
- Input encoders: For video inputs, 2-second clips are converted into predicted action-and-object instruction embeddings that condition program generation.DilatedNet provides semantic segmentation, and a Temporal Relation Network uses four-frame relations.
6. Experiments
The experiments evaluate program prediction from text and video, action and instruction classification, and whether generated programs execute plausibly in VirtualHome. Reinforcement learning improves program quality, but balancing sequence similarity with executability remains important.
- Experimental setup: The experiments use ActivityPrograms and VirtualHome Activity, the latter providing synthesized programs, natural descriptions, and videos.
- Program generation: Program induction is evaluated with longest-common-subsequence accuracy, alongside separate action and object accuracies.The LCS-based score preserves matching-step order while allowing gaps and normalizes by the longer program.
- Program generation: RL with an LCS reward outperforms MLE on both LCS and executability metrics.
- Program generation: RL with both LCS and simulator rewards slightly lowers LCS relative to LCS-only training but significantly improves executability.
- Video-based prediction: Video-based program generation is the most challenging evaluated task, and LCS-only RL improves accuracy while decreasing executability.The paper attributes this trade-off partly to MLE producing shorter programs, which are more likely to be executable.
- Executing programs in VirtualHome: Human scores generally agree with the program-based metrics, although imperfect animation produces examples with high metric performance but lower human scores.The authors identify simulator improvement as a remaining need.
7. Conclusion
The paper concludes by presenting a household-activity knowledge base, a symbolic program representation, and VirtualHome for simulating and evaluating activities. It also demonstrates program inference from text or video to drive agents, while identifying reinforcement-learning-based task performance as future work.
- The dataset combines natural-language activity descriptions with symbolic programs containing all steps necessary to perform an activity.
- VirtualHome is a 3D household simulator used to create a large video activity dataset with rich ground-truth.
- A model infers programs from video or textual descriptions, allowing naive users to drive robots through language or video demonstrations.
- The paper identifies training agents to perform tasks from visual observation alone using reinforcement learning as a future direction.