Source-linked AI summary
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh H. McDermott, Daniel L. K. Yamins
TL;DR
Embodied AI research needs scalable training and benchmarking environments because human annotation and real-world trial-and-error are costly and risky. ThreeDWorld provides a general-purpose simulator combining high-fidelity multimodal rendering, realistic physics, procedural scene generation, and flexible agent interaction. Its example applications span visual and sound recognition transfer, physical-dynamics prediction, multi-agent interaction, and human attention, while the platform remains limited in support for some articulated objects and humanoid-agent capabilities.
Problem
Embodied AI requires perception and physical understanding, but large annotated datasets and real-world trial-and-error are laborious, expensive, slow, and potentially dangerous.
Method
TDW combines high-fidelity video and audio rendering, realistic rigid and non-rigid physics, procedural environments, customizable assets, and embodied or human-controlled agents in one flexible simulator.
Results
TDW enables applications in visual and sound recognition transfer, multimodal physical scene understanding, physical-dynamics prediction, multi-agent interaction, and human-neural attention studies.
Takeaways & Limitations
TDW provides a shared platform for studying perception, interaction, navigation, and physical inference with multimodal simulated data and interactions.
Takeaways & Limitations
Support for user-modifiable articulatable objects is currently limited, and fully articulated humanoid agents are future development directions.
Abstract
from arXiv · showhide
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable agents that embody AI agents; and support for human interactions with VR devices. TDW's API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, physical dynamics predictions, multi-agent interactions, models that learn like a child, and attention studies in humans and neural networks.
1 Introduction
ThreeDWorld addresses the cost and logistical difficulty of training embodied AI by providing a general-purpose simulator that combines high-fidelity multimodal rendering, realistic physics, and flexible agent interaction. The paper describes TDW and demonstrates applications across perception, physical prediction, multi-agent interaction, and human attention.
- Training embodied AI requires large annotated datasets and real-world trial-and-error that can be slow, expensive, and potentially dangerous.
- Virtual simulation can remove real-time and error costs while providing controllable synthetic data, including physical quantities difficult for humans to label.
- TDW is a general-purpose simulator combining high-fidelity video and audio rendering, realistic physics, and a single flexible controller for multimodal interactions.
- The paper presents applications including transferable visual representations, impact-sound classification, physical-dynamics prediction, multi-agent behavior, and attention studies involving humans and neural networks.
- TDW supports near-photorealistic indoor and outdoor scenes, physics-based sound, procedural environments, realistic rigid and non-rigid interactions, embodied agents, and user-extensible assets.
2 ThreeDWorld Platform
TDW integrates rendering, audio synthesis, physics, and interaction through a flexible simulation architecture. Its platform supports procedural scenes, multimaterial physics, multimodal audio, embodied agents, and human VR interaction.
- Design Principles and System Overview: TDW integrates advanced physics engines with high-quality visual and auditory rendering under flexible scenario and asset design principles.
- Design Principles and System Overview: The platform separates a Unity-based Build, which renders and simulates, from an external Python Controller that sends commands and receives simulation data.
- Rendering and Scene Creation: TDW creates scenes from a model library using procedural or scripted population, without restricting which models can be combined with which environments.
- High-fidelity Audio Rendering: Its audio engine supports physics-driven impact sounds, reverberation, spatialization, attenuation, occlusion, and directional cues.
- Physical Simulation: TDW combines rigid-body PhysX with NVIDIA Flex to simulate rigid, soft-body, cloth, and fluid interactions.
- Interactions and Agents: Objects can be controlled through API commands, AI-agent embodiments, or human VR interaction, while agents range from cameras and simple avatars to articulated robots.
3 Example Applications
TDW supports example applications spanning visual and auditory transfer, multimodal physical scene understanding, physical dynamics prediction, and human–agent attention studies in VR. These applications show how its simulated data, physics, and interactive environments can support learning and evaluation across perception, prediction, and social behavior.
- Visual and Sound Recognition Transfer: TDW-generated images support transfer learning to fine-grained visual recognition tasks across Aircraft, Birds, CUB, Cars, Dogs, Flowers, and Food datasets.Representations were pretrained using a TDW image dataset comparable in size to ImageNet, then fine-tuned on downstream fine-grained classification tasks.
- Visual and Sound Recognition Transfer: More than 30% better accuracy gains were achieved by models trained on TDW audio than by models trained on Sound20K for real-world material classification.The authors attribute this improvement plausibly to TDW’s more diverse range of simulated sounds.
- Visual and Sound Recognition Transfer: Audio was more diagnostic than video for material and mass classification, while audiovisual features achieved the best performance.The comparison used visual-only, audio-only, and concatenated audiovisual representations, underscoring the utility of multimodal rendering.
- Training and Testing Physical Dynamics Understanding: TDW provides a varied physical prediction benchmark containing trajectories with complex geometries, photorealistic textures, and rigid, soft-body, cloth, and fluid materials.The benchmark targets issues including object permanence, shadows, and sliding versus rolling in physical scene understanding.
- Training and Testing Physical Dynamics Understanding: DRHRN clearly outperformed HRN and DPI across all evaluated scenarios, achieving lower local MSE and physically plausible predictions without unnatural deformations.The study first replicated HRN comparisons against simpler baselines, then evaluated DRHRN against HRN and DPI.
- Social Agents and Virtual Reality: TDW’s VR integration enabled comparisons between human and socially curious neural-agent attention patterns in environments containing multiple robots and static objects.Human head movements were mapped to a TDW sensor camera, and the agents’ aggregate attentional gaze pattern was reported as quite similar to that of human adults.
4 Future Directions
TDW’s future directions target higher-level physical interaction through robotic integration and articulatable objects. Planned extensions include broader object support, humanoid agents, and higher-level API layers.
- Articulatable Objects: TDW plans to expand support for articulatable objects, including lidded containers, chests with removable drawers, and doors with functional handles.Only a small number of TDW objects currently support user modification.
- Humanoid Agents: Humanoid agent types are planned for actionable-object interaction and fine-motor tasks such as solving jigsaw puzzles.The proposed agents would use motion-capture-driven body movement and separate gesture control for hand and finger articulation.
- Robotic Systems Integration: The authors envision additional ultra-high-level API layers for specific physical interaction scenarios and a PyBullet wrapper for cross-system physics behavior replication.The wrapper would convert PyBullet API commands into comparable TDW commands.
Supplementary Material
The supplement documents TDW’s broader research scope and provides implementation details for datasets, benchmarks, models, lighting, and simulator comparisons. It also identifies current limitations in audio materials, robotic simulation, and photorealistic rendering.
- Supplementary Coverage: The supplement covers TDW image-dataset implementation, Advanced Physical Prediction Benchmark scenarios, HRN and DRHRN training, lighting, and simulator comparisons.These topics are organized across Sections B through E.
- Broader Impact: TDW is presented as a flexible platform spanning neuroscience, cognitive science, psychology, engineering, and machine learning or AI.The authors envision research on visual, auditory, and tactile sensory processing, physical inference, and scene understanding.
- Audio Research: TDW and PyImpact are envisioned for human and machine audio-perception research, although PyImpact’s diversity of audio materials is not yet adequate for the longer-term goal.The authors plan to increase the scope of audio materials significantly.
- Current Limitations: TDW does not yet adequately support robotic simulation tasks, while its image generation remains below the authors’ desired level of photorealism.The authors describe ongoing efforts to improve rendering and image generation.
B.1 TDW-image Dataset
The TDW-image dataset is generated through a two-loop image-capture process that first evaluates scene composition before producing final images. Example dataset scenes are released with the TDW package.
- Image Generation: TDW first caches camera and object positions and rotations, then replays them to generate images.Separating the loops avoids transmitting image data for frames rejected for poor composition.
- Composition Filtering: The controller uses IdPassGrayscale data to assess image composition from a segmentation-color pass reduced to one grayscale pixel.The positional loop begins with a 32 × 32 window and minimal render quality to accelerate processing.
- Dataset Examples: Figure 6 presents examples from the TDW pre-training dataset, which is to be released as part of the TDW package.The figure provides visual examples rather than a quantitative evaluation.
B.2 Advanced Physical Prediction Benchmark
The Advanced Physical Prediction Benchmark contains varied scenarios designed to test physical understanding across collisions, occlusion, motion, stability, containment, cloth, deformation, and fluids. Randomized object properties, forces, and scene configurations generate diverse physical events.
- Collisions: The benchmark includes collision scenarios ranging from randomized binary impacts to multiple objects dropped with varied starting positions and orientations.Binary collisions apply a randomized force to one toy aimed at another, while complex collisions involve multiple falling objects.
- Visibility and Permanence: Occlusion and object-permanence scenarios test visual reasoning about hidden objects using randomized models, camera arrangements, and ball trajectories.In the object-permanence scenario, a ball rolls behind an occluder and reemerges.
- Physical Reasoning: The benchmark tests lighting-dependent appearance, balance, containment, and sliding-versus-rolling through randomized physical and visual conditions.These scenarios include motion through bright and shadowed areas, stacks that fall or remain stable, objects rattling inside containers, and objects sliding or rolling on a table.
- Deformation and Fluids: Additional scenarios model bouncing, squishing, and submerging objects in fluids with varied physical properties.The fluid examples include water, chocolate, honey, oil, and glycerin; squishy objects deform and recover according to applied forces.
- Cloth Interaction: Cloth scenarios cover draping, folding, and dragging rigid objects by pulling a cloth beneath them.Cloth physics values and pulling forces are randomized.
C Training a Learnable Intuitive Physics Simulator
The simulator predicts physical dynamics from hierarchical particle graphs while adapting its environment representation to object–surface contact. Training combines motion and structure losses and recurrently exposes the model to its own predictions.
- HRN Architecture: Pairwise graph convolutions compute particle interaction effects, which hierarchical graph convolutions propagate across leaves, ancestors, siblings, and descendants before predicting future particle states.The HRN uses particle or particle-group nodes arranged hierarchically.
- Dynamic Recurrent HRN (DRHRN): DRHRN represents large environment components with sparse triangular meshes and spawns surface particles near contacting object particles.Particles are created at the small-object resolution when contact points lie within distance d.
- Dynamic Recurrent HRN (DRHRN): DRHRN deletes environment nodes when objects move away, enabling efficient dynamic-resolution representations of large TDW environments.The spawned nodes are connected to the relevant object particles and removed after contact ends.
- Training: The training loss combines L2 position-delta and pairwise-structure losses as L = αLStructure + (1 −α)LDelta.The structure term preserves pairwise distances between predicted particles within each object.
- Training: Recurrent, state-invariant training improves robustness to prediction errors accumulated when one-step physics models are unrolled.The model is trained without a hidden state because the authors characterize physical dynamics as state-free.
F.4 Benchmarks
The benchmark reports the hardware configuration used for evaluation.
- F.4 Benchmarks: Benchmarks ran on an Intel i7-7700K @4.2GHz CPU with an NVIDIA GeForce GTX 1080 GPU.These are the only benchmark implementation details supplied here.
F.5.1 Implementation Overview
TDW uses an inheritance-based command API whose JSON messages are deserialized into executable commands and whose completion can span multiple frames. Documentation is generated from command comments and examples.
- Command API: Every command subclasses Command and must implement Do(), while IsDone() reports completion for commands requiring multiple frames.The base class tracks completion with the isDone field.
- Command API: Controllers send JSON dictionaries that the build deserializes into command objects and executes through their Do() methods.An example JSON command carrying integer 15 produces console output 15.
- Command API: The inheritance-based converter allows new commands to be added without changing the rest of the API and helps isolate bugs affecting multiple commands.The API defines specialized object commands through the ObjectCommand subclass.
- Object commands: ObjectCommand targets a cached model by unique ID and delegates object-specific behavior to DoObject().ObjectLookAtPosition is an example that applies a LookAt transformation toward a target position.
- Documentation: TDW generates markdown API documentation with example JSON by scraping command summary comments.The generated documentation covers commands and their parameters.