Source-linked AI summary
AI2-THOR: An Interactive 3D Environment for Visual AI
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, Ali Farhadi
TL;DR
Visual AI research needs environments where agents can learn through interaction rather than only from still images or videos, while real-robot experimentation is costly, unsafe, and difficult to scale. AI2-THOR addresses this with a customizable, near-photo-realistic interactive simulation platform containing diverse scenes, agents, actions, and metadata. Its broad adoption and reported state-of-the-art ObjectNav transfer demonstrate its research utility across Embodied AI.
Problem
Existing vision models primarily learn from still images or videos, while real-robot experiments are costly, unsafe, mechanically constrained, and difficult to scale across varied environments.
Method
AI2-THOR provides a near-photo-realistic interactive simulation platform with diverse procedurally generated and artist-designed scenes, object interactions, agent embodiments, and environment metadata.
Results
AI2-THOR has supported experimentation in over 150 publications, while ProcTHOR pre-training alone achieved state-of-the-art ObjectNav performance on RoboTHOR, iTHOR, and ArchitecTHOR without additional training data.
Takeaways & Limitations
AI2-THOR serves as a broad, customizable platform for research spanning Embodied AI, robotics, and computer vision.
Abstract
from arXiv · showhide
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor.allenai.org. AI2-THOR consists of near photo-realistic 3D indoor scenes, where AI agents can navigate in the scenes and interact with objects to perform tasks. AI2-THOR enables research in many different domains including but not limited to deep reinforcement learning, imitation learning, learning by interaction, planning, visual question answering, unsupervised representation learning, object detection and segmentation, and learning models of cognition. The goal of AI2-THOR is to facilitate building visually intelligent models and push the research forward in this domain.
1 What is AI2-THOR?
AI2-THOR is an interactive, near-photo-realistic simulation framework designed to support visual AI research through environment interaction. It combines diverse scenes, object interactions, scalable experimentation, and a Python interface to Unity.
- Interactions: AI2-THOR supports object state changes, arm-based manipulation, and causal interactions such as opening, slicing, toasting, and filling objects.These interactions let agents perform more than passive visual recognition.
- Scenes: Procedural generation and professionally designed scenes provide substantially more interactive training environments, including 120 iTHOR rooms, 89 RoboTHOR scenes, and 10 ArchitecTHOR houses.The platform combines scalable generation with manually authored environments.
- Quality: Near photo-realistic objects and scenes are intended to improve transfer of learned models to the real world.The paper contrasts this visual complexity with games such as ATARI and GO.
- API: A Python API connects researchers to Unity 3D functionality for navigation, applying forces, object interaction, and physics modeling.The API provides a programmable interface for interacting with the simulated environment.
- Motivation: AI2-THOR provides a scalable, fast, and inexpensive proxy for real-world robot experiments across different scenarios.Simulation avoids the scalability, cost, safety, and actuator constraints of training physical robots directly.
2 What does AI2-THOR feature?
AI2-THOR combines diverse interactive scenes, embodied agents, action spaces, image modalities, and object assets in a unified environment for Embodied AI research.
- Scenes: AI2-THOR provides interactive scene datasets, including iTHOR, RoboTHOR, ProcTHOR-10K, and ArchitecTHOR, usable through the same API with different agents.iTHOR contains 120 room-sized scenes, RoboTHOR 89 dorm-sized apartments, ProcTHOR-10K 10K procedurally generated houses, and ArchitecTHOR 10 evaluation houses.
- Scenes: ProcTHOR procedurally scales training-scene number and diversity, while ArchitecTHOR provides real-world-distribution evaluation houses for testing generalization beyond procedural layouts.ArchitecTHOR contains 5 validation and 5 testing houses developed with ProcTHOR.
- Agents: AI2-THOR supports multiple embodiments, including ManipulaTHOR, StretchRE1, LoCoBot, Abstract, and Drone agents, each with distinct executable actions.All agents navigate and perform environment queries and state changes; ManipulaTHOR and StretchRE1 additionally support arm-based grasping and opening.
- Actions: Its actions span navigation, interactive manipulation and state changes, environment queries, and environment modifications such as material or lighting randomization.Interactive behavior includes abstracted actions, arm-based manipulation, and causal effects between objects.
- Image Modalities: Each camera can render RGB, depth, semantic segmentation, instance segmentation, and surface normals, with additional cameras and modalities supported.More modalities can be added by modifying the Unity back-end, often through shaders.
- Objects: The object database contains 3,578 hand-modeled interactive objects designed to support actions and state changes such as opening, breaking, and cooking.Examples span alarm clocks, side tables, plants, and chairs.
3 What has AI2-THOR been used for?
AI2-THOR has supported a broad range of embodied AI, vision-and-language, navigation, interaction, and computer-vision research. Its uses include procedural scene synthesis, affordance learning, sim2real transfer, interpretability, and multi-agent coordination.
- Since 2017, AI2-THOR has supported experimentation in over 150 publications and has been downloaded over 500k times.
- Visual Navigation: Visual navigation includes ImageNav, which targets an image, and ObjectNav, which targets a semantic category such as a bed.
- Vision-and-Language: AI2-THOR supports embodied vision-and-language datasets for natural-language instruction following, human-robot dialogue, and interactive question answering.
- Additional applications include learning object affordances and functional relationships between objects to guide downstream tasks and activities.
- Scene Synthesis: ProcTHOR procedurally generated 10K houses for training, and pre-training on ProcTHOR alone achieved state-of-the-art ObjectNav performance on RoboTHOR, iTHOR, and ArchitecTHOR.
- Research uses AI2-THOR for interaction-based learning, computer vision, interpretability, human-robot interaction, sim2real transfer, and multi-agent coordination.
4 Why use AI2-THOR?
AI2-THOR is designed as a large-scale interactive simulator with first-class interaction support and convenient extensibility. Its benchmarked training throughput is lower than Habitat 1.0 in the reported setup.
- AI2-THOR provides a comparison framework for embodied AI simulators, as summarized in Table 1.
- AI2-THOR is significantly larger in scale than other simulators while providing first-class support for interaction and easy addition of new capabilities through Unity.
- 167.7 average training FPS was measured for AI2-THOR, compared with 230.5 average training FPS for Habitat 1.0 under the same reported setup.
5 Conclusion
AI2-THOR is a large-scale, customizable interactive simulation platform for Embodied AI that supports diverse research tasks and continues to evolve.
- AI2-THOR is a large-scale interactive simulation platform for Embodied AI, used across over 150 publications and a wide variety of tasks and research areas.
- The platform supports diverse scenes, agent embodiments, actions, and metadata, while its capabilities continue to evolve.
- AI2-THOR is highly customizable and is intended to support new improvements and use cases.
A Contributions
The AI2-THOR project combined engineering, simulation, manipulation, infrastructure, documentation, and project-management contributions. Contributors developed the platform’s backend, agents, scenes, APIs, and related frameworks.
- Eric Kolve built the Python–Unity API, optimized runtime performance, added headless rendering, and contributed to RoboTHOR, ProcTHOR, and ManipulaTHOR.
- Winson Han developed Unity backend functionality for physics-based interactions, state changes, visibility, repositioning, and annotations.
- Eli VanderBilt built the 3D scenes, created thousands of interactive assets, modeled agents, and implemented arm-based manipulation.
- Luca Weihs contributed frontend and backend actions, tests, processes, and led development of the AllenAct training framework.
- Alvaro Herrasti developed Unity and Python API infrastructure, graphics and shaders, WebGL integration, continuous-action physics, and ProcTHOR functionality.
- Matt Deitke led ProcTHOR, built the website and demo, wrote documentation, and contributed to RoboTHOR, the backend, and accessibility infrastructure.
B Performance Comparison
Comparing embodied-AI simulators is difficult because agent capabilities, parallel scalability, and reinforcement-learning bottlenecks complicate raw speed comparisons. The profiling experiments control several of these factors by matching hardware, agents, policies, actions, and training settings.
- Comparison challenges: Simulator comparisons are difficult because supported agents differ in action spaces and capabilities.AI2-THOR includes agents with different embodiment complexity, making simulation costs agent-dependent.
- Comparison challenges: Random action sampling can underestimate costs of rare but important interactions, such as arm movements among many objects.These interactions may dominate computation for well-trained agents despite being uncommon under random sampling.
- Comparison challenges: Single-process simulation speed can misrepresent scalability when simulators parallelize differently across processes on one GPU.Therefore, raw single-process throughput does not capture ease of scaling.
- Comparison challenges: Reinforcement-learning training speed is also limited by model passes and environment resets, not simulator stepping alone.For AI2-THOR, Habitat, and iGibson, changing scenes can be orders of magnitude more expensive than taking one agent step, creating synchronization costs.
- Controlled profiling: The profiling setup matches a 2-GPU machine, LoCoBot agents, action spaces, actor-critic networks, rollout lengths, and training hyperparameters across environments.Agents are trained on ObjectNav, always take 500 steps by removing the End action, and use 128-step rollouts.