Source-linked AI summary
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, Ali Farhadi
TL;DR
Embodied AI needs stronger evidence about whether models trained in simulation generalize to reality, a question made difficult by interaction, alignment, physical replication, and cost. RoboTHOR provides an open platform pairing simulated scenes with physical counterparts for training, benchmarking, and remote robot testing. Its initial experiments show a significant simulation-to-real performance drop and identify visual and control discrepancies as transfer challenges.
Problem
Simulation-trained embodied models have uncertain real-world generalization, while physical robots, environments, and experiments are costly and difficult to standardize.
Method
RoboTHOR pairs diverse simulated environments with carefully matched physical counterparts and provides an interface for deploying models on real robots.
Results
Models trained in simulation show a significant performance drop when transferred to the real world, with feature-space and control discrepancies affecting transfer.
Takeaways & Limitations
RoboTHOR offers an open, modular, and remotely accessible benchmark for systematically studying simulation-to-real transfer in embodied AI.
Takeaways & Limitations
Real-world evaluation is constrained by slow robot motion, with episodes lasting as long as 10 minutes and limiting testing volume.
Abstract
from arXiv · showhide
Visual recognition ecosystems (e.g. ImageNet, Pascal, COCO) have undeniably played a prevailing role in the evolution of modern computer vision. We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems. Recently, various synthetic environments have been introduced to facilitate research in embodied AI. Notwithstanding this progress, the crucial question of how well models trained in simulation generalize to reality has remained largely unanswered. The creation of a comparable ecosystem for simulation-to-real embodied AI presents many challenges: (1) the inherently interactive nature of the problem, (2) the need for tight alignments between real and simulated worlds, (3) the difficulty of replicating physical conditions for repeatable experiments, (4) and the associated cost. In this paper, we introduce RoboTHOR to democratize research in interactive and embodied visual AI. RoboTHOR offers a framework of simulated environments paired with physical counterparts to systematically explore and overcome the challenges of simulation-to-real transfer, and a platform where researchers across the globe can remotely test their embodied models in the physical world. As a first benchmark, our experiments show there exists a significant gap between the performance of models trained in simulation when they are tested in both simulations and their carefully constructed physical analogs. We hope that RoboTHOR will spur the next stage of evolution in embodied computer vision. RoboTHOR can be accessed at the following link: https://ai2thor.allenai.org/robothor
1. Introduction
Embodied AI lacks the open, standardized, and affordable ecosystem that accelerated progress in computer vision, while simulation-to-real transfer remains insufficiently understood. RoboTHOR addresses this gap with paired simulated and physical environments, remote access, and benchmarks focused on real-world transfer.
- Embodied AI remains challenging despite major advances in computer vision and natural language processing.
- High physical costs, limited standardized benchmarks, and data- and time-inefficient learning algorithms hinder progress in real-world embodied AI.
- Synthetic environments lower research barriers and support progress on embodied tasks, but how well simulation-trained models transfer to reality remains an open question.
- RoboTHOR pairs agents in simulated environments with robotic counterparts in the physical world as an open and accessible benchmarking platform.
- RoboTHOR uses diverse simulated scenes with corresponding physical counterparts, including modular assets and reconfigurable layouts that support scalable testing.The platform includes 731 unique assets, and physical scenes can be reconfigured in roughly 30 minutes.
- The platform is designed to broaden participation through open-source resources, remote no-cost deployment, replicable physical environments, and transfer-focused challenges.The physical environment costs roughly $10,000 in materials and assets to reproduce.
- Initial experiments identify a disparity between simulation and reality, including feature-space differences, real-world control variation, and no improvement from off-the-shelf image correction.
- These findings show that generalizing from simulated environments to the real world remains a daunting challenge while motivating further research.
2. Related Work
Prior embodied-AI research has benefited from inexpensive synthetic environments that abstract away low-level physical complexity, but real-world transfer remains insufficiently established. RoboTHOR extends this line of work by pairing simulation with physical robots and focusing on indoor semantic navigation and sim-to-real transfer.
- Embodied AI Environments: Synthetic frameworks support visual navigation, task completion, and question answering by abstracting control, manipulation, and appearance noise.
- Robotics research platforms: Robotics platforms have historically been expensive, although low-cost robots have broadened access to research.
- Robotics research platforms: Simulator–robot combinations have traditionally addressed tasks such as robotic-arm manipulation and autonomous driving.
- Visual Navigation: This paper focuses on indoor semantic navigation, directing an agent toward an instance of a specified object category.
- Visual Navigation: Existing language-navigation work has primarily been evaluated in simulation, leaving transferability to the real world open.
- Sim2Real Transfer: Sim-to-real methods include source randomization, source–target mapping, learned-manifold interpolation, and generative adversarial target-domain synthesis.
3. RoboTHOR
RoboTHOR is built to study simulation-to-real transfer using diverse, carefully aligned apartments, modular assets, matched robot sensing, noisy simulated control, and remote connectivity. Its scenes and interface support both scalable simulation training and physical deployment.
- Scenes: RoboTHOR contains 89 apartments: 75 for train/validation, 4 for test-dev, and 10 for a blind physical test-standard set.Training uses 60 apartments and validation uses 15 from the train/validation set.
- Scenes: Apartments sharing wall layouts receive different room assignments and furniture placements, while train/validation and test scenes avoid overlap in wall layouts and object instances.
- Scenes: Simulation layouts, furniture, objects, and lighting are designed to closely resemble physical counterparts while preserving distinctions across splits.
- Assets: The modular asset library includes 11 furniture types and 32 small-object types, with 14 object categories designated as semantic-navigation targets in every scene.
- Robot and API: The physical LoCoBot uses an Intel RealSense RGB-D camera, and simulation replicates its physical and camera properties.
- Robot and API: Simulation adds controller noise estimated from repeated measurements of real-robot orientation and displacement errors.
- Robot and API: A PyRobot-based API supports seamless switching between synthetic and physical environments, while HTTP connectivity and scheduling enable remote access.
4. Visual Semantic Navigation
Visual semantic navigation requires an agent to find an instance of a specified object category using egocentric observations, while handling obstacles, appearance variation, and noisy motion. The task involves exploration before directed movement, making shortest-path statistics an incomplete practical benchmark.
- Task definition: The agent must combine obstacle avoidance and movement with semantic reasoning about object instances, occlusion, scale, and appearance variation.
- Task definition: Visual semantic navigation directs an agent toward an instance of a noun-specified object category using egocentric sensory inputs.Inputs may include RGB, depth, or both; success requires seeing the object, reaching a distance threshold, and reporting completion.
- Task challenges: Stochastic motion makes navigation harder and is intended to reflect real-world motor noise and wheel slippage.Deterministic movement during training can produce suboptimal performance when policies are tested in the real world.
- Task challenges: Shortest-path lengths underestimate practical difficulty because agents must first explore to locate the target before moving toward it.Human trajectories illustrate wandering during search followed by direct movement after the target becomes visible.
- Evaluation: Baseline models include Random, Instant Done, and Blind variants, alongside models evaluated with Success Rate, SPL, action count, and travel distance.These baselines test scene simplicity and trivial starting locations, while path measures quantify exploration.
5. Experiments
Experiments train noisy-motion navigation policies in simulation and evaluate them across simulated and physical scenes. Results show a substantial sim-to-real performance drop, with remaining real-world behavior affected by scene difficulty, appearance, and control variation.
- Training and evaluation: Models are trained with A3C reinforcement learning on simulated scenes using noisy virtual-agent movement to mimic real-robot dynamics.Training uses a four-action space for Television navigation, with rewards for successful completion and penalties per step.
- Training and evaluation: Evaluation trains on 50 scenes and reports performance on two test-dev scenes categorized as easy, medium, or hard by shortest-path length.
- Sim-to-sim results: In sim-to-sim evaluation, trivial baselines perform poorly, while the image-only model exceeds 50% success on easy trajectories but performs poorly on hard ones.Adding object detection does not improve performance in this benchmark.
- Sim-to-real results: The best sim-to-sim model shows a sizeable drop in real-robot performance, especially in SPL, although over 80% of trajectories avoid collisions.Real episodes can last up to 10 minutes because of the robot’s slow motion, limiting physical-world testing.
- Sim-to-real results: When trained directly on a test-dev scene, the robot performs well on easy targets but remains strongly affected on hard targets and has substantially reduced SPL.Appearance and control variations can place the robot away from the target despite simulation overfitting.
6. Analysis
The analysis examines appearance and camera mismatches between RoboTHOR’s real and simulated environments, and evaluates image translation for transfer. These studies show that modality differences can substantially hinder sim-to-real policies.
- Appearance disparities: t-SNE embeddings compare ResNet-18 features from real and simulated apartment images, alongside feature-space and spatial nearest neighbors.The ResNet-18 model was pre-trained on ImageNet; the two nearest-neighbor views differ slightly in agent viewpoint.
- Appearance disparities: At IoU 0.5, object detection mAP was 0.338 for real images and 0.255 for simulated images.The comparison used 761 corresponding images, Faster R-CNN, and ground-truth annotations for 10 object classes.
- Appearance disparities: Real-to-simulation transfer is also challenging because detection probabilities differ across the two modalities.The passage notes that this transfer direction is opposite to the paper’s current sim-to-real setup, but remains informative for models exploiting detection probabilities.
- Modifying camera parameters: Camera-parameter changes were studied to test the sensitivity of policies trained in simulation before deployment on the physical robot.The real and simulated apartments were designed by the authors, allowing the simulation camera to approximate the LoCoBot camera.
- Image translation: CycleGAN flattened textures and adjusted lighting and shadows, but the translated images produced 0% robot accuracy.The authors attribute the poor performance to spurious errors that substantially affect image embeddings and the resulting policy.
- Image translation: Figure 9 illustrates real-to-simulation translation that changes textures, lighting, and shadows toward a simulated appearance.The model was trained with paired images from three test-dev scenes and evaluated on the remaining test-dev scene.
7. Conclusion
RoboTHOR is an open, modular, re-configurable, and replicable embodied AI platform with simulation and real-world counterparts. Its preliminary findings show that performance drops significantly when models transfer from simulation to reality.
- Conclusion: RoboTHOR lets researchers remotely deploy models on physical robots and test embodied AI algorithms in the real world.The platform pairs simulated environments with physical counterparts.
- Conclusion: Performance drops significantly when models trained in simulation are transferred to the real world.This is presented as a preliminary finding from the platform’s benchmark experiments.