RoboTHOR: An Open Simulation-to-Real Embodied AI Platform

• Embodied AI remains challenging despite major advances in computer vision and natural language processing.Source: §1 · p. 1
• High physical costs, limited standardized benchmarks, and data- and time-inefficient learning algorithms hinder progress in real-world embodied AI.Source: §1 · p. 2
• Synthetic environments lower research barriers and support progress on embodied tasks, but how well simulation-trained models transfer to reality remains an open question.Source: §1 · p. 2
• RoboTHOR pairs agents in simulated environments with robotic counterparts in the physical world as an open and accessible benchmarking platform.Source: §1 · p. 2
• RoboTHOR uses diverse simulated scenes with corresponding physical counterparts, including modular assets and reconfigurable layouts that support scalable testing.The platform includes 731 unique assets, and physical scenes can be reconfigured in roughly 30 minutes.Source: §1 · p. 2
• The platform is designed to broaden participation through open-source resources, remote no-cost deployment, replicable physical environments, and transfer-focused challenges.The physical environment costs roughly $10,000 in materials and assets to reproduce.Source: §1 · p. 2
• Initial experiments identify a disparity between simulation and reality, including feature-space differences, real-world control variation, and no improvement from off-the-shelf image correction.Source: §1 · pp. 2–3
• These findings show that generalizing from simulated environments to the real world remains a daunting challenge while motivating further research.Source: §1 · p. 3

Embodied AI Environments

• Synthetic frameworks support visual navigation, task completion, and question answering by abstracting control, manipulation, and appearance noise.Source: §2 · p. 3

Robotics research platforms

• Robotics platforms have historically been expensive, although low-cost robots have broadened access to research.Source: §2 · p. 3
• Simulator–robot combinations have traditionally addressed tasks such as robotic-arm manipulation and autonomous driving.Source: §2 · p. 3

Visual Navigation

• This paper focuses on indoor semantic navigation, directing an agent toward an instance of a specified object category.Source: §2 · p. 3
• Existing language-navigation work has primarily been evaluated in simulation, leaving transferability to the real world open.Source: §2 · p. 3

Sim2Real Transfer

• Sim-to-real methods include source randomization, source–target mapping, learned-manifold interpolation, and generative adversarial target-domain synthesis.Source: §2 · p. 3

Scenes

• RoboTHOR contains 89 apartments: 75 for train/validation, 4 for test-dev, and 10 for a blind physical test-standard set.Training uses 60 apartments and validation uses 15 from the train/validation set.Source: §3 · p. 3
• Apartments sharing wall layouts receive different room assignments and furniture placements, while train/validation and test scenes avoid overlap in wall layouts and object instances.Source: §3 · pp. 3–4
• Simulation layouts, furniture, objects, and lighting are designed to closely resemble physical counterparts while preserving distinctions across splits.Source: §3 · p. 4

Assets

• The modular asset library includes 11 furniture types and 32 small-object types, with 14 object categories designated as semantic-navigation targets in every scene.Source: §3 · p. 4

Robot and API

• The physical LoCoBot uses an Intel RealSense RGB-D camera, and simulation replicates its physical and camera properties.Source: §3 · p. 4
• Simulation adds controller noise estimated from repeated measurements of real-robot orientation and displacement errors.Source: §3 · p. 4
• A PyRobot-based API supports seamless switching between synthetic and physical environments, while HTTP connectivity and scheduling enable remote access.Source: §3 · pp. 4–5

Task definition

• The agent must combine obstacle avoidance and movement with semantic reasoning about object instances, occlusion, scale, and appearance variation.Source: §4 · p. 5
• Visual semantic navigation directs an agent toward an instance of a noun-specified object category using egocentric sensory inputs.Inputs may include RGB, depth, or both; success requires seeing the object, reaching a distance threshold, and reporting completion.Source: §4 · p. 5

Task challenges

• Stochastic motion makes navigation harder and is intended to reflect real-world motor noise and wheel slippage.Deterministic movement during training can produce suboptimal performance when policies are tested in the real world.Source: §4 · p. 5
• Shortest-path lengths underestimate practical difficulty because agents must first explore to locate the target before moving toward it.Human trajectories illustrate wandering during search followed by direct movement after the target becomes visible.Source: §4 · p. 5, Figure 5 · p. 5, Figure 6 · p. 5

Evaluation

• Baseline models include Random, Instant Done, and Blind variants, alongside models evaluated with Success Rate, SPL, action count, and travel distance.These baselines test scene simplicity and trivial starting locations, while path measures quantify exploration.Source: §4.1 · pp. 5–6, §4.2 · p. 6

Training and evaluation

• Models are trained with A3C reinforcement learning on simulated scenes using noisy virtual-agent movement to mimic real-robot dynamics.Training uses a four-action space for Television navigation, with rewards for successful completion and penalties per step.Source: §5 · p. 6
• Evaluation trains on 50 scenes and reports performance on two test-dev scenes categorized as easy, medium, or hard by shortest-path length.Source: §5 · p. 6

Sim-to-sim results

• In sim-to-sim evaluation, trivial baselines perform poorly, while the image-only model exceeds 50% success on easy trajectories but performs poorly on hard ones.Adding object detection does not improve performance in this benchmark.Source: §5 · p. 6

Sim-to-real results

• The best sim-to-sim model shows a sizeable drop in real-robot performance, especially in SPL, although over 80% of trajectories avoid collisions.Real episodes can last up to 10 minutes because of the robot’s slow motion, limiting physical-world testing.Source: §5 · p. 6
• When trained directly on a test-dev scene, the robot performs well on easy targets but remains strongly affected on hard targets and has substantially reduced SPL.Appearance and control variations can place the robot away from the target despite simulation overfitting.Source: §5 · p. 6

Appearance disparities

• t-SNE embeddings compare ResNet-18 features from real and simulated apartment images, alongside feature-space and spatial nearest neighbors.The ResNet-18 model was pre-trained on ImageNet; the two nearest-neighbor views differ slightly in agent viewpoint.Source: Figure 7 · p. 7
• At IoU 0.5, object detection mAP was 0.338 for real images and 0.255 for simulated images.The comparison used 761 corresponding images, Faster R-CNN, and ground-truth annotations for 10 object classes.Source: §6 · p. 7
• Real-to-simulation transfer is also challenging because detection probabilities differ across the two modalities.The passage notes that this transfer direction is opposite to the paper’s current sim-to-real setup, but remains informative for models exploiting detection probabilities.Source: §6 · p. 7

Modifying camera parameters

• Camera-parameter changes were studied to test the sensitivity of policies trained in simulation before deployment on the physical robot.The real and simulated apartments were designed by the authors, allowing the simulation camera to approximate the LoCoBot camera.Source: §6 · p. 8

Image translation

• CycleGAN flattened textures and adjusted lighting and shadows, but the translated images produced 0% robot accuracy.The authors attribute the poor performance to spurious errors that substantially affect image embeddings and the resulting policy.Source: §6 · p. 8
• Figure 9 illustrates real-to-simulation translation that changes textures, lighting, and shadows toward a simulated appearance.The model was trained with paired images from three test-dev scenes and evaluated on the remaining test-dev scene.Source: Figure 9 · p. 8, §6 · p. 8

Conclusion

• RoboTHOR lets researchers remotely deploy models on physical robots and test embodied AI algorithms in the real world.The platform pairs simulated environments with physical counterparts.Source: §7 · p. 8
• Performance drops significantly when models trained in simulation are transferred to the real world.This is presented as a preliminary finding from the platform’s benchmark experiments.Source: §7 · p. 8