Source-linked AI summary
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, Vladlen Koltun
TL;DR
Realistic indoor navigation requires scalable environments and flexible sensory inputs beyond simple maze settings. MINOS provides such a simulator and benchmark, finding that current deep reinforcement learning struggles in large realistic scenes while multimodality benefits navigation in clutter.
Problem
Navigation remains challenging, while physical-world development is costly and existing deep-learning methods had mainly succeeded in simple 3D mazes.
Method
MINOS combines large SUNCG and Matterport3D environments with configurable multimodal sensors, procedural scene variation, and navigation benchmarks.
Results
Current deep reinforcement learning approaches fail in large realistic indoor environments, while multimodality benefits learning to navigate cluttered scenes.
Takeaways & Limitations
MINOS provides an open-source framework for sensorimotor learning research in realistic indoor environments.
Abstract
from arXiv · showhide
We present MINOS, a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments. The simulator leverages large datasets of complex 3D environments and supports flexible configuration of multimodal sensor suites. We use MINOS to benchmark deep-learning-based navigation methods, to analyze the influence of environmental complexity on navigation performance, and to carry out a controlled study of multimodality in sensorimotor learning. The experiments show that current deep reinforcement learning approaches fail in large realistic environments. The experiments also indicate that multimodality is beneficial in learning to navigate cluttered scenes. MINOS is released open-source to the research community at http://minosworld.org . A video that shows MINOS can be found at https://youtu.be/c0mL9K64q84
1. Introduction
Indoor navigation remains difficult in realistic environments, motivating simulation for scalable development and evaluation. MINOS provides large indoor scenes, configurable multimodal sensing, and benchmarks showing that current methods struggle with complexity while multimodality helps in clutter.
- Physical-world sensorimotor development is difficult because operation is real-time, failures can damage systems, supervision may be required, and generalization requires varied environments.
- MINOS provides large realistic indoor environments, flexible multimodal sensors, procedural scene reconfiguration, and high rendering rates for extensive training.It uses SUNCG and Matterport3D and supports vision, depth, surface normals, touch, and semantic segmentation.
- The benchmark uses fixed splits of varying complexity across SUNCG and Matterport3D with PointGoal, ObjectGoal, and RoomGoal tasks.
- 20% or less of PointGoal trials were completed in furnished medium-scale Matterport3D scenes, while RoomGoal completion reached only 14% in small scenes.
- Depth and touch can individually outperform vision, while combined modalities are more effective, especially in cluttered environments.
- MINOS is released open-source to support further sensorimotor learning research.
2. Related Work
MINOS builds on simulation platforms for sensorimotor learning while targeting realistic, cluttered indoor navigation. Its distinction is large interconnected environments, flexible sensor suites, and support for varied control configurations.
- Existing simulators support sensorimotor control, games, autonomous driving, UAV control, and robotics research across two- and three-dimensional settings.
- VizDoom and DeepMind Lab provide immersive 3D labyrinths but lack realistic layouts, appearance, and scene objects.
- MINOS focuses on realistic, cluttered indoor scenes for developing and validating sensorimotor control models.
- Compared with AI2-THOR, MINOS uses thousands of furnished houses with interconnected layouts of up to dozens of rooms rather than 32 single-room environments.
- MINOS supports flexible sensor suites, including vision, depth, surface normals, segmentation, and touch, with configurable numbers and parameters.
- MINOS supports navigation with both continuous and discrete state spaces in SUNCG and Matterport3D environments.
3. Simulation Framework
MINOS is a flexible client-server simulator for large indoor environments, with configurable scenes, controls, sensors, goals, and tasks. It draws on SUNCG and Matterport3D to support realistic, varied navigation experiments.
- Simulation Framework: MINOS uses a flexible configuration API for environment selection and variation, agent controls, and arbitrary multimodal sensor specifications.
- Simulation Framework: The server-client architecture provides a Python RL API, a web client, and WebSocket communication for distributed training.
- Datasets: SUNCG supplies approximately 45,000 furnished houses and more than 750K rooms with complex inter-room and intra-room navigation.
- Datasets: Matterport3D contains 90 multi-floor residences and approximately 2,000 annotated room regions, providing a realistic testbed for reinforcement-learning navigation.
- Agent: The agent uses parameterized commands and physical dynamics that support continuous navigation or effectively discretized motion.
- Sensors: MINOS supports vision, depth, surface normals, contact forces, semantic segmentation, and measurements such as velocity, acceleration, and goal distance.
- Customization: Users can vary materials and object clutter while preserving split separation, and can specify point, object, or room goals.
- Customization: Tasks are arbitrary Python functions that compute rewards and episode success or failure from observations, measurements, and state.
4. Methods
MINOS benchmarks four end-to-end navigation algorithms, including A3C variants and Direct Future Prediction, using discrete actions and episodic sensorimotor interaction. The methods differ in memory, auxiliary supervision, and whether they predict rewards or future measurements.
- Interaction model: The simulator represents observations as multimodal sensory inputs and lets agents choose actions from a discrete action set.The benchmark treats each interaction as an episodic sequence with observations and rewards at discrete time steps.
- Algorithms: The benchmark evaluates three A3C-based agents and Direct Future Prediction in continuous-state indoor navigation with discretized actions.The compared methods are Feedforward A3C, LSTM A3C, UNREAL, and DFP.
- A3C variants: Feedforward A3C learns a value function and policy with a feedforward convolutional network and policy-gradient training.The value function estimates discounted future rewards, while the policy favors actions associated with larger-than-average rewards.
- A3C variants: LSTM A3C augments the feedforward network with memory that can support internal environmental representations and temporally extended action sequences.The LSTM units are trained by backpropagation through time.
- A3C variants: UNREAL augments LSTM A3C with auxiliary unsupervised tasks that provide additional training signals for convergence and stability.The auxiliary tasks include value-function replay, reward prediction, and pixel control.
- Direct Future Prediction: DFP predicts future low-dimensional sensory measurements rather than explicitly maximizing future rewards, then selects actions using an objective defined over those measurements.The method is described as Monte Carlo reinforcement learning with a decomposed reward.
5. Experiments
MINOS evaluates navigation algorithms on unseen indoor environments while varying goals, dataset, scene complexity, clutter, and sensor suite. Performance declines sharply in large realistic environments, whereas depth and multimodal sensing improve navigation, especially in cluttered scenes.
- Experimental design: The experiments test generalization to previously unseen environments across spatial and semantic goals, environmental complexity, and sensory modalities.The benchmark uses PointGoal, ObjectGoal, and RoomGoal tasks and compares different sensor combinations.
- Environments: MINOS benchmarks SUNCG and Matterport3D scenes with controlled variation in house size, furniture, and realism.SUNCG includes empty and furnished variants, while Matterport3D provides larger interconnected environments.
- Evaluation: Agents are evaluated by episode success rate on fixed test episodes using presampled starting configurations spanning distances from the goal.Testing uses 10 episodes per scene and reports the percentage of episodes in which the agent reaches the goal.
- Algorithm comparisons: UNREAL generally performs best, while DFP is strongest in smaller, less cluttered SUNCG PointGoal settings and UNREAL dominates larger Matterport3D and RoomGoal settings.The authors associate UNREAL’s advantage in challenging settings with memory and auxiliary-learning supervision.
- Environment complexity: 80% success occurs in the simplest two-room empty SUNCG PointGoal environments, but all agents reach 20% or less in the most complex Matterport3D PointGoal setup.The complex setup contains houses with up to 24 rooms, and performance declines as environments become larger and more cluttered.
- Goal specification: RoomGoal is harder than PointGoal for all algorithms, likely because its reward only indicates whether the agent occupies a matching room type.The semantic task therefore provides a sparser reward signal than the spatial task.
- Multimodal sensing: Depth is particularly effective in empty multi-room houses, while the full multimodal agent performs best in furnished settings.Among individual modalities in furnished environments, depth provides the strongest advantage; modality combinations are especially useful in clutter.
6. Discussion
MINOS provides a flexible multimodal simulator built from large indoor-environment datasets for developing and evaluating navigation models. Its experiments show that current deep reinforcement learning approaches fail in large realistic environments, while multimodality benefits learning in cluttered scenes.
- Contribution: MINOS is a multimodal simulation platform for developing multisensory models for goal-directed indoor navigation.Its sensory input modules can be flexibly combined.
- Contribution: The simulator leverages large indoor datasets and controlled variation in appearance and clutter to provide orders of magnitude more environments for training and testing.The platform is built around large-scale indoor-environment data and configurable sensory inputs.
- Findings: Current deep reinforcement learning approaches fail in large, realistic indoor environments, while multimodality benefits learning to act in cluttered scenes.These are the paper’s principal experimental conclusions.