PaperlayerView exact revision on arXiv

Source-linked AI summary

iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes

Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D'Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, Silvio Savarese

arXiv:2012.02924v6cs.AIcs.CVcs.RO

TL;DR

Robotic learning needs simulation environments that combine realistic large scenes with rich, continuous interaction rather than narrow tasks or simplified object transitions. iGibson 1.0 provides 15 interactive home scenes with 108 rooms, realistic sensors, randomization, planners, and demonstration tools; experiments report stronger navigation generalization, successful imitation learning, and faster downstream manipulation training. The environment also extends to more than 12,000 additional interactive home scenes through layout integration.

  • Problem

    Existing environments often use clean small scenes, disable interaction in large scenes, or simplify interactions, limiting end-to-end sensorimotor control for rich continuous tasks.

  • Method

    iGibson 1.0 combines physics-based interactive home scenes, realistic virtual sensors, domain randomization, sampling-based motion planners, and a human-iGibson demonstration interface.

  • Results

    Experiments show more robust navigation policies, effective imitation learning for manipulation and mobile manipulation, and visual representations that accelerate downstream manipulation training.

  • Takeaways & Limitations

    Full scene interactivity, realistic sensing, randomization, planning, and demonstrations support navigation and manipulation learning in large realistic scenes.

Abstract

from arXiv · show

We present iGibson 1.0, a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes. Our environment contains 15 fully interactive home-sized scenes with 108 rooms populated with rigid and articulated objects. The scenes are replicas of real-world homes, with distribution and the layout of objects aligned to those of the real world. iGibson 1.0 integrates several key features to facilitate the study of interactive tasks: i) generation of high-quality virtual sensor signals (RGB, depth, segmentation, LiDAR, flow and so on), ii) domain randomization to change the materials of the objects (both visual and physical) and/or their shapes, iii) integrated sampling-based motion planners to generate collision-free trajectories for robot bases and arms, and iv) intuitive human-iGibson interface that enables efficient collection of human demonstrations. Through experiments, we show that the full interactivity of the scenes enables agents to learn useful visual representations that accelerate the training of downstream manipulation tasks. We also show that iGibson 1.0 features enable the generalization of navigation agents, and that the human-iGibson interface and integrated motion planners facilitate efficient imitation learning of human demonstrated (mobile) manipulation behaviors. iGibson 1.0 is open-source, equipped with comprehensive examples and documentation. For more information, visit our project website: http://svl.stanford.edu/igibson/

I. INTRODUCTION

iGibson 1.0 addresses the lack of large, realistic, continuously interactive simulation environments for end-to-end robotic learning. It combines interactive home scenes with realistic sensing, randomization, planning, and demonstration tools, and experiments show benefits for navigation, manipulation, and representation learning.

  • Motivation: Existing simulators often provide narrow tasks, clean small scenes, or simplified interaction, limiting end-to-end sensorimotor control for rich continuous interactions.Large-scene simulators may disable interaction or use simplified state transitions, creating difficulties for tasks requiring detailed physical interaction and transfer.
  • Contribution: iGibson 1.0 enables embodied-agent development in large-scale realistic scenes through physics simulation of rigid and articulated objects.The environment integrates physics-based interaction, robot control, virtual sensors, reinforcement-learning frameworks, and realistic indoor scenes.
  • Assets: 15 fully interactive scenes contain 108 rooms modeled after real-world homes, with object layouts and category distributions preserved from reconstructed scans.The scenes are converted from static 3D reconstructions into interactive models populated with articulated objects.
  • Tools: iGibson provides realistic virtual sensors, domain randomization, human demonstrations, and sampling-based motion planners for navigation and manipulation.Supported signals include RGB, depth, point clouds, LiDAR, and flow; randomization covers visual textures, dynamics, and object instances.
  • Evaluation: Experiments show more robust navigation policies, effective imitation learning for manipulation and mobile manipulation, and interaction-aware visual representations that accelerate downstream training.The platform is open-source and supports navigation, manipulation, and mobile manipulation experiments.

II. RELATED WORK

Related simulation environments differ in their physics, scene scale, interaction abstractions, and robotics tooling. iGibson is positioned as a large-scene environment that combines continuous physics with sensors and robot-learning support.

  • Definitions: Simulation environments include physics engines, renderers, and reusable scene, object, and robot assets for embodied-AI and robotics research.The paper distinguishes physics simulators from broader simulation environments that package physics, virtual signals, and task-ready assets.
  • Scene scale: Compared with manipulation simulators centered on stationary arms, iGibson targets house-sized scenes with 15 interactive scenes and 108 rooms for navigation, manipulation, and mobile manipulation.Its scenes include settings such as kitchens and bedrooms.
  • Prior large-scene environments: Gibson v1 provides many reconstructed floors but represents each scene as one rigid mesh, preventing object interaction beyond collisions.This restricts its use to navigation-oriented interaction with the static mesh.
  • Interaction models: Predefined actions transition objects between states and support high-level planning, but they limit access to the granular physical interactions needed for full robotics tasks and sim2real transfer.AI2Thor and VirtualHome use predefined-action abstractions; rigid-body physics can instead simulate continuous object behavior.
  • iGibson positioning: iGibson unifies LiDAR and PBR rendering, reinforcement-learning speed, robot integration, motion planners, and continuous physics for agents and objects.This combination supports physics-based mobile manipulation across the entire task.

A. Simulation Characteristics and API

iGibson uses a Gym-style interface and configurable environments built from scenes, objects, robots, and tasks. Its asset pipeline preserves realistic layouts while expanding access to interactive home scenes.

  • API: iGibson follows the OpenAI Gym convention: actions produce observations, rewards, and episode metadata, while config files specify scenes, tasks, robots, and sensors.A configured Environment contains a Task and Simulator, whose Scene contains interactive objects and robots.
  • Tooling: The platform includes configurations, demonstrations, Docker files, parallelization, off-screen rendering, and prior use in real-world-transfer robotics research.Virtual LiDAR has supported planning and reinforcement learning for social navigation and mobile manipulation.
  • Core assets: iGibson provides 15 high-quality interactive scenes with 108 rooms, created by converting reconstructed Gibson v1 scenes while preserving real-world object layouts and distributions.Scene generation uses semi-automatic annotation of structural elements and objects.
  • Core assets: The scene-generation pipeline annotates floors, walls, doors, windows, and 57 object classes using 3D bounding boxes and class labels.Classes include furniture and smaller objects such as plants, laptops, and speakers.
  • Object modeling: Object models from ShapeNet, PartNet Mobility, and SketchFab receive photorealistic materials and mappings between visual and physical material properties.These annotations support both visual realism and physics realism in rendering and simulation.
  • Additional assets: Compatibility with CubiCasa5K and 3D-Front adds more than 12,000 interactive home scenes, although these scenes contain fewer objects than the fifteen primary iGibson scenes.Their layouts are populated with iGibson’s annotated object models.
  • Learning benefit: Fully interactive scenes support learning interaction-aware visual representations that accelerate downstream manipulation-task training.The paper demonstrates this use in its experiments.

C. Virtual Sensors

iGibson generates high-quality virtual sensor signals for simulated robots across image, point-cloud, LiDAR, and motion-related modalities. These signals support realistic perception and sensor-guided policy research.

  • Sensor suite: Virtual sensor generation is a central component of iGibson, producing images and point clouds for simulated robots.The sensor suite is presented as a foundation for navigation and manipulation experiments.
  • Rendering: The physics-based renderer approximates BRDF models using spatially varying roughness, metallic, and tangent-space normal maps.These material maps provide visual and physical surface information for rendering.
  • LiDAR: Virtual LiDAR supports both 1-beam and 16-beam configurations, includes dropout noise, and can convert scans into local bird’s-eye-view occupancy maps.The configurations emulate sensors such as Hokuyo and Velodyne VLP-16.
  • Additional modalities: Beyond RGB and LiDAR, iGibson supports depth, optical and scene flow, normals, and semantic, instance, material, and movable-part segmentation.These modalities support perception, pose estimation, flow prediction, and visuomotor policy research.

D. Domain Randomization

iGibson provides domain randomization that varies object instances, visual materials, dynamics properties, and shapes while preserving scene semantics, supporting robust sensor-based policies.

  • Purpose: iGibson’s randomized environments are intended to make policies more robust and more likely to generalize to unknown evaluation environments.The evaluation environment may be the real world when policies are transferred from simulation.
  • Object randomization: Object randomization preserves semantic object locations while enabling near-infinite combinations of object instances.The resulting variation strengthens depth-map and LiDAR-based policy robustness.
  • Scene and sensor context: The scenes preserve real-world object layout, distribution, and size while supporting interactive robot sensing.The environment exposes RGB, depth, segmentation, normals, point-cloud LiDAR, and 1D LiDAR signals.
  • Material randomization: Material randomization changes plausible visual appearances and associated dynamics properties for object parts.It produces strong RGB variation while retaining plausible material combinations.
  • Motion planning: Integrated sampling-based planners generate collision-free trajectories for robot bases and arms, with acceleration-bounded shortcuts for smoother paths.Implemented planners include RRT, BiRRT, and lazyPRM.

F. Human-iGibson Interface

The human-iGibson interface supports simple mouse-and-keyboard navigation and object interaction for demonstrations, while experiments assess generalization across held-out and real-world settings.

  • Human-iGibson Interface: Users can navigate scenes and pull, push, pick, and place objects through a mouse-and-keyboard viewer interface.The interface avoids requiring virtual-reality or 3D-mouse hardware.
  • Experimental aims: The experiments examine whether iGibson supports navigation generalization, imitation learning for manipulation, and interaction-based visual pretraining.The stated evaluation spans held-out scenes, manipulation tasks, and downstream interaction learning.
  • Navigation evaluation: Navigation policies are evaluated in held-out scenes and textures using depth, RGB, or LiDAR observations.The LiDAR experiment tests transfer from iGibson to a real-world counterpart without adaptation.
  • Results: Domain randomization improves held-out navigation performance, while simulated LiDAR transfers with a 9% success-rate drop from iGibson to the real-world apartment.PointGoal depth navigation rises from 0.27 to 0.40 SPL and from 31.25% to 44.75% success rate; RGB object navigation rises from 49.75% to 57.5%.
  • Imitation learning: The human-iGibson interface and integrated planners support imitation learning for manipulation and mobile manipulation.The reported evaluation achieves 98% success for pick-and-place and 70% for mobile manipulation.

B. Imitation Learning of Human Demonstrated (Mobile) Manipulation

iGibson supports imitation learning from both interface-collected and teleoperated demonstrations for manipulation and mobile-manipulation tasks.

  • Manipulation: Fifty interface-collected demonstrations train a behavioral-cloning policy for picking mugs and placing them in a sink.The demonstrations use 20 mug models and map object-position states to desired end-effector translations at 20 Hz.
  • Mobile manipulation: Two hundred teleoperated demonstrations train a behavioral-cloning policy for mobile search-and-pick tasks.The simulated Fetch robot navigates, opens drawers, finds a bowl, and picks it under randomized initial poses.
  • Policy formulation: The mobile-manipulation policy combines proprioception and RGB-D observations with base, arm, extension, and gripper actions.The policy operates at 20 Hz and predicts discrete and continuous action components.
  • Results: 98% success rate is achieved over 100 evaluation episodes for the imitation-learned manipulation-only policy.The result is reported for the interface-demonstration experiment.
  • Results: 70% success rate is achieved over 20 evaluation episodes for the imitation-learned mobile-manipulation policy.The result is reported for the teleoperated-demonstration experiment.

C. Pretraining in Fully Interactive Scenes

iGibson’s fully interactive scenes support pretraining visual representations that predict interaction outcomes and accelerate downstream manipulation-task learning.

  • Motivation: The pretraining objective is to learn an intermediate visual representation encoding expected interaction outcomes for different objects.The representation is intended to associate visual observations with promising interaction regions.
  • Pretraining procedure: A virtual agent interacts at image-coordinate points and labels success or failure after simulated pushing interactions.The selected pixel’s corresponding 3D location is displaced by 30 cm opposite the surface normal.
  • Visual representation: A U-Net-based encoder predicts heatmaps of expected interaction success from RGB input.These heatmaps represent where interactions are likely to succeed.
  • Downstream policies: The pretrained heatmaps gate Q-value maps while DQN policies select push points for PushDrawer and PushCabinet tasks.Integrated motion planners then generate arm motions for the selected interaction points.
  • Results: Pretraining identifies door edges as stronger push locations than areas near hinges and accelerates training on both downstream tasks.The model also assigns weak activation to closed cabinets.
  • Scope of interactivity: Fully interactive scenes are necessary for successful representation learning because non-interactable objects create false negatives during pretraining.The paper identifies continuous robot actions in fully interactive scenes as a specialty of iGibson.

APPENDIX

The appendix details navigation, manipulation, and mobile-manipulation experiments, including observation spaces, demonstrations, and behavioral-cloning evaluation. A simulated Fetch policy achieved 98% success on mug pick-and-place.

  • Navigation: The appendix specifies PointGoal observations, including depth maps, robot velocities, goal location, and ten shortest-path waypoints, with stacked frames.Depth and sensor observations use four-frame stacks; the policy encoders combine convolutional depth features with multilayer-perceptron sensor features.
  • Navigation: LiDAR-based navigation uses 512-ray scans, robot velocities, and goal location, with an 8-frame stack and a controller running at 10 Hz.Separate multilayer perceptrons encode LiDAR and sensor observations before fusion.
  • Imitation Learning: The human-iGibson interface collected 50 mug pick-and-place demonstrations using 20 mug models, split into 35 training, 10 validation, and 5 test demonstrations.The recorded state contains object position, while actions contain desired translation changes and gripper opening.
  • Interactive Tasks: The appendix illustrates successful PushDrawer and PushCabinet executions, including pushing drawers in and closing cabinet doors.
  • Imitation Learning: Behavioral cloning maps states to actions with a three-layer MLP, using mean-squared error for translation and cross-entropy for the binary gripper action.Training used ADAM with learning rate 0.1 for 1000 epochs after validation loss plateaued.
  • Imitation Learning: 98% success rate was achieved across 100 evaluation episodes for mug pick-and-place on the simulated Fetch robot.The two failures involved premature gripper opening and dropping the mug outside the sink, within a 500-step time budget.

4) Pretraining in Fully Interactive Scenes:

The pretraining experiment learns an interactability representation from RGB images and evaluates it on pushing tasks. With pretraining, agents learn faster in fully interactive scenes.

  • Pretraining: The pretraining network is a UNet that maps 128×128 RGB images to binary masks identifying interactable areas.Its encoder uses ResNet9, followed by four upsampling-convolution blocks and a two-convolution readout.
  • Downstream Tasks: The downstream pushing tasks use 128×128 RGB observations and image-point actions, with a DQN predicting dense Q-values over image pixels.The policy uses a six-layer convolutional network and selects the pixel with the highest predicted value.
  • Results: With pretraining, the agent learns faster during the pushing interactions shown in Fig. 8.The figure depicts stages of interaction learned using the intermediate representation.
  • Physics-Based Rendering: iGibson rendering represents materials with metallic, roughness, albedo, and tangent-space normal layers combined by a physics-based model.Environment maps and pre-integrated Cook-Torrance BRDF components support light and specular-effect rendering.
  • Physics-Based Rendering: Specular highlights use pre-filtered environment cube maps with GGX importance sampling, while diffuse effects use quasi-Monte Carlo Hammersley sampling.
  • Physics-Based Rendering: Shadow mapping uses orthogonal uniform light in the +z direction, with all scene objects except ceilings acting as shadow casters.The resulting shadows need not match real indoor lighting but are intended to provide a realistic depth cue.

C. iGibson Performance

iGibson’s performance evaluation benchmarks rendering, physics simulation, and combined execution under VisualRL and HighFidelity settings. The appendix defines the image resolutions and rendering features used for these benchmarks.

  • Evaluation Scope: The evaluation measures rendering time, physics simulation time, and combined simulation performance.
  • Rendering Settings: VisualRL renders 128×128 images with physically based rendering enabled and anti-aliasing and shadow mapping disabled.This setting targets fast generation of low-resolution simulated sensor data.
  • Rendering Settings: HighFidelity renders 512×512 images with physically based rendering, multi-sample anti-aliasing, and shadow mapping enabled.This setting targets photorealistic images while maintaining a minimum rendering speed.
  • Reported Metrics: Table IV reports rendering speed in frames per second, while Table V reports simulator-step and full-step speed in hertz.

2) Physics Simulator and Full Simulator Performance:

The appendix describes how iGibson converts CubiCasa5K and 3D-Front layouts into interactive scenes and characterizes their scale and remaining differences from curated iGibson scenes.

  • Integration of CubiCasa5K: CubiCasa5K layouts are converted into 6297 fully interactive indoor iGibson scenes by generating structures and populating them with objects according to annotated poses and sizes.The source dataset contains annotated floor plans of real-world homes in Finland.
  • Integration of 3D-Front: 3D-Front static scenes are converted into interactive scenes by retaining structural meshes visually and procedurally generating collision meshes.
  • Integration of 3D-Front: 3D-Front integration faces undefined object categories, corrupted geometry, uninteractive kitchen-cabinet annotations, and significant furniture overlap.Alternative scene versions omit interaction for kitchen cabinets or omit the cabinets entirely.
  • Integration of 3D-Front: 6049 3D-Front scenes remained after filtering and fixing object placements to remove overlaps.
  • Scene Comparison: Curated iGibson scenes contain 75 objects per room, compared with 37 in 3D-Front rooms and 32 in CubiCasa5K rooms.The curated scenes also include baked lighting effects unavailable for the imported datasets without additional light annotations.
Loading 2012.02924v6…