Source-linked AI summary
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, Silvio Savarese
TL;DR
Embodied-AI simulators have largely focused on motion and physical contact, limiting support for household tasks involving richer object states. iGibson 2.0 addresses this gap with extended state simulation, semantic predicates and generative scene sampling, and a VR interface for demonstrations. The paper evaluates these capabilities on novel tasks and imitation learning for bimanual operations, while identifying limitations in scalable soft-body and human-behavior simulation.
Problem
Existing simulators primarily model motion and physical contact, while many household tasks require object temperature, wetness, and cleanliness states.
Method
iGibson 2.0 combines extended object-state simulation, logical predicates with generative physical-state sampling, semantic scene population, and VR demonstration collection.
Results
The authors evaluate iGibson 2.0 on six novel embodied-AI tasks and use its VR interface to collect demonstrations for imitation learning in bimanual operations.
Takeaways & Limitations
iGibson 2.0 supports research on diverse household activities and provides a simulator and dataset for developing embodied-AI solutions.
Takeaways & Limitations
iGibson 2.0 does not yet support scalable soft bodies or accurate human behavior modeling, limiting large interactive flexible-material and human-robot-interaction tasks.
Abstract
from arXiv · showhide
Recent research in embodied AI has been boosted by the use of simulation environments to develop and train robot learning approaches. However, the use of simulation has skewed the attention to tasks that only require what robotics simulators can simulate: motion and physical contact. We present iGibson 2.0, an open-source simulation environment that supports the simulation of a more diverse set of household tasks through three key innovations. First, iGibson 2.0 supports object states, including temperature, wetness level, cleanliness level, and toggled and sliced states, necessary to cover a wider range of tasks. Second, iGibson 2.0 implements a set of predicate logic functions that map the simulator states to logic states like Cooked or Soaked. Additionally, given a logic state, iGibson 2.0 can sample valid physical states that satisfy it. This functionality can generate potentially infinite instances of tasks with minimal effort from the users. The sampling mechanism allows our scenes to be more densely populated with small objects in semantically meaningful locations. Third, iGibson 2.0 includes a virtual reality (VR) interface to immerse humans in its scenes to collect demonstrations. As a result, we can collect demonstrations from humans on these new types of tasks, and use them for imitation learning. We evaluate the new capabilities of iGibson 2.0 to enable robot learning of novel tasks, in the hope of demonstrating the potential of this new simulator to support new research in embodied AI. iGibson 2.0 and its new dataset are publicly available at http://svl.stanford.edu/igibson/.
1 Introduction
iGibson 2.0 extends household-task simulation beyond motion and contact by modeling richer object states, logical representations, generative scene construction, and VR demonstrations. These capabilities support diverse task instances and robot-learning experiments, including imitation learning for bimanual operations.
- Existing simulators mainly model motion and physical contact, leaving household activities involving temperature, dirtiness, and wetness insufficiently supported.
- iGibson 2.0 maintains and updates temperature, wetness, cleanliness, toggled, and sliced object states that affect rendered sensor signals.
- Logical predicates map continuous simulated states to semantic conditions such as Cooked, while generative functions sample valid physical states from those conditions.This enables logical scene specifications and diverse instances of the same task.
- The VR interface collects human demonstrations, logs executions, and supports replay for imitation learning and additional sensor or interaction visualizations.
- The authors evaluate iGibson 2.0 on six novel embodied-AI tasks and use VR-collected demonstrations to train an imitation-learning policy for bimanual operations.Keyboard and mouse interfaces were described as insufficient for bimanual manipulation.
- A rule-based mechanism uses logical predicates and generative functions to populate scenes densely with small objects in semantically meaningful locations.
2 Related Work
Related simulators have expanded embodied-AI research through kinodynamic, object-centric, and VR-based environments, but important differences remain in physical realism and interaction interfaces. iGibson 2.0 is positioned among these approaches as a simulator for additional object states and physically realistic VR interaction.
- Simulation environments with (mostly) kinodynamic simulation: Many robotics simulators use physics engines to model kinodynamic interactions for navigation, manipulation, pick-and-place, and rearrangement tasks.
- Simulation environments with (mostly) kinodynamic simulation: Everyday activities also require modifiable object states such as temperature, wetness, and cleanliness, motivating extensions beyond mostly kinodynamic simulation.
- Simulation environments with object-centric representation: Object-centric simulators such as AI2Thor and VirtualHome use extended states, but their predefined discrete symbolic actions rely on preconditions and postconditions.
- Simulation environments with virtual reality interfaces: VR has been used to collect demonstrations in simulation, while prior systems differ in realism, with some relying on primitive precondition/postcondition actions and others providing more physically realistic interaction.
3 Extended Physical States for Simulation of Everyday Household Tasks
iGibson 2.0 extends object-centric simulation beyond pose and contact by maintaining temperature, wetness, cleanliness, toggled, and sliced states. These states approximate household-relevant processes while affecting object appearance and task interactions.
- Extended object states: iGibson 2.0 maintains five additional object states: temperature, wetness, cleanliness, toggled, and sliced state.The simulator uses one value for each extended state per simulated object, including rigid, flexible, and articulated objects.
- Temperature: Temperature is updated through annotated heat sources and sinks, including proximal, containment-based, and toggle-dependent mechanisms.Sources raise temperature toward a heating temperature, while sinks lower it toward a cooling temperature at category-specific rates.
- Temperature: Historical maximum temperature preserves cooked or burned appearance even after an object's current temperature decreases.The simulator records the maximum temperature reached over time and uses it to determine persistent appearance changes.
- Wetness: Wetness is represented by absorbed droplets, while droplets can be created, destroyed, absorbed, contained, and poured.This particle-based system approximates liquid behavior for soakable objects and receptacles.
- Cleanliness: Dust and stains decrease as particles are removed, with dust requiring dry cleaning and stains requiring soaked cleaning tools.Cleanliness reaches 0% when no particles remain on the object's surface.
- Functional states: Toggling changes binary functional states, while slicing replaces a sliceable object with two halves that inherit its extended states.Toggled states can alter appearance or activate processes such as heating food in a microwave.
4 Logical Representation of Physical States
iGibson 2.0 bridges continuous physical simulation and household-language descriptions with logical predicates, then uses those predicates to generate valid physical states and populated scenes.
- Logical representation: Logical predicates map extended physical states to semantically meaningful properties such as Cooked, Soaked, Dusty, and Stained.The predicate set also covers spatial relations, articulated-object states, temperature states, wetness, cleanliness, toggling, and slicing.
- Logical representation: The logical engine is both discriminative and generative: it evaluates physical states and samples physical instances satisfying symbolic predicates.This supports task initialization from logical descriptions rather than manually specifying every physical state.
- Generative system: Sampling procedures depend on predicate type, using state values or particles for extended-state predicates and validated poses for kinematic predicates.Kinematic sampling checks conditions such as non-penetration and stationarity using ray-casting and analytical methods.
- Scene generation: The generative system simplifies dense scene creation by allowing users to specify logical predicates representing realistic object distributions.Semantic rules assign probabilities to object categories in container and room types, producing more populated scenes.
5 Virtual Reality Interface
iGibson 2.0 adds VR for immersive human control and demonstration collection in the same household scenes used by AI agents. Its evaluation applies the interface to novel extended-state tasks and bimanual imitation learning.
- VR interface: The VR interface supports human demonstrations for new household tasks and is compatible with major commercially available VR headsets through OpenVR.The interface is intended to support imitation-learning solutions.
- Evaluation: Evaluation creates six novel tasks requiring extended-state manipulation and uses VR demonstrations to train an imitation-learning policy for a bimanual task.The logical engine detects task completion and generates multiple task instances for training.
- Immersive control: Humans control an avatar's head and hands through the headset and hand controllers while receiving stereo images from the avatar's viewpoint.An optional tracker can control the avatar's main body.
- Immersive control: Assistive grasping adds a palm-to-object constraint after a 50% actuation threshold when the object contacts the hand.The constraint helps grasp small objects and can break during excessive load or acceleration.
- Navigation: A controller touchpad translates the avatar between rooms because the physical VR space is smaller than typical iGibson scenes.Avatar navigation otherwise follows the human's locomotion.
6 Evaluation
The evaluation tests iGibson 2.0 on six novel household tasks using reinforcement learning and on bimanual imitation learning from VR demonstrations. Results show strong performance on simpler tasks, but substantial challenges for slicing, bimanual coordination, realistic grasping, and long-horizon imitation.
- Experimental Setup: The evaluation covers six novel tasks spanning kinematic, liquid, cleanliness, temperature, slicing, and bimanual manipulation capabilities.The tasks include Grasping Book, Soaking Towel, Cleaning Stained Shelf, Cooking Meat, Slicing Fruit, and Bimanual Pick and Place.
- RL experiments: SAC-trained bimanual humanoid and Fetch agents achieve 100% success on Grasping Book, Soaking Towel, Cleaning Stained Shelf, and Cooking Meat.The agents receive RGB-D and proprioception observations and output hand velocities.
- RL experiments: 15% and 0% success on Slicing Fruit for the bimanual humanoid and Fetch robots, respectively, reflects the precision required to align the knife blade with the fruit.The bimanual humanoid also achieves 0% on Bimanual Pick and Place because controlling and coordinating both hands is difficult.
- RL experiments: Without simplified grasping, the Fetch robot achieves 25% success on 2 tasks and 0% on the other 3, highlighting diverse-object grasping as a major challenge.This evaluation uses more realistic grasping conditions than the simplified-grasping experiments.
- RL experiments: Training with no, pose-only, or object-and-pose variability yields 19%, 79%, and 87% success on an unseen Soaking Towel setup.The ablation supports training with diverse object models and initial states for robust generalization.
- IL experiments in Bimanual Pick and Place with VR demonstrations: The full Bimanual Pick and Place imitation policy fails after 30 demonstrations, but succeeds 19% and 46% when initialized 6 seconds and 3 seconds before completion.The authors attribute full-task divergence to covariate shift from differing long-task demonstration strategies.
7 Conclusion
The paper concludes that iGibson 2.0 broadens household-task simulation through extended object states, logical and generative mechanisms, and VR demonstration collection. Experiments demonstrate new research opportunities for embodied AI and robot learning.
- 7 Conclusion: iGibson 2.0 combines object-centric extended states, logical predicates with generative world construction, and a VR interface for collecting human demonstrations.The listed states include temperature, wetness, and cleanliness level.
- 7 Conclusion: The authors present these capabilities as facilitating novel embodied AI solutions and making iGibson 2.0 a useful open-source tool for the community.
A.1 The iGibson 2.0 Virtual Reality Interface
The VR interface maps human motion to a bimanual virtual embodiment, provides haptic and assistive grasping support, and logs demonstrations for deterministic replay. These mechanisms target natural interaction and imitation-learning data collection.
- Mapping human motion to the virtual embodiment: VR controls a bimanual embodiment consisting of a head, two hands, and a torso using a headset, hand controllers, and optionally a torso tracker.Head pose is directly set from the headset pose, bypassing physics simulation for head motion.
- Haptic feedback: Collisions trigger controller vibrations, with strong feedback for body contact and lower-strength feedback when the hands contact objects.
- Assistive Grasping: Assistive grasping activates when a trigger exceeds 50% depression and creates a joint between the hand and a nearby contacted object meeting containment and force criteria.The mechanism uses ray casting, palm proximity, and hand-contact force to select the assisted object.
- Assistive Grasping: The assistive grasping connection breaks below 50% trigger depression or beyond a 10 cm object-to-palm distance, limiting unrealistic pulling and enabling small-object dexterity.
- Hardware compatibility: The interface supports HTC Vive, Oculus Rift S, and Oculus Quest, rendering 1296 x 1440 images at 30 Hz for the immersive experience.The renderer is capable of up to 90 Hz, but the displayed frame rate was reduced for the tested devices.
- Logging and replaying demonstrations: Logged kinematics, extended states, and VR actions can be replayed deterministically and reused to analyze human strategies or train imitation-learning systems.
A.2 Extended Object States, Logical Predicates and Generative System in iGibson 2.0
iGibson 2.0 represents object properties and extended states, updates them during simulation, and connects them to semantic logical predicates. Generative functions then sample physical configurations satisfying logical descriptions, including constrained object poses.
- Extended states associated to object categories: Object categories specify which extended states apply, because states such as Sliced and Cooked are relevant only to particular object types.Object instances are annotated with category properties indicating which states the simulator updates.
- Object model annotations: Object models require annotations such as shape, kinematic structure, weight, center of mass, and moment of inertia to simulate interactions and logical states.
- Updating object state: During simulation, Bullet updates kinematic states while custom rules update non-kinematic states such as temperature and wetness.
- Logical predicates as discriminative functions: Discriminative logical predicates map extended physical states to semantic states for symbolic planning and intermediate reinforcement-learning success checks.Examples include Cooked and Sliced.
- Logical predicates as generative functions: Generative functions sample valid physical states from logical requirements, such as placing a book OnTopOf a table or sampling stain particles on a shelf.This supports automatic creation of concrete task instances from logical descriptions.
- Pose Sampling Algorithm: Pose sampling must satisfy stable-orientation, support-surface, non-penetration, and other kinematic constraints, making it more involved than sampling scalar extended states.The algorithm begins by querying stable orientations for the sampled object.
- Appendix reference tables: The appendix organizes the system into tables of discriminative predicates, category-associated states, annotated model properties, maintained object states, and generative predicates.
A.3 Experimental Setup and Additional Results
The experiments specify reinforcement-learning setups for Bimanual Humanoid and Fetch robots, plus imitation learning from human demonstrations. The imitation policy is also tested under task-specific initialization because long-horizon execution causes divergence.
- Reinforcement Learning: RL observations combine 128 × 128 RGB-D head-camera images with proprioceptive information, while actions specify hand or end-effector linear and angular velocities.The Bimanual Humanoid action space controls one hand; the Fetch setup uses end-effector control.
- Reinforcement Learning: Success rewards are based on satisfying task predicates, with distance shaping and cleaning-progress rewards added to encourage intermediate progress.The Cleaning Stained Shelf task additionally rewards each stain particle that is cleaned.
- Reinforcement Learning: Soft Actor-Critic encodes RGB-D observations with a convolutional network and proprioception with an MLP before producing actions through additional MLP layers.Each encoder produces a 256-dimensional feature vector before concatenation.
- Imitation Learning: The imitation-learning setup gives the bimanual humanoid ground-truth task-object poses and proprioception, with both hands controlled through 12 degrees of freedom.In Bimanual Pick and Place, relevant objects include the cauldron, table, and agent.
- Imitation Learning: 30 human demonstrations produced 6500+ state-action pairs for behavior cloning, using separate MLP encoders for proprioception and object poses.The encoded features are concatenated and passed through additional MLP layers to generate actions.
- Imitation Learning: The long-horizon task exceeds 300 steps, and the policy diverges under covariate shift; evaluation therefore tests initialization a few seconds before successful completion.A successful sequence is shown after rewinding 2 seconds from task completion.
A.4 Performance Benchmark of iGibson 2.0
The benchmark compares iGibson 2.0 with iGibson 1.0 under a controlled idle simulation setup and also reports additional task-learning results. iGibson 2.0 improves simulation speed despite tracking extended physical states, while task difficulty varies across robots and tasks.
- Benchmark Setup: The benchmark runs 15 scenes with a stationary TurtleBot, zero actions, physics and extended-state simulation, and 512 × 512 RGB-D rendering on fixed hardware.The setup uses action timestep ta = 1/30s and physics timestep ts = 1/120s in a single process.
- Benchmark Results: 25% higher average simulation performance is achieved by iGibson 2.0 than iGibson 1.0 despite adding extended physical states.The reported speedup mainly comes from improved object sleeping and lazy renderer pose updates.
- Additional RL Results: Bimanual Humanoid RL achieves perfect reward on tasks 1)-4), but Slicing Fruit reaches 15% success because precise knife-fruit alignment is required.Bimanual Pick and Place fails, although the agent receives partial reward for approaching the cauldron.
- Additional RL Results: Without the sticky-mitten grasping simplification, Fetch RL reaches around 25% success for Grasping Book and Soaking Towel and 0% for the other three tasks.With the simplification, Fetch achieves results similar to the Bimanual Humanoid experiments.
- Additional IL Results: A bimanual imitation sequence successfully places a heavy object on a table when the policy takes over after replay begins 2 seconds before completion.The sequence uses a blue replay segment followed by a green policy-control segment.
- Limitations: iGibson 2.0 does not yet support scalable soft-body simulation or accurate human behavior modeling, limiting folding, bed-making, and intrinsically rich human-robot interaction tasks.The authors identify these capabilities as future work alongside additional object states and real-world policy transfer.