Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments

Fei Xia; William B. Shen; Chengshu Li; Priya Kasimbeg; Micael Tchapmi; Alexander Toshev; Li Fei-Fei; Roberto Martín-Martín; Silvio Savarese

Source-linked AI summary

Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments

Fei Xia, William B. Shen, Chengshu Li, Priya Kasimbeg, Micael Tchapmi, Alexander Toshev, Li Fei-Fei, Roberto Martín-Martín, Silvio Savarese

arXiv:1910.14442v3Oct 30, 2019cs.ROcs.AIcs.CVcs.LG

TL;DR

Interactive Navigation lacks benchmarks that jointly evaluate reaching goals and physically disturbing cluttered environments. The paper introduces Interactive Gibson and the Interactive Navigation Score, then evaluates learning-based baselines that reveal trade-offs between path efficiency and interaction effort. The benchmark supports realistic, scalable study of these behaviors across indoor environments.

Problem
Interactive Navigation requires metrics and benchmarks that evaluate both navigation quality and the effort of manipulating objects, which conventional navigation metrics do not capture.
Method
The paper builds a photorealistic Interactive Gibson Environment with interactable objects, defines the Interactive Navigation Score, and evaluates reinforcement-learning baselines.
Results
Different interaction penalties produce distinct behaviors trading path efficiency against effort efficiency, while baseline performance generalizes to unseen environments without a statistically significant difference.
Takeaways & Limitations
Interactive Gibson provides a benchmark for comparing Interactive Navigation solutions across navigation success, path quality, and disturbance effort.

Abstract

from arXiv · show

We present Interactive Gibson Benchmark, the first comprehensive benchmark for training and evaluating Interactive Navigation: robot navigation strategies where physical interaction with objects is allowed and even encouraged to accomplish a task. For example, the robot can move objects if needed in order to clear a path leading to the goal location. Our benchmark comprises two novel elements: 1) a new experimental setup, the Interactive Gibson Environment (iGibson 0.5), which simulates high fidelity visuals of indoor scenes, and high fidelity physical dynamics of the robot and common objects found in these scenes; 2) a set of Interactive Navigation metrics which allows one to study the interplay between navigation and physical interaction. We present and evaluate multiple learning-based baselines in Interactive Gibson, and provide insights into regimes of navigation with different trade-offs between navigation path efficiency and disturbance of surrounding objects. We make our benchmark publicly available(https://sites.google.com/view/interactivegibsonenv) and encourage researchers from all disciplines in robotics (e.g. planning, learning, control) to propose, evaluate, and compare their Interactive Navigation solutions in Interactive Gibson.

I. INTRODUCTION

Interactive Navigation extends robot navigation to cluttered indoor environments where physical interaction can be necessary. The paper introduces a realistic, scalable benchmark, metrics, and learning-based baselines to study trade-offs between path efficiency and environmental disturbance.

Motivation: Interactive Navigation allows robots to push objects or open doors when interaction is necessary to reach goals in cluttered homes and offices.This differs from classic navigation, which primarily emphasizes reaching goals while avoiding collisions.
Motivation: Physics simulation can reduce safety risks and support research on robot–object dynamics, but handcrafted models lack the visual realism and complexity of real-world spaces.The paper motivates combining physical interaction simulation with photorealistic indoor environments.
Contributions: The Interactive Gibson Environment preserves Gibson’s photorealism and scale while adding realistic object interactions such as pushing and opening doors.Its faster rendering also makes large-scale reinforcement-learning training possible.
Contributions: Interactive Navigation Score unifies navigation success and path quality with disturbance effort proportional to displaced mass and applied force.INS captures the trade-off between taking a longer path and disturbing the environment.
Baselines: Learning-based baselines modulate interaction penalties in the reward to produce different Interactive Navigation behaviors and scores.Agents can trade longer paths against pushing heavier objects.

II. RELATED WORK

Prior navigation benchmarks and metrics largely assume collision avoidance, while Interactive Navigation requires reasoning about object manipulation and the trade-off between interaction effort and detours. The paper situates its benchmark between geometric planning, realistic simulation, and reproducible evaluation.

Interactive Navigation: Interactive Navigation has received less attention than autonomous navigation, while prior mobile-manipulation work often isolates individual interactions such as opening doors.The paper frames Interactive Navigation as an entire task combining movement and environmental interaction.
Interactive Navigation: NAMO plans pushing and pulling in 2D with known object and environment geometry, whereas this work studies realistic 3D interaction under partial observation.The comparison identifies the paper’s broader physical and perceptual setting.
Benchmarking: Existing navigation benchmarks face trade-offs among perceptual isolation, costly physical reproduction, limited remote-robot scalability, and infrequent competitions.These constraints motivate a more accessible simulation-based benchmark.
Navigation Metrics: Conventional success, safety, and path-quality metrics label object manipulation as collision and cannot compare interaction effort with taking a longer route.Interactive Navigation therefore requires metrics that represent both path quality and environmental disturbance.
Robot Simulation Environments: Simulation environments offer scalable and reproducible development settings, but game-engine approaches may sacrifice physical fidelity and often script interactions.Scanned environments provide photorealistic real-world layouts but can contain reconstruction artifacts.

III. THE INTERACTIVE GIBSON ENVIRONMENT

Interactive Gibson is designed as a reproducible simulation test bed for Interactive Navigation, combining realistic rendering with dynamic, interactable scene assets. It extends Gibson V1 to support changing environmental states during navigation.

Motivation: Interactive Navigation benefits from a reproducible and controllable simulation environment because physical testing can create risks for robots.Simulation provides the paper’s proposed setting for safe evaluation and training.
Gibson V1: Gibson V1 provides photorealistic visual navigation and sim2real transfer but cannot model environmental state changes needed for Interactive Navigation.Its rendering and reconstructed assets do not support object interaction.
Interactive Gibson: Interactive Gibson adds a faster rendering engine for dynamic environments and interactable assets including doors, chairs, and tables.These changes support faster reinforcement-learning training and physical interaction with relevant object classes.

A. Interactive Gibson Renderer

Interactive Gibson replaces Gibson V1’s slow, interaction-incompatible image-based rendering with fast mesh rendering for dynamic Interactive Navigation environments.

Gibson V1’s image-based rendering achieves high photo-realism but runs at only 25-40 fps on modern GPUs.Rendering requires loading and processing images from available viewpoints on the fly, limiting speed for reinforcement learning.
Image-based rendering cannot support environments whose visual state changes when objects move or are added.This makes it unsuitable for interaction-driven tasks such as Interactive Navigation.
Interactive Gibson replaces image-based rendering with mesh rendering so visual agents can navigate and interact with objects.The mesh-based approach is designed for dynamic environments and faster training.
The Python and C++ rendering pipeline reaches up to 1000 fps at 256×256 resolution on common computers.The implementation uses PyOpenGL, PyOpenGL-accelerate, pybind11, and GPU tensor transfer to reduce overhead and memory copies.

B. Interactive Gibson Assets

Interactive Gibson converts static Gibson V1 reconstructions into photo-consistent scenes with separate CAD objects that can be physically interacted with in simulation.

Interactive Gibson augments 106 scenes with 1984 interactable CAD model alignments spanning chairs, desks, doors, sofas, and tables.Original textures are transferred to the CAD models to maintain visual fidelity.
Automated semantic segmentation and instance analysis generate coarse object proposals that human annotators refine and align to CAD models.The pipeline combines pretrained Minkowski segmentation, connected-component analysis, and Amazon Mechanical Turk annotation.
Over 4,000 object proposals resulted from the automated proposal-generation stage.
Annotators select ShapeNet models and align them using at least six keypoint correspondences, optimizing scale, position, and rotation.The alignment transformation is solved over seven parameters, while pitch and roll are predefined.
Aligned CAD models replace corresponding mesh regions, with holes filled and original textures transferred to produce photo-consistent interactable objects.Articulated objects such as doors are represented with links and joints in URDF models.

C. Interactive Gibson Agents

Interactive Gibson provides diverse embodied agents and multimodal observations for studying navigation with physical interaction.

The environment includes ten fully functional robotic agents, including eight models of real robot platforms.The collection spans humanoid, ant, wheeled, legged, mobile-manipulator, and quadrotor embodiments.
Agents can observe configuration, localization, velocity, collisions, odometry, and visual signals including RGB, semantics, surface normals, and depth.

IV. INTERACTIVE GIBSON BENCHMARK

The Interactive Gibson Benchmark evaluates agents navigating between random locations on a floor amid existing and randomly placed interactable objects under fixed episode conditions.

Episodes sample an environment, ten additional object locations, and start and goal locations, with the start and goal on the same floor at least 1 m apart.
Interactive scenes contain five replaced object categories—chairs, desks, doors, sofas, and tables—and additional scanned objects such as baskets, shoes, pots, and toys.The additional objects are randomly placed on the floor as obstacles.
Figure 4 compares annotated CAD replacements across ten reconstructed scenes, their retextured ShapeNet models, and randomly placed common objects.
Episodes terminate when the agent reaches the goal or runs out of time, with a 1,000-step limit.Goal convergence is defined by a distance below the agent’s body width.

Interactive Navigation Score

Interactive Navigation Score (INS) evaluates navigation by combining path efficiency with the effort spent disturbing the environment. It captures the trade-off between taking a shorter path and minimizing object displacement and applied interaction forces.

Path Efficiency measures how efficiently the robot reaches the goal relative to the ideal shortest path without movable obstacles.Unsuccessful navigation receives zero path efficiency, and the ideal score may be unattainable when immovable objects block the shortest path.
Effort Efficiency measures interaction cost using displaced mass and force applied to the environment, excluding locomotion forces from the floor.The metric captures both kinematic effort from displaced mass and dynamic effort from applied force.
INS combines Path Efficiency and Effort Efficiency through a convex combination.Both component scores lie in [0, 1].
Varying α controls the trade-off between path quality and interaction effort, with α = 1 corresponding to classical SPL.A short path can require pushing objects, whereas avoiding disturbance can require a longer route.

V. EVALUATING BASELINES ON INTERACTIVE GIBSON

The evaluation studies how reinforcement-learning baselines trade path efficiency against interaction effort in Interactive Gibson. Varying interaction penalties produces distinct behaviors, while SAC generally achieves the strongest TurtleBot scores and the baselines generalize to unseen environments.

Path–effort trade-off: For Fetch, kint = 1 yields higher Effort Efficiency but much lower Path Efficiency, whereas kint = 0.1 nearly preserves unpenalized Path Efficiency while improving Effort Efficiency.High penalties can cause agents to deviate from the shortest path or fail when blocked.
Baseline setup: The baselines use PPO, DDPG, and SAC on TurtleBot v2 and Fetch, training in eight environments and testing in two unseen environments.The robots differ substantially in size and weight, producing different interactions with scene objects.
Baseline setup: The baseline observations combine goal location, robot velocities, ten shortest-path waypoints, depth, and a 68 × 68 semantic segmentation mask.Actions are wheel joint velocities.
Reward and interaction control: The reward combines success, geodesic-distance progress, and an interaction penalty controlled by kint.The interaction term penalizes object interaction through Rint = −kint1int, while success provides a one-time reward of 10.
Interactive Navigation Score: For TurtleBot, SAC generally achieves the highest INS, with SAC at kint = 0.1 performing best across balances except when α = 0.At α = 0, interaction-minimizing agents perform best; at α = 0.5, SAC leads unless kint = 1.
Learned navigation behaviors: Higher interaction penalties make TurtleBot agents avoid progressively smaller objects, while lower penalties favor shorter paths through interactions.TurtleBots avoid very large objects even without penalties, whereas Fetch agents can move larger objects because of their greater size and power.
Generalization: The baselines generalize to unseen environments, with no statistically significant seen–unseen INS0.5 difference and a reported one-sample t-test p-value of 0.171.The authors relate this behavior to movable objects sharing the same classes across environments.

VI. CONCLUSION AND FUTURE WORK

Interactive Gibson Benchmark provides a photo-realistic, physically interactive benchmark for training and evaluating Interactive Navigation agents. Its INS metric captures the trade-off between path efficiency and effort efficiency, and the benchmark is intended for standardized comparison and future interactive tasks.

Conclusion and future work: Interactive Gibson includes a photo-realistic simulation environment with a new renderer and more than one hundred reconstructed real-world environments.Relevant object classes are annotated and replaced with high-resolution CAD models.
Conclusion and future work: The Interactive Navigation Score evaluates the trade-off between path efficiency and effort efficiency.The benchmark is publicly available for researchers to test and compare solutions under equal conditions.
Conclusion and future work: Future extensions will annotate additional object classes for interactive tasks such as interactive object search and retrieval.