PaperlayerView exact revision on arXiv

Source-linked AI summary

Gibson Env: Real-World Perception for Embodied Agents

Fei Xia, Amir Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, Silvio Savarese

arXiv:1808.10654v1cs.AIcs.CVcs.GRcs.LGcs.RO

TL;DR

Active-agent perception lacks suitable passive datasets, and simulation leaves real-world transfer in question. Gibson trains embodied agents in scanned real spaces with Goggles; its perceptual stair-climbing agent outperformed its non-perceptual counterpart by 70% under test-location shifts.

  • Problem

    Active-agent perception lacks suitable observations in conventional passive, static datasets, while simulation raises the question of transfer to real-world images.

    Source: §1 · p. 2

  • Method

    Gibson places embodied agents in scanned real spaces under physics and spatial constraints, using Goggles to align rendered and real-camera images.

    Source: §1 · p. 2

  • Results

    70%: in stair climbing, the perceptual agent outperformed the non-perceptual agent under randomized test-location shifts.

    Source: §5.4 · p. 8

  • Takeaways & Limitations

    The authors present Gibson as a basis for learning complex navigation and locomotion.

    Source: §6 · p. 8

    Quantitative evaluations of Goggles transfer to the real world were conducted mostly on static recognition tasks.

    Source: §6 · p. 8

Abstract

from arXiv · show

Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we are concerned with the problem of developing real-world perception for active agents, propose Gibson Virtual Environment for this purpose, and showcase sample perceptual tasks learned therein. Gibson is based on virtualizing real spaces, rather than using artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism, "Goggles", enabling deploying the trained models in real-world without needing further domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.

1. Introduction

Gibson is a virtual environment for training and testing real-world perceptual agents, built from scanned real spaces and designed to help models transfer from simulation to real cameras. It combines physically embodied agents and fast visual rendering with Goggles, a mechanism for bridging rendered and real images.

Introduction

  • Developing principled perception models for active agents is motivated by their compound perceptual and physical capabilities and proposed links between perception and physical activity.The introduction cites arguments from evolutionary and computational biology, neuroscience, and robotics.Source: §1 · p. 1
  • Simulation offers an alternative to passive datasets and costly, real-time-limited physical learning, but transfer requires semantic complexity and visual realism.Gibson addresses these concerns by basing environments on scanned real spaces and reducing differences between simulated and camera images.Source: §1 · p. 2
  • Gibson embodies arbitrary agents in diverse scanned real spaces, enforces physical and spatial constraints, and streams camera-like views from arbitrary viewpoints.Its rendering engine runs faster than real-time and works with sparse scans, such as one panorama per 5–10 m².Source: §1 · p. 2
  • Goggles bridges Gibson renderings and real-camera images by jointly training forward and backward rendering functions to produce equal outputs.The backward function acts as deployment-time corrective glasses for the agent.Source: §1 · p. 2
  • The paper showcases local obstacle-avoidance planning, distant navigation, and visual stair climbing learned in Gibson, while focusing on vision rather than claiming generality to physics simulation.The authors caution that their statements need not generalize to other aspects of virtual-environment learning, such as physics simulation.Source: §1 · p. 2

2. Related Work

Gibson positions itself as a virtual environment for active-agent perception, using real spaces, a custom neural view synthesizer, and a built-in mechanism to reduce the need for custom domain adaptation. Its related work spans active-agent control, virtual environments, domain adaptation, and view synthesis.

Active Agents and Control

  • Active-agent control includes non-learning methods and learned visuomotor policies using imitation [73], reinforcement [78] [44] [77], prediction [30], or self-supervision [67] [30] [66] [46].These methods are potential users of Gibson and other virtual environments.Source: §2 · p. 2

Virtual Environments for Learning

  • Gibson targets perception with real spaces, a custom neural view synthesizer, and baked-in Goggles adaptation, addressing visual-world oversimplification in many virtual environments.Static vision datasets [61] and passive video datasets [101] are limited for active agents; many simulators use synthetic databases or deficient rendering pipelines.Source: §2 · p. 2

Domain Adaptation and Transferring to Real-World

  • Prior domain-transfer work includes domain randomization [74] [93] and joint spaces; Gibson uses paired source-target data to train forward and backward models for baked-in adaptation.This mechanism is intended to minimize the need for additional, custom adaptation.Source: §2 · p. 2

View Synthesis and Image-Based Rendering

  • Gibson combines geometric rendering of a target view with neural correction of artifacts and dis-occlusions, plus a jointly trained backward mapping from real images to synthesized ones.Figure 2 shows the pipeline starting from sparse RGB-D panoramas and selecting and interpolating pixels before neural completion.Source: §2 · p. 3, Figure 2 · p. 3

3. Real-World Perceptual Environment

Gibson builds perception for embodied agents from scans of real spaces, combining novel-view synthesis and real-to-rendered image alignment with agents that experience physical constraints. It also provides geometric, semantic, and proprioceptive sensor modalities.

3.1. Gibson Database of Spaces

  • The database contains 572 full buildings, 1,447 floors, and 211k m², represented by RGB panoramas with global camera poses and reconstructed 3D meshes.Its base format resembles 2D-3D-Semantics [10], but it includes two orders of magnitude more spaces.Source: §3.1 · p. 3

3.2. View Synthesis

  • Gibson renders arbitrary novel viewpoints from sparse RGB-D panoramas by combining geometric point-cloud rendering with a neural network that repairs artifacts and fills dis-occlusions.The geometric output can retain stitching marks, deformed objects, and large dis-occluded regions; the network uses perceptual loss with color-moment matching.Source: §3.2 · pp. 3–4

3.2.1 Closing the Gap with Real-World: Goggles

  • Goggles jointly aligns rendered and real-image domains by correcting real images toward corresponding Gibson renderings, while a one-to-one constraint prevents collapse.The approach addresses the remaining domain gap because imperfections in 3D inputs and geometric renderings make fully photorealistic output implausible.Source: §3.2.1 · pp. 4–5

3.3. Embodiment and Physics Integration

  • Gibson integrates PyBullet and supports arbitrary URDF agents, while its controllers can abstract low-level control and agent motion dynamics.The platform provides rigid- and soft-body simulation, collision handling, and PID, nonholonomic, and ideal positional controllers.Source: §3.3 · p. 5

3.4. Additional Modalities

  • Gibson adds mesh-rendered depth and surface normals, semantic annotations for 52,561 m², and configurable proprioceptive sensory data.Semantic annotations come from 2D-3D-S [10] and Matterport3D [18].Source: §3.4 · p. 5

4. Tasks

Gibson defines Gym-style agent interaction and validates the environment with active perceptual tasks and static-recognition tasks. Together, these tasks test navigation, visually controlled movement, and transfer of image-based predictions to real images.

4. Tasks

  • At each timestep, agents act in the environment, which advances with physics and returns a rendered observation, reward, and termination signal through a Gym-style interface [14].Utilities also support keyboard control and visualization of recorded runs.Source: §4 · p. 5

4.1. Experimental Validation Tasks

  • Agents must navigate to a nearby relative-coordinate target using continuous depth and/or RGB frames, without other environmental information, and avoid obstacles such as couches.Source: §4.1 · p. 5
  • For distant navigation to a fixed target, agents start at randomized locations without odometry or GPS and must form a contextual map to succeed.The task resembles robot auto-docking from a distant location.Source: §4.1 · p. 5
  • An ant agent uses visual inputs to learn a controller for descending stairs without flipping.Source: §4.1 · p. 5
  • Static-recognition evaluation trains on Gibson renderings paired with ground truth and tests on real images to assess rendering similarity and Goggles’ impact.The protocol compares u(It) with Is, f(Is), and It.Source: §4.1 · pp. 5–6
  • Depth estimation predicts depth from a single RGB image [33], while scene classification predicts classes generated by a Places-trained network.Depth models use Is, f(Is), u(It), or It as input, with performance compared in Section 5.3.Source: §4.1 · p. 6

5. Experimental Results

Gibson’s experiments evaluate its space database, view synthesis, transfer to real imagery, and perception-enabled active tasks. They report close real-image depth-transfer performance with Goggles and successful learning across navigation, planning, and stair-climbing tasks.

5.1. Benchmarking Space Databases

  • Gibson’s scene-diversity entropy was 3.72, versus 2.89 for SUNCG and 3.32 for THOR, using 10k images and a maximum entropy of 5.90.The database benchmarks also define specific surface area as a clutter measure, navigation complexity using A* path length relative to straight-line distance, and real-world transfer error using depth estimation on 2D-3D-S images.Source: §5.1 · p. 6

5.2. Evaluation of View Synthesis

  • Post-network renderings partially resolve geometric artifacts, and Goggles makes real images more like renderings while f and u suppress poorly reconstructed fixtures and whiten window scenes.Post-network images still have lower contrast and different color distributions than real images; Figure 6 compares renderings before and after correction with Goggles-transformed and ground-truth images.Source: §5.2 · p. 7, Figure 6 · p. 7

5.3. Transferring to Real-World

  • Depth error was 0.91 when training on f(Is) and testing on u(It), closest to the 0.86 real-image train-and-test gold standard.Figure 7(a) shows depth-estimation errors for feasible train-test combinations; the passage identifies the 0.91 pairing as the closest to the gold standard.Source: §5.3 · p. 7, Figure 7 · p. 8
  • MMD and CORAL distances were smaller for f(Is)–u(It) than for other pairs, while retrieval reached 91.6%, 93.5%, 95.6% for f(Is)→Is and 85.9%, 87.2%, 89.6% for u(It)→It.The authors use these nearest-neighbor accuracies to indicate that correspondence is preserved and the networks do not collapse images to a few points.Source: §5.3 · pp. 7–8, Figure 7 · p. 8

5.4. Validation Tasks Learned in Gibson

  • The perceptual husky agent achieved a higher score than the non-perceptual agent and developed obstacle avoidance to reach local-planning goals faster.Both agents were trained for 150 episodes; local planning and visual obstacle avoidance included a collision penalty.Source: §5.4 · p. 8
  • Global navigation emerged after 1700 episodes (680k frames), and only the agent with visual state completed the distant-navigation task.Figure 9 shows sample initial locations and the target alongside reward curves.Source: §5.4 · p. 8, Figure 9 · p. 8
  • For the active-task policy, training on f(Is) and testing on u(It) yielded a 0.204 action-logit discrepancy, versus 0.300 for testing on It and 0.242 for training on Is and testing on It.The discrepancy is measured as L2 distance between output action logits at locations with real images available.Source: §5.4 · p. 8
  • The perceptual ant learned stair-climbing more slowly but generalized better to shifted initial and target locations, outperforming the non-perceptual agent by 70%.Both agents began acquiring stair-climbing skills after 1700 episodes (700k time steps); the test locations were shifted slightly and randomly.Source: §5.4 · p. 8

6. Limitations and Conclusion

Gibson Environment was presented for developing real-world perception for active agents and validated on a set of tasks. The authors identify limitations in its content, physics fidelity, and evaluation of Goggles on real robots.

6. Limitations and Conclusion

  • Gibson provides a basis for learning complex navigation and locomotion, but currently lacks dynamic content and manipulation support.The authors suggest integrating synthetic objects [19] as a potential solution.Source: §6 · p. 8
  • Gibson lacks full material properties, and no existing physics simulator is optimal, potentially creating physics-related domain gaps.Source: §6 · p. 8
  • Goggles was evaluated quantitatively for real-world transfer mostly on static recognition tasks; its ultimate test is evaluation on real robots.Source: §6 · p. 8

Related papers

Loading 1808.10654v1…