Source-linked AI summary
Gibson Env: Real-World Perception for Embodied Agents
Fei Xia, Amir Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, Silvio Savarese
TL;DR
Active-agent perception is difficult to learn in the physical world, while simulation must address transfer to real cameras. The paper proposes Gibson, which virtualizes scanned real spaces and uses embodied agents, fast rendering, and Goggles to bridge rendered and real images. It demonstrates learned active perceptual tasks, including stair climbing, while acknowledging limitations in dynamics, manipulation, physics fidelity, and the breadth of real-world transfer evaluation.
Problem
Learning visual perception for active agents in the physical world is costly and constrained, while simulation raises questions about semantic and visual similarity to the real world.
Method
Gibson trains and tests embodied agents in scanned real spaces using faster-than-real-time rendering and Goggles, jointly learned forward and backward functions that bridge rendered and real images.
Results
The environment supports active perceptual tasks, and a perceptual stair-climbing agent outperformed a non-perceptual agent by 70% under shifted initial and target locations.
Takeaways & Limitations
Gibson provides a real-space-based environment for developing and evaluating active perceptual agents with physics and spatial constraints.
Takeaways & Limitations
Gibson lacks dynamic content and manipulation, has incomplete material properties and possible physics-related domain gaps, and evaluates Goggles transfer mostly on static recognition tasks.
Abstract
from arXiv · showhide
Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we are concerned with the problem of developing real-world perception for active agents, propose Gibson Virtual Environment for this purpose, and showcase sample perceptual tasks learned therein. Gibson is based on virtualizing real spaces, rather than using artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism, "Goggles", enabling deploying the trained models in real-world without needing further domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.
1 Stanford University 2 University of California, Berkeley
Gibson is named after J. J. Gibson and depicts embodied agents receiving visual observations in real-world spaces. The paper also acknowledges equal author contributions.
- Agents in Gibson are active, embodied, and constrained by physics and space while receiving visual observations from real-world spaces.They can also receive depth, semantic labels, or surface normals.
- Gibson is named after J. J. Gibson, author of Ecological Approach to Visual Perception.The passage quotes Gibson’s view that perception and movement are mutually related.
- The authors contributed equally.
1. Introduction
The paper motivates Gibson as a real-world virtual environment for training active perceptual agents, addressing the limits of physical learning and simulation-to-reality transfer. Gibson uses scanned real spaces, embodied agents, fast rendering, and Goggles, then demonstrates active perceptual tasks while limiting claims primarily to vision.
- Motivation: Physical-world learning is constrained by real-time speed, logistical cost, rare-event coverage, and the fragility and expense of robots.Static computer-vision datasets are also passive and unsuitable for active agents.
- Motivation: Simulation raises whether semantic complexity and rendered observations sufficiently match the real world.
- Gibson Environment: Gibson places arbitrary embodied agents in diverse scanned real spaces, enforces physics and spatial constraints, and supplies arbitrary-viewpoint visual observations.Its rendering engine operates faster than real-time and can use sparse scans such as one panorama per 5–10 m2.
- Goggles: Goggles bridges rendered and real-image domains through jointly trained forward and backward neural rendering functions.The backward function maps real images toward renderings and serves as deployment-time corrective processing.
- Validation and Scope: Gibson showcases local planning, distant navigation, and visual stair climbing, while the paper focuses on vision rather than general claims about physics simulation.
2. Related Work
Related work spans active-agent control, virtual environments, domain adaptation, and view synthesis. Gibson combines geometric rendering from sparse RGB-D panoramas with neural correction and domain-bridging functions for real-world transfer.
- Active Agents and Control: Active-agent research includes classical control and learned visuomotor policies using imitation, reinforcement learning, prediction, and self-supervision.
- Virtual Environments for Learning: Virtual environments address the limitations of passive static and prerecorded video datasets for active-agent learning.Prior environments include computer games such as Minecraft, Doom, and GTA5.
- Domain Adaptation and Transferring to Real-World: Simulation-to-real transfer has used domain randomization and joint spaces; Gibson leverages paired source-target data for forward and backward models.
- View Synthesis and Image-Based Rendering: Gibson’s view synthesis combines geometric rendering, image-based selection, neural artifact correction, dis-occlusion filling, and a backward mapping for real images.The pipeline starts from sparse RGB-D panoramas and renders arbitrary novel viewpoints.
3. Real-World Perceptual Environment
Gibson combines real-space assets, geometric and neural view synthesis, and physics-based embodiment to support perceptual agents in realistic environments. Its view-synthesis pipeline addresses artifacts and domain differences between rendered and real images.
- Real-space database: Gibson’s database contains 572 full buildings and 1,447 floors, with RGB panoramas, camera poses, and reconstructed 3D meshes.The spaces cover 211k m2 and provide the underlying real-world assets for the environment.
- View synthesis: The view-synthesis module converts sparse RGB-D panoramas into panoramas from arbitrary novel camera viewpoints.It combines geometric point-cloud rendering with a neural network that repairs artifacts and fills dis-occluded regions.
- View synthesis: Geometric rendering aggregates transformed point clouds from nearby reference views using density-based weights before bilinear interpolation.Adaptive selection reduces lighting inconsistency and misalignment artifacts compared with blindly superimposing points.
- View synthesis: The neural filler network repairs stitching marks, deformed objects, and large dis-occluded regions remaining after geometric rendering.The approach uses stochastic identity initialization and color-moment matching within a perceptual loss.
- Closing the gap with real-world: Goggles trains forward and backward image mappings jointly so rendered and real images correspond despite a non-bijective domain mapping.The backward function transforms real images toward the rendered domain, while the forward function transforms renderings toward target images.
- Physics and embodiment: Gibson integrates a physics engine and embodied agents, including physically plausible humanoid motion and corresponding visual observations.The environment’s integrated agents and physics engine are described as core components alongside the scene database and view synthesis.
4. Tasks
Gibson evaluates active perceptual control and static recognition using agents operating from visual observations in diverse spaces. Its static tasks test whether models trained on renderings transfer to real images and whether Goggles helps bridge the domains.
- Active Perceptual Tasks: The active tasks cover local obstacle avoidance, distant visual navigation, and visually controlled stair descent.Agents must plan or control movement from visual inputs under task-specific spatial or dynamic constraints.
- Active Perceptual Tasks: Local planning uses depth and/or RGB streams to reach a nearby relative-coordinate target while avoiding obstacles.The agent receives no other environment information and may need to route around objects such as couches.
- Active Perceptual Tasks: Distant navigation fixes the target farther away and requires the agent to form a contextual map without odometry or GPS.The randomized initial location makes the task analogous to auto-docking from a distant position.
- Static Recognition Tasks: Static recognition comprises monocular depth estimation and scene classification, trained on rendering-ground-truth pairs and tested on real-image-ground-truth pairs.This setup assesses rendering-to-real transfer; scene labels are generated with a network trained on Places because Gibson images lack scene-class annotations.
- Static Recognition Tasks: Depth estimation compares networks receiving pre-neural renderings, post-neural renderings, real images processed by Goggles, or unprocessed real images.The four inputs are Is, f(Is), u(It), and It, respectively.
5. Experimental Results
Gibson is evaluated as a diverse real-space database, a view-synthesis system, and a platform for transferring perception to real images and active tasks. Across these evaluations, Goggles narrows the rendering-to-real gap, while perceptual agents outperform or generalize better than non-perceptual agents on several tasks.
- 5.1. Benchmarking Space Databases: Gibson’s database spans diverse real spaces and is benchmarked using clutter, navigation complexity, real-world transfer error, and scene diversity.The database includes offices, garages, stadiums, grocery stores, gyms, hospitals, and houses; scene diversity entropy is 3.72 for Gibson versus 2.89 for SUNCG and 3.32 for THOR.
- 5.2. Evaluation of View Synthesis: The view-synthesis pipeline combines geometric rendering with neural correction and Goggles to reduce artifacts and align rendered and real imagery.Qualitative results show improved post-correction renderings, while the smallest appearance gap is between f(Is) and u(It).
- 5.3. Transferring to Real-World: Goggles produced the closest cross-domain depth-transfer result to real-image training, with 0.91 error versus the 0.86 real-image train-test reference.Distributional measurements also found f(Is)-u(It) smaller than other compared pairs for both MMD and CORAL.
- 5.3. Transferring to Real-World: Nearest-neighbor retrieval preserved correspondence after Goggles, with u(It)→It accuracies of 85.9%, 87.2%, and 89.6% for top-1, top-2, and top-5.These results were used to indicate that the synthesis networks did not collapse images to a few points.
- 5.4. Validation Tasks Learned in Gibson: Perceptual agents learned effective active behaviors: obstacle avoidance improved local-planning reward, visual state enabled distant navigation, and stair-climbing generalization exceeded the non-perceptual agent by 70%.Global navigation emerged after 1700 episodes, and perceptual stair climbing showed better test-time robustness to location shifts despite slower learning.
6. Limitations and Conclusion
Gibson is presented as a basis for real-world perception in active agents, but its current scope and transfer evaluation remain limited.
- Gibson lacks dynamic content and does not support manipulation, limiting its current coverage of active-agent tasks.The authors suggest integrating synthetic objects as a potential solution.
- Incomplete material properties and non-optimal physics simulation may create physics-related domain gaps.
- Goggles has been quantitatively evaluated mostly on static recognition tasks rather than real-robot deployment.The authors identify evaluation on real robots as the ultimate test.