Source-linked AI summary
Phone2Proc: Bringing Robust Robots Into Our Chaotic World
Matt Deitke, Rose Hendrix, Luca Weihs, Ali Farhadi, Kiana Ehsani, Aniruddha Kembhavi
TL;DR
Simulation-trained embodied agents often struggle to generalize to physical environments. Phone2Proc conditions procedural training scenes on a phone scan of the target space. Across 234 physical-world ObjectNav episodes, average success rate rises from 34.7% to 70.7% compared with ProcTHOR.
Problem
Simulation-trained embodied agents have shown limited success when deployed on physical robots, and must generalize across real scenes with varied layouts, objects, clutter, people, and lighting.
Source: §1 · pp. 1–2Method
Phone2Proc uses a phone scan of a target environment to condition a procedurally generated distribution of semantically similar training scenes.
Source: §1 · p. 2Results
Success rate improves from 34.7% to 70.7% on average across 234 physical-world ObjectNav episodes in five environments, compared with ProcTHOR.
Source: §1 · p. 2Takeaways & Limitations
The robot navigated effectively despite clutter, lighting changes, furniture shifts, and human movement.
Source: §1 · p. 2The model design targets RGB observations and avoids reliance on depth or localization inputs.
Source: §3.3 · p. 5
Abstract
from arXiv · showhide
Training embodied agents in simulation has become mainstream for the embodied AI community. However, these agents often struggle when deployed in the physical world due to their inability to generalize to real-world environments. In this paper, we present Phone2Proc, a method that uses a 10-minute phone scan and conditional procedural generation to create a distribution of training scenes that are semantically similar to the target environment. The generated scenes are conditioned on the wall layout and arrangement of large objects from the scan, while also sampling lighting, clutter, surface textures, and instances of smaller objects with randomized placement and materials. Leveraging just a simple RGB camera, training with Phone2Proc shows massive improvements from 34.7% to 70.7% success rate in sim-to-real ObjectNav performance across a test suite of over 200 trials in diverse real-world environments, including homes, offices, and RoboTHOR. Furthermore, Phone2Proc's diverse distribution of generated scenes makes agents remarkably robust to changes in the real world, such as human movement, object rearrangement, lighting changes, or clutter.
1. Introduction
Phone2Proc addresses the difficulty of transferring simulation-trained embodied agents to real environments by generating training scenes conditioned on a target environment. In 234 physical-world episodes, it improves average ObjectNav success over ProcTHOR while remaining effective amid real-world changes.
1. Introduction
- Simulation-trained agents struggle to generalize from limited simulated environments to real scenes with novel layouts, objects, clutter, moving people, and changing lighting.The paper frames the challenge as generalizing from small training sets to a distribution of real test environments that may be semantically distant.Source: §1 · pp. 1–2
- Phone2Proc scans a target environment and uses its layout and large-furniture placements to condition a procedurally generated distribution of training scenes.The generated worlds randomize objects and their placements, materials, textures, lighting, and clutter.Source: §1 · p. 2
- Success rate improves from 34.7% to 70.7% on average across 234 physical-world ObjectNav episodes in five diverse environments, compared with ProcTHOR.The LoCoBot evaluations included apartments, a RoboTHOR scene, a conference room, and a cafeteria; the policy used an RGB-only camera without depth, localization, or explicit mapping components.Source: §1 · p. 2
- The paper reports robustness experiments covering lighting, clutter, and human presence alongside its real-world evaluations.Its contribution summary also reports experiments on changes in the environment.Source: §1 · p. 2
2. Related Works
Prior navigation work explores mapping, scene priors, and RGB-only models, while real-robot sim-to-real evaluation remains less common. Phone2Proc contrasts with methods requiring depth or localization and with approaches that tune simulation randomization for narrower tasks.
2. Related Works
- Mapping-based navigation methods use agent pose and depth sensors, whereas Phone2Proc relies on RGB without additional sensors.Other approaches include scene priors, meta-learning, paired grid worlds, scene-memory transformers, passive videos, and expert trajectories.Source: §2 · p. 2
- Many navigation policies are evaluated in simulation rather than on real robots, and sim-to-real transfer can be brittle under realistic system noise.Prior work also reports that greater simulation fidelity did not improve real-world performance in a four-legged robot comparison.Source: §2 · p. 2
- Simulation randomization can help transfer, but excessive randomization may hurt training and hand-tuning does not scale; prior real-world-tuned approaches target narrower tasks.Those approaches randomize subsets of parameterized dynamics rather than entire visual environments and object instances.Source: §2 · pp. 2–3
- Phone2Proc differs from real-to-sim work by conditioning procedural generation on a real scene rather than reconstructing observations, and it focuses on interactive environments for embodied-agent training.The paper also contrasts its real-robot evaluation with mapping- or localization-dependent navigation work.Source: §2 · p. 3
3. Approach
Phone2Proc turns a phone scan of a target space into a distribution of interactive training environments, then trains and transfers an RGB-based navigation policy to a physical robot. The scan captures room structure and large objects, while procedural generation varies scene details and the real-world evaluation uses uncalibrated environments.
3. Approach
- Phone2Proc scans the target, conditions procedural scene variations on that scan, and transfers a trained policy to a LoCoBot navigating the physical environment.The scan-to-transfer workflow is presented as the three stages of the approach.Source: §3 · p. 3
3.1. Scanning
- A few-minute iPhone or iPad scan using RoomPlan produces a USDZ template with room layouts and visible large-object placements.The template also records walls, large objects, windows, and doors; smaller surface objects are ignored.Source: §3.1 · p. 3
3.2. Environment-Conditioned Procedural Generation
- Procedural generation conditions scene layout and semantic objects on the scan, then varies small objects, materials, lighting, and clutter to create diverse scenes.The method supports scanned wall layouts beyond orthogonal axes and samples varied assets matching scanned object categories.Source: §3.2 · pp. 4–5, Figure 3 · p. 4
3.3. Transfer to Real World
- The transfer design uses RGB observations and a CLIP encoder followed by a GRU, and real-world evaluation requires no training or calibration in the test environments.The five test settings included apartments, a RoboTHOR scene, a conference room, and a cafeteria; the paper's model-design goals exclude reliance on localization and depth inputs.Source: §3.3 · p. 5
4. Experiments
Across 234 episodes in five diverse real-world environments, the experiments evaluate PHONE2PROC against ProcTHOR, a privileged reconstruction baseline, and under environmental variation. They also analyze statistical significance and illustrate how agents navigate toward large and small objects.
4.1. How Well Does Phone2Proc Work?
- Across real-environment evaluation, PHONE2PROC achieved an aggregate Success Rate of 70.68%, compared with 34.68% for PROCTHOR, and performed effectively in challenging multi-room scenes requiring long-range exploration.Figure 5 reports fifteen trajectories per number: five objects from three starting locations.Source: §4.1 · p. 6, Figure 5 · p. 6
4.2. How Does Phone2Proc Compare To A Privileged Upper Bound?
- PHONE2PROC matches RECON in Success and Episode length across 15 RoboTHOR-Real episodes, despite RECON's privileged, manually reconstructed simulation.The reconstruction took five days to produce and was fine-tuned on this one scene.Source: §4.2 · pp. 6–7, Figure 6 · p. 7
4.3. How Robust is Phone2Proc to Chaos?
- PHONE2PROC remains robust across tested scene disturbances and camera-FOV changes, while RECON fails under object rearrangement and other variations.RECON's performance drops drastically when a model trained with wide vertical FOV is evaluated with narrow FOV; PHONE2PROC is robust to this change.Source: §4.3 · p. 7, Figure 7 · p. 8
4.4. Statistical Analysis
- PHONE2PROC is associated with 1.39 times greater odds of success than PROCTHOR, with a 95% confidence interval of [1.2, 1.62].The logistic regression models success across 150 datapoints and finds the PHONE2PROC coefficient significant at the 0.0001 level.Source: §4.4 · p. 7
4.5. Qualitative Analysis
- For large targets, agents use familiar landmarks and may double back to regain their bearings; for small targets absent from the scan, they explore broadly to find them.These behaviors occur without a map, ground-truth localization, or additional memory support.Source: §4.5 · pp. 7–8, Figure 8 · p. 8
5. Conclusion
The conclusion presents PHONE2PROC as an approach for training performant agents robust to unpredictable real-world conditions, with improved sim-to-real performance demonstrated across five diverse environments.
5. Conclusion
- PHONE2PROC's environment-conditioned procedural scenes are fully interactable, and the paper reports significant sim-to-real improvements across five diverse environments.The authors describe further exploration of these capabilities as future work.Source: §5 · p. 8
A. Implementation Details
The implementation uses an RGB-based visual encoder and recurrent policy, with modified training settings and multi-node computation. A Habitat baseline is trained separately for comparison.
A. Implementation Details
- A frozen CLIP-ResNet-50 processes 3x224x224 RGB images; a CNN-compressed visual embedding is combined with goal and previous-action embeddings and passed to a 512-unit GRU policy.The policy uses actor and critic heads to produce action probabilities and state-value estimates.Source: A. Implementation Details · p. 11
- Policy fine-tuning lowers the learning rate to 3e-5, penalizes collisions by -0.05, and adds randomized small movements near objects to approximate unintended interactions.Neck actions are limited to looking 30° above and below the horizon.Source: A. Implementation Details · pp. 11–12
- The horizontal field of view is randomized from 48° to 65° in 0.2° increments, and vertical FOV and aspect ratio are adjusted toward the Intel RealSense D435.Source: A. Implementation Details · p. 12
- Training uses three or four AWS g4dn.12xlarge machines with 16 processes each; the Habitat baseline uses 80 processes and eight A100 GPUs for 200M steps.The Habitat model selected for evaluation had the best performance on a 200-episode validation set.Source: A. Implementation Details · p. 12
B. Failure Cases
PHONE2PROC most often failed through semantic confusion, including misidentifying object instances or categories. The authors suggest limited visual diversity may contribute and that greater diversity could improve results.
B. Failure Cases
- Semantic confusion was PHONE2PROC's most common failure, involving missed object instances or confusion between object categories.Source: B. Failure Cases · p. 12
- Examples included a covered couch mistaken for a bed and spare water-cooler jugs mistaken for a vase.Source: B. Failure Cases · p. 12
- Scene generation used only six vase models, and the authors suggest greater visual diversity might improve PHONE2PROC further.Source: B. Failure Cases · p. 12