PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoPush: Learning End-to-End Egocentric Multi-Object Rearrangement for Mobile Robots

Boyuan An, Zhexiong Wang, Yipeng Wang, Jiaqi Li, Sihang Li, Jing Zhang, Chen Feng

arXiv:2602.18071v1cs.RO

TL;DR

Long-horizon multi-object rearrangement from a single egocentric camera is difficult because occlusions and dynamic interactions make global state estimation unreliable. EgoPush uses object-centric relative representations, visibility-constrained teacher–student distillation, and stage-local rewards to train a visual policy. It improves success rate over the reported ablation and transfers zero-shot from simulation to a mobile robot.

  • Problem

    Long-horizon multi-object rearrangement requires egocentric operation under occlusions and dynamic scenes where global state estimation can fail.

  • Method

    EgoPush combines object-centric relative representations with a visibility-constrained privileged RL teacher, visual student distillation, and temporally decayed stage-local rewards.

  • Results

    70% success rate is achieved by the student, compared with 21% for w/o C-GV in the reported ablation, alongside successful zero-shot sim-to-real transfer.

  • Takeaways & Limitations

    Egocentric relative representations and recoverable active perception support perception-driven rearrangement without explicit global state estimation.

Abstract

from arXiv · show

Humans can rearrange objects in cluttered environments using egocentric perception, navigating occlusions without global coordinates. Inspired by this capability, we study long-horizon multi-object non-prehensile rearrangement for mobile robots using a single egocentric camera. We introduce EgoPush, a policy learning framework that enables egocentric, perception-driven rearrangement without relying on explicit global state estimation that often fails in dynamic scenes. EgoPush designs an object-centric latent space to encode relative spatial relations among objects, rather than absolute poses. This design enables a privileged reinforcement-learning (RL) teacher to jointly learn latent states and mobile actions from sparse keypoints, which is then distilled into a purely visual student policy. To reduce the supervision gap between the omniscient teacher and the partially observed student, we restrict the teacher's observations to visually accessible cues. This induces active perception behaviors that are recoverable from the student's viewpoint. To address long-horizon credit assignment, we decompose rearrangement into stage-level subproblems using temporally decayed, stage-local completion rewards. Extensive simulation experiments demonstrate that EgoPush significantly outperforms end-to-end RL baselines in success rate, with ablation studies validating each design choice. We further demonstrate zero-shot sim-to-real transfer on a mobile platform in the real world. Code and videos are available at https://ai4ce.github.io/EgoPush/.

I. INTRODUCTION

EgoPush addresses long-horizon multi-object non-prehensile rearrangement in texture-sparse, partially observable environments where global localization is unreliable. It combines object-centric relative representations, visibility-constrained teacher–student learning, and stage-local rewards, achieving stronger simulation performance and zero-shot real-world transfer.

  • Texture-sparse scenes, object motion, and occlusions prevent reliable global localization, requiring action coordination from perceived relative relations.
  • EgoPush consistently improves success rate and sample efficiency over alternative designs and demonstrates successful zero-shot sim-to-real transfer on a Turtlebot mobile platform.The framework is evaluated through extensive simulation benchmarks, baseline comparisons, and ablation studies.
  • EgoPush learns object-centric latent representations that encode relative spatial relations, enabling rearrangement from egocentric vision without explicit maps or global positioning.
  • Constrained Teacher RL limits privileged observations to egocentric, visibility-restricted cues, inducing active perception behaviors recoverable by the visual student.
  • Stage-aligned, temporally decayed, local completion rewards decompose long-horizon rearrangement into manageable subproblems and improve temporal credit assignment.

B. Visual Policy Learning of Mobile Robots

EgoPush combines egocentric visual policy learning with constrained teacher reinforcement learning and staged rewards for long-horizon mobile rearrangement. The teacher uses structured sparse observations and stage-local supervision, while the student is distilled from this behavior.

  • End-to-end pixel-based RL remains sample-inefficient and sensitive to reward sparsity and partial observability in mobile manipulation.
  • EgoPush uses a two-phase pipeline: constrained privileged RL trains a teacher, then imitation learning distills its behavior into an egocentric visual student.
  • The teacher represents active, anchor, and obstacle objects with shared-weight, group-wise latent embeddings for task-relevant scene structure.
  • Sparse keypoints encode relative poses of the active object, anchor, obstacles, and target references, reducing observation dimensionality while supporting pushing decisions.
  • Virtual egocentric field-of-view masking and center-gated target visibility constrain teacher observations to cues recoverable by the student.
  • Stage-aligned completion rewards use per-stage budgets and remaining-budget weighting to shorten credit assignment and encourage timely completion.

C. Phase 2 (Student): Supervised Learning

The Phase 2 student learns from egocentric RGB-D observations through online teacher querying, group-wise depth processing, and relational latent distillation. These choices address closed-loop error and the teacher–student representation gap.

  • RGB supplies instance masks for semantic grouping, while the student policy receives depth-only inputs organized into fixed-dimensional group-wise layers.
  • Spatially disjoint instance masks are summed within semantic groups, producing constant-dimensional depth representations without pixel-wise interference.
  • Online DAgger-style distillation queries the teacher at visited states and immediately updates the student with supervised action labels.
  • A relational distillation loss aligns pairwise structure among active, anchor, and obstacle groups instead of directly matching incompatible teacher and student latent spaces.
  • Because anchor–reference relative pose is invariant within each task category, the student can inherit target-seeking behavior without an explicit goal observation.
  • Teacher-policy weight initialization and camera-pose domain randomization support faster convergence and robustness during real-world deployment.

IV. EXPERIMENTS

EgoPush is evaluated in simulation and on a real mobile platform across diverse rearrangement settings. The reported experiments emphasize precise rearrangement, accessible teacher supervision, and staged task decomposition.

  • EgoPush achieves precise multi-object rearrangement for diverse object geometries and target formations in simulation and the real world.
  • Restricting the teacher’s observation space produces more accessible and distillable supervision for the student.
  • Sequential sub-task decomposition is evaluated as part of the approach to long-horizon rearrangement.

A. Experimental Setup

The experiments assess teacher observation constraints, long-horizon credit assignment, and relational distillation through controlled ablations on rearrangement tasks. Results show that constrained observations improve student recoverability, while stage-aligned rewards and relational supervision support learning and performance.

  • Teacher Observation Ablation: Teacher Success Rate remains above 98% across all three observation settings, while stronger observation constraints increase execution time and trajectory length.ExecTime and TrajLen are computed only over successful episodes.
  • Teacher Observation Ablation: The constrained teacher’s student reaches 70% success, compared with 21% without center-gated visibility and zero successful episodes from the Global teacher.The shorter trajectories of the w/o C-GV student reflect success on trivial instances rather than stronger performance.
  • Teacher Observation Ablation: The constrained teacher learns anchor-facing pushing behaviors that preserve visibility, whereas Global observation produces behaviors unavailable from the student’s egocentric viewpoint.Without center-gated visibility, the teacher tracks partial reference cues and provides sparse actionable supervision.
  • Credit Assignment Ablation: Stage-wise rewards, temporal decay, and stage-timer resets progressively improve long-horizon credit assignment, with the full method reaching near-saturated performance using half the training steps.The stage timer resets decay at each stage boundary, and Ours converges at about ∼45k steps versus ∼90k for the extended ablations.
  • Relational Distillation Ablation: Removing relational distillation completely fails on LINE-SHAPE, where asymmetric targets cause small sequential errors to compound.The ablation also produces a noticeably larger converged action loss.

C. Baseline Comparisons

Baseline comparisons test mapping-based and end-to-end visual RL methods under egocentric sensing, followed by a zero-shot real-world deployment. EgoPush solves the simulated long-horizon objective and transfers to hardware with substantial success.

  • Simulation Baselines: End-to-end egocentric RL baselines achieve below 1% success, despite several obtaining non-trivial Reach rates.RGB, RGB-D, semantic-mask, and recurrent variants are compared under the same reward, termination conditions, and training budget.
  • Simulation Baselines: Even RGBD-Sem with ground-truth semantic masks fails, indicating that perfect object-level perception alone does not solve the rearrangement task.The task requires completing a long-horizon push-and-align objective under egocentric sensing.
  • Simulation Baselines: EgoPush achieves 100% Success Rate and 100% Reach rate under the same egocentric sensing constraints.The reported metrics are from the simplified baseline-comparison task.
  • Real-World Transfer: In 10 real-world cross-shape episodes with five boxes, the TurtleBot completes the rearrangement with an 80% success rate under a loose evaluation metric.Minor orientation and centering deviations are allowed, and the robot completes successful trials within two minutes.
  • Limitations: The discussion identifies reactive memory as a remaining boundary: alternating between goal-seeking and path-seeking views can cause oscillation or deadlock in narrow passages.This limitation arises when the robot must coordinate contact while traversing constrained corridors.

APPENDIX

The appendix documents baseline protocols, qualitative failure modes, and implementation details for end-to-end visuomotor and SIM comparisons. It shows that long-horizon rearrangement remains difficult even with oracle semantic perception, while SIM is limited by accumulated state inconsistency and contact-sensitive control.

  • Baseline Configurations: E2E-RGB uses RGB, E2E-RGBD adds depth, and the E2E policies output differential-drive commands without teacher supervision or explicit mapping.
  • E2E Failure Modes: Even when agents transiently reach an object, they often fail to maintain stable interaction long enough to transport it to the anchor region.Observed failures include spinning, freezing, repeated re-approach attempts, and unstable contact.
  • E2E Failure Modes: Even with oracle semantic masks, RGBD-Sem remains poor, indicating that object recognition alone does not provide persistent spatial consistency after occlusion and configuration changes.
  • SIM Baseline: SIM uses a robot-centric, heading-aligned spatial intention map and map-based target selection rather than directly regressing velocities from pixels.Its adapted setup constructs local semantic maps from oracle segmentation and integrates SIM into PPO.
  • SIM Baseline: Accumulated odometry and contact errors can misalign SIM’s semantic map, undermining local planning and the map-update/action-selection feedback loop.The resulting discrepancies produce compounding errors over long episodes.
  • SIM Baseline: Pushing also requires fine alignment and stable contact, which a purely map-based decision layer struggles to achieve without structured, stage-aware supervision.Unfavorable approach angles can cause repeated attempts, wasted steps, and increased disturbances.

A.4 Qualitative Visualizations

The qualitative appendix presents full-episode filmstrips for recurring baseline failures and specifies a smoothness reward that penalizes abrupt action changes. These visualizations compare three E2E failures with SIM-Odom’s worsening map mismatch.

  • Qualitative Visualizations: Figure 7 contains four representative failure patterns: three E2E rows and one SIM-Odom row.Figure 8 complements this with a ground-truth-versus-map sequence whose alignment degrades over time.
  • Reward Function Details: The smoothness term penalizes sudden changes in consecutive linear and angular actions.
  • Reward Function Details: εsmooth is the pre-scaling threshold used to determine whether consecutive action differences incur the smoothness penalty.

B.2 World Box

The world-box and visibility design isolates environments and constrains privileged observations to information recoverable from an egocentric camera. It uses masked keypoints, field-of-view checks, and center-gated access to reference targets.

  • World Box: Gray cylindrical world boxes isolate parallel environments from visual crosstalk in Isaac Lab’s shared 3D coordinate system.
  • FOV Design: The teacher receives egocentric, visibility-limited information through a virtual frustum mask and center-gated visibility for reference target keypoints.
  • Virtual Egocentric FOV Masking: A keypoint is visible only when it lies in front of the camera, within horizontal and vertical field-of-view bounds, and between minimum and maximum depths.
  • Virtual Egocentric FOV Masking: Masked keypoints use a constant value ϵ when they are not visible, while image-plane coordinates are used only for gating.
  • Fixed-size Grouping: Obstacles are sorted by increasing robot distance, padded to a fixed maximum with ϵ, and truncated when exceeding the maximum count.
  • Center-gated Visibility: Reference keypoints encode the active object’s target configuration relative to the anchor and are instantiated using the anchor pose and a predefined task transform.
  • Center-gated Visibility: The reference target is revealed only when the anchor is inside the virtual field of view and a central image-plane gate.

B.4 Front Pusher on Robot

A front-mounted pusher keeps objects within the depth camera’s reliable sensing range but introduces a lever-arm-induced yaw disturbance. Despite this difficulty, the learned agent compensates for the resulting contact dynamics.

  • Robot Hardware: The integrated pusher addresses the depth camera’s approximately 15 cm sensing dead zone with a 7.5 cm protrusion and 14 cm × 10 cm frontal profile.Its form factor remains consistent with the robot’s width.
  • Contact Dynamics: The pusher’s longitudinal offset from the robot’s instantaneous center of rotation amplifies off-center contact into parasitic yaw moments.Minor pushing-vector misalignments therefore create substantial rotational disturbances.
  • Learned Compensation: The learning-based agent suppresses parasitic yaw moments by modulating control signals and achieves precise manipulation despite the constrained contact dynamics.
  • Restricted-Observation Ablation: With unconstrained global perception, the teacher converges to pushing with the rear chassis instead of the integrated pusher.
  • Restricted-Observation Ablation: The rear-chassis strategy reduces the moment arm, alignment sensitivity, and snaking effect, enabling faster and more stable completion.This behavior supports interpreting the pusher as a perceptual scaffold rather than merely a contact device.

C. REAL WORLD ROBOT SYSTEM SETUP

The real-world system uses a TurtleBot3 Burger with a front-mounted pusher and top-mounted RGB-D egocentric sensing. Processing combines onboard or server-side depth handling with color-based box segmentation in a 3m × 3m arena.

  • Mobile platform: A TurtleBot3 Burger carries a 3D-printed front pusher and provides maximal translation and rotation velocities of 0.22 m/s and 2.84 rad/s.The robot is reconfigured to place the passive ball caster at the front, providing a longer actuator lever arm.
  • Sensing: An Intel RealSense D435i mounted above the robot provides an RGB-D egocentric view, with the RGB sensor positioned 20cm above and 5cm behind the ball caster.The camera is pitched downward by 11.5◦.
  • Sensing: RGB images are captured at 320 × 240 and depth images at 480 × 270 before alignment and joint down-sampling to 240 × 180.The camera operates at a base frequency of 30Hz, with processing-dependent delays.
  • Computation pipeline: The robot streams egocentric RGB-D observations over Wi-Fi to a server, where images are decoded, post-processed, segmented, and converted into control-policy inputs.The server uses an Intel Ultra 9 275HX CPU and NVIDIA RTX 5080 GPU, while the robot runs ROS1 on a Jetson Nano.
  • Scene processing: The arena contains five differently colored boxes, which HSV thresholding separates from the gray background to create masks for filtering depth images.The HSV ranges are specified using OpenCV conventions in Table VI.
  • Depth processing: Depth processing compares server-side reconstruction with onboard filtering because raw RealSense depth contains noise and frequent top-surface dropouts.CDM preserves top-surface details but incurs approximately 50ms per frame, while onboard filtering produces temporal flickering and disappearing surfaces.

E. ADDITIONAL EXPERIMENT RESULTS

Additional experiments test student-policy generalization across object geometries and evaluate cuboid pushing with a nearest-distance accuracy measure. The student transfers its reach-and-approach behavior across shapes and achieves an average cuboid error rate of approximately 13.3%.

  • Object geometry generalization: The student policy is evaluated on cylinders and triangular prisms while keeping sensing, action bounds, termination conditions, and evaluation protocol unchanged.This probes generalization beyond the training cuboid.
  • Object geometry generalization: Table VII reports success rate, reach-box rate, execution time per episode, and trajectory length for different object geometries.The table defines SR, Reach, ExecTime, and TrajLen as the evaluated metrics.
  • Object geometry generalization: Fig. 13 presents qualitative results for boxes with different geometries.The figure complements the quantitative geometry-generalization evaluation.
  • Accuracy evaluation: The cuboid pushing metric measures Euclidean distance between the final box position and its corresponding invisible target tag.The student does not access the target tag during execution.
  • Accuracy evaluation: The normalized accuracy bounds distance error by the training-time threshold, and the student achieves an average error rate of approximately 13.3% on cuboid pushing.The threshold serves as the maximum admissible error for normalization.
Loading 2602.18071v1…