PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoReveal: Learning Active Vision and Object Manipulation under Partial Occlusions from Egocentric Demonstrations

Hiromu Kurihara, Kazutoshi Tanaka, Masashi Hatano, Tadashi Kozuno, Hideo Saito, Mariko Isogawa

arXiv:2610.12999v1cs.RO

TL;DR

Prior egocentric demonstration methods do not explicitly learn viewpoint control for partial occlusions, despite the scalability of easily collected first-person data. EgoReveal learns viewpoint repositioning and manipulation from those demonstrations, switching policies according to target visibility. In initially occluded tasks, it completed tasks more reliably than fixed-camera baselines.

  • Problem

    Egocentric demonstrations are easy to collect, but prior methods typically do not explicitly learn viewpoint control for handling partial occlusions.

    Source: §I · p. 1, A. Learning Manipulation from… · p. 2

  • Method

    EgoReveal learns Active Vision and Manipulation policies from egocentric demonstrations and switches between them based on target visibility.

    Source: §I · pp. 1–2

  • Results

    EgoReveal completed initially occluded tasks more reliably than fixed-camera baselines.

    Source: §I · p. 1

  • Takeaways & Limitations

    EgoReveal addresses partial-occlusion resolution and object manipulation using easily obtainable human demonstration data.

    Source: §VIII · p. 8

    The Occlusion Resolution Assessment Module relies on a reference image and HSV color matching, assuming consistent lighting and reliable color separability between objects.

    Source: §VII · p. 8

Abstract

from arXiv · show

Learning from Demonstrations (LfD) enables robots to learn object manipulation directly from human demonstrations. In contrast to resource-intensive and labor-intensive teleoperation, egocentric demonstrations are easy to collect, enabling scalable data collection. Manipulated objects are often partially occluded in everyday environments, yet prior egocentric demonstration-based methods that require no robots or specialized teaching devices do not explicitly address occlusions. To fill this gap, we propose EgoReveal, a framework that learns viewpoint-repositioning together with manipulation actions from egocentric human demonstrations to reposition the viewpoint and obtain target visibility under partial occlusions. EgoReveal instantiates this as two imitation policies: one predicts the future camera-arm pose sequence from the current view and camera-arm pose, and the other predicts the future control commands from object keypoints and a history of control commands. EgoReveal switches between the two based on target visibility. In experiments where the manipulated object is partially occluded initially, EgoReveal repositions the camera to obtain visibility and completes the task more reliably than fixed-camera baselines.

I. INTRODUCTION

Egocentric demonstrations are easier to collect at scale than teleoperation, but prior methods using them do not explicitly learn viewpoint control for partial occlusions. EgoReveal couples viewpoint repositioning and manipulation, switching policies based on target visibility; in initially occluded tasks, it completed tasks more reliably than fixed-camera baselines.

I. INTRODUCTION

  • Egocentric demonstrations are easy to collect, whereas teleoperation requires specialized hardware and skilled operators, limiting scalable data collection.Teleoperation records robot observations and actions directly; egocentric demonstrations use first-person video,,,.Source: §I · p. 1
  • Egocentric methods typically do not explicitly learn viewpoint control for handling partial occlusions, unlike EgoReveal.The target is often partially occluded in everyday settings, while many prior methods assume a fixed inference-time viewpoint,.Source: §I · p. 1
  • EgoReveal repositions the camera to obtain target visibility and completes initially occluded tasks more reliably than fixed-camera baselines.It switches between viewpoint-repositioning and manipulation policies based on target visibility.Source: §I · p. 1
  • EgoReveal learns manipulation from egocentric demonstrations without robotic teleoperation or handheld teaching devices.Its framework couples Active Vision and Manipulation policies through visibility-based switching to handle partial occlusions.Source: §I · pp. 1–2

A. Learning Manipulation from Egocentric Demonstrations

Prior egocentric manipulation work includes both human-only learning and approaches that combine human videos with robot demonstrations, but these methods typically do not explicitly learn viewpoint control for partial occlusions. Viewpoint-control methods use teleoperation or dedicated capture interfaces, while EgoReveal learns from natural egocentric demonstrations; the paper also establishes notation for coordinate-frame transformations.

A. Learning Manipulation from Egocentric Demonstrations

  • Egocentric-demonstration manipulation methods typically do not explicitly learn viewpoint control policies for handling partial occlusions.The surveyed approaches include human-only learning such as EgoZero, MAPLE, and AINA, as well as methods combining human and robot data –.Source: A. Learning Manipulation from… · p. 2

B. Learning Viewpoint Control from Demonstrations

  • Viewpoint-control approaches use HMD teleoperation or dedicated capture interfaces, whereas EgoReveal learns viewpoint motions from natural egocentric demonstrations.AV-ALOHA and ViA collect robot-operated demonstrations; ActiveUMI and EgoMI use dedicated capture devices and interfaces.Source: B. Learning Viewpoint Control from… · p. 2

A. Notation and Coordinate Frames

  • The notation aTb denotes a rigid transformation from coordinate frame b to frame a, mapping a point expressed in b into a.The paper represents rigid-body transformations with homogeneous transformation matrices and composes chained transformations by the standard rule.Source: A. Notation and Coordinate Frames · pp. 2–3

B. Problem Formulation

The task is tabletop picking when the target is in view but partially occluded; fully occluded or out-of-view targets are outside the stated scope. EgoReveal uses egocentric demonstrations to train two policies and switches from active vision to manipulation when the target is sufficiently visible.

B. Problem Formulation

  • The method targets reliable tabletop picking when the target remains in the camera view but is partially occluded; fully occluded or out-of-view targets are not addressed.The task couples viewpoint repositioning for visibility with manipulation once the target is sufficiently visible.Source: B. Problem Formulation · p. 3
  • The robot uses a dedicated RGB-D camera-arm and an independent manipulation-arm; RGB images condition viewpoint planning and estimate keypoints, which depth unprojects into 3D.Both RGB and depth images come from the same camera viewpoint at each timestep.Source: B. Problem Formulation · p. 3
  • Training uses markerless egocentric demonstrations without handheld teaching devices or object markers, with optional fiducial calibration to align coordinate frames.At execution, the target is specified by a reference image and optionally annotated reference keypoints.Source: B. Problem Formulation · p. 3
  • A visibility-based module switches from receding-horizon Active Vision to Manipulation when the target is sufficiently visible, after which manipulation continues for the episode.The Active Vision policy replans while visibility remains insufficient; the Manipulation policy replans from new observations after the switch.Source: B. Problem Formulation · p. 3

C. Action Chunking with Transformers (ACT)

ACT predicts future actions in chunks, while EgoZero implements manipulation by predicting future end-effector actions from object keypoints and past actions. The section describes each policy’s inputs, architecture, and training objective.

C. Action Chunking with Transformers (ACT)

D. EgoZero

  • EgoZero predicts future end-effector actions from 3D object keypoints and a history of past end-effector actions.Each past action comprises a 3D position and binary gripper state; the inputs also specify the keypoint count and past and future time horizons.Source: D. EgoZero · p. 3
  • EgoZero uses a closed-loop Transformer trained by behavior cloning, modeling predictions as a normal distribution and minimizing negative log likelihood with fixed variance.Source: D. EgoZero · p. 3

IV. METHOD

EgoReveal learns viewpoint repositioning and manipulation from egocentric demonstrations, using separate policies and an occlusion-resolution module to decide when to hand off between them. Its Active Vision policy predicts future camera poses from the current visual observation and camera-arm state.

A. Overview

  • EgoReveal combines Active Vision and Manipulation policies, switching between them when the target is adequately visible; the policies imitate demonstrations collected with smart glasses.The Active Vision policy repositions the viewpoint, while the Manipulation policy acts on object keypoints from the resulting camera view.Source: A. Overview · pp. 3–4

B. Active Vision Policy

  • The Active Vision policy adapts ACT to a single-view setup, taking the current rectified image and camera-arm pose as input and predicting a sequence of future camera poses.Source: B. Active Vision Policy · p. 4
  • Training observations use rectified RGB frames from smart-glasses fisheye video, with corresponding camera-arm poses obtained offline.Rectification reprojects fisheye images onto a virtual pinhole camera model.Source: B. Active Vision Policy · p. 4

C. Occlusion Resolution Assessment Module

The Occlusion Resolution Assessment module detects and segments the target, then switches to manipulation when its observed area stops changing significantly. The Manipulation policy estimates object keypoints from the newly visible view and predicts actions using those keypoints and robot-state history.

C. Occlusion Resolution Assessment Module

  • ORA detects the target using OwlViT with a reference image, filtering candidates by embedding similarity and then by HSV color matching.SAM3 segments the selected box, and the method proceeds only when the region’s Bhattacharyya-distance matching score exceeds a threshold.Source: C. Occlusion Resolution Assessment… · p. 4
  • ORA tracks the segmented target’s masked area across recent frames and switches to manipulation when its slope falls below a threshold.A sufficiently small slope indicates the observed area has converged and no longer changes significantly.Source: C. Occlusion Resolution Assessment… · p. 4

D. Manipulation Policy

  • The Manipulation policy uses EgoZero, which learns tasks from first-person human demonstrations without robot data.Source: D. Manipulation Policy · p. 4
  • After ORA switches policies, GroundingDINO detects the target and DIFT finds template-referenced 2D keypoints, which are unprojected into 3D and combined with robot-state history to predict actions.Source: D. Manipulation Policy · pp. 4–5

E. Coordinate Transformation

EgoReveal aligns egocentric camera observations with the robot arms using a marker-defined frame and coordinate transformations. At runtime, it repositions the camera until the target is visible, then switches to manipulation.

E. Coordinate Transformation

  • The marker-defined virtual base frame relates egocentric camera trajectories to the robot base, aligning data collection, policy training, and inference.The camera trajectory is supplied by SLAM relative to a fixed world frame, with a static world-to-marker transform.Source: E. Coordinate Transformation · p. 5
  • Camera poses are transformed into the manipulation-arm base frame through camera calibration, active-vision arm kinematics, and inter-arm calibration.The camera-arm and manipulation-arm are separate, so their coordinate frames must be connected for grasp execution.Source: E. Coordinate Transformation · p. 5

F. Execution Pipeline

  • In active-vision mode, the policy uses the rectified view and current camera pose to predict short-horizon camera poses, replanning as the chunk runs down.Predicted poses are converted to camera-arm joint targets via inverse kinematics.Source: F. Execution Pipeline · p. 5
  • ORA switches to manipulation when the target is sufficiently visible; the switch occurs at most once, after which the camera holds its last active-vision pose.Handling re-occlusion during manipulation is beyond the work's scope.Source: F. Execution Pipeline · p. 5
  • After switching, the system extracts 3D object keypoints once, then predicts manipulation action chunks from those keypoints and recent control history.The next action is transformed to the manipulation-arm base and executed via inverse kinematics, with replanning from updated robot-state history.Source: F. Execution Pipeline · p. 5

A. Scenarios and Tasks

The experiments test tabletop picking when targets are initially partially occluded, with object and occluder positions randomized. They use two robot arms and egocentric demonstrations for active vision and manipulation.

A. Scenarios and Tasks

  • The unseen-occlusion evaluation trains a unified policy across three occluders and tests it on two novel occluders, including a book and a bag.The other described scenarios are a SoftToy behind a pencil case and a towel behind a book, manipulated after lateral camera motion.Source: Figure 3 · p. 6, Figure 4 · p. 6
  • The study evaluates a pick task with targets partially visible at the initial viewpoint and randomizes target-occluder positions across episodes.Active vision succeeds when the target is entirely unoccluded at the switch; task success requires a stable grasp and lift.Source: A. Scenarios and Tasks · pp. 5–6

B. Hardware

  • Two 7DoF robot arms divide the roles: one manipulates the object and the other carries the camera; Project Aria supplies RGB and a RealSense D435 supplies depth.The arms are arranged with a 90-degree angular offset.Source: B. Hardware · p. 6

C. Data Collection

  • Human operators recorded 100 active-vision episodes with randomized object positions and varied head trajectories to demonstrate moving around occlusions.The demonstrations use first-person views.Source: C. Data Collection · p. 6
  • The manipulation dataset contains 100 grasp-and-lift demonstrations without occluders, with target positions and operator head pose randomized.Source: C. Data Collection · p. 6

D. Baseline Methods

EgoZero is the baseline for the combined task of resolving occlusions and manipulating targets from human demonstrations. EgoReveal achieves higher success rates than the baseline, and the full method leads its ablations.

D. Baseline Methods

  • Because no existing method directly addresses the combined task, the study adopts EgoZero's demonstration-learned manipulation policy as its baseline.Source: D. Baseline Methods · p. 6

E. Implementation Details

  • Both policies were trained on one RTX 6000 Ada GPU with batch size 8; Active Vision ran 100,000 steps and Manipulation 300,000 steps.The learning rates were 1 · 10−5 and 1 · 10−4, respectively.Source: E. Implementation Details · p. 6

A. Comparison with the Baseline Method

B. Ablation Study

  • Ours, including Active Vision and ORA, achieved the highest success rate among the tested ablations.The variants omit Active Vision or replace ORA's visibility-triggered switch with a fixed-time interval.Source: Table I · p. 7, B. Ablation Study · p. 7

C. Evaluation on Unseen Occlusion Scenarios

EgoReveal achieves higher success rates on unseen occlusions than the baseline and its version without ORA. The discussion identifies scope boundaries and extensions for embodiments, occlusion types, and future evaluation.

C. Evaluation on Unseen Occlusion Scenarios

  • On unseen occlusions, EgoReveal achieves higher success rates than the baseline and the version without the Occlusion Resolution Assessment module.The experiments use occluding objects not present during training.Source: C. Evaluation on Unseen Occlusion… · p. 7

VII. DISCUSSION

  • Experiments cover a fixed-base manipulator with a gripper under partial occlusions, while other embodiments could use a replacement manipulation policy.The active-vision and occlusion-assessment components could remain unchanged for a different end-effector policy.Source: §VII · p. 7
  • Complete occlusion and targets outside the camera’s field of view are not addressed directly, but an added search stage could locate them before the occlusion-aware pipeline begins.The proposed extension estimates or searches for an initial arm and camera configuration, then uses a viewpoint where the target is at least partially visible.Source: §VII · pp. 7–8
  • ORA relies on a reference image and HSV color matching, assuming consistent lighting and reliable color separation between objects.The authors identify these requirements as assumptions that limit broader applicability.Source: §VII · p. 8
  • Future work targets broader occlusion scenarios, object categories, and manipulation tasks with a larger, more diverse demonstration dataset, alongside unified cross-task policies.The current reliance on scenario-specific policies limits scalability.Source: §VII · p. 8

VIII. CONCLUSIONS

  • The conclusion frames EgoReveal as a three-component approach combining active vision, manipulation, and ORA, with experiments showing higher task success than the baseline.The paper presents this as a step toward robot learning for more complex real-world manipulation problems.Source: §VIII · p. 8

Related papers

Loading 2610.12999v1…