PaperlayerView exact revision on arXiv

Source-linked AI summary

ActiveMimic: Egocentric Video Pretraining with Active Perception

Xingyao Lin, Guojin Zhong, Tianyi Lu, Ziyi Ye, Yichen Zhu, Zuxuan Wu, Yu-Gang Jiang

arXiv:2606.06194v1cs.ROcs.CV

TL;DR

Egocentric human video is scalable for robot pretraining but lags robot data, partly because standard pipelines miss humans’ active viewpoint motion. ActiveMimic models camera and wrist actions jointly, surpassing human-video baselines and matching robot-data models across real-world tasks.

  • Problem

    Egocentric human video is scalable but pretrained models underperform robot-data models, while existing proxy-action approaches omit humans’ active viewpoint repositioning during manipulation.

  • Method

    ActiveMimic recovers synchronized camera and wrist trajectories from egocentric video, models viewpoint motion as action, and jointly pretrains active perception and manipulation before robot-specific adaptation.

  • Results

    ActiveMimic consistently surpasses human-video baselines and matches state-of-the-art robot-data models, exceeding π0 on Restocking (90.1% vs. 86.4%).

  • Takeaways & Limitations

    The analysis indicates that active perception is acquired during egocentric video pretraining and transfers from human perception to robot control.

  • Takeaways & Limitations

    Extending ActiveMimic to human-robot interaction scenarios remains a future direction.

Abstract

from arXiv · show

Egocentric human video offers a scalable alternative to robot data for pretraining, yet models pretrained on such video consistently underperform those pretrained on robot data. We attribute this gap to a missing signal, the active perception behavior in egocentric videos, where humans continuously reposition their viewpoint during manipulation, inducing camera motion that standard pipelines treat as noise. To address this, we present ActiveMimic, a pretraining framework that recovers synchronized camera and wrist trajectories from a single body-worn RGB camera, models camera motion as a viewpoint action, and jointly learns active perception and manipulation from in-the-wild egocentric human video before adapting to a target robot. Empirically, real-world experiments across tasks with diverse active perception demands show that ActiveMimic consistently surpasses baselines pretrained on human video and matches state-of-the-art models pretrained on robot data. Further analysis provides evidence that active perception capability originates from egocentric human video pretraining rather than robot-specific fine-tuning, confirming active perception as the key to unlocking egocentric human video for robot pretraining.

1 Introduction

ActiveMimic addresses the underperformance of egocentric-video-pretrained robot models by treating human viewpoint repositioning during manipulation as an active-perception signal rather than camera noise. It recovers synchronized camera and wrist trajectories, jointly models viewpoint and manipulation actions, and achieves strong real-world transfer to robots.

  • Motivation: Egocentric human videos are cheaper, more scalable, and more diverse than robot data, but robot foundation-model pretraining commonly relies on costly, limited robot datasets.Robot data remains expensive to collect, difficult to scale, and limited in task diversity.
  • Problem: Existing proxy-action approaches miss humans’ continuous viewpoint repositioning during manipulation, whose substantial camera motion standard pipelines treat as noise.Prior proxies include hand trajectories, hand point clouds, and object motion signals.
  • Method: ActiveMimic recovers synchronized camera and wrist trajectories, resolves their coupling, and encodes viewpoint plus bimanual wrist motion as a unified 27D action.The representation uses a common reference frame and is learned through a single flow matching objective.
  • Method: ActiveMimic jointly pretrains active perception and manipulation from in-the-wild egocentric video before adapting to a target robot, without dedicated capture hardware.The framework computes its unified action space on Ego4D.
  • Results: Real-world experiments show ActiveMimic consistently surpasses human-video-pretrained baselines and matches state-of-the-art robot-data-pretrained models across tasks with diverse active-perception demands.The analysis attributes active perception to egocentric-video pretraining rather than robot-specific fine-tuning and finds camera-motion supervision facilitates transfer to robot control.

2 Related Work

Prior work uses human videos for scalable robot pretraining but typically supervises only hand or object motion, omitting viewpoint actions. ActiveMimic addresses this gap with vision-based active-perception learning from a single body-worn RGB camera, unlike approaches requiring dedicated capture hardware or additional sensors.

  • Learning from human videos: Human-video pretraining commonly uses proxy hand trajectories, hand point clouds, or object motion, but provides no viewpoint-action signal.These methods supervise hand or object motion rather than active perception.
  • Active perception: Classical active perception optimizes viewpoint selection independently of downstream manipulation, whereas recent work jointly models camera motion and manipulation in a shared action space.Active perception controls viewpoint to reduce perceptual uncertainty rather than passively receiving images.
  • Learning active perception from egocentric human videos: Existing egocentric-video models usually leave active perception unmodeled or require additional cameras or wearable sensors beyond a single body-worn RGB camera.This contrasts with in-the-wild egocentric video captured using only one body-worn RGB camera.
  • Learning active perception from egocentric human videos: ActiveMimic introduces a purely vision-based approach that recovers active perception from in-the-wild egocentric human video using a single body-worn RGB camera.The passage positions this approach against prior methods relying on extra capture hardware or sensors.

3 Method

ActiveMimic treats egocentric camera motion as a viewpoint action and jointly models it with manipulation by recovering synchronized camera and wrist trajectories from RGB video. It decouples these motions into a unified 27-dimensional action space and trains a two-stage model to predict active-perception and manipulation actions before robot adaptation.

  • Trajectory recovery: ActiveMimic recovers synchronized camera and wrist trajectories from a single body-worn RGB camera without additional sensors or controlled capture.The pipeline uses off-the-shelf vision models, a common reference frame, and a unified action representation.
  • Trajectory recovery: The egocentric camera trajectory serves as a viewpoint action, encoding active perception independently of whether the camera is head-, chest-, or glasses-mounted.Wrist poses are estimated by SAM-3D-Body, while VGGT recovers a scale-normalized camera path.
  • Camera–wrist decoupling: ActiveMimic resolves camera–wrist coupling by expressing both motions in a common spatial reference frame, preventing wrist supervision from conflating actual wrist movement with camera motion.Wrist poses are initially expressed in the current-frame camera frame, whereas the camera trajectory is anchored to the first frame.
  • 27D action representation: The decoupled camera and two wrist poses form a unified 27-dimensional action vector that jointly captures viewpoint action and bimanual manipulation.The unified 27D space lets the model learn camera–wrist dynamics within one prediction objective.
  • Architecture and training objective: ActiveMimic uses a two-stage mix-of-transformers strategy combining a visual-language prefix with an action-expert suffix to inject active perception capability.The action chunk is the unified 27D action during egocentric human-video pretraining and the robot action chunk during robot-specific fine-tuning.

4 Experiments

Experiments evaluate whether active perception unlocks egocentric human video for robot pretraining through real-world tasks, trajectory-label validation, and ablations of camera supervision and viewpoint inputs. ActiveMimic surpasses all baselines across four tasks, while analyses support the effectiveness and pretraining origin of active perception.

  • Evaluation questions: Experiments assess camera-motion supervision, trajectory pretraining signals, the source and use of active perception, and transfer to robots.These four questions organize the evaluation across Sections 4.2–4.5.
  • Experimental setup: The robot is an AGI-BOT G1 humanoid upper body with a 2-DoF head, 2-DoF waist, two 7-DoF arms, and head- and wrist-mounted RGB cameras.The head camera and head/waist joints form the active perception subsystem for viewpoint repositioning.
  • Tasks: ActiveMimic is evaluated on four real-world tasks spanning active perception demands: Restocking, Reaching, Finding, and Pouring.Restocking requires crouching, scanning a three-tier shelf, and placing a bottle; Reaching involves leaning over a 24 cm obstacle.
  • Pretraining data: 2,561 episodes from Ego4D provide roughly 10 hours of filtered egocentric manipulation video for pretraining at 10 fps.The episodes average 130 frames each and are filtered to retain clips suitable for active perception supervision.
  • Real-world results: 90.1% on Restocking, 88.9% on Reaching, 91.7% on Finding, and 93.3% on Pouring are ActiveMimic’s success rates, surpassing all baselines on every task.ActiveMimic also achieves comparable or higher success rates than π0 across all four tasks, while MotoVLA falls behind by a substantial margin.
  • Trajectory validation: HOT3D validation shows predicted camera and wrist trajectories closely follow ground-truth trends, confirming that the 27D labels carry effective pretraining signals.Ego4D lacks ground-truth head and wrist pose annotations, so label fidelity is evaluated on the external HOT3D dataset.
  • Active perception ablation: On Restocking placement, ActiveMimic scores 24 out of 27, while pickup is reliably completed across all three tested inference conditions.The conditions compare all cameras, a zeroed head camera, and ActiveMimicsft-only with all cameras.

5 Conclusion · A From Egocentric Video to Unified Action Space

ActiveMimic is an active-perception-aware pretraining framework for in-the-wild egocentric video that consistently surpasses human-video baselines across real-world tasks. The results identify active perception as key to robot pretraining, with evidence that it originates from egocentric pretraining and that camera-motion supervision supports transfer to robots.

  • 5 Conclusion: ActiveMimic introduces active-perception-aware pretraining for in-the-wild egocentric video.The framework is designed to address active perception during robot pretraining.
  • 5 Conclusion: Across real-world tasks, ActiveMimic consistently surpasses baselines pretrained on human video.The comparison is explicitly against human-video-pretrained baselines.
  • 5 Conclusion: The results confirm active perception as the key to unlocking egocentric human video for robot pretraining.This conclusion follows from ActiveMimic’s performance across real-world tasks.
  • 5 Conclusion: The analysis provides evidence that active perception originates from egocentric pretraining.The finding attributes the capability to pretraining rather than leaving its source unspecified.
  • 5 Conclusion: Camera motion supervision facilitates representational transfer from human perception to robot control.The passage specifically links camera-motion supervision with transfer from human perception to robot c.
  • 5 Conclusion: Together, the findings support egocentric human video as a viable source for robot pretraining when active perception is modeled.This point synthesizes the framework’s purpose, comparative result, and transfer evidence.

A.1 Metric Scale Recovery · A.2 Video Filtering and Segmentation

The method recovers metric camera trajectories by aligning VGGT depth with UniDepth and applies two-stage VLM/LLM filtering to extract high-confidence indoor hand-object manipulation segments from Ego4D. The resulting corpus preserves diverse manipulation verbs and object categories for general-purpose pretraining.

  • A.1 Metric Scale Recovery: VGGT recovers a scale-normalized camera path whose translational component is defined only up to a global scale factor.
  • A.1 Metric Scale Recovery: Metric scale recovery aligns VGGT’s per-pixel normalized depth map with UniDepth’s per-pixel metric depth map.
  • A.1 Metric Scale Recovery: A per-frame scale is computed as the median depth ratio over pixels with valid positive depth values in both maps.
  • A.1 Metric Scale Recovery: Per-frame scales are aggregated into an episode-level scale λ = mediank∈{1,...,K} λk.
  • A.1 Metric Scale Recovery: The metric camera trajectory is obtained by scaling only the translational component of the scale-normalized transform.
  • A.2 Video Filtering and Segmentation: Ego4D manipulation segments are identified through two-stage filtering that combines VLM-based temporal segmentation with LLM-based semantic filtering.
  • A.2 Video Filtering and Segmentation: Qwen3-VL-8B-Instruct proposes segments containing purposeful hand manipulation of physical objects while excluding observation, walking, waiting, and pure camera motion.
  • A.2 Video Filtering and Segmentation: Qwen3-30B-A3B-Instruct retains candidates involving artificial physical objects, hand-object manipulation, and indoor scenes, removing nonqualifying segments.

B Training Details · C Experimental Details · C.1 Task Setup

ActiveMimic uses staged pretraining and robot-specific fine-tuning, with a 3B visual-language prefix and 0.6B action expert. The evaluation covers four tasks on a common robot platform, while the filtered pretraining corpus spans diverse actions and objects.

  • B Training Details: The model combines a 3B visual-language prefix with a 0.6B action expert.This architecture supports the staged training recipe described for ActiveMimic.
  • B Training Details: Egocentric human-video pretraining begins with warm-up training that freezes the visual-language prefix and updates only the randomly initialized action expert.Warm-up brings the action expert to a reasonable operating point before joint optimization.
  • B Training Details: The final filtered pretraining corpus broadly covers manipulation actions and object categories.Figure 7 visualizes this coverage with word clouds of action verbs and manipulated objects.
  • B Training Details: Full training then unfreezes all parameters for end-to-end optimization, followed by task-specific robot-data fine-tuning from the pretrained checkpoint.ActiveMimic and ActiveMimicwrist-only share the same training configuration, differing only in wrist-action supervision.
  • B Training Details: Stage 2 trains for approximately 5 epochs on each task.Table 1 summarizes the hyperparameters for each training phase.
  • C.1 Task Setup: All four evaluation tasks are executed on the same robot platform described in Sec. 4.1.Table 2 consolidates the detailed specifications for each evaluation task.

C.2 Robustness Evaluation

ActiveMimic shows the strongest robustness under both flashing lighting and unseen-object conditions. It achieves the highest success rate and smallest performance drop in each evaluation.

  • Restocking under flashing lighting: Under flashing red/green/blue light, ActiveMimic achieves the highest success rate, 79.0%, with the smallest drop, −11.1% from 90.1%.The evaluation uses 81 trials per condition and the same checkpoint as the main results.
  • Restocking under flashing lighting: Camera motion supervision improves robustness: ActiveMimicwrist-only drops 24.7% to 58.0% under flashing light.
  • Finding with unseen objects: With two unseen yogurt variants, ActiveMimic maintains the highest success rate, 72.2%, and the smallest drop, −19.5% from 91.7%.The variants differ in packaging but have identical shape and size; evaluation uses 36 trials per condition.
  • Finding with unseen objects: On unseen yogurt variants, π0 drops 22.2% to 63.9%, while ActiveMimicwrist-only drops 33.4% to 47.2%.

C.3 Failure Case Analysis

Removing ActiveMimic’s head camera causes placement failures in Restocking by severing the visual loop coordinating head and hand movements. Failures range from imprecise placement to incorrect lateral position or shelf tier.

  • Robustness evaluation: ActiveMimic achieves the highest success rate and smallest absolute drop under alternating flashing light and unseen yogurt objects.These perturbations evaluate Restocking under red/green/blue flashing light and Finding with unseen yogurt objects.
  • Head-camera ablation: Without the head camera, all three Restocking failures occur at the placement point.The failures stem from severing the pretrained model’s visual loop for coordinating head and hand movements during active perception.
  • Head-camera ablation: One failure reaches the correct shelf tier and lateral position but uses an imprecise placement motion that knocks over the shelf.
  • Head-camera ablation: The other failures place the bottle at the wrong lateral position on the correct tier or target the wrong tier entirely.

C.4 Representational Transfer: K Sensitivity Analysis

ActiveMimic’s representational-transfer advantage over ActiveMimicwrist-only remains consistent across K ∈ {5, 10, 15, 20}, indicating robustness to the choice of K.

  • K Sensitivity Analysis: Across K ∈ {5, 10, 15, 20}, ActiveMimic consistently achieves higher top-K% activation overlap than ActiveMimicwrist-only.This extends the K = 10 analysis reported in Sec. 4.5 and confirms the conclusion is robust to K selection.

D Limitations and Future Directions

ActiveMimic’s limitations concern corpus scale, embodiment coverage, label fidelity, and evaluation scope. Future work can expand pretraining data and validate the framework for loco-manipulation and human-robot interaction.

  • Data scale: The current corpus contains approximately 10 hours of filtered Ego4D egocentric manipulation video, while larger corpora could further strengthen pretrained representations.The full Ego4D and Ego-Exo4D datasets can be incorporated using the same automated procedure.
  • Embodiment diversity: Real-world experiments use a single humanoid platform, so broader embodiment validation remains a natural next step.Because pretraining is embodiment-agnostic, extending to other embodiments requires only robot-specific training with corresponding demonstrations.
  • Label fidelity: Vision-based pose estimation provides action labels but introduces estimation noise, although downstream performance indicates robustness to moderate label noise.Better off-the-shelf pose estimation methods could further improve label quality.
  • Loco-manipulation: ActiveMimic could extend beyond stationary tabletop and shelf manipulation to humanoid loco-manipulation using walking-while-manipulating footage already present in egocentric datasets.Such settings require coordinating locomotion and manipulation simultaneously.
  • Human-robot interaction: Extending ActiveMimic to human-robot interaction could leverage egocentric video’s coverage of daily activities involving people and shared-workspace responses.These scenarios require robots to perceive and respond to human actions.
Loading 2606.06194v1…