PaperlayerView exact revision on arXiv

Source-linked AI summary

MobileEgo Anywhere: Open Infrastructure for long horizon egocentric data on commodity hardware

Senthil Palanisamy, Abhishek Anand, Satpal Singh Rathore, Pratyush Patnaik, Shubhanshu Khatana, Ekaksh Janweja

arXiv:2605.05945v7cs.CVcs.CL

TL;DR

Long-horizon egocentric datasets are limited by short episodes and high collection-hardware barriers. MobileEgo Anywhere uses commodity smartphone sensing, open tooling, and a free capture app to produce long-form data, while mid-training a VLA lowers held-out action-prediction loss by 19%.

  • Problem

    Existing egocentric datasets have short episodes and high collection-hardware barriers, limiting evidence for long-horizon robotic task dependencies.

  • Method

    MobileEgo Anywhere combines smartphone visual-inertial pose tracking with a free capture app, the open-source STERA pipeline, and a 200-hour egocentric dataset.

  • Results

    Held-out action-prediction loss decreases from 0.265 to 0.214 (−19%) after mid-training a VLA on the egocentric trajectories.

  • Takeaways & Limitations

    The released framework and dataset lower the barrier to creating VLA-ready egocentric datasets and support work toward more generalizable robot policies.

  • Takeaways & Limitations

    The training-signal validation is open-loop and demonstrates trainability rather than policy-level task success.

Abstract

from arXiv · show

Vision-language-action (VLA) models have driven demand for large-scale egocentric datasets, yet the hardware and infrastructure to collect long-horizon data remain inaccessible. Datasets today typically have episodes only a few minutes long, which fails to capture the long-horizon temporal dependencies that complex robotic task execution requires. We present MobileEgo Anywhere, a framework for collecting hour-plus egocentric trajectories on commodity mobile hardware that uses modern smartphone sensors for long-term pose tracking without the hardware barriers of traditional robotics data collection. We release three components: (1) STERA, an open-source video-processing pipeline that converts raw mobile captures into standardized, training-ready formats for VLA and foundation-model research; (2) a free mobile app that lets any user record egocentric activity; and (3) a 200-hour dataset of diverse, long-form egocentric data with persistent state tracking across 584 sessions. We further show this data is a usable training signal:mid-training a VLA on it lowers held-out action-prediction error.

I. INTRODUCTION · II. RELATED WORK · Egocentric Datasets for Robotics

VLA progress depends on larger, more varied egocentric datasets, while existing resources remain limited by short episodes and collection-hardware barriers. MobileEgo Anywhere addresses these constraints with smartphone-based pose tracking, open tools, a free capture app, and a 200-hour dataset validated for robotics learning.

  • I. INTRODUCTION: VLA training combines internet video, simulation, and egocentric human interfaces, but only egocentric data provides rich interaction signals for varied, long-horizon pretraining.Internet video lacks contact dynamics, while simulation retains a sim-to-real gap.
  • I. INTRODUCTION: MobileEgo Anywhere uses smartphone visual-inertial odometry, specifically ARKit on iPhone Pro, for 6 DoF pose tracking without specialized peripherals.The framework directly targets short episodes and high collection-hardware barriers in existing egocentric datasets.
  • I. INTRODUCTION: The release includes a free capture app, open-source processing suite STERA, and a 200-hour household-activity dataset with continuous episodes up to 108 minutes.The dataset also includes 3D hand trajectories and three-level hierarchical language annotations.
  • I. INTRODUCTION: The pipeline is evaluated through ARKit pose accuracy, ground-truth-free hand-pose consistency, hierarchical label quality, and vision-language-action training utility.These validation axes cover tracking, hand-pose consistency, annotation quality, and downstream learning signal.
  • Egocentric Datasets for Robotics: Early egocentric datasets such as Ego4D and EPIC-KITCHENS offered thousands of hours for recognition and human-object interaction but were largely passive and lacked continuous 6 DoF pose tracking.The rise of Foundation Models and VLA architectures has increased demand for actionable egocentric data for robotic policy learning.

Scalable Data Collection Interfaces · Long-Term Egocentric SLAM and State Estimation · III. SYSTEM OVERVIEW

MobileEgo Anywhere combines commodity mobile capture with ARKit-based 6 DoF tracking and offline processing for long-horizon egocentric data collection. Its system uses a head-worn LiDAR iPhone, hands-free voice control, synchronized sensor logging, and STERA conversion into structured trajectories and action annotations.

  • Scalable Data Collection Interfaces: Prior demonstration interfaces scale poorly because each sample requires an operator and robot, while UMI still needs specialized mounts and calibrated setups.
  • Long-Term Egocentric SLAM and State Estimation: Long-term egocentric SLAM is difficult because COLMAP becomes computationally intractable on hour-long trajectories and ORB-SLAM3 accumulates drift in dynamic or texture-poor indoor scenes.
  • III. SYSTEM OVERVIEW: The system uses a LiDAR-enabled iPhone Pro on a head-worn rig to capture a first-person view of the participant’s hands and workspace.
  • III. SYSTEM OVERVIEW: ARKit captures synchronized RGBD streams and 6 DoF camera poses, while the mobile app exports RGBD frames, IMU readings, and camera intrinsics in MCAP format.
  • III. SYSTEM OVERVIEW: STERA transforms raw mobile logs offline into 3D hand trajectories, atomic action labels, and a hierarchical instruction tree.
  • III. SYSTEM OVERVIEW: Contributors record from a consistent egocentric field of view using a head-worn mount, with any sufficiently elevated mounting hardware compatible with the setup.
  • III. SYSTEM OVERVIEW: Voice commands trigger recording hands-free, while ARKit performs real-time sensor fusion to generate 6 DoF camera poses during capture.

3D Hand Trajectory Estimation · Atomic Action Labels

The paper estimates biomechanically constrained 3D hand trajectories and anchors them globally for imitation learning, while automatically generating detailed atomic action labels across 200 hours of video. Automated labels are more descriptive and temporally cleaner than human annotations, with no temporal defects across 5,249 labels.

  • 3D Hand Trajectory Estimation: WiLoR with MANO parameterization estimates 3D hand joints under biomechanical constraints for VLA training.The method is designed to handle partial occlusions common in first-person manipulation more reliably than alternatives such as MediaPipe.
  • 3D Hand Trajectory Estimation: WiLoR’s relative 3D coordinates are localized into a global frame by sampling ARKit depth maps at detected joint locations.This supplies depth information at the estimated joint positions before global transformation.
  • 3D Hand Trajectory Estimation: Applying the extrinsic camera transformation yields world-anchored trajectories for imitation learning.The transformation converts camera-relative estimates into a global coordinate system.
  • Atomic Action Labels: Action-conditioned VLA policies require labels specifying the manipulated object, action, and destination rather than generic descriptions such as “pick up object.”These labels are intended to capture the object and motion details needed for action conditioning.
  • Atomic Action Labels: The automated annotation pipeline partitions raw video into contiguous, non-overlapping temporal spans and prompts a VLM to produce constrained imperative sentences.The pipeline was used to generate labels across 200 hours of video.
  • Atomic Action Labels: 7.95 words versus 2.94 average label length demonstrates that automated labels are more descriptive than human annotations.Automated labels also averaged 1.09 descriptive modifiers versus 0.09 for human annotations.
  • Atomic Action Labels: 0 temporal defects occurred across all 5,249 automated labels, whereas human annotations contained 63 nonpositive-duration segments and 877 overlapping consecutive pairs.The 877 overlaps represented 9.9% of 8,821 adjacent human-annotation pairs and could propagate as corrupted training samples.

Hierarchical Task Instructions … Motion Capture Ground Truth Comparison

MobileEgo Anywhere organizes long-horizon egocentric activity into validated hierarchical instructions and evaluates consumer-device pose tracking against Vicon motion-capture ground truth. The dataset provides 584 sessions totaling 200 hours, while ARKit maintains strong relative and local pose consistency across varied motion profiles.

  • Hierarchical Task Instructions: Atomic span captions form a three-level instruction tree spanning session-level goals, sub-goals, and episodes.A language model groups temporally contiguous spans into episodes, clusters related episodes into sub-goals, and synthesizes session-level goals.
  • Hierarchical Task Instructions: Six of seven evaluated language models produced fully valid hierarchical outputs under three enforced invariants.The invariants require unique span assignment, exact timestamp boundaries, and full session coverage without gaps.
  • IV. DATA QUALITY VALIDATION: 584 sessions totaling 200 hours come from 20 contributors, averaging 20.5 minutes and reaching a maximum of 108 minutes.The dataset combines consumer hardware with continuous 6 DoF pose, LiDAR depth, and MANO annotations, while sessions exceed one hour.
  • Motion Capture Ground Truth Comparison: ARKit pose accuracy was evaluated against ground truth from a 30-camera Vicon motion-capture system across representative motion profiles.The trajectories included household tasks, a slow walk, a closed-loop traversal, and a spinning sequence.
  • Motion Capture Ground Truth Comparison: Table II reports ATE RMSE, relative ATE, and translational and rotational RPE against Vicon ground truth.The evaluation covers per-sequence translational and rotational trajectory errors.
  • Motion Capture Ground Truth Comparison: Relative ATE stays below 1% for nine of ten sequences, rotational RPE below 4°, and translational RPE below 5 cm throughout.The short spinning sequence is the lone exception, with elevated values attributed to rapid rotational motion; local pose consistency remains strong.

Long-Term Drift Evaluation · V. TRAINING SIGNAL VALIDATION

MobileEgo Anywhere exhibits low long-term pose drift on commodity mobile hardware and provides a usable training signal for vision-language-action models. Mid-training VITRA on the collected trajectories reduces held-out action-prediction loss, especially for finger articulation.

  • Long-Term Drift Evaluation: Drift stays below 1 cm in five of six environments, with 1.5 cm at whole-house traversal session end.ArUco markers were revisited at session start, midpoint, and end to assess long-term drift.
  • Long-Term Drift Evaluation: Drift remains below 0.1% of trajectory length in every tested environment.The evaluation spans six environments and supports ARKit tracking for downstream VLA applications.
  • V. TRAINING SIGNAL VALIDATION: VITRA is mid-trained on MobileEgo Anywhere trajectories and evaluated on 27 held-out sessions disjoint from training.The setup initializes from VITRA-VLA-3B, freezes the vision encoder, conditions on hierarchical instructions, and supervises MANO hand-pose prediction for 10,000 steps.
  • V. TRAINING SIGNAL VALIDATION: Held-out action-prediction loss decreases monotonically from 0.265 to 0.214 (−19%).Because evaluation sessions are never seen during training, the reported decrease reflects generalization rather than memorization.
  • V. TRAINING SIGNAL VALIDATION: Finger articulation loss decreases 24% per hand, compared with a 14% decrease for the wrist root.The larger finger-articulation improvement indicates learning of fine, dexterity-relevant hand structure carried by the labels.
  • V. TRAINING SIGNAL VALIDATION: The action-prediction improvement is concentrated in finger articulation rather than the wrist root.The metric is computed on held-out sessions, and the comparison is reported as −24% per hand for fingers versus −14% for the wrist root.

VI. LIMITATIONS

The current system depends on iPhone Pro hardware and ARKit, with additional constraints on field of view and recording duration. Downstream validation demonstrates trainability through open-loop held-out prediction, but not closed-loop robot task success.

  • Platform dependency: The capture pipeline currently requires an iPhone Pro because STERA relies on ARKit’s visual-inertial odometry and LiDAR depth sensing.Android/ARCore offers lower VIO accuracy and lacks LiDAR depth, while depth-free fallback modes remain future work.
  • Platform dependency: The ultrawide lens is inaccessible during active ARKit sessions, limiting the field of view for wide-workspace activities.This restriction reduces coverage of broad workspaces during capture.
  • Thermal constraints on session length: Approximately two hours of continuous recording can trigger thermal throttling on iPhone Pro hardware.A heat sink attachment mitigates throttling but adds deployment friction in warm environments.
  • Preliminary downstream validation: The training-signal experiment is open-loop, evaluating held-out hand-action prediction without closed-loop robot rollout.It demonstrates trainability rather than policy-level task success, which remains future work.

VII. CONCLUSION

MobileEgo Anywhere provides an accessible, commoditized framework for large-scale, long-horizon egocentric data collection using consumer hardware. Its releases include a free mobile app, the open-source STERA pipeline, and a 200-hour dataset with rich spatial and hand-tracking signals.

  • Framework: MobileEgo Anywhere enables large-scale, long-horizon egocentric data collection through an accessible framework built for standard consumer hardware.The framework is intended for researchers and contributors worldwide.
  • Released components: The project releases a free mobile application and open-source STERA pipeline for generating VLA-ready datasets.These components let contributors use standard consumer hardware for data collection and processing.

APPENDIX · VIII. 3D HAND POSE CONSISTENCY · IX. HIERARCHICAL INSTRUCTION QUALITY

MobileEgo Anywhere’s hand-pose evaluation shows stable geometry, biomechanically plausible angles, and smooth wrist dynamics across 98 sessions. Its hierarchical annotations cover all 584 sessions with timestamp-aligned, gap-free spans organized across distinct temporal scales.

  • VIII. 3D HAND POSE CONSISTENCY: Hand detection succeeds on 86.2% of frames, while 247 zero-depth frames (0.02%) are discarded before ground-truth-free pose metrics.Evaluation covers 98 sessions, 1.19 M frames, and 25.2 hours using bone length constancy, joint angle plausibility, and wrist dynamics.
  • VIII. 3D HAND POSE CONSISTENCY: 1.27% and 1.43% are the median bone-length CVs for left and right hands, respectively, dropping below 1% when the noisy pinky distal phalanx is excluded.The pinky’s elevated approximately 7.5% CV reflects its short approximately 2 cm length and fixed absolute noise.
  • VIII. 3D HAND POSE CONSISTENCY: Over 99.99% of the 15 flexion angles fall within published biomechanical limits across all sessions, with unimodal distributions across grasp types.The angles cover MCP, PIP, and DIP joints for each finger.
  • VIII. 3D HAND POSE CONSISTENCY: 0.34 m/s and 0.27 m/s are the median left- and right-wrist velocities, while median accelerations are 2.7 and 1.5 m/s2, with smooth unimodal dynamics.Velocity medians remain below the approximately 0.62 m/s peak hand velocity reported for healthy adults during standardized drinking, with no teleportation-scale discontinuities.
  • IX. HIERARCHICAL INSTRUCTION QUALITY: 75,857 atomic spans form 9,922 episodes and 2,212 sub-goals across 584 sessions, with one-episode span assignments, exact timestamps, and complete gap-free coverage.The hierarchy was generated using DeepSeek V4 Flash with high reasoning.
  • IX. HIERARCHICAL INSTRUCTION QUALITY: 5 s atomic spans, 39 s episodes, 3.9 min sub-goals, and 16.8 min sessions occupy distinct temporal bands, with 80% of episodes containing ≤10 atomic spans.Adjacent levels show a natural 4–8× scale separation, and episode and sub-goal counts scale roughly linearly with session length.

X. MOCAP TRAJECTORIES AND RESULTS

Motion-capture trajectories are presented to validate ARKit accuracy, as discussed in Section IV-A1 and shown in Figure 6.

  • Mocap Trajectories and Results: Motion-capture trajectories validate ARKit accuracy and are shown in Figure 6.The validation setting is discussed in Section IV-A1.

XI. ETHICS AND PRIVACY

The study obtained informed consent for capture, processing, and public release, instructed contributors to avoid recording non-consenting individuals, and blurred accidentally captured faces during post-processing.

  • All contributors signed informed consent covering capture, processing, and public release.
  • Contributors were instructed to avoid recording non-consenting individuals, and accidentally captured faces were blurred during post-processing.
Loading 2605.05945v7…