PaperlayerView exact revision on arXiv

Source-linked AI summary

Ego-OSCAR: Egocentric Open source Stereo CAptuRe System

Gunjan Paul, Senthil Palanisamy, Satpal Singh Rathore, Pratyush Kumar Patnaik, Shubhanshu Khatana, Abhishek Anand

arXiv:2608.08285v2cs.CVcs.ARcs.RO

TL;DR

Egocentric data collection lacks an open, reproducible stereo-inertial platform suitable for distributed deployment. Ego-OSCAR provides one with an accompanying dataset and achieves a 96% usable-session rate across approximately 550 camera-hours.

  • Problem

    Existing egocentric capture ranges from monocular consumer cameras without synchronized inertial streams to high-fidelity closed platforms, leaving no freely reproducible stereo-inertial option for scaled collection.

  • Method

    Ego-OSCAR combines an open, COTS-based head-mounted synchronized stereo camera, 6-axis IMU, embedded recorder, watchdog controller, and open processing pipeline.

  • Results

    96% of sessions produced usable end-to-end data across 1,462 sessions, yielding approximately 550 hours per camera from 25 contributors across 40+ environments.

  • Takeaways & Limitations

    The system and dataset validate low-cost, open stereo-inertial capture at distributed deployment scale.

  • Takeaways & Limitations

    The paper does not evaluate whether policies trained on Ego-OSCAR outperform policies trained on existing corpora.

Abstract

from arXiv · show

We present Ego-OSCAR, an open-hardware, low-cost, head-mounted stereo-inertial capture device for egocentric data collection in the wild. EgoOSCAR pairs a hardware-synchronized global-shutter stereo camera with a 6- axis IMU, an embedded Linux SBC for on-device video encoding, and a realtime microcontroller for user feedback and watchdog functions. The complete bill of materials is under USD 200 per unit, using only commercially available components and 3D-printed parts. Alongside the device, we release a complete software stack (hardware-accelerated recording pipeline, IMU sampling daemon, time-synchronization tooling, and watchdog firmware) and roughly 550 hours of egocentric stereo video per camera with synchronized IMU, collected by a distributed contributor network across everyday indoor environments. The release is annotated rather than raw: free-form action captions cover essentially the entire recorded timeline with an open vocabulary, and per-frame 3D hand reconstructions ship alongside per-session stereo calibration. Ego-OSCAR does not aim to match the per-unit fidelity of research-grade systems such as Project Aria; it aims to be the cheapest defensible substrate for crowdsourced egocentric capture, and to lower the activation energy for any team that wants to collect egocentric data at scale. All hardware designs, software, and the dataset are open-sourced

1 Introduction

Egocentric capture offers scalable, diverse data for embodied AI, but existing systems trade away precision, reproducibility, or both. Ego-OSCAR addresses this gap with open hardware, an open-source capture pipeline, and the approximately 550-hour Ego-OSCAR-550h dataset.

  • Motivation: Growing VLA and world-model data demands have shifted the bottleneck from model architecture toward diverse, large-scale, multimodal interaction datasets.The introduction emphasizes data collection, curation, and annotation as increasingly critical alongside model development.
  • Egocentric data collection: Egocentric video provides scalable coverage of diverse environments, objects, and interactions, but lacks direct robot-state and end-effector-pose information.Instrumented handheld interfaces such as UMI partially address this precision limitation by recording end-effector trajectories alongside egocentric video.
  • Capture-system gap: Existing capture options leave a reproducibility gap: Ego4D uses monocular rolling-shutter cameras without hardware-synchronized inertial streams, whereas Project Aria is closed.The gap is consequential because modern VLA pipelines increasingly condition on camera pose in a world frame.
  • Contributions: Ego-OSCAR contributes an open-hardware capture device with full CAD, bill of materials, wiring, and assembly documentation.The device contribution is presented as the first of three concrete contributions.
  • Contributions: 209,315 free-form action segments accompany the approximately 550-hour Ego-OSCAR-550h stereo-video dataset, which includes synchronized IMU and per-frame hand detections rather than raw streams.The release also includes an open-source recording daemon, IMU sampler, time-synchronization tooling, and watchdog firmware.

2 Related Work

Related work spans large-scale egocentric datasets, research-grade and commodity capture devices, downstream value for robotics, and open-hardware efforts that lower replication barriers. Ego-OSCAR is positioned within this landscape as an accessible, open approach to head-mounted egocentric capture.

  • Egocentric datasets: Ego4D established large-scale unscripted egocentric data collection, followed by paired egocentric-exocentric, kitchen-domain, and motion-capture or 3D-pose datasets.The cited lineage includes Ego-Exo4D, EPIC-KITCHENS, Aria Everyday Activities, and Nymeria.
  • Capture devices: Project Aria offers a high-quality research-grade sensor suite but is closed and not freely reproducible or adaptable by the community.HoloLens and Magic Leap provide comparable sensor suites at substantially higher cost, while GoPro has been widely used for egocentric capture.
  • Downstream value: 51% improvement over robot-only baselines was reported by EgoHumanoid on humanoid loco-manipulation in unseen environments.Its alignment pipeline combines robot-free egocentric human demonstrations with limited robot data to reduce the view and action gap.
  • Accessible capture: 200 hours across 584 sessions were released by MobileEgo Anywhere using commodity smartphone sensing for long-horizon egocentric capture.Smartphones remove the hardware barrier but provide no hardware-level stereo synchronization and no control over shutter or exposure timing.
  • Open hardware: ALOHA, Mobile ALOHA, and UMI exemplify open robotic hardware designed for rapid replication, a lineage Ego-OSCAR extends to head-mounted observation-only capture.The shared thesis is that commoditizing capabilities enables scale.

3 Method

Ego-OSCAR is an open, reproducible head-mounted stereo-inertial system built around affordable commercial components, hardware-synchronized sensing, and deployment robustness. Its method combines stereo capture, embedded recording, inertial timestamping, clock bridging, and watchdog supervision in a lightweight wearable device.

  • Design principles: The system is engineered around affordability, reproducibility, sensor fidelity, and deployment robustness using commercial components, open designs, synchronized sensing, and real-time fault detection.The enclosure is 3D-printable, and the hardware uses no custom PCBs or proprietary silicon.
  • Hardware architecture: The device integrates hardware-synchronized stereo cameras, an SBC for encoding and storage, a 6-axis IMU, and an MCU for timestamping, feedback, and watchdog functions.Total assembled mass is approximately 280 g and the device mounts via a commodity sport visor.
  • Stereo sensing: The stereo module captures synchronized 1280×720 video at 30 FPS per camera, with a 126° per-sensor FOV and 42 mm baseline.Global shutter avoids per-row temporal offsets that can corrupt visual-inertial estimation during rapid head motion.
  • Embedded recording: The RK3588-based Radxa Rock 5C uses hardware MJPEG decoding and H.264 encoding to sustain 30 FPS stereo without frame drops within a wearable power budget.The design selects the RK3588 as a compromise between media acceleration, power draw, and physical size.
  • Synchronization: The ESP32-S3 timestamps every camera exposure, samples the IMU at 120 Hz, forwards the merged stream over UART, and enables offline alignment of sensor timestamps to video frames.The recorded IMU contains 3-axis accelerometer and 3-axis gyroscope data, while the camera and IMU operate on independent clocks.
  • Deployment robustness: The MCU provides heartbeat-driven user feedback and detects missing SBC heartbeats within two seconds, while USB-PD power supports 5–6 hours of capture with hot-swappable batteries.Field deployment identified strap pressure points, forward center-of-mass droop, and lack of moisture resistance as durability issues.

4 Evaluations

Ego-OSCAR meets its core sensor requirements, supports stereo depth and hand-centric downstream use, and achieved 96% usable data across a six-month distributed deployment. Visual-inertial odometry remained limited by egocentric motion, scene dynamics, and IMU bias, so the reported 12/20 result is a convergence rate rather than a trajectory-accuracy measure.

  • Sensor Fidelity: 0.4 px mean per-pixel epipolar error was achieved across all 13 deployed devices after rectification.Per-session calibration removes distortion from the 126° per-sensor FOV, and both SGBM and RAFT-Stereo recover disparity across the full FOV.
  • Sensor Fidelity: 700 µs residual visual-inertial lag remained after per-session offset correction.The lag was validated using Kalibr’s Cam-IMU offset test.
  • Data Utility: 12/20 held-out sequences produced stable VINS-Fusion trajectories, while 8 diverged.The result reflects convergence rather than metric pose accuracy: no motion-capture or surveyed ground truth was available, so ATE and RPE were not reported.
  • Data Utility: 94% of decoded video frames contained at least one detected hand above the default confidence threshold.This full-corpus rate measures hand visibility and detector coverage, not detection accuracy.
  • Deployment Scale: 96% of sessions produced usable data end-to-end across the six-month deployment.The watchdog detected thermal shutdowns, SD-card I/O errors, and cable strain, while added heatsinking, per-batch validation, and connector reinforcement addressed the dominant failure modes.

5 The Ego-OSCAR-550h Dataset

Ego-OSCAR-550h is a large, annotated egocentric stereo-inertial corpus collected across distributed indoor environments. It combines full-timeline open-vocabulary action segments and per-frame hand detections with calibrated global-shutter stereo, and releases the complete hardware, software, and dataset stack.

  • Dataset scale and composition: 1,462 sessions provide approximately 550 hours of egocentric stereo video per camera across 40+ predominantly residential indoor environments.Collection ran across a contributor network in India over roughly six months, ending in Q1 2026.
  • Annotations: The release covers essentially the entire recorded timeline with open-vocabulary action segments and per-frame hand detections over the full corpus.The corpus is annotated rather than raw, using two complementary annotation layers.
  • Comparison with existing corpora: Ego-OSCAR reports roughly 380 labeled segments per recorded hour, denser per hour than the compared egocentric corpora.Compared datasets include closed-taxonomy or incomplete-coverage annotations, while Ego-OSCAR uses open-vocabulary full-timeline coverage.
  • Sensing and labels: 550/cam (1,100 cam-h) is paired with calibrated RGB stereo, global shutter, per-session calibration, and 120 Hz inertial data.The release reports 209,315 segments with approximately 100% coverage.
  • Open release: CAD, assembly, bill of materials, firmware, capture, calibration, synchronization, and dataset artifacts are all released under permissive licensing.The released software includes recording, hardware-accelerated encoding, segmentation, upload, calibration, and time-synchronization components.

6 Limitations

Ego-OSCAR validates its sensor package rather than robot-learning gains, lacks pose ground truth, and is limited by a concentrated contributor base and domestic activity mix.

  • Evaluation scope: The evaluation does not show that policies trained on Ego-OSCAR outperform policies trained on existing corpora.The authors identify closing this robot-learning gap as the most important follow-on work.
  • Measurement limitations: Without pose ground truth, the paper reports visual-odometry convergence but no ATE or RPE.This limits quantitative assessment of trajectory accuracy.
  • Dataset coverage: The corpus is concentrated among 25 contributors in India using 13 shared devices, with domestic rather than open-domain activities.These sampling characteristics constrain the breadth of the released data.

7 Conclusion

Ego-OSCAR is a low-cost, open-hardware stereo-inertial system for scalable egocentric data collection. At a complete BoM cost of ∼USD 200, it captures hardware-synchronized stereo video and 6-axis IMU data and includes an integrated recording pipeline.

  • Conclusion: ∼USD 200 complete BoM cost enables Ego-OSCAR’s low-cost, open-hardware stereo-inertial design for egocentric data collection at scale.The device uses commercially available components and 3D-printed parts.
  • Conclusion: Hardware-synchronized stereo video and 6-axis IMU data are captured by the Ego-OSCAR device.The system is built for egocentric data collection.
  • Conclusion: The accompanying pipeline provides hardware-accelerated encoding, cross-domain time synchronization, watchdog-protected recording, and post-session upload.These components support the complete capture workflow.

A Released Dataset: Detailed Description

Ego-OSCAR-550h is a first-person calibrated stereo RGB corpus of everyday human manipulation, with synchronized camera pairs and per-session stereo calibration. Its strongest differentiator is dense, free-form action captioning on second-scale spans, supported by sessions selected for hand visibility and broad task coverage.

  • Dataset composition: Ego-OSCAR-550h is a first-person calibrated stereo RGB corpus of everyday human manipulation across objects, materials, tools, and multi-step activities.The corpus captures visual geometry of hands, object interaction, state change, and task progression.
  • Dataset composition: Every session contains a synchronized left/right camera pair with per-session stereo calibration.This provides calibrated stereo data for analyzing hand and object interactions.
  • Action annotations: Every recording is segmented into second-scale spans, each carrying a descriptive natural-language caption.The captions use a free-form vocabulary and are densely applied across the recordings.
  • Action annotations: Hand-visible sessions combine repeated coverage of foundational skills with a long tail of rare task expressions.The selection produces consistently rich hand–object interaction data.

A.1 Headline Statistics

The dataset comprises 1,462 calibrated stereo sessions with near-complete action coverage, broad verb–object diversity, and synchronized 6-axis IMU in most sessions. It also exhibits a substantial long tail and distributed collection across contributors and devices.

  • Dataset scale: 1,462 stereo sessions yield 2,924 video files, with per-session stereo calibration.Each stereo session contributes left and right video files.
  • Annotation coverage: Labeled action segments cover ≈100% of the recorded timeline, with a median of 94 labeled segments per session.95.8% of sessions show 10+ distinct task expressions.
  • Annotation diversity: 460 action verbs and 32,630 object phrases form 57,104 distinct verb–object combinations.This vocabulary supports broad open-world action coverage.
  • Annotation diversity: The top-20 expressions account for only 1.5% of all instances, indicating a pronounced long tail.The distribution is characterized as a genuine long tail for open-world learning.
  • Sensor synchronization: Per-session 6-axis IMU is synchronized to video in 1,271 of 1,462 sessions (86.9%).Synchronization coverage is reported at the session level.
  • Collection distribution: Collection spans 25 contributors across 13 shared devices.The contributor count uses unique user IDs.

A.2 Per-Session Contents and Release Format

Each session releases synchronized stereo video, annotations, calibration, and session metadata in a uniform per-session structure. IMU data is provided for most sessions, while video remains in the device-recorded H.264 format at 30 FPS.

  • Annotations and calibration: Each session includes ordered free-form action captions, per-frame hand localizations, and per-session stereo calibration.Calibration contains pinhole and radial-tangential intrinsics plus stereo extrinsics.
  • Inertial data: ∼120 Hz 6-axis IMU data is present in 1,271 of 1,462 sessions (86.9%).The IMU is distributed as a CSV file where present.
  • Release format: Video is released as original device-recorded H.264 in 5-minute MP4 clips, with one uniformly structured directory per session.Each directory contains left and right video, labels, hand detections, calibration.json, and the IMU CSV where available.

A.3 Annotation Layers … A.13 Summary of Distinguishing Properties

Ego-OSCAR releases a large, calibrated stereo-inertial corpus with dense free-form action and hand annotations spanning the recorded timeline. Its documented scope emphasizes long-tailed domestic activity, session-level geometric metadata, multimodal files, and explicit limitations around unavailable object pose and unreliable camera trajectories.

  • A.3 Annotation Layers: ≈100% of the recorded timeline is covered by 209,315 second-scale free-form action segments, while per-frame hand detections cover the full corpus at a 94% detection rate.The annotations provide dense temporal supervision rather than clip-level tags, with a median of 94 segments per session.
  • A.4 Task, Object and Contributor Diversity; A.5 Task-Diversity Structure: 460 action verbs and 32,630 object phrases form 57,104 distinct verb–object pairs, with the top-20 task expressions representing only 1.5% of activity instances.The corpus retains broad manipulation coverage and a genuine long tail rather than imposing a fixed taxonomy.
  • A.6 Activity and Environment Composition: Keyword-derived activity families show a kitchen-centric corpus, while roughly a third of labeled time falls outside cooking and dishwashing across textile, laundry, craft, cleaning, and organizing activities.These family assignments are approximate orientation categories derived from free-form captions, not ground-truth labels.
  • A.9 Per-Session Stereo Calibration: Per-session calibration supplies metric stereo geometry with a ≈42 mm baseline and sub-0.03 px per-camera reprojection error.The release includes camera intrinsics, distortion coefficients, stereo extrinsics, and calibration metadata for each session.
  • A.10 Session Length; A.13 Summary of Distinguishing Properties: Sessions have a median duration of 14.9 minutes and extend to 211 minutes, supporting both focused episodes and long-horizon activity.The distinguishing-properties summary also reports ≈550 labeled hours per camera across 1,462 stereo sessions.
Loading 2608.08285v2…