EgoExplorer: A Vision-Motion-Touch Observatory for Human Loco-Manipulation In Situ

1. Introduction

• Existing portable datasets often omit parts of loco-manipulation: egocentric video leaves much of the body unobserved, body tracking often lacks finger and contact measurements, and hand datasets typically omit whole-body movement.These partial records obscure the coordination central to loco-manipulation.Source: §1 · p. 2
• EgoExplorer aligns egocentric video, reconstructed body motion, glove-measured finger articulation, and tactile observations across continuous tasks, with step-level descriptions.The recordings emphasize professional activities and preserve transitions among approaching targets, adjusting posture, manipulating objects, and relocating.Source: §1 · p. 2
• C3 separates fixed body-sensor geometry from motion inference through reusable calibration, temporal completion of unobserved joints, and refinement under layered constraints.The constraints promote measurement agreement, temporal continuity, and geometric consistency.Source: §1 · p. 2
• 71.9%: EgoExplorer reduces body-point error relative to the strongest evaluated body-reconstruction baseline, with better hand stability, kinematic target recovery, and regional contact estimation than evaluated alternatives.Preliminary studies explore multimodal prediction and human-to-robot motion transfer in simulation.Source: §1 · p. 3
• The contributions combine a lightweight, multimodal dataset with C3 reconstruction and experiments on annotation quality, design choices, and embodied-learning utility.The system uses sparse trackers to recover metric whole-body motion.Source: §1 · p. 3

Multimodal human datasets

• Calibrated capture offers detailed body, hand, and interaction geometry but requires instrumented spaces; egocentric video broadens activity coverage, while wearable body and hand datasets typically omit complementary signals.EgoExplorer combines these signals in portable recordings of continuous loco-manipulation.Source: §2 · p. 3

Motion reconstruction from partial observations

• Video-based reconstruction is sensitive to occlusion, depth ambiguity, and camera motion, while sparse-tracking methods use learned priors and physical regularization to complete motion.Sparse anchors still leave intermediate joints ambiguous, and mount errors can be absorbed into articulation.Source: §2 · pp. 3–4

Human data for embodied learning

• EgoExplorer preserves visual context, body coordination, and motion-contact relationships throughout continuous tasks, connecting the dataset to human-demonstration learning and whole-body control.Figure 2 summarizes its capture, calibration, completion, constraint, and annotation stages.Source: §2 · p. 4, Figure 2 · p. 4

3. The EgoExplorer Dataset

• EgoExplorer contains 2,715 recordings totaling 100 hours across ten task templates and five environments, with an emphasis on professional activities.The recordings include regular and recovery episodes, including responses to interruptions or failed attempts.Source: §3 · p. 4

3.1. Capture Hardware

• The capture system combines sparse body-joint tracking with direct finger sensing and optional touch in a compact, reusable rig suited to collection across environments.The rig uses five trackers for the head, hands, and ankles; calibration is reused while the sensors remain mounted.Source: §3.1 · p. 5

3.2. Whole-Body Reconstruction from Sparse Measurement

• C3 separates fixed body-sensor geometry from time-varying motion, using protocol-guided calibration, prior-guided completion, and joint refinement of skeletal agreement, surface clearance, and ground consistency.Glove-measured finger articulation and touch provide direct interaction evidence.Source: §3.2 · p. 5

3.2.1. Protocol-guided calibration

• Protocol-guided calibration matches prescribed participant movements to a reference motion library and fits body shape, scale, and sensor-to-anchor transforms for reuse with unchanged mounts.The protocol uses varied poses and motions, with phase matching to accommodate differences in execution speed.Source: §3.2.1 · pp. 5–6

3.2.2. Prior-guided motion completion

• With calibrated geometry fixed, a temporal prior completes unobserved articulation from the five anchors to produce an initial whole-body motion proposal.The proposal includes root orientation and local joint rotations; pelvis position is aligned using the calibrated head anchor.Source: §3.2.2 · p. 7

3.2.3. Layered constraints

• Three refinement layers fit tracker measurements and regularize motion, discourage participant-specific self-intersection, and penalize body vertices below the estimated ground plane.The ground term does not impose foot-contact or foot-locking constraints.Source: §3.2.3 · pp. 7–8

3.3. Multimodal integration and annotation

• Glove-measured finger geometry is fused to the reconstructed hands, while tactile observations are time-aligned and associated with hand regions; expert-guided annotations segment tasks and undergo manual multimodal review.Uncertain outcomes are left unspecified, and video timestamps align annotations with motion and tactile streams.Source: §3.3 · p. 8

4. Data Statistics

• 2,715 recordings totaling 100 hours span ten task templates across five environments, emphasizing professional activities.Figure 3 summarizes session durations and the balance between regular and recovery episodes.Source: §4 · p. 8

5. Experiments

• The Evaluation Subset measures kinematic accuracy, temporal quality, and contact accuracy against modality-specific references; Natural Recordings lack matched accuracy references and are assessed for sensor agreement, temporal quality, and geometric consistency.Full-body errors use tracker-fitted labels.Source: §5 · pp. 8–9

5.1. Annotation Reliability

• 18.0 mm body-point error and 20.3 mm error at non-tracked points; EgoExplorer outperforms all three baseline settings across every reported metric.The non-tracked points are 17 unobserved joints.Source: §5.1 · p. 9, Table 2 · p. 10
• 0.4 mm holding RMS versus 4.5 mm for HaWoR and 11.3 mm for HandFlow; regional contact IoU is 0.915 versus 0.621 for HACO and 0.324 for hand-pose-only inference.The glove setting also achieves 100% kinematic target recovery.Source: §5.1 · p. 9, Table 3 · p. 10, Table 4 · p. 10
• Expert review finds factual errors in 11.09% of segments, unsupported claims in 2.46%, missed steps in 8.16%, extra steps in 2.05%, and boundary corrections in 3.08%.These residual errors motivate manual verification and correction before release.Source: §5.1 · p. 9, Table 5 · p. 10

5.2. Preliminary Training Study

• Joint generation raises video PSNR from 15.94 to 16.62 dB and contact F1 from 0.610 to 0.694; video conditioning lowers action MSE from 0.0478 to 0.0277, and touch lowers it to 0.0264.The study uses sampled predictions at 5K training steps.Source: §5.2 · p. 10, Table 6 · p. 10
• Retargeting two recordings to a Unitree G1 with a ZeroWBC-style controller provides a preliminary demonstration of human-to-robot motion transfer in simulation.Figure 6 contrasts human motion in green with the G1 whole-body control policy in gold.Source: Figure 6 · p. 10, §5.2 · p. 11

5.3. Ablation Study

• Adding reconstruction Layers 1–3 lowers body-point error from 79.0 to 16.1 mm, crossing pairs from 72.8 to 38.7 per frame, and floor violations from 61.4% to 6.0%.Replacing calibrated shape and scale with the mean body raises body-point error from 18.0 to 59.5 mm and floor violations from 6.0% to 50.9%; omitting mount calibration raises body-point error to 95.0 mm during training or 98.1 mm at test time. Full has the lowest Avg. (28.4; mean of nine reported values), despite higher body-point error and jerk than Layer 1 alone. Figure 7 shows reduced self-intersection and floor penetration.Source: Table 7 · p. 10, §5.3 · p. 11, Figure 7 · p. 11

• EgoExplorer is an egocentric dataset aligning whole-body motion, hand articulation, contact, and descriptions for human loco-manipulation.Source: §6 · p. 11
• C3 recovers metric body motion from sparse wearable sensing through calibration, temporal completion, and layered constraints; evaluations show benefits of direct sensing.The paper presents this portable approach for capturing and studying coordinated movement and manipulation.Source: §6 · p. 11
• Reconstruction cannot guarantee exact recovery of all joints, tactile sensing has variable reliability, and preliminary studies leave real-world humanoid-learning benefits unverified.Source: §6 · p. 11

A. Capture Hardware

• Calibration parameters are reused while mounts remain fixed and recomputed after remounting; the rig combines five body trackers, articulation gloves, a head camera, and Wuji touch sensing.Native timestamps are preserved for alignment, while resampling interpolates existing measurements rather than adding observations.Source: A. Capture Hardware · p. 19, Figure 8 · p. 20

B. Data Format

• The release aligns episode rows to a 30 Hz grid, stores numerical data and annotations in Parquet and video in MP4, and preserves native timing, validity, and provenance.Episode-relative timestamps reset per episode; unavailable streams use validity masks rather than measured zeros, and interpolation or held values retain source-time information.Source: B. Data Format · pp. 19–21

C. Whole-Body Reconstruction from Sparse Measurement

• The reconstruction supplement retains the C3 profile, calibration variables, motion state, and losses, with optimization distances expressed in meters.It defines MS as the mean of squared scalar entries and specifies Euclidean and Frobenius norm conventions.Source: C. Whole-Body Reconstruction from… · p. 21

C.1. Protocol-guided calibration

• Calibration uses 17 staged body, hand, and root-motion actions; monotonic DTW matches tracker-distance and orientation patterns to reference phases despite differing speeds and pauses.Stable observations from static holds and observations within ±0.11 s of matched dynamic instants are aggregated; finger, turning, and stepping stages are retained for motion checks but excluded from body-profile fitting.Source: §C.1 · p. 21, READING THE PLATE · p. 23

READING THE PLATE

• The calibration fit jointly estimates body shape and scale, shared attachment translations, and recording alignment, then obtains attachment rotations through robust averaging with outlier rejection.A Huber position loss limits the influence of large residuals; measured height and arm span and geometry regularization help constrain body–mount compensation.Source: READING THE PLATE · pp. 23–24

C.2. Prior-guided motion completion

• A 40-frame transformer prior predicts 22 body rotations and two foot-contact logits, trained on public, synthetic, guide, and teacher-fitted motion sequences.Teacher fits provide task-domain supervision rather than independent ground truth; video and glove signals are not network inputs.Source: §C.2 · p. 25

C.3. Constraint layer 1: skeletal motion

• The skeletal constraint layer combines tracker agreement with temporal and pose regularizers; its torso-angle threshold is an engineering bias that may penalize genuine deep bends.The active configuration does not couple temporal constraints across chunk boundaries or use overlapping chunks.Source: §C.3 · pp. 25–26

C.4. Constraint layer 2: surface clearance

• The surface-clearance layer samples interior vertices from six body parts and penalizes nearby signed inter-part surface violations, aggregating the eight largest penalties per part pair.The 0.10 m support gate limits penalties to nearby points, and sampled vertices are model outputs rather than additional measurements or optimization variables.Source: §C.4 · p. 26
• The 0.018 m surface-clearance proxy can omit deep intersections and does not guarantee collision-free motion.It is not an exact signed distance field.Source: §C.4 · p. 27

C.5. Constraint layer 3: body-ground consistency

• The body-ground term uses a horizontal plane at estimated height g, set by default from ankle-tracker heights rather than scene geometry.The launcher uses the fifth percentile of valid ankle heights minus 0.12 m.Source: §C.5 · p. 27

C.6. Joint refinement

• Joint refinement keeps the calibrated profile fixed and jointly updates all three constraint layers, returning a finite-iteration estimate without a global-optimum guarantee.Chunks use Adam for 60 iterations, with bounded retries for failed chunks.Source: §C.6 · p. 27

D.1. Hand and touch fusion

• Hand fitting aligns glove landmarks to the calibrated model, then optimizes wrist-local finger articulation while preserving the reconstructed body and its attachments.The fitting uses 20 non-wrist landmarks per hand; hand scale is separate from body scale.Source: §D.1 · pp. 27–28
• Tactile outputs retain native values, baseline-relative changes, and regional contact with separate health or validity; contact is temporally associated with motion, not used to constrain body IK.The release rule ORs valid native-rate detections within each centered 1/30 s bin, masks bins without valid samples, and treats these as contact indicators rather than calibrated force or pressure.Source: §D.1 · p. 28, Table 10 · p. 28

D.2. Expert-provided procedural context

• Expert-provided procedural context supplies terminology and likely transitions, while observed video determines which steps are annotated and their order.The supplied context covers preparation, reaction, monitoring, workup, extraction, concentration, and purification.Source: §D.2 · p. 28

D.3. Example text prompt

• The example annotation prompt asks for time-ordered procedural segments grounded in video, using background context without inserting unobserved steps or outcomes.The requested fields include times, skill, description, object roles, and observed outcome.Source: §D.3 · pp. 29–30

D.4. Human verification and experimental interpretation

• Operators check action and object identification, omitted or extra steps, unsupported statements, and temporal boundaries during review.They review the egocentric video, reconstructed motion, available touch, and draft text together.Source: §D.4 · p. 30
• The procedural-document ablation holds the annotation model and other inputs fixed, testing professional background knowledge without model retraining.Source: §D.4 · p. 30
• Automated draft assessment is distinct from the expert review used for annotation verification.Source: §D.4 · p. 30

E. Data Statistics

• 2,715 recordings total 100 hours across five environments; glove streams cover 58% and 41% of regular recordings, and measured touch accompanies 41%.Laboratory activities account for 41% of recordings and 46% of duration; these percentages describe stream availability, not reconstruction accuracy.Source: E. Data Statistics · p. 30
• Short locomotion bouts alternate with sustained manipulation; study and living-room recordings include longer head paths, while ankle dynamics distinguish locomotion from stationary manipulation.Figure 11 summarizes head paths, bout durations, and five-tracker dynamics; Figure 12 aligns ego frames, speed, behavior, and posture across a reaction-setup episode.Source: E. Data Statistics · pp. 30–31, Figure 11 · p. 31, Figure 12 · p. 32
• Across healthy tactile streams, median contact-event frequency is 722 per hour; bimanual contact occupies 62% of industrial recording time, while laboratory recordings show 52% left-only contact.These summaries use 545 recordings for bilateral occupancy and 565 recordings for event frequency; events last at least 0.1 s, with gaps under 0.15 s merged.Source: E. Data Statistics · p. 31, Figure 13 · p. 33
• Regional contact values are median active fractions among contact frames, and regions with insufficient live-taxel support are marked n/a.Multiple regions can be active together, so values need not sum to one; n/a includes regions lacking three recordings with at least 30% live taxels.Source: E. Data Statistics · p. 31, Figure 14 · p. 34
• Caption intervals cover 99.9998% of video duration, with median segment duration 7.486 s and a 90th percentile of 24.000 s.Captions average 98.87 characters, and the exact duplicate rate is 0.0402%; these metrics describe coverage and granularity, while semantic quality is assessed separately.Source: E. Data Statistics · p. 32, Table 12 · p. 32

F.1. Evaluation setup

• The Evaluation Subset uses repeated static and dynamic body actions, held-out third repetitions, ten right-hand configurations, matched video recordings, and button-marked contact intervals.The controlled comparisons use modality-specific references and settings; those detailed settings do not apply to the entire collection.Source: §F.1 · p. 37, Table 13 · p. 37

F.2. Whole-body reconstruction: inputs and references

• Full-body evaluation labels are tracker-fitted references, not independent ground truth, and their shared model, calibration, and constraints can favor related methods.The labels were reviewed frame by frame and agree with trackers within 5 mm; the evaluation scores the same 22 body joints for all methods.Source: §F.2 · p. 38

F.3. Hand motion and contact capture

• Hand comparisons test pose retention and kinematic target recovery, while contact scoring uses task-specific regions and button-marked press intervals.Glove-derived prototypes exclude the tested repetition; contact masks exclude transition frames and regions outside the task mask from negative examples.Source: §F.3 · pp. 38–39

F.6. Preliminary training study

• Action MSE measures error in a normalized 57-dimensional hand-pose feature space, not millimeter-scale body-point or fingertip error.The pilot reports 5,000-step checkpoints using a sampled-evaluation implementation.Source: §F.6 · pp. 41–42
• Joint generation reports higher video PSNR and contact F1, and lower action MSE, than separate-model generation.The separate-model row combines scores from three independently trained models, not an ensemble; all output comparisons receive video, action, and touch history.Source: §F.6 · pp. 41–42
• The conditional-action study supplies recorded video, or video plus touch, from the same interval as the generated action, so it tests cross-modal conditioning rather than history-only prediction.All rows still receive video/action/touch history.Source: §F.6 · p. 42

F.7. Calibration and layered reconstruction ablations

• Layer ablations share five-anchor inputs, proposal checkpoint, calibrated profile, and iteration budget; the Full variant activates the corresponding refinement terms without retraining.Mean-body and no-mount-calibration variants change geometry or attachment handling, while uncalibrated-at-test omits attachment correction during inference and refinement.Source: §F.7 · p. 42
• The 64.1 mm AvatarPoser comparison against the 79.0 mm proposal-only row does not isolate refinement because those rows use different training pools and checkpoints.Within-cohort layer rows provide the appropriate comparison; lower crossing or floor violations can occur without improved articulation.Source: §F.7 · pp. 42–43

F.8. Professional-skill ablation

• Adding professional Skills documents lowers factual-error, unsupported-claim, and event-omission rates by 6.52, 5.52, and 7.05 percentage points, respectively.The annotation model, videos, task instructions, and decoding settings are held fixed; generated drafts still undergo human verification and correction.Source: §F.8 · p. 43, Table 15 · p. 43

F.9. Humanoid retargeting

• Kinematic retargeting has mean position errors of 6.8 and 7.0 cm for reaction-setup and reagent-weighing clips, with wrist errors of 7.5 and 7.8 cm.The robot-reference comparisons use nine corresponding body origins; residuals include differences between human joint centers and robot link origins.Source: §F.9 · p. 43
• Mean 14-body tracking errors are 4.6 and 3.6 cm with torso-position and yaw alignment, versus 6.6 and 4.8 cm without alignment, for the two clips.Against the same human targets used for kinematic retargeting, simulated trajectories instead have mean errors of 8.5 and 6.8 cm without evaluation-time alignment; these references differ and the errors should not be added.Source: §F.9 · p. 43
• Floor correction raises the root by 2.7–6.7 cm, while wrist error rises from approximately 5.6 cm to 7.5 and 7.8 cm after correction.The authors characterize the results as preliminary simulated motion transfer rather than precise object interaction; tighter tracking rewards and joint optimization remain future directions.Source: §F.9 · pp. 43–44
• The hand transfer matches fingertip vectors to a three-finger robot hand with 1.1–2.2 cm fingertip errors, and thumb–index aperture correlations of 0.991–0.993.The second rollout remains fall-free under the reported random base-velocity perturbations.Source: §F.9 · pp. 44–45
• The simulated tracking results replay training clips, so they do not establish generalization to unseen tasks or hardware deployment.Deployment would additionally require state estimation and safety constraints, which are not evaluated; fingers are replayed kinematically while the policy controls 29 body joints.Source: §F.9 · p. 45

F.10. Qualitative examples

• Qualitative examples show closer recovery of forward bends, lunges, and leg raises than six baselines, while temporal examples pair body motion with synchronized touch.Figure 21 shows matched instants from the Evaluation Subset; Figure 22 presents synchronized sequences across ten sessions.Source: §F.10 · p. 45, Figure 21 · p. 46, Figure 22 · p. 47

G. Discussion on Capture Trade-offs

• Table 16 summarizes capture arrangements and deployment requirements, while the annotation comparisons cover body, hand, and contact against evaluated baselines.The supplied discussion does not specify Table 16's individual arrangements or comparison outcomes.Source: G. Discussion on Capture Trade-offs · p. 48, Table 16 · p. 48
• Multi-view RGB capture needs overlapping views, synchronization, and camera calibration, which must be renewed when cameras move.Occlusion and fingers' small image scale make fine hand articulation challenging for multi-view methods.Source: G. Discussion on Capture Trade-offs · p. 48
• EgoExplorer avoids external camera-array and base-station setup and reuses body–sensor calibration while mounts stay fixed, but inside-out trackers require suitable visual conditions.Its wearable rig supports collection in existing environments and combines reconstructed body motion with hand and contact observations.Source: G. Discussion on Capture Trade-offs · p. 48
• The cost figures exclude computers, software licenses, accessories, taxes, shipping, and setup labor; prices are public listings recorded in September 2026.The MANUS subtotal uses a reseller listing excluding its software license; the Wuji price is converted to USD.Source: G. Discussion on Capture Trade-offs · p. 48
• At USD 69.99 per C920s, four to six RGB cameras cost less than the USD 1,184 tracker subsystem, while MANUS and Wuji subtotals are approximately USD 7,979 and USD 5,397.The wearable options trade added hardware for deployment flexibility and direct finger/contact measurements.Source: G. Discussion on Capture Trade-offs · p. 48, Table 17 · p. 49