PaperlayerView exact revision on arXiv

Source-linked AI summary

Everybody Tracking Every Body

Daeyun Shin, Yunhan Zhao, Shu Kong, Alexander C. Berg, Charless Fowlkes

arXiv:2608.29927v1cs.CV

TL;DR

The paper addresses multi-person 3D pose estimation when egocentric head-motion signals are continuous but exocentric visual observations are sparse and unreliable. It uses conditional diffusion to fuse these signals, trained on motion-capture and multi-person video data. Experiments report improved absolute and relative pose accuracy over motion-only and vision-only baselines.

  • Problem

    Multi-person egocentric pose estimation lacks a sufficient single source because head-motion estimates have limited pose accuracy while visual observations have sparse temporal coverage.

  • Method

    A conditional diffusion model fuses egocentric camera trajectories with variable noisy exocentric observations, using motion-capture and multi-person video training data.

  • Results

    The proposed fusion approach improves estimation accuracy and outperforms approaches that address only individual aspects of the problem.

  • Takeaways & Limitations

    The framework jointly estimates consistent multi-person 3D motion in a global coordinate frame, including people not visible in any camera view.

  • Takeaways & Limitations

    Generalization to other activity types, participant counts, and device configurations remains to be established because available data are limited and evaluation ground truth is imperfect.

Abstract

from arXiv · show

We address the problem of 3D body pose estimation of multiple interacting people from their egocentric views with centralized coordination. Each individual wears a camera recording egocentric video and IMU data. Processing this video with VIO SLAM provides high-quality tracking of each egocentric camera through space. The first-person view from one individual provides third-person observations of other people, although these exocentric observations are sparse, intermittent, and of highly variable reliability as both cameras and subjects move. To integrate these synchronized data streams, we propose a diffusion-based approach that fuses estimates of pose based on head motion derived from egocentric camera motion with exocentric pose observations, conditioning on both observation content and reliability. Our model is trained on a mixture of single-person motion-capture data and multi-person video in order to learn rich priors for body motion trajectories and video observation reliability. Evaluation on challenging multi-person datasets suggests our fusion approach improves over motion-only and vision-only baselines in terms of both absolute and relative pose accuracy.

1. Introduction

The paper formulates multi-person egocentric pose estimation as fusing complementary ego and exo signals. It introduces conditional diffusion to combine continuous head-motion information with sparse visual observations and reports improved fusion performance.

  • Motivation: Multi-person egocentric cameras create an ego–exo duality: each wearer is mostly unseen by their own camera but observed by others.These complementary views support estimating all participants in a shared global coordinate frame.
  • Motivation: Head-motion priors provide full temporal coverage and global positioning but weak relative limb-pose accuracy, whereas visual observations are more accurate but intermittent.Approximately 24% of person-frames lack an observation from any camera in the evaluation setting.
  • Approach: The proposed conditional diffusion model integrates an egocentric camera trajectory with variable numbers of noisy exocentric observations.Training combines AMASS motion-capture data with smaller multi-person datasets to learn motion priors and calibrate sparse observation reliability.
  • Results: The proposed conditional model outperforms approaches addressing only individual aspects of the fusion problem.The evaluation compares against direct-imputation and diffusion-posterior-sampling alternatives.
  • Contributions: The method formulates joint tracking of multiple people, including individuals not visible in any camera view.It predicts consistent multi-person 3D motion in a global coordinate frame from variable egocentric and exocentric observations.

2. Related Work

Prior work separately models pose from moving-camera or exocentric observations and uses generative priors for partial conditioning. The paper’s setting combines their complementary strengths while addressing intermittent visibility and global-position errors.

  • Pose estimation with moving cameras: Moving-camera pose methods jointly estimate camera trajectories and human pose, while egocentric priors use head motion to estimate the wearer’s body.These approaches provide dense wearer estimates but weakly constrain detailed limb pose.
  • Exocentric pose estimation: Exocentric methods estimate visible people from third-person images or videos, providing direct body evidence only when subjects are visible.In multi-person egocentric settings, limited fields of view cause people to frequently appear and disappear from observation.
  • Conditional body motion generation: Diffusion and other generative motion models combine learned motion priors with partial observations, including noisy or sparse conditioning signals.Examples include MDM, PriorMDM, HumanMAC, OmniControl, CondMDI, HuMoR, and RoHM.
  • Paper setting: The paper’s fusion setting targets the complementary weaknesses shown in Figure 2: ego estimates have limb-pose errors, while exo estimates have scale, offset, and coverage problems.The reported evaluation setting has 24% missing exocentric predictions for person-frames.

3. Method

The method represents each person’s body and root pose in a shared world frame, combining egocentric camera motion with sparse exocentric observations in a conditional diffusion model. It accommodates variable observation availability while using camera geometry and pose priors to estimate complete trajectories.

  • Problem formulation and representation: The system estimates every person’s body state and root-joint pose at every timestep in a synchronized common world coordinate frame.It uses synchronized camera feeds and pose estimates, with the world frame defined as z-up and the ground plane at z = 0.
  • Problem formulation and representation: Each body is represented with the 496-dimensional SMPL-H state, including shape, pose, contact flags, and hand rotations.The body root is anchored through the Central Pupil Frame, defined from the midpoint between the pupils and aligned with the SMPL head joint.
  • Egocentric and exocentric conditioning: Egocentric camera trajectories provide full temporal coverage and absolute positioning, but leave substantial uncertainty in detailed limb pose.The camera-to-body mapping uses the head-anchored Central Pupil Frame and its inverse to relate camera and root-joint poses.
  • Egocentric and exocentric conditioning: Exocentric detections are transformed into the target person’s root frame using relative camera motion, producing accurate poses when visible but incomplete temporal coverage.Data association assigns detections using the distance between estimated exocentric CPF locations and the tracked camera trajectories.
  • Conditional diffusion fusion: A unified conditional diffusion model integrates variable numbers of sparse or intermittent exocentric observations with the egocentric trajectory.The model estimates pθ(Si|Ci, ˆSi1, . . . , ˆSiN), while attentional pooling handles changing observation counts and zero active observers.
  • Conditional diffusion fusion: The DiT denoiser encodes noisy body trajectories and observer sequences, fuses pooled observations across layers, and integrates ego motion through cross-attention.Training drops observations so the model learns to operate with sparse inputs; alternative conditioning methods include imputation and diffusion posterior sampling.

4. Experiments

Experiments evaluate fusion on EgoHumans and Harmony4D using absolute, relative, and motion-quality metrics. The conditional fusion model combines complementary egocentric and exocentric signals, improving accuracy across observation conditions and over strong conditioning baselines.

  • Evaluation setup: The evaluation compares Fusion with egocentric motion, CoMotion+SLAM, RePaint, CondMDI, and DPS on EgoHumans and Harmony4D.All methods are assessed in a shared world coordinate frame using absolute positioning, relative body structure, and motion quality.
  • Headline results: 90.7mm MPJPE and 45.9mm PA-MPJPE on EgoHumans show Fusion improves both global accuracy and relative body pose.The egocentric prior reaches 111.9mm MPJPE and 82.6mm PA-MPJPE, while CoMotion+SLAM reaches 130.3mm MPJPE and 50.9mm PA-MPJPE.
  • Headline results: 73.4mm MPJPE and 40.9mm PA-MPJPE on Harmony4D follow the same improvement pattern across both datasets.The pooled visibility analysis reports consistent fusion improvements on the second dataset.
  • Observation conditions: Fusion performs at least as well as the egocentric prior at every observation fraction and improves further with additional observations without low-coverage degradation.The learned conditional model achieves the best performance across observation fractions, while the egocentric estimate remains roughly flat.
  • Observation conditions: At 100% coverage, Fusion reduces exocentric-only error from 107.2mm MPJPE to 88.2mm MPJPE, a 19mm improvement, while PA-MPJPE worsens by 1.5mm.This gain occurs despite complete observation because exocentric pose estimates remain noisy.
  • Ablations and baselines: Fusion improves over RePaint, CondMDI, and DPS on MPJPE, PA-MPJPE, and MPJAE on both datasets, with temporal smoothness comparable to DPS.Attentional pooling over all observations slightly outperforms using only the highest-confidence observation on EgoHumans: 90.7 / 45.9 versus 91.8 / 46.3.

5. Discussion and Conclusion

The approach combines visual tracking with generative motion modeling for multi-person egocentric pose estimation, while supporting modular deployment and highlighting privacy and generalization boundaries.

  • Discussion and Conclusion: The framework integrates head-motion priors, visually grounded pose estimates, and SLAM alignment to improve global positioning.Its diffusion model conditions on SMPL body parameters, allowing tracker or motion-prior components to be replaced as better models become available.
  • Discussion and Conclusion: Per-camera detection can run on-device while participants exchange only low-bandwidth pose summaries instead of raw video.This modular design contrasts with end-to-end approaches requiring access to all raw video.
  • Limitations and Future Work: Training and evaluation remain constrained by limited, imperfect real-world data and distribution differences from motion-capture data.Generalization to other activities, participant counts, and device configurations remains to be established.
  • Broader Impacts: Responsible deployment requires explicit agreement from everyone being tracked because egocentric cameras capture people and environments beyond the wearer.

A.1. Dataset pre-processing

The preprocessing calibrates differences between a mesh-derived virtual head reference used in training and Aria’s sensor-tracked head trajectory at inference.

  • Dataset pre-processing: Training computes the CPF from the ground-truth body mesh, whereas inference tracks the head trajectory with Aria’s visual-inertial sensors.The two references differ because of head shape and device-placement variation.
  • Dataset pre-processing: A per-person rigid-transform prior estimated from calibration data captures the mismatch between training and inference references.The prior assumes left-right symmetry.

A.2. Training and inference details

The motion prior and conditional models use distinct optimization schedules, with large-batch pretraining and staged fine-tuning for the conditional model.

  • Training and inference details: The motion prior trains with global batch size 2048 using Muon for two-dimensional weight matrices and AdamW for remaining parameters.Muon uses learning rate 0.02; AdamW uses 3×10^-4, with weight decay 10^-4 for both.
  • Training and inference details: Fine-tuning uses AdamW at 3×10^-4 for the conditional model and 10^-4 for the CondMDI baseline.The conditional model uses separate warmup schedules for frozen-backbone and unfrozen phases, followed by cosine decay after unfreezing.

A.3. Observation noise characteristics

Observation errors are substantial, systematic, and temporally correlated, with the spine2 joint contributing disproportionately to downstream positional error.

  • Observation noise characteristics: 17.1° per-joint mean geodesic error is observed in CoMotion body-rotation predictions across 180,931 frames.Approximately 49% of the error is systematic bias.
  • Observation noise characteristics: 0.826 lag-1 autocorrelation indicates that observation noise persists across frames.
  • Observation noise characteristics: 24.6° spine2 rotation error propagates to 43.8mm joint-position error through forward kinematics.The spine2 joint dominates both rotation error and its downstream positional effect.
  • Observation noise characteristics: Targeted improvement of trunk estimation in monocular trackers could disproportionately improve downstream fusion.

A.4. Justification of variables for conditioning

Visual tracker confidence is informative about pose accuracy, supporting its use as a conditioning variable for fusion.

  • Detection confidence correlates with frame-level prediction accuracy, with higher confidence associated with lower per-frame MPJPE.The reported Spearman correlation is ρ = −0.434.
  • The relationship between detection confidence and per-frame MPJPE provides evidence that tracker reliability varies across frames.
  • Frame-level confidence can distinguish more and less accurate visual pose observations for conditioning.

A.5. Motion prior backbone vs EgoAllo

The motion-prior backbone is compared with EgoAllo on the AMASS test split, where the proposed model reports lower pose errors.

  • 24mm MPJPE and 18mm PA-MPJPE improvements are reported over EgoAllo on the AMASS test split.The comparison uses the released pretrained EgoAllo model.
  • The reported comparison evaluates absolute and aligned pose error using MPJPE and PA-MPJPE.
  • Our motion-prior backbone is compared against EgoAllo on the AMASS test split.Table 3 reports all values in millimeters.
Loading 2608.29927v1…