PaperlayerView exact revision on arXiv

Source-linked AI summary

EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild

Manuel Kaufmann, Jie Song, Chen Guo, Kaiyue Shen, Tianjian Jiang, Chengcheng Tang, Juan Zarate, Otmar Hilliges

arXiv:2308.16894v1cs.CV

TL;DR

Accurate in-the-wild 3D pose and shape datasets with global body and camera trajectories are scarce. EMDB combines body-worn EM sensors, a hand-held iPhone, and staged pose refinement to construct such a dataset. Volumetric-capture evaluation reports higher accuracy than the cited 3DPW results, and the paper evaluates RGB pose estimators on EMDB.

  • Problem

    Accurate 3D pose-and-shape datasets with global trajectories in fully in-the-wild settings are lacking, despite the need to estimate human and camera locations for moving-camera applications.

    Source: §1 · p. 1

  • Method

    EMP constructs EMDB by fitting SMPL to body-worn EM measurements and refining global poses with image observations and camera poses.

    Source: §1 · p. 2

  • Results

    2.3 cm positional and 10.6° angular error: EMP is more accurate than the reported 3DPW results of 2.6 cm and 12.1°.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EMDB provides a dataset for evaluating camera-relative and global RGB pose estimators, while the paper concludes that sensor-based performance capture is needed to procure high-quality 3D human motion.

    Source: §1 · p. 2, §8 · p. 9

    EMDB has no multi-person sequences because multiple EM systems require nontrivial changes to avoid sensor cross-talk and interference.

    Source: §8 · p. 9

Abstract

from arXiv · show

We present EMDB, the Electromagnetic Database of Global 3D Human Pose and Shape in the Wild. EMDB is a novel dataset that contains high-quality 3D SMPL pose and shape parameters with global body and camera trajectories for in-the-wild videos. We use body-worn, wireless electromagnetic (EM) sensors and a hand-held iPhone to record a total of 58 minutes of motion data, distributed over 81 indoor and outdoor sequences and 10 participants. Together with accurate body poses and shapes, we also provide global camera poses and body root trajectories. To construct EMDB, we propose a multi-stage optimization procedure, which first fits SMPL to the 6-DoF EM measurements and then refines the poses via image observations. To achieve high-quality results, we leverage a neural implicit avatar model to reconstruct detailed human surface geometry and appearance, which allows for improved alignment and smoothness via a dense pixel-level objective. Our evaluations, conducted with a multi-view volumetric capture system, indicate that EMDB has an expected accuracy of 2.3 cm positional and 10.6 degrees angular error, surpassing the accuracy of previous in-the-wild datasets. We evaluate existing state-of-the-art monocular RGB methods for camera-relative and global pose estimation on EMDB. EMDB is publicly available under https://ait.ethz.ch/emdb

1. Introduction

The paper addresses the shortage of accurate in-the-wild datasets with global trajectories by introducing EMDB and EMP, a sensor-and-image capture pipeline. EMP is evaluated against volumetric capture, and existing RGB pose estimators are evaluated on EMDB.

1. Introduction

  • EMDB provides 58 minutes of challenging motion across 81 sequences and 10 participants, with pose, shape, body-root, and camera-trajectory annotations.The sequences were recorded with a hand-held mobile phone.Source: §1 · p. 1
  • EMDB addresses the lack of comprehensive in-the-wild datasets with accurate 3D poses, shapes, and global trajectories needed for global pose estimation.The paper identifies mobile capture as difficult and contrasts EM sensors’ direct positional measurements with IMU drift and limited positional information.Source: §1 · p. 2
  • EMP combines EM measurements, monocular RGB-D images, and camera poses in three stages to estimate SMPL parameters and global body and camera trajectories.Its final stage fits a neural body model to RGB images using pixel-level supervision for smoothness and pose-to-image alignment.Source: §1 · p. 2
  • 2.3 cm positional and 10.6° angular error: EMP is more accurate than the reported 3DPW results of 2.6 cm and 12.1°.Global SMPL root trajectories have an estimated 5.1 cm error against the indoor MVS.Source: §1 · p. 2
  • The paper evaluates camera-relative and global RGB-based pose estimators on EMDB and reports baseline results alongside EMP’s accuracy evaluation.The dataset is presented as a resource for future local and global pose-estimation research.Source: §1 · p. 2

2. Related Work

Prior datasets and sensor-based systems leave limitations in mobile capture, global trajectories, or accuracy evaluation. EMDB provides an in-the-wild dataset with global camera and body-root trajectories and evaluates its annotations against volumetric capture.

2. Related Work

  • Earlier IMU-based methods can yield accurate local poses, but IMUs lack direct positional measurements and their position estimates drift over time.Some approaches mitigate missing positional information through more sensors or fusion with other modalities.Source: §2 · p. 2
  • EM-POSE uses EM sensing for body-worn pose estimation, but is limited to a small indoor space, requires external root tracking, and is not aligned with image observations.The paper presents EMDB’s mobile, in-the-wild capture and pose-to-image alignment as moving beyond these limits.Source: §2 · pp. 2–3
  • Most RGB-based pose research estimates camera-local poses, while only a few approaches estimate realistic global human and camera trajectories.The paper evaluates methods on both camera-relative and global pose-estimation tasks.Source: §2 · p. 3
  • Common pose datasets are often indoor with static backgrounds, while prior mobile or outdoor capture approaches have constraints such as fixed spaces, line-of-sight limits, or missing accuracy evaluation.The cited examples include H3.6M, MPI-INF-3DHP, HumanEva, TotalCapture, 3DPW, HPS, HSC4D, RGB-D capture, and EgoBody.Source: §2 · p. 3
  • EMDB is described as the first unrestricted, mobile, in-the-wild 3D pose-and-shape dataset with global camera and SMPL root trajectories; its annotations are reported as more accurate than 3DPW’s.The paper also reports EMDB as twice as large as 3DPW.Source: §2 · p. 3

3. Overview

The paper’s workflow moves from recording and estimation to accuracy evaluation, then uses the resulting method to build EMDB and assess existing estimators.

3. Overview

  • The dataset targets accurate 3D body poses and shapes plus global trajectories for both the body root and moving camera, using EM measurements and RGB-D from one hand-held iPhone.Source: §3 · p. 3
  • EMP estimates global SMPL parameters, and its accuracy is evaluated against ground truth recorded with a multi-view volumetric system.Source: §3 · p. 3
  • The authors apply EMP to newly captured in-the-wild sequences to introduce EMDB and evaluate existing state-of-the-art methods on it.Source: §3 · p. 3

4. Capture Setup

The capture setup combines body-worn electromagnetic sensing with a hand-held iPhone, then calibrates participant shape and sensor-to-body offsets before in-the-wild recording.

4.1. Sensing Hardware

  • Body-worn EM sensors measure position and orientation relative to a field source mounted on the participant’s lower back.Sensors are placed on the extremities, head, and torso, and stream wirelessly to a laptop.Source: §4.1 · p. 3
  • A hand-held iPhone 13 Pro Max supplies depth and ARKit-estimated 6D camera pose; a hand clap synchronizes phone and EM streams.Source: §4.1 · p. 3

4.2. Body Calibration

  • Participants are scanned in minimal clothing with an MVS to register SMPL shape parameters β from surface scans and 53 RGB views.Source: §4.2 · pp. 3–4
  • A 3-second calibration sequence determines subject-specific skin-to-sensor offsets, with sensor measurements aligned spatially and temporally to the MVS.The offsets are computed per sensor from selected SMPL mesh anchor points; calibration is repeated only when sensor placement changes.Source: §4.2 · p. 4

5. Method (EMP)

EMP estimates SMPL pose and shape in the iPhone’s world coordinate frame by combining EM sensor measurements with RGB-D observations and camera poses. Its three optimization stages progress from local sensor fitting to world alignment and pixel-level refinement.

5.1. Notations and Preliminaries

  • The method takes EM sensor measurements, skin-to-sensor offsets, shape parameters, RGB images, depth point clouds, and camera parameters to estimate SMPL pose and root translation in the iPhone’s coordinate frame.Shape parameters are obtained a priori rather than optimized.Source: §5.1 · p. 4

5.2. Multi-stage Optimization

  • Stage 1 fits SMPL parameters to EM sensor positions and orientations in EM-local space, with a regularizer for impossible joint angles.Virtual sensor measurements are computed from the current SMPL mesh and skin-to-sensor offsets; the optimization is batched over all frames.Source: §5.2 · pp. 4–5
  • Stage 2 aligns the initial poses with RGB-D observations and camera poses, balancing 2D keypoint reprojection against EM-based pose constraints.It adds a prior to the first-stage pose and uses noisy point clouds mainly to regularize translation.Source: §5.2 · p. 5
  • Stage 3 fits an implicit model of clothed human shape and appearance to RGB images, jointly refining SMPL poses through a dense pixel-level rendering objective.The objective also includes Eikonal, scene-decomposition, and pose-regularization terms; the human and background fields are optimized over the sequence.Source: §5.2 · pp. 5–6

6. Evaluation

The evaluation compares EMP’s pose accuracy with monocular methods and tests its staged refinement. It also measures iPhone pose accuracy and global trajectory error, including drift on in-the-wild sequences.

6.1. Pose Accuracy

  • EMP outperforms recent monocular methods in pose accuracy comparisons, including HybrIK, ROMP, V2A, and LGD.The comparison includes methods using ground-truth bounding boxes and methods that do not.Source: §6.1 · p. 6
  • Stage 3 marginally improves pose while improving smoothness and image alignment; naïvely smoothing stage-2 outputs harms alignment.The evaluation uses 21 MVS-recorded sequences, approximately 13,000 frames, and 53 high-resolution RGB views to obtain ground-truth registrations.Source: §6.1 · pp. 6–7, Figure 4 · p. 6

6.2. Global Trajectories

  • 1.8 ± 0.9 cm and 0.4 ± 0.2 deg are the position and orientation differences between iPhone and Apriltag trajectories over 15 seconds.Source: §6.2 · p. 7
  • 5.1 ± 3.2 cm mean trajectory error was measured over five sequences; endpoint drift was 23.4 cm (0.3%) indoors and 73.0 cm (0.7%) outdoors.Source: §6.2 · p. 7

7. EMDB

EMDB combines diverse in-the-wild motion data with evaluations of camera-relative pose estimation and global trajectory prediction. The RGB baselines show substantial remaining errors, and GLAMR trajectories drift from the provided tracks.

7.1. Dataset Overview

  • EMDB contains 104,963 frames across 81 sequences and 10 participants; approximately 85% of frames were recorded in-the-wild.The recordings total 58.3 minutes at 30 fps, with indoor and outdoor sequences.Source: §7.1 · p. 7

7.2. Baselines on EMDB

  • >65 mm MPJPE-PA for best-performing HybrIK and >23° average angular error across methods indicate substantial room for improvement.MPJPE-PA is forgiving because Procrustes alignment removes rotation, translation, and scale; high jitter and MPJPE also affect all baselines.Source: Table 3 · p. 8, §7.2 · p. 8

7.1. Dataset Overview

  • In VPoser latent space, EMDB covers more area than 3DPW, with additional coverage comprising complex, diverse poses.The authors also report less diverse poses near 3DPW’s lower boundary and broken-pose outliers in 3DPW for which the closest EMDB pose remains valid.Source: §7.1 · p. 8

7.2. Baselines on EMDB

  • The camera-relative benchmark evaluates eight methods on EMDB 1, while global trajectory prediction uses GLAMR on EMDB 2.EMDB 1 contains 17 challenging sequences; EMDB 2 contains 25 sequences with meaningful global trajectories.Source: §7.2 · p. 8
  • A common baseline failure is missing lower-arm rotations, which MPJPE does not capture and motivates reporting joint angular errors.The paper shows selected baseline outputs in Figure 7 and this failure case in Figure 8.Source: §7.2 · p. 8, Figure 7 · p. 9, Figure 8 · p. 9
  • GLAMR achieves G-MPJPE of 3 193 mm, G-MVE of 3 203 mm, and acceleration of 12.6 mm s−2, while its prediction drifts significantly from the provided trajectories.The beginning of each window is aligned to the ground-truth trajectory.Source: §7.2 · p. 9

8. Conclusion

EMDB provides SMPL poses, shapes, and trajectories in an unrestricted, mobile, in-the-wild setting. Its capture design has specific boundaries, including no multi-person sequences, no foot sensors, and camera trajectories limited by Apple’s AR toolkit.

8. Conclusion

  • EMDB provides accurate SMPL poses, shapes, and trajectories in an unrestricted, mobile, in-the-wild setting; its limitations include no multi-person sequences and camera trajectories bounded by Apple’s AR toolkit.Multiple EM systems would require nontrivial changes to avoid sensor cross-talk and interference; feet lack sensors because indoor floors often contain metal beams that could disturb readings.Source: §8 · p. 9

A. Sensor Placement

  • Sensors worn under regular clothing measure position and orientation relative to a lower-back EM source; the source-mounted Apriltag is required only for body calibration.Wireless sensors communicate with receivers connected to a recording laptop, usually kept within 3–4 meters to minimize packet loss.Source: A. Sensor Placement · p. 14, Figure 10 · p. 14

B. Details on EMDB’s Contents

  • In EMDB, 15.5% of all data was recorded on the multi-view volumetric capture system (MVS,), 12.7% indoors but not on MVS, and the remaining 71.8% outdoors.Source: B. Details on EMDB’s Contents · p. 14

C. SMPL Registration to Multi-View Data

The registration pipeline fits SMPL to multi-view scans and triangulated keypoints, then calibrates EM measurements to the capture system and participant-specific sensor offsets. A final light smoothing pass reduces jitter while retaining pose-to-image alignment.

C. SMPL Registration to Multi-View Data

  • The MVS supplies watertight 40k-vertex scans and RGB images from 53 camera views for fitting gender-specific or neutral SMPL models.The same registration procedure provides ground-truth SMPL shapes and pose registrations for accuracy evaluation.Source: C. SMPL Registration to Multi-View Data · p. 14

C.1. 3D Keypoint Triangulation

  • OpenPose detections on scan renderings are triangulated by ordinary least squares into 25 COCO-format 3D keypoints per time step.Rendering scans against a white background avoids isolating the iPhone holder’s detections when multiple people appear in camera images.Source: §C.1 · p. 14

C.2. SMPL Fitting

  • SMPL fitting combines triangulated-keypoint and scan-surface terms with spine and joint-angle regularizers, then fixes shape from a minimally clothed A-pose scan.Sequences are optimized frame by frame using the previous frame’s output for initialization; the second optimization stage refines pose parameters only.Source: §C.2 · pp. 14, 16

D. Body Calibration Details

D.1. EM and MVS Alignment

  • EM–MVS alignment estimates the rigid offset between the source Apriltag and EM source origin using synchronized EM and MVS measurements.The calibration moves five sensors for 15 seconds; because the Apriltag is rigidly attached to the source, this procedure is done once.Source: §D.1 · pp. 16–17

D.2. Computing Skin-To-Sensor Offsets

  • Skin-to-sensor offsets are optimized by matching virtual sensor poses on registered SMPL meshes to world-space EM measurements.The method uses a short three-second calibration sequence and computes participant- and session-specific offsets.Source: §D.2 · p. 17

E. Stage 3 Smoothing

  • Stage 3 permits light smoothing without breaking pose-to-image alignment, unlike stronger smoothing applied directly to stage 2 outputs.The paper uses a Savitzky-Golay filter with a seven-frame window and second-order polynomial; Figure 14 illustrates the alignment trade-off.Source: E. Stage 3 Smoothing · p. 18, Figure 14 · p. 18

F. Visual Comparison to 3DPW

  • In a visual comparison on a similar walking-and-occlusion sequence, EMDB’s SMPL fits show higher fidelity than 3DPW’s.The comparison includes brief occlusion while the participant walks around poles.Source: F. Visual Comparison to 3DPW · p. 18

G. Fine-tuning with EMDB

The authors fine-tune ROMP with EMDB and evaluate it on 3DPW. Joint errors on 3DPW decrease after fine-tuning.

G. Fine-tuning with EMDB

  • The experiment fine-tunes an existing human pose estimator using EMDB and tests its effect on performance on 3DPW.The authors use ROMP and its publicly available code.Source: G. Fine-tuning with EMDB · p. 18
  • Joint errors on 3DPW decrease after fine-tuning ROMP with EMDB.The comparison uses ROMP’s official pretrained model as the baseline that has never seen 3DPW.Source: Table 6 · p. 18, G. Fine-tuning with EMDB · p. 18
  • The reported improvement is on the 3DPW test set, using ROMP’s official pretrained model as the comparison baseline.Table 6 presents the effect of fine-tuning ROMP on EMDB.Source: Table 6 · p. 18, G. Fine-tuning with EMDB · p. 18

H. Evaluation of Global Trajectories

The evaluation aligns the iPhone’s self-localized camera trajectory with optical tracking, then extends the alignment procedure to compare EMP’s SMPL root trajectory with MVS ground truth.

H.1. Camera Trajectory

  • The iPhone alignment optimizes a world-frame transform and constant rigid offset against optically tracked AprilTag poses; its objective value is reported as alignment error.Figure 16 visualizes the optically tracked and self-localized iPhone trajectories.Source: §H.1 · pp. 18–19, Figure 16 · p. 19

H.2. SMPL Root Trajectory

  • The iPhone-to-world transform cannot be reused for root evaluation because the iPhone coordinate frame changes between recordings.Evaluation takes also contain fewer iPhone movements, leaving Eq. (17) underconstrained and sometimes unable to converge to meaningful solutions.Source: §H.2 · p. 19
  • Adding an EMP-predicted SMPL root position term removes global rigid misalignment; the remaining Euclidean distance is reported as root-trajectory alignment error.The root term is compared with the registered ground-truth SMPL root trajectory.Source: §H.2 · p. 19

I. EMP Implementation Details

EMP’s three stages have distinct optimization procedures and runtimes, while the paper frames EMDB as paired sensor-and-pose data for research use and notes potential societal risks from pose estimation.

I. EMP Implementation Details

  • Stage 1 uses two L-BFGS passes, first aligning SMPL root parameters and then optimizing all SMPL parameters; it typically takes 1–2 minutes.The paper uses PyTorch for computation and fits an entire sequence in one batch on a 24 GB GPU.Source: §I · p. 19
  • Stage 2 optimizes frames sequentially with Adam, taking about 5 seconds per frame and roughly 2 hours for a typical 45-second sequence.Each frame receives 100 optimization iterations.Source: §I · p. 19
  • Stage 3 fits a neural implicit human model for 48 hours on one 24 GB GPU, splitting very long sequences across GPUs to speed convergence.This is the computationally heaviest stage.Source: §I · p. 19

J. Socetial Impact

  • Accurate body-pose extraction is connected to applications including remote telepresence, rehabilitation, and fitness or health coaching.The paper also cites potential benefits such as lower-cost medical treatment and reduced environmental impact.Source: J. Socetial Impact · p. 20
  • The authors describe paired input sensor data and 3D human pose as a resource for training pose estimators or evaluating them on EMDB.They state that this work does not directly improve pose-estimation methods.Source: J. Socetial Impact · p. 20
  • The authors flag possible misuse of image-based pose estimation for surveillance or identifying people through gait or face recognition, while stating that EMDB publishes no identifiable information.They judge the benefits of accurate pose estimators to outweigh these risks.Source: J. Socetial Impact · p. 20

Related papers

Loading 2308.16894v1…