PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching

Minghao Liu, Tutian Tang

arXiv:2610.19857v1cs.CV

TL;DR

Sparse, noisy, and unstable observations make 3D motion recovery in the wild difficult, while existing approaches are often configuration-specific and computationally expensive. OmniPoser unifies heterogeneous inputs with masked conditioning and Conditional Flow Matching. Across three settings and seven benchmarks, it performs on par with state-of-the-art methods at over 700 FPS.

  • Problem

    Real-world motion recovery must infer 3D motion from noisy, fragmented, and unstable signals, while existing methods are often tied to fixed sensor configurations and too expensive for real-time use.

    Source: §1 · pp. 1–2

  • Method

    OmniPoser combines Conditional Flow Matching with cross-modal masking and a Masked Conditioning Encoder that represents each missing joint with a learnable embedding.

    Source: §1 · pp. 2–3

  • Results

    Across three settings and seven benchmarks, OmniPoser performs on par with state-of-the-art methods while running at over 700 FPS.

    Source: §5 · p. 15

  • Takeaways & Limitations

    One framework covers diverse sensor setups while enabling real-time, single-step ODE inference.

    Source: §5 · p. 15

    Nymeria contains 260M frames with dense body MVNX annotations, but resource constraints led the authors to leave its preprocessing and large-scale training for future work.

    Source: Setting 3 (Mocap data from 6 IMUs with… · p. 10

Abstract

from arXiv · show

Recovering 3D human motion from sparse wearable sensors is challenging due to the inherent noise, occlusion, and instability of real-world tracking signals. Existing approaches are restricted to fixed sensor configurations and, when built on diffusion models, are computationally prohibitive for real-time deployment. We propose OmniPoser, a universal framework that recovers full-body SMPL motion from a flexible combination of heterogeneous inputs, such as 3D keypoints, sparse IMU rotations, head-mounted 6-DoF poses, or any subset thereof, within a single model. OmniPoser introduces a cross-modal masking mechanism paired with a Masked Conditioning Encoder, enabling one architecture to incorporate diverse sensor configurations through a masked design. A decoupled dual-stream generative backbone generates complete motion via Conditional Flow Matching, requiring only a single ODE integration step at inference. Across three settings and seven benchmarks, including in-the-wild Nymeria and Ego-Exo4D datasets, OmniPoser performs on par with state-of-the-art methods while running at over 700 FPS. The project page is available at https://jamesbridgewater51.github.io/omniposer-eccv/.

1 Introduction

Real-world motion recovery must work from noisy, fragmented sensor signals, but existing methods are configuration-specific and often too slow for real time. OmniPoser combines heterogeneous inputs in one framework and reports competitive performance across diverse settings at over 700 FPS.

1 Introduction

  • Real-world keypoints and sparse sensor readings are noisy, fragmented, and unstable, with occlusion, lighting, drift, and disconnections affecting inputs.These signals include 3D keypoints, wearable-sensor rotations, and head-mounted or handheld 6-DoF poses.Source: §1 · pp. 1–2
  • At over 700 FPS, OmniPoser performs on par with state-of-the-art methods across three sensor settings.Fig. 1 also reports matching or surpassing baselines with lower MPJPE.Source: Figure 1 · p. 2, §1 · p. 3
  • Prior methods typically require fixed sensor configurations and retraining for new inputs, while diffusion-based approaches are often too computationally expensive for real-time use.The paper frames these constraints against the need to recover motion from limited, imperfect signals.Source: §1 · pp. 1–2
  • OmniPoser uses Conditional Flow Matching and cross-modal masking to recover full-body motion from heterogeneous sparse observations in one framework.Supported inputs include 3D keypoint positions, joint rotations, and combinations of these signals.Source: §1 · pp. 2–3
  • The Masked Conditioning Encoder replaces each unobserved joint with a learnable per-joint embedding so the network can distinguish which joints are missing.At inference, a single Euler integration step maps Gaussian noise to motion conditioned on masked observations.Source: §1 · p. 3

2 Related Work

Prior work spans sparse-sensor capture, monocular motion recovery, and masked or decomposed motion modeling, but many systems remain tied to particular sensor configurations. OmniPoser adapts masking to unify heterogeneous sparse observations within one recovery framework.

Human Motion Capture with Sparse Sensors

  • Sparse-sensor capture includes IMU systems, head-mounted cameras, and VR devices, with many methods tailored to fixed placements or a single sensor configuration.Examples include sparse IMU setups, egocentric systems, and HMD-and-controller tracking.Source: §2 · pp. 3–4

Human Motion Recovery

  • Monocular recovery commonly extracts keypoints first and then models their sequences; existing frameworks remain limited by keypoint inputs and lack of topological flexibility.RoHM is identified as a primary baseline for different keypoint patterns.Source: §2 · p. 4

Masked Motion Modeling and Decomposition

  • Related work uses masked modeling and local-global decomposition for motion or shape recovery, including explicitly masked variable conditioning signals.OmniPoser adapts masking to indicate whether a joint is observed through position, rotation, or a 6-DoF signal.Source: §2 · p. 4
  • OmniPoser processes heterogeneous 3D keypoints and rotations with cross-modal masking and an encoder, then generates SMPL parameters through a decoupled dual-stream flow-matching model.The streams model global trajectory and local body pose, with inference in one ODE step.Source: Figure 2 · p. 5

3 Method

OmniPoser treats heterogeneous sparse observations as partial views of a shared body representation and recovers complete SMPL motion. Its masked conditioning and decoupled flow-matching streams handle varied inputs and generate motion in one inference step.

3 Method

  • OmniPoser frames motion recovery as completing partial views of a shared canonical body representation, with heterogeneous observations conditioning a unified generative model.Source: §3 · p. 5

3.1 Motion Representation

  • Each motion frame concatenates root trajectory, joint positions and rotations, shape, and foot contacts; the model outputs all feature groups and joints.Source: §3.1 · pp. 5–6

3.2 Cross-Modal Masking and Masked Conditioning Encoder

  • A configurable feature-level mask represents visible joints and global features, including intermittent visibility and sensor layouts from full optical capture to head-only input.Source: §3.2 · p. 6

Masked Conditioning Encoder

  • The Masked Conditioning Encoder replaces missing joints with learnable joint embeddings, then uses temporal convolutions to smooth conditioning and interpolate short occlusion gaps.Source: Masked Conditioning Encoder · p. 7

Conditional Flow Matching.

  • Conditional Flow Matching interpolates Gaussian noise and clean motion along an affine path, training a denoiser to predict clean motion and using one Euler step at inference.Source: Conditional Flow Matching · p. 7

Decoupled Architecture

  • Independent streams share the same conditioning interface: a convolutional UNet generates global trajectory, while a Transformer-based stream generates local pose, shape, and contacts.Source: Decoupled Architecture · pp. 7–8
  • At inference, predicted global trajectory conditions local pose generation; during training, the local stream instead receives ground-truth trajectory.Source: Decoupled Architecture · p. 8

Training Objectives

  • Local-stream losses combine flow matching with representation, kinematic, and physics-grounded supervision, including foot-contact labels and penalties for sliding during predicted contact.Source: Training Objectives · p. 8

4 Experiments

The experiments assess performance across heterogeneous sparse-observation settings and test generation diversity and masking-based generalization. They use one-step inference and compare against specialist methods across sparse-joint and head-pose benchmarks.

4 Experiments

  • The experiments assess state-of-the-art comparisons across heterogeneous sparse-observation settings, generation diversity, and whether masking and the Masked Conditioning Encoder support generalization.Source: §4 · p. 8

4.1 Implementation Details

  • At inference, one Euler ODE step yields over 16× speedup compared with diffusion baselines using 100 or more steps.Source: §4.1 · p. 9

Benchmarks and Baseline Methods

  • Benchmarks cover sparse upper-body or randomly masked joints and head-only HMD signals, using AMASS, Nymeria, and RICH with RoHM, HuMoR, EgoEgo, and EgoAllo comparisons.Source: Benchmarks and Baseline Methods · p. 9

Setting 2 (Egocentric Head 6D Pose):

The evaluation spans six-IMU and keypoint settings, plus qualitative real-world examples generated with a cross-setting checkpoint. The Nymeria evaluation omits some metrics because of resource constraints.

Setting 2 (Egocentric Head 6D Pose):

Setting 3 (Mocap data from 6 IMUs with keypoint positions):

Metrics

OmniPoser is evaluated across sparse observations, egocentric head-pose input, and six-IMU-plus-keypoint input using multiple motion-quality metrics. The reported results include comparisons with specialist methods across AMASS, Nymeria, AIST++, TotalCapture, and RICH.

Metrics

  • The section reports MPJPE, PA-MPJPE, acceleration error, jerk, per-vertex error, contact accuracy, foot skating, ground penetration, translation error, and hip/shoulder rotation error.Lower values are better except for contact accuracy, for which higher values are better.Source: Metrics · p. 10

Setting 1: Sparse Observation

  • 32.1 mm MPJPE on AMASS and 21.1 vs. 59.3 mm MPJPE on Nymeria are reported for upper-body observations, with the latter comparison against RoHM.The random-joint-masking evaluation reports 35.7 mm MPJPE for Ours-S, with up to 15 of 22 joints masked.Source: Setting 1: Sparse Observation · p. 10, Table 1 · p. 11, Table 2 · p. 11
  • 83.2 mm MPJPE on AMASS for Ours-S is compared with 119.7 mm for EgoAllo and 167.4 mm for EgoEgo in the head-6D-pose setting.On Nymeria, OmniPoser reports 40.7 vs. 103.0 mm MPJPE against EgoAllo; on RICH, Ours-S reports 131.6 mm MPJPE, a stated 25% improvement over specialist methods.Source: Setting 1: Sparse Observation · p. 11, Table 3 · p. 12
  • 26.0/29.3 mm MPJPE and 19.4/21.4 mm PA-MPJPE are reported for Ours-S/Ours-U on AIST++, while Ours-S jerk is 0.29 km/s3.The reported jerk is 5× lower than SIP and 8× lower than RobustCap.Source: Setting 1: Sparse Observation · p. 12, Table 4 · p. 13
  • 36.2 vs. 42.3 mm MPJPE is reported for Ours-S and RobustCap3D on TotalCapture; SIP has lower translation error (5.42 vs. 15.9 mm) and hip/shoulder rotation error (10.6° vs. 13.3°).On Nymeria, the reported OmniPoser values are 16.6 mm MPJPE and 0.048 m/s3 jerk.Source: Setting 1: Sparse Observation · p. 12, Table 4 · p. 13

4.4 Qualitative Results

Qualitative results show one Ours-U model producing reconstructions under five masking configurations and recovering motion in outdoor and ego-centric scenes. The section also identifies an AMASS upper-body ablation table.

4.4 Qualitative Results

  • One Ours-U model produces plausible speed-vault reconstructions under five masking configurations; spatially distributed inputs track ground truth most faithfully, while upper-body-only input yields a less dynamic but coherent trajectory.The same figure also shows AIST++ dance and TotalCapture examples.Source: §4.4 · p. 13
  • Table 5 is scoped to ablation studies on AMASS with upper-body observations.The supplied passage identifies the table's scope but gives no ablation outcomes.Source: Table 5 · p. 14
  • OmniPoser recovers whole-body motion in Ego-Exo4D basketball and bicycle-repair activities involving body-object interactions.On 3DPW, it produces coherent walking gaits and seated interactions in outdoor scenes.Source: §4.4 · pp. 13–14

4.5 Ablation Studies

The ablations examine the Masked Conditioning Encoder and sampling diversity in the conditional generative network. Sampling diversity is described in relation to input constraints and ODE steps.

4.5 Ablation Studies

  • The ablation study examines the effectiveness of the MCE module and sampling diversity from the generative conditional network.Source: §4.5 · p. 14
  • Figure 4 concerns diversity under input constraints and ODE steps.Source: Figure 4 · p. 14
  • Table 6 measures sampling diversity as joint-position standard deviation in millimeters over 100 samples.Source: Table 6 · p. 14

MCE learnable embeddings.

Replacing per-joint learnable embeddings with zero-padding worsens MPJPE, indicating that the embeddings contribute to recovery performance.

MCE learnable embeddings

  • Replacing per-joint learnable embeddings with zero-padding worsens MPJPE by +56.0 mm.The passage attributes the difference to weaker ability to distinguish which joints are missing.Source: MCE learnable embeddings · p. 14

Diversity, input constraints, and inference steps.

OmniPoser’s output diversity varies with the amount of input and the number of flow-matching steps, while additional steps offer little performance gain but trade speed for diversity.

Diversity, input constraints, and inference steps.

5 Conclusion

OmniPoser treats different sensor inputs as partial views of one shared body model and covers diverse sensor setups with a single-step, real-time pipeline. It performs on par with state-of-the-art methods, while remaining limited to single-person SMPL 22-joint motion without hand or face articulation.

5 Conclusion

  • OmniPoser treats flexible inputs, including partial keypoints, head-mounted 6-DoF poses, and sparse IMUs, as partial views of one shared body model.Source: §5 · p. 15
  • Across three settings and seven benchmarks, OmniPoser performs on par with state-of-the-art methods while running at over 700 FPS.Source: §5 · p. 15
  • Modular masking with a Masked Conditioning Encoder, a decoupled dual-stream architecture, and Conditional Flow Matching support diverse setups with real-time, single-step ODE inference.Source: §5 · p. 15
  • The method models one person and lacks body-object or body-body contact forces, while operating on SMPL’s 22-joint skeleton without hand or face articulation.The cascaded pipeline propagates trajectory errors into pose recovery without built-in correction, although evaluated trajectories are reported as smooth and drift-free.Source: §5 · p. 15
Loading 2610.19857v1…