PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionBERT: A Unified Perspective on Learning Human Motion Representations

Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, Yizhou Wang

arXiv:2210.06551v5cs.CV

TL;DR

Human-centric video tasks are often treated separately despite shared motion patterns and heterogeneous data sources. The paper pretrains a motion encoder to recover 3D motion from corrupted partial 2D skeletons, then adapts it to downstream tasks with simple heads. The resulting representations transfer across tasks, while the approach remains limited by its focus on single-person skeleton sequences.

  • Problem

    Existing human-centric video tasks are modeled in isolation, while available motion data differ in fidelity, diversity, labels, and annotation availability.

  • Method

    The framework pretrains DSTformer to recover 3D motion from randomly masked and noisy 2D skeleton sequences, then finetunes the learned representation for each downstream task.

  • Results

    The learned motion representations transfer across multiple downstream tasks and outperform task-specific state-of-the-art methods.

  • Takeaways & Limitations

    A shared motion encoder can support diverse human-centric video tasks through simple task-specific regression heads.

  • Takeaways & Limitations

    The representation focuses on single-person skeleton sequences and is therefore insensitive to appearance, surroundings, and interactions.

Abstract

from arXiv · show

We present a unified perspective on tackling various human-centric video tasks by learning human motion representations from large-scale and heterogeneous data resources. Specifically, we propose a pretraining stage in which a motion encoder is trained to recover the underlying 3D motion from noisy partial 2D observations. The motion representations acquired in this way incorporate geometric, kinematic, and physical knowledge about human motion, which can be easily transferred to multiple downstream tasks. We implement the motion encoder with a Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. Furthermore, our proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with a simple regression head (1-2 layers), which demonstrates the versatility of the learned motion representations. Code and models are available at https://motionbert.github.io/

1 Peking University 2 Shanghai AI Laboratory

The section provides contact email addresses for contributors associated with Peking University and Shanghai AI Laboratory.

  • The section lists the email address wtzhu, maxiaoxuan, libin.liu, and yizhou.wang at pku.edu.cn.
  • The listed contacts use institutional and personal email domains.
  • The section lists the email addresses zyliumy and wuwenyan0503 at gmail.com.

1. Introduction

The paper targets the fragmentation of human-centric video tasks and heterogeneous motion data by learning a shared motion representation through corrupted-2D-to-3D pretraining and task-specific adaptation.

  • Motivation: Existing human-centric video tasks are commonly modeled in isolation despite sharing typical human motion patterns.
  • Motivation: Heterogeneous data trade off fidelity, semantic labels, appearance diversity, and annotation availability across motion-capture, action, and in-the-wild sources.
  • Approach: The pretraining task corrupts 2D skeleton sequences and trains a motion encoder to recover the underlying 3D motion.
  • Approach: The pretext task requires inferring 3D structure and recovering erroneous or missing observations, encouraging representations of joint linkages, anatomical constraints, and temporal dynamics.
  • Approach: The pretrained representations are adapted to downstream tasks with task-specific supervision and a simple regression head.
  • Contribution: The experiments report a transferable motion representation that can be used across multiple human-centric video tasks and outperforms task-specific state-of-the-art methods.

2. Related Work

Related work spans motion representation learning, 3D pose estimation, skeleton-based action recognition, and human mesh recovery, while this paper uses DSTformer within a unified pretrain-finetune framework.

  • Learning Human Motion Representations: Earlier motion representation methods use probabilistic models, temporal encoders, future prediction, action memory, and varied self-supervised pretext tasks.
  • Learning Human Motion Representations: Prior representation-learning methods generally use homogeneous motion data and target a specific downstream task.
  • 3D Human Pose Estimation: 3D pose estimation methods either regress poses directly from images or lift extracted 2D poses using fully connected, convolutional, graph, or Transformer networks.
  • Skeleton-based Action Recognition: Skeleton-based action recognition models human-joint relationships with LSTMs, GCNs, 3D-CNNs, and metric-learning approaches for one-shot settings.
  • Human Mesh Recovery: Human mesh recovery methods regress parametric meshes from images, while video methods add denoising, smoothing, or temporal cues to improve temporal coherence.
  • Model Architecture: DSTformer is presented as a general human-motion backbone using spatial and temporal attention streams with adaptive fusion.

3. Method

The method unifies pretraining and task-specific finetuning around 2D skeleton sequences, using DSTformer to lift corrupted 2D observations into 3D motion. The learned motion embeddings are transferred to pose estimation, action recognition, and mesh recovery with shallow task heads.

  • Unified framework: The framework trains a motion encoder for 2D-to-3D lifting, then finetunes it with a few new layers for downstream tasks.2D skeleton sequences are used as input in both stages because they can be extracted from diverse motion sources.
  • Network architecture: DSTformer projects 2D skeleton sequences into features, adds spatial positional encoding, produces motion embeddings, and estimates 3D motion.The encoder outputs a representation E before a final linear transformation predicts 3D motion.
  • Network architecture: Spatial MHSA models relationships among joints within each time step, while temporal MHSA models relationships across time for each joint.The spatial block includes MHSA, LayerNorm, an MLP, and residual connections; the temporal block is parallelized over the spatial dimension.
  • Network architecture: DSTformer uses parallel spatial-temporal branches whose outputs are fused with input-dependent adaptive weights and repeated across network depth.The branches use different block orders, and an attention regressor predicts the fusion weights.
  • Unified pretraining: Unified pretraining corrupts projected or detected 2D skeletons with masks and noise, then optimizes 3D reconstruction and velocity losses where 3D ground truth is available.For in-the-wild RGB videos without 3D ground truth, the framework uses 2D reprojection supervision and a combined pretraining loss.
  • Downstream tasks: The learned embedding is finetuned with a linear layer or one-hidden-layer MLP for 3D pose estimation, action recognition, or human mesh recovery.Mesh recovery regresses SMPL pose and shape parameters, while action recognition applies pooling followed by classification or clip-level representation learning.

4. Experiments

Experiments evaluate the framework on 3D pose estimation, action recognition, and human mesh recovery, including limited-data settings and ablations of pretraining and architecture. Across these evaluations, pretrained representations improve performance, convergence, transfer, and computational sharing.

  • Experimental setup: Human3.6M and AMASS provide diverse 3D human motion data for pretraining, while Human3.6M is used to evaluate 3D pose estimation.Human3.6M contains 3.6 million frames of professional actors performing daily actions; subjects 1, 5, 6, 7, 8 are used for training and 9, 11 for testing.
  • 3D Pose Estimation: 3D pose estimation is evaluated with MPJPE for root-aligned joint accuracy and MPJVE for temporal smoothness using detected 2D skeletons.The experiments finetune the entire network on Human3.6M and also compare against a model trained from scratch.
  • 3D Pose Estimation: The proposed models outperform previous approaches on both MPJPE and MPJVE, while pretraining further lowers errors.The comparison includes detected and ground-truth 2D pose inputs; the authors attribute the scratch-model advantage to DSTformer and the additional improvement to pretraining.
  • Skeleton-based Action Recognition: On NTU-RGB+D and NTU-RGB+D-120, the methods are comparable or superior to state of the art, with pretraining producing a large performance gain.The evaluation covers cross-subject and cross-view recognition and one-shot recognition; NTU-RGB+D contains 57K videos across 60 classes, while NTU-RGB+D-120 contains 114K videos across 120 classes.
  • Skeleton-based Action Recognition: In one-shot recognition, the model uses one labeled video per class and supervised contrastive learning on an auxiliary set of 100 classes.Class exemplars are compared with test examples using cosine distance and 1-nearest neighbor classification.
  • Human Mesh Recovery: For mesh recovery, the finetuned model surpasses video-based state-of-the-art methods on Human3.6M and 3DPW, especially on MPVE, while sparse 2D keypoints leave shape estimation ill-posed.The authors therefore introduce a hybrid refiner combining the framework’s coherent motion with RGB-based shape estimation.
  • Finetuning: Pretrained models show superior performance and faster convergence than scratch models on all three tasks, achieving optimal performance after only 1–2 finetuning epochs.These results are interpreted as evidence that pretraining learns transferable motion representations that generalize to novel tasks with limited annotations.
  • Partial Finetuning: With the motion encoder frozen, partial finetuning remains competitive and improves substantially over random initialization across pose, mesh, and action tasks.Sharing one pretrained backbone across tasks significantly reduces computation for applications requiring multi-task inference.

5. Conclusion

The paper presents a unified framework for learning human motion representations from heterogeneous data and transferring them across human-centric video tasks.

  • The framework learns human motion representations through a shared approach across various human-centric video tasks.

A.1. Experimental Details

The experiments use multiple motion datasets and task-specific training procedures, with pretraining combining 3D motion capture and 2D video data.

  • Pretraining: Pretraining uses Human3.6M and AMASS for diverse 3D motion data, alongside PoseTrack and InstaVariety for in-the-wild 2D motion.The pipeline uses projected or extracted skeletons and introduces random masking and noise.
  • 3D Pose Estimation: The 3D pose estimation experiments compare training from scratch with finetuning initialized from pretrained weights.The finetuning setup loads pretrained weights and trains for 30 epochs.
  • Skeleton-based Action Recognition: Skeleton-based action recognition uses HRNet-extracted 2D skeleton sequences and task-specific MLP or contrastive-learning configurations.NTU-RGB+D uses an MLP head, while one-shot recognition uses supervised contrastive loss.
  • Human Mesh Recovery: Human mesh recovery uses skeleton sequences, SMPL ground-truth parameters, and additional COCO data for training.Human3.6M and 3DPW use different sources for SMPL ground-truth parameters.

A.2. Additional Experiments

The additional experiments include architectural analysis, further action-recognition results, a pretraining mask-ratio ablation, and qualitative results.

  • The supplementary experiments cover model architecture, skeleton-based action recognition, mask-ratio ablation, and qualitative evaluation.

A.2.1 Analysis of DSTformer

The analysis probes DSTformer’s adaptive fusion mechanism and examines how its two streams are weighted across actions and body joints.

  • The analysis probes fusion weights αST and αTS to understand how DSTformer balances its dual streams across inputs.
  • Action-dependent fusion: Higher-movement actions receive lower S-T fusion weights, whereas static actions receive higher S-T weights.The reported examples are walking for higher movement and sitting or eating for more static motion.
  • Visualization: Figure 4 summarizes average S-T fusion weights grouped by actions and joints on the Human3.6M test set.The probing uses the third layer of a 3D pose estimation model trained from scratch.
  • Stream specialization: The S-T stream is associated with spatial modeling, while the T-S stream is associated with temporal modeling.
  • Joint-dependent fusion: Fusion weights vary across joints: limb joints favor T-S, while more stable torso joints receive greater S-T importance.
  • Visualization: Figure 5 presents the correlation matrix of joint-wise fusion weights on the Human3.6M test set.The probing again uses the third layer of a model trained from scratch.

A.2.2 Architecture Hyperparameter Analysis

The study varies DSTformer’s depth, attention heads, and channel size for 3D human pose estimation, then transfers the selected architecture across tasks without further adjustment.

  • Architecture choices: DSTformer performance is evaluated while varying stacking-module count, self-attention heads, and embedding channels.The measurements use the 3D human pose estimation task.
  • Architecture choices: The authors select the optimal combination of depth, heads, and channels from the architecture sweep.
  • Cross-task transfer: The selected DSTformer design is applied to all tasks without additional architecture adjustment.

A.2.3 Comparison with Self-supervised Action Recognition Approaches

The framework learns motion representations from 2D skeleton sequences and transfers them to action recognition, including novel-class one-shot recognition. Its analysis also examines representation quality under masking and identifies a single-person scope limitation.

  • Self-supervised action recognition: The method uses 2D-to-3D lifting as a pretext task to learn 3D-aware motion representations from 2D skeleton sequences.
  • Self-supervised action recognition: 2D motion representations are easier to apply to in-the-wild videos than prior representations requiring 3D skeleton sequences.The comparison considers videos from mobile devices or the Internet and reports scratch, finetune, and diff performance.
  • One-shot action recognition: 67.4% top-1 accuracy is achieved for one-shot action recognition using 1-nearest-neighbor exemplars on novel classes.Meaningful clusters emerge although the finetuned model has not seen these actions during training.
  • Mask ratio analysis: Masking ratios below 45% allow effective 2D-to-3D lifting during pretraining and improve finetuning over the scratch baseline.The study measures MPJPE on the Human3.6M test set.
  • Limitations: The representation is limited to single-person skeleton sequences and is therefore insensitive to appearance, surroundings, and interactions.The authors suggest combining it with generic video or scene representations in future work.
Loading 2210.06551v5…