PaperlayerView exact revision on arXiv

Source-linked AI summary

MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation

Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, Luc Van Gool

arXiv:2111.12707v4cs.CVcs.AIcs.LG

TL;DR

Monocular 3D human pose estimation is ill-posed because depth ambiguity and self-occlusion permit multiple feasible poses, while existing methods often estimate only one solution. MHFormer models and communicates multiple pose hypotheses before synthesizing the final pose, achieving state-of-the-art results on Human3.6M and MPI-INF-3DHP, including a 3% improvement over PoseFormer on Human3.6M.

  • Problem

    Monocular 2D-to-3D pose lifting is ill-posed because self-occlusion and missing depth allow multiple feasible pose hypotheses.

  • Method

    MHFormer generates multiple pose hypotheses, refines and communicates their features through self- and cross-hypothesis interactions, then aggregates them into a final 3D pose.

  • Results

    MHFormer achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP, outperforming PoseFormer by 3% with a 1.3 mm error reduction on Human3.6M.

  • Takeaways & Limitations

    Explicitly processing multiple hypotheses enhances the representation and produces a more accurate synthesized 3D pose.

  • Takeaways & Limitations

    Prior multi-hypothesis methods often fail to build relationships among different hypothesis features.

Abstract

from arXiv · show

Estimating 3D human poses from monocular videos is a challenging task due to depth ambiguity and self-occlusion. Most existing works attempt to solve both issues by exploiting spatial and temporal relationships. However, those works ignore the fact that it is an inverse problem where multiple feasible solutions (i.e., hypotheses) exist. To relieve this limitation, we propose a Multi-Hypothesis Transformer (MHFormer) that learns spatio-temporal representations of multiple plausible pose hypotheses. In order to effectively model multi-hypothesis dependencies and build strong relationships across hypothesis features, the task is decomposed into three stages: (i) Generate multiple initial hypothesis representations; (ii) Model self-hypothesis communication, merge multiple hypotheses into a single converged representation and then partition it into several diverged hypotheses; (iii) Learn cross-hypothesis communication and aggregate the multi-hypothesis features to synthesize the final 3D pose. Through the above processes, the final representation is enhanced and the synthesized pose is much more accurate. Extensive experiments show that MHFormer achieves state-of-the-art results on two challenging datasets: Human3.6M and MPI-INF-3DHP. Without bells and whistles, its performance surpasses the previous best result by a large margin of 3% on Human3.6M. Code and models are available at \url{https://github.com/Vegetebird/MHFormer}.

1. Introduction

Monocular 3D human pose lifting is ill-posed because self-occlusion and depth ambiguity permit multiple feasible poses. MHFormer addresses this inverse problem with a three-stage Transformer that generates, refines, communicates across, and synthesizes multiple pose hypotheses.

  • Problem: Monocular 2D-to-3D lifting is inherently ill-posed because self-occlusion and depth ambiguity create multiple feasible solutions.Existing methods typically detect 2D keypoints and lift them into 3D joint locations.
  • Limitation: Prior multi-hypothesis methods use multiple output heads with shared features but fail to model relationships among different hypothesis features.This limits feature expressiveness and model performance.
  • MHFormer: MHFormer performs one-to-many mapping followed by many-to-one synthesis, enriching intermediate hypothesis diversity for a better final 3D pose.The method learns spatio-temporal representations of diverse pose hypotheses through a three-stage Transformer framework.
  • MHFormer: Self-Hypothesis Refinement independently models and refines each hypothesis, while Cross-Hypothesis Interaction captures mutual correlations for cross-hypothesis communication.These modules provide independent and mutual processing of multi-hypothesis features in an end-to-end manner.
  • Contributions: MHFormer is a Transformer-based method that learns multiple pose hypotheses and communicates among their features independently and mutually.The resulting representation is enhanced and the synthesized pose is more accurate.

2. Related Work

Related work covers two-stage 2D-to-3D lifting for monocular pose estimation, Transformer-based modeling of spatial and temporal dependencies, and multi-hypothesis approaches for the ill-posed inverse problem. MHFormer extends multi-hypothesis modeling by learning one-to-many generation followed by many-to-one aggregation.

  • 3D Human Pose Estimation: Single-view 3D pose estimation methods are broadly divided into one-stage approaches and two-stage methods that lift detected 2D keypoints to 3D poses.Two-stage methods benefit from strong 2D pose detectors to regress 3D poses efficiently and accurately.
  • Vision Transformers: Vision Transformer methods apply global self-attention, while pose-specific Transformers model human-joint correlations and temporal dependencies.PoseFormer uses a pure Transformer, and Strided Transformer lifts a long 2D pose sequence to a single 3D pose.
  • Multi-Hypothesis Methods: Because monocular 3D pose estimation is ill-posed, multi-hypothesis methods generate diverse feasible pose solutions instead of assuming a single solution.Prior work includes candidates constrained by 2D keypoints and anatomy, as well as posterior modeling of 3D pose hypotheses.
  • Multi-Hypothesis Methods: Unlike prior one-to-many methods, MHFormer learns one-to-many mapping first and many-to-one mapping afterward to model diverse hypothesis features.This design improves representation ability by modeling features corresponding to different hypotheses.

3. Multi-Hypothesis Transformer

MHFormer reconstructs the center-frame 3D pose from consecutive 2D poses by learning multi-hypothesis spatio-temporal feature hierarchies. Its three-stage design generates, refines, and cross-communicates pose hypotheses before aggregating them for regression.

  • Multi-Hypothesis Generation: MHFormer uses a cascaded Transformer architecture to generate multiple latent features from consecutive 2D pose sequences, initializing diverse pose-hypothesis representations.The Multi-Hypothesis Generation module models human joint relations and produces multi-level features with diverse semantic information.
  • Self-Hypothesis Refinement: Self-Hypothesis Refinement independently models temporal dependencies within each hypothesis, then uses a hypothesis-mixing MLP to merge information across hypotheses.The mixing operation explores relationships among channels of different hypotheses after within-hypothesis self-attention.
  • Cross-Hypothesis Interaction: Cross-Hypothesis Interaction applies parallel multi-head cross-attention to capture correlations among hypotheses and improve cross-hypothesis message passing.Its efficient configuration uses different inputs and requires only three MCA blocks for three hypotheses, rather than six.
  • Prediction and Training: The final CHI layer aggregates all hypothesis features into a single representation, which a linear regression head maps to a 3D pose sequence before selecting the center-frame prediction.The aggregated representation has dimension N×(C·M), and the model is trained end-to-end with mean squared error.

4. Experiments

Experiments evaluate MHFormer on Human3.6M and MPI-INF-3DHP, showing strong performance against state-of-the-art and prior multi-hypothesis methods. Ablation experiments further examine the impact of the model’s components and design choices.

  • Datasets: MHFormer is evaluated on Human3.6M and MPI-INF-3DHP, following established Human3.6M training and testing protocols.Human3.6M contains 3.6 million images from four synchronized cameras, with training on subjects S1, S5, S6, S7, and S8 and testing on two subjects.
  • Results on Human3.6M: 43.0 mm under Protocol 1 and 34.4 mm under Protocol 2 demonstrate that MHFormer outperforms previous state-of-the-art methods on Human3.6M.These results use a 351-frame receptive field with 2D detected inputs and are reported without bells and whistles.
  • Results on Human3.6M: 30.5 mm in MPJPE gives MHFormer the best performance among compared methods using 2D inputs on Human3.6M.The comparison results are reported in Table 8 (bottom).
  • Multi-hypothesis comparison: With 3 hypotheses versus 200, MHFormer consistently outperforms prior multiple-3D-pose-hypothesis methods while reporting a deterministic specific solution.Prior methods report the best hypothesis under one-to-many mapping, whereas MHFormer learns a deterministic mapping.
  • Ablation studies: Extensive ablation experiments assess the impact of each component and design choice in MHFormer.The reported studies include different receptive fields and MHG parameters such as the number of MHG layers and hypotheses.

M L1 Params (M) FLOPs (G) MPJPE (mm)

The ablations show that MHFormer benefits from longer temporal receptive fields, carefully chosen SHR/CHI depth and embedding size, and multi-level features. The optimal configuration is L2=2, L3=1, and C=512, while additional layers or larger dimensions provide no further gains.

  • Impact of Receptive Fields: 16.7% error reduction occurs when increasing GT-input frames from 9 to 351, demonstrating the value of long-range temporal dependencies.The method obtains larger gains as more frames are fed into the model.
  • Parameter Ablation: Embedding dimension 512 improves performance over 256, whereas dimensions above 512 and additional SHR or CHI layers yield no further gains.The optimal parameters are L2=2, L3=1, and C=512.
  • Effect of Model Components: The component study compares MHFormer against a baseline model to quantify the influence of the proposed components.The supplied passage introduces the baseline comparison but does not report its numerical result.
  • Effect of Model Components: Multi-level features in MHG improve MHFormer’s performance, indicating that they provide valuable information for final pose estimation.The comparison uses MHFormer ∗, which lacks multi-level features in MHG.

5. Qualitative Results

MHFormer is visualized through finetuned intermediate hypotheses, showing multiple plausible 3D pose solutions for ambiguous observations and a final pose synthesized by aggregating them.

  • Visualization setup: Additional regression layers and finetuning visualize MHFormer’s intermediate pose hypotheses.The method itself is not designed to produce multiple final 3D pose predictions.
  • Multiple hypotheses: MHFormer generates different plausible 3D pose solutions, particularly for ambiguous body parts.The qualitative results are presented in Figure 6.
  • Sources of ambiguity: The qualitative examples target ambiguity from depth uncertainty, self-occlusion, and 2D detector uncertainty.
  • Final synthesis: The final 3D pose is synthesized by aggregating the intermediate hypotheses.

6. Conclusion

MHFormer is a Transformer-based, three-stage framework for monocular-video 3D human pose estimation that addresses the problem’s ambiguity through multiple pose hypotheses. It generates spatial hypothesis representations and enables independent and mutual communication across them temporally.

  • 6. Conclusion: MHFormer introduces a Transformer-based three-stage framework for the ambiguous inverse problem of 3D human pose estimation from monocular videos.The framework is explicitly designed around multiple pose hypotheses.
  • 6. Conclusion: The method first generates representations of multiple pose hypotheses in the spatial domain.This is the initial stage of MHFormer’s hypothesis-based processing.
  • 6. Conclusion: It then communicates across hypotheses in the temporal domain through both independent and mutual interactions.These two communication modes constitute the subsequent temporal processing stages.
  • 6. Conclusion: Experiments show that MHFormer has a fundamental advantage over single-hypothesis Transformers.The supplied conclusion passage reports this comparative finding without specifying a numerical margin.

A. Multi-Head Cross-Attention

The section defines multi-head cross-attention among three tensors by separately projecting inputs into queries, keys, and values. Unlike the common shared-key/value configuration, it uses distinct inputs for all three projections for greater efficiency.

  • Definition: MCA operates on tensors x, y, and z, which are linearly mapped to queries Qx, keys Ky, and values Vz, respectively.All inputs and projected tensors have dimensions R^n×d.
  • Attention computation: The MCA uses scaled dot-product attention to relate the query tensor to key and value tensors.This extends the attention formulation described for the multi-head self-attention block.
  • Input configuration: Instead of the common configuration x̸ = y = z, the proposed strategy uses different inputs, x̸ = y̸ = z.The distinct-input configuration is presented as more efficient.

B. Additional Quantitative Results

MHFormer delivers strong quantitative performance on Human3.6M under Protocol 2 using CPN-estimated 2D poses. With a pose refinement module, it achieves 42.4 mm MPJPE and surpasses all other approaches by a large margin.

  • Human3.6M Protocol 2: MHFormer outperforms state-of-the-art approaches on Human3.6M under Protocol 2 using CPN-estimated 2D poses.These results are achieved without bells and whistles.
  • Pose Refinement: 42.4 mm MPJPE is achieved by MHFormer with a pose refinement module, surpassing all other approaches by a large margin.The refinement module follows the design first proposed by ST-GCN.

C. Additional Ablation Studies

This section defines ablation variants of MHFormer by modifying or removing its MHG, SHR, and CHI components. The variants compare a standard Transformer baseline with alternative layer configurations.

  • Effect of Model Components: The Baseline uses three standard Transformer encoder layers, following the same architecture as ViT [6].
  • Effect of Model Components: SHR-CHI removes the MHG module and uses L2=2 SHR layers plus L3=1 CHI layer.
  • Effect of Model Components: MHG-SHR replaces MHFormer’s CHI layers with SHR layers, using L1=4 MHG layers and L3=3 SHR layers.
  • Effect of Model Components: MHG-CHI replaces MHFormer’s SHR layers with CHI layers, using L1=4 MHG layers and L3=3 CHI layers.
  • Effect of Model Components: MHFormer∗ simplifies the MHG by building it upon several parallel Transformers.

D. Additional Visualization Results

MHFormer produces qualitative 3D poses across benchmark and in-the-wild videos, outperforming comparison methods particularly on complex and rare actions. Its intermediate outputs show diverse pose hypotheses whose differences are visualized alongside the final synthesized estimation.

  • 3D Reconstruction Visualization: MHFormer generates qualitative 3D pose results on Human3.6M, MPI-INF-3DHP, and challenging in-the-wild videos.These results are shown in Figures 8 and 9.
  • 3D Reconstruction Visualization: On challenging wild videos, MHFormer produces more accurate and reasonable poses than the baseline and PoseFormer, especially for complex and rare actions.Figure 10 marks wrong estimations with yellow arrows.
  • Hypothesis Visualization: MHFormer generates diverse intermediate 3D pose hypotheses and visualizes them from two perspectives alongside the final synthesized estimation.The final synthesized estimation is color-coded green.
  • Hypothesis Visualization: Multi-head attention maps visualize attention patterns from the Multi-Hypothesis Generation and Self-Hypothesis Refinement modules across three hypotheses.The maps use 9 heads for Multi-Hypothesis Generation and 8 heads for Self-Hypothesis Refinement in the 351-frame model.
Loading 2111.12707v4…