PaperlayerView exact revision on arXiv

Source-linked AI summary

Mesh Graphormer

Kevin Lin, Lijuan Wang, Zicheng Liu

arXiv:2104.00272v2cs.CV

TL;DR

Single-image 3D human mesh reconstruction requires modeling both global interactions and local neighborhood structure, which transformers and GCNNs address differently. Mesh Graphormer combines graph convolutions, self-attention, and image-grid features in a transformer, achieving state-of-the-art performance on Human3.6M, 3DPW, and FreiHAND. Its main limitation is performance when most of the body is outside the input view.

  • Problem

    Transformers model non-local interactions, while GCNNs exploit neighborhood interactions; the paper studies how to combine them for human mesh reconstruction.

  • Method

    Mesh Graphormer injects graph convolutions into transformer blocks and lets joints and mesh vertices attend to image-grid features.

  • Results

    Mesh Graphormer achieves new state-of-the-art performance on Human3.6M, 3DPW, and FreiHAND datasets.

  • Takeaways & Limitations

    Combining graph convolutions with self-attention effectively models both local and global interactions for 3D human pose and mesh reconstruction.

  • Takeaways & Limitations

    The method may fail when most of the human body is outside the input image, likely because the training set lacks out-of-view 3D data.

Abstract

from arXiv · show

We present a graph-convolution-reinforced transformer, named Mesh Graphormer, for 3D human pose and mesh reconstruction from a single image. Recently both transformers and graph convolutional neural networks (GCNNs) have shown promising progress in human mesh reconstruction. Transformer-based approaches are effective in modeling non-local interactions among 3D mesh vertices and body joints, whereas GCNNs are good at exploiting neighborhood vertex interactions based on a pre-specified mesh topology. In this paper, we study how to combine graph convolutions and self-attentions in a transformer to model both local and global interactions. Experimental results show that our proposed method, Mesh Graphormer, significantly outperforms the previous state-of-the-art methods on multiple benchmarks, including Human3.6M, 3DPW, and FreiHAND datasets. Code and pre-trained models are available at https://github.com/microsoft/MeshGraphormer

1. Introduction

Mesh Graphormer combines self-attention and graph convolution to model global and local interactions for single-image 3D human pose and mesh reconstruction. It reports improved performance across several benchmarks and uses image-grid features to refine coordinate predictions.

  • Motivation: Transformers capture long-range dependencies, whereas convolutional layers extract local features but require many layers for global context.This complementarity motivates combining self-attention with graph convolution for human mesh reconstruction.
  • Architecture: The Graphormer encoder uses a stack of N = 4 identical blocks incorporating graph-convolutional processing.The architecture is presented as a transformer encoder reinforced with graph convolutions.
  • Contribution: Mesh Graphormer combines graph convolutions and self-attention in a transformer to model both local and global interactions.The graph-convolution-reinforced encoder is designed for 3D human pose and mesh reconstruction.
  • Method: Mesh Graphormer allows joints and mesh vertices to attend freely to image-grid features for refining 3D coordinate predictions.The grid features provide detailed local information that complements graph convolutions.
  • Results: Mesh Graphormer outperforms previous state-of-the-art methods on Human3.6M, 3DPW, and FreiHAND datasets.The paper also reports extensive experiments and ablations over model-design options.

2. Related Works

Prior mesh-reconstruction methods emphasize either local mesh topology through graph convolutions or global interactions through self-attention. Mesh Graphormer addresses this division by injecting graph convolutions into transformer encoder blocks and adding image-grid features.

  • Approaches: Parametric approaches regress human-model coefficients, while non-parametric approaches regress mesh vertices directly from an image.The cited discussion contrasts these two broad reconstruction strategies.
  • Graph Convolutions: GCNNs model local interactions between neighboring vertices using a given adjacency matrix but are less efficient at capturing global vertex-joint interactions.This limitation motivates transformer-based alternatives.
  • Transformers: Transformer-based methods use self-attention to freely attend among vertices and body joints, addressing global interactions.The passage positions self-attention as complementary to GCNN-based local modeling.
  • Relation to Prior Work: Compared with METRO, Mesh Graphormer adds graph-convolutional reinforcement and image-grid tokens that joints and vertices can attend to.These design differences target local interactions and detailed visual information.
  • Research Gap: Unlike prior transformer combinations focused on language or speech, Mesh Graphormer injects graph convolutions into transformer blocks for 3D human mesh data.The model is intended to capture local and global interactions in complex mesh structures.

3. Graphormer Encoder

Graphormer combines transformer self-attention with graph convolution in stacked encoder blocks to model global context and fine-grained local interactions. The encoder processes contextualized token features and explicitly incorporates graph structure to improve spatial locality.

  • The Graphormer encoder stacks four identical blocks containing normalization, multi-head self-attention, a graph residual block, and an MLP.Graph convolution is inserted into a transformer-like encoder to model fine-grained local interactions.
  • 3.1. Multi-Head Self-Attention: Multi-head self-attention projects input tokens into queries, keys, and values, then computes contextual representations across parallel subspaces.The resulting subspace outputs are concatenated into the final representation.
  • The encoder combines self-attention for long-range dependencies with graph convolution to address MHSA's weaker fine-grained modeling of complex 3D mesh structure.This design explicitly encodes graph structure within the network and improves spatial locality in the features.
  • 3.2. Graph Residual Block: Graph convolution refines MHSA features by using an adjacency matrix and trainable parameters to encode spatially local interactions.The graph residual block applies graph convolution to contextualized features and adds nonlinear transformation.
  • 3.2. Graph Residual Block: The graph residual block follows a multi-layer graph-convolution design aligned with transformer normalization and activation choices.It replaces group normalization with layer normalization and ReLU with GeLU.

4. Graphormer for Mesh Reconstruction

For single-image mesh reconstruction, the framework combines CNN image-grid and global features with joint and vertex queries, then predicts 3D joints and mesh coordinates through a multi-layer Graphormer. It uses coarse-mesh prediction, upsampling, and multiple training losses to produce efficient mesh reconstruction.

  • 4. Graphormer for Mesh Reconstruction: The end-to-end framework tokenizes image features and predicts 3D mesh-vertex and body-joint coordinates simultaneously.The input is a 224x224 image, and the image features are processed by a multi-layer Graphormer encoder.
  • 4.1. CNN and Image Grid Features: A pre-trained CNN supplies 49 tokenized 7 × 7 × 1024 grid features plus a 2048-Dim global image feature vector.Grid features provide fine-grained local details, while positional encoding uses 3D coordinates from a human template mesh; all tokens are projected to 2051-Dim.
  • 4.2. Multi-Layer Graphormer Encoder: The Graphormer jointly processes 49 grid-feature tokens, 14 joint queries, and 431 vertex queries through three blocks with hidden dimensions 1024, 256, and 64.The encoder sequentially reduces feature dimensions while maintaining the same number of tokens.
  • 4.2. Multi-Layer Graphormer Encoder: The model predicts a 431-vertex coarse mesh and linearly upsamples it to the original SMPL resolution of 6K vertices.Coarse prediction and upsampling reduce redundancies and make training more efficient.
  • 4.3. Training Details: Training applies L1 losses to 3D vertices, 3D joints, and projected 2D joints, with masked vertex modeling and intermediate coarse-mesh supervision.These objectives improve image-mesh alignment, robustness, and convergence.
  • 4.3. Training Details: The model trains for 200 epochs with Adam, reducing the learning rate by 10 after 100 epochs while using ImageNet-pre-trained CNN weights.Graphormer weights are randomly initialized, and reported results use an HRNet backbone.

5. Experiments

Experiments compare Mesh Graphormer with prior methods across human and hand reconstruction benchmarks, then analyze architectural choices through ablations. Results support combining image grid features, graph convolutions, and self-attention to model local and global interactions.

  • Main Results: Mesh Graphormer outperforms previous state-of-the-art methods on Human3.6M and 3DPW, including both GCNN-based and transformer-based approaches.The comparison reports more accurate mesh reconstruction and attributes the result to modeling local and global interactions better.
  • Main Results: Mesh Graphormer also outperforms prior methods on FreiHAND and reconstructs hands with novel, complicated gestures.The paper presents this as evidence of generalization to 3D hand reconstruction.
  • Ablation Study: Image grid features improve reconstruction performance over a transformer baseline using a single global feature vector.The authors identify the single global feature vector as a performance bottleneck in existing techniques.
  • Ablation Study: The latest HRNet feature map, labeled F, performs better than the other evaluated feature maps.The results suggest that the feature pyramid helps improve transformer-encoder performance.
  • Ablation Study: Adding graph convolution to Encoder3 improves PA-MPJPE by 0.9, whereas adding it to Encoder1 or Encoder2 does not improve performance.The authors suggest lower layers focus on global posture interactions while upper layers emphasize local shape interactions.
  • Ablation Study: A graph convolution layer placed directly after MHSA performs better than the other tested Graphormer block designs.The study compares parallel placement, graph convolution followed by MHSA, and MHSA followed by graph convolution.
  • Ablation Study: A Graph Residual Block works better than a basic graph convolution layer in the evaluated encoder configuration.The ablation replaces a graph convolution layer with a Graph Residual Block to increase model capacity.
  • Ablation Study: Enabling both grid features and graph convolution improves PA-MPJPE by 2.2, exceeding their individual improvements of 0.8 and 0.1.The paper describes this as mutual reinforcement between grid features and graph convolutions.

6. Conclusion

The paper concludes that Mesh Graphormer combines graph convolutions and self-attention for single-image human pose and mesh reconstruction. Experiments report state-of-the-art performance across Human3.6M, 3DPW, and FreiHAND.

  • 6. Conclusion: Mesh Graphormer incorporates graph convolutions and self-attention to model local and global interactions in single-image human pose and mesh reconstruction.The conclusion also reports that graph convolutions and grid features improve transformer performance.
  • 6. Conclusion: Mesh Graphormer achieves new state-of-the-art performance on Human3.6M, 3DPW, and FreiHAND datasets.The conclusion bases this claim on the paper’s experimental results.

Supplementary Material

Under heavy occlusion, both METRO and Graphormer are described as robust, while Graphormer produces more favorable head and body poses.

  • Supplementary Material: Graphormer generates better head and body poses than METRO in qualitative examples with heavy occlusions.The comparison includes cases where almost half of the subject is occluded and where a car door occludes the subject.

B. Additional Qualitative Results

Additional qualitative tests show that Mesh Graphormer reconstructs plausible hand meshes under severe artificial occlusions and interactions with novel objects. The section also reports an architectural comparison in which graph convolutions outperform similarly sized MLP replacements.

  • Occlusion and novel-object robustness: Artificial black-stripe occlusions were applied only during inference, progressively covering fingers or palm regions in Internet-collected hand images.The evaluation used no artificial occlusions during training.
  • Occlusion and novel-object robustness: Graphormer reconstructs reasonable hand meshes despite severe artificial occlusions, indicating some robustness to the tested occlusion patterns.Figure 9 covers a hand holding an orange with increasingly severe stripe occlusions.
  • Occlusion and novel-object robustness: Graphormer reconstructs hand meshes for novel-object interactions, including banana and ice-cream-cone grasps with substantial finger occlusion.The ice-cream-cone example also preserves an object-specific grasp, while the banana was unseen during training.
  • Occlusion and novel-object robustness: With most fingers invisible, Graphormer still produces a hand mesh with the correct pose for a hand holding half an orange.This result is reported for Figure 12.
  • MLP versus graph residual block: Replacing the graph residual block with MLPs did not improve the baseline, whereas graph convolutions improved PAMPJPE from 35.9 to 34.5 with only 0.04M additional parameters.The comparison increased MLP hidden size to similar or larger model sizes before evaluating Graphormer.
  • Encoder design: Among three encoder designs, placing graph convolutions after MHSA performed better than the other studied options.The design alternatives were compared for human mesh reconstruction.

E. Discussion of Attention Map

The attention-map discussion cautions that color normalization can obscure absolute differences between methods. Without normalization, Graphormer shares attention patterns with METRO in some regions while additionally attending to the left lower leg.

  • Color normalization: Attention colors are normalized by each diagram’s maximum value, making Graphormer’s colors lighter because its maximum attention value is smaller.The authors provide unnormalized diagrams to support direct visual comparison.
  • Regional attention comparison: Both Graphormer and METRO attend similarly to the left arm and right foot, while Graphormer also attends to the left lower leg.The unnormalized comparison is the basis for this regional observation.

H. Computational Costs

Adding graph convolutions introduces little computational overhead while substantially improving reported performance across multiple benchmarks. The comparison includes parameter count and GFLOPs, with the HRNet backbone reported separately for reference.

  • Computational overhead: Adding graph convolutions increases the transformer by 0.04M parameters and 0.01 GFLOPs relative to the conventional transformer.Table 10 compares the conventional transformer, Graphormer, and the HRNet CNN backbone.
  • Performance relative to cost: Graphormer significantly improves state-of-the-art performance across multiple benchmarks despite the small added complexity.The authors interpret the comparison as evidence for the effectiveness of the proposed method.
  • End-to-end accounting: The total end-to-end pipeline parameter count is the sum of the HRNet and Graphormer parameters.This distinguishes the full pipeline from the transformer-module comparison.

I. Limitation

Mesh Graphormer may fail when much of the reconstruction target lies outside the input view. The authors attribute this limitation probably to insufficient out-of-view 3D training data and identify it for future work.

  • Out-of-view targets: When most of the human body is outside the input image, Mesh Graphormer may fail to estimate a correct human mesh.A case with only two visible hands and the rest of the body out of view also performs poorly.
  • Out-of-view targets: The authors probably attribute these failures to a lack of out-of-the-view 3D training data and plan to address the issue in future work.The stated explanation is qualified as probable rather than definitive.
Loading 2104.00272v2…