PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformers

Junhyeong Cho, Kim Youwang, Tae-Hyun Oh

arXiv:2207.13820v1cs.CVcs.AIcs.LG

TL;DR

Monocular 3D human mesh recovery is accurate but computationally expensive with encoder-based transformers, limiting practical deployment. FastMETRO disentangles image encoding and mesh estimation through an encoder-decoder architecture with human-body structural priors, achieving improved accuracy-efficiency trade-offs and strong results across human and hand datasets.

  • Problem

    Encoder-based transformers achieve strong monocular mesh-recovery accuracy but use many parameters and expensive computations, limiting practical deployment.

  • Method

    FastMETRO uses an encoder-decoder architecture with cross-attention, topology-based attention masking, and mesh upsampling to separate image encoding from mesh estimation.

  • Results

    FastMETRO improves the accuracy-efficiency Pareto front, with FastMETRO-L achieving state-of-the-art results on Human3.6M and 3DPW and generalizability demonstrated on FreiHAND.

  • Takeaways & Limitations

    FastMETRO provides a more practical transformer-based approach to monocular 3D human mesh recovery by reducing parameters and inference time while retaining strong accuracy.

  • Takeaways & Limitations

    Human3.6M training uses pseudo 3D meshes from SMPLify-X because ground-truth 3D meshes are unavailable under the dataset license.

Abstract

from arXiv · show

Transformer encoder architectures have recently achieved state-of-the-art results on monocular 3D human mesh reconstruction, but they require a substantial number of parameters and expensive computations. Due to the large memory overhead and slow inference speed, it is difficult to deploy such models for practical use. In this paper, we propose a novel transformer encoder-decoder architecture for 3D human mesh reconstruction from a single image, called FastMETRO. We identify the performance bottleneck in the encoder-based transformers is caused by the token design which introduces high complexity interactions among input tokens. We disentangle the interactions via an encoder-decoder architecture, which allows our model to demand much fewer parameters and shorter inference time. In addition, we impose the prior knowledge of human body's morphological relationship via attention masking and mesh upsampling operations, which leads to faster convergence with higher accuracy. Our FastMETRO improves the Pareto-front of accuracy and efficiency, and clearly outperforms image-based methods on Human3.6M and 3DPW. Furthermore, we validate its generalizability on FreiHAND.

1 Introduction

FastMETRO addresses the efficiency bottleneck of encoder-based transformers for monocular 3D human mesh recovery by disentangling image encoding from mesh estimation. Its encoder-decoder design and morphological priors improve efficiency, convergence, and benchmark performance.

  • Motivation: Monocular 3D mesh recovery remains difficult because articulated bodies, occlusions, and depth ambiguity must be handled from a single image.The task estimates both 3D body joints and mesh vertices for applications such as motion analysis and human-computer interaction.
  • Method: FastMETRO uses an encoder-decoder architecture to separate image encoding from mesh estimation and reduce costly token interactions.The decoder uses learnable joint and vertex tokens with cross-attention to image features.
  • Motivation: Encoder-based methods concatenate the same global image feature with every joint and vertex token, causing spatial-information loss, duplicated computation, and quadratic layer complexity.Only 0.15% of the input-token information is distinctively informative according to the supplied passage.
  • Results: FastMETRO-L achieves state-of-the-art results on Human3.6M and 3DPW while using fewer parameters and shorter inference time than encoder-based methods.FastMETRO-S uses about 9% of the transformer parameters and is faster while remaining competitive.
  • Method: FastMETRO imposes human-body structure through mesh-topology attention masks and mesh upsampling, supporting faster convergence and higher accuracy.The attention design preserves non-local joint-vertex relations and local vertex-vertex relations.

2 Related Work

Related work spans parametric and non-parametric approaches to monocular human mesh recovery. FastMETRO builds on transformer-based non-parametric methods by decoupling image encoding and mesh estimation while retaining human-body structural priors.

  • Human Mesh Reconstruction: Parametric methods exploit human-body models such as SMPL for stable monocular recovery but require learning difficult 3D joint rotations.Their pose-space regression targets are expressed as 3D rotations of human body joints.
  • Human Mesh Reconstruction: Non-parametric methods use image features and mesh topology to regress vertex coordinates directly, including graph convolutional and transformer-based designs.GCNN-based methods learn local vertex-vertex relations from the topology of the SMPL human triangle mesh.
  • Transformer-Based Methods: METRO and Mesh Graphormer use encoder architectures whose token design introduces unnecessary parameters and computation while learning image-to-mesh and mesh-vertex relations together.Their encoders process tokens formed from global image features and mesh coordinates.
  • Transformer-Based Methods: FastMETRO decouples heterogeneous 2D-image and 3D-mesh relations with an encoder-decoder architecture that supports structural attention masking.This separate decoder design enables human-body priors to be imposed through masked self-attention, producing more stable optimization and higher accuracy.

3 Method

FastMETRO separates image encoding from mesh estimation with a transformer encoder-decoder, then uses structured token interactions, mesh-topology masking, and coarse-to-fine upsampling to recover 3D joints and vertices.

  • 3.1 Feature Extractor: FastMETRO encodes image features separately, while decoder joint and vertex tokens use self-attention and cross-attention to estimate mesh features.The encoder extracts spatially positioned image features; the decoder focuses tokens on relevant image regions.
  • 3.1 Feature Extractor: The encoder processes a camera token and flattened image features, while joint and vertex tokens represent body joints and mesh vertices.The camera feature predicts weak-perspective camera parameters and is not provided to the decoder.
  • 3.2 Transformer Decoder: Attention masking uses SMPL mesh adjacency to capture local vertex-vertex and non-local joint-vertex relations.This imposes human-body morphological structure on decoder self-attention.
  • 3.3 Mesh Regression: FastMETRO predicts a coarse mesh and upsamples it with a precomputed matrix, reducing redundancy from the spatial locality of mesh vertices.The fine mesh is computed as ˆV′3D = U ˆV3D.

4 Implementation Details

FastMETRO is implemented in three transformer-depth variants, and the evaluation compares these configurations with encoder-based transformers on Human3.6M.

  • 4 Implementation Details: FastMETRO-S, FastMETRO-M, and FastMETRO-L share the architecture but differ in the number of transformer encoder and decoder layers.Table 1 reports configurations for the transformer components.
  • 4 Implementation Details: On Human3.6M, the small FastMETRO variant has much faster inference, while the large variant converges faster than encoder-based transformers.The comparison is shown for encoder-based transformers and the proposed FastMETRO variants.

5 Experiments

FastMETRO is evaluated on Human3.6M, 3DPW, COCO, and FreiHAND using standard training protocols, quantitative comparisons, visualizations, and ablations. The experiments show competitive or state-of-the-art accuracy, efficient inference, meaningful attention behavior, and benefits from topology-aware design choices.

  • 5.3 Experimental Results: FastMETRO-S uses 8.99% of METRO's transformer parameters while achieving competitive results and 1.86× faster overall inference.FastMETRO-L-R50 achieves similar results to METRO-H64 while running 2.58× faster; FastMETRO-L uses 25.30% of Mesh Graphormer's transformer parameters.
  • 5.3 Experimental Results: FastMETRO-L-H64 achieves state-of-the-art performance on every metric on 3DPW and on PA-MPJPE on Human3.6M among image-based methods.The comparison is reported without iterative fitting procedures or test-time optimizations.
  • 5.3 Experimental Results: Self-attention captures non-local joint-vertex relations, while cross-attention focuses joint and vertex tokens on relevant image regions.These visualizations support the decoder's use of disentangled mesh and image modalities for 3D coordinate regression.
  • 5.4 Ablation Study: Attention masking of non-adjacent vertices improves regression accuracy, while graph convolutions achieve similar results with more parameters.Masking only half the attention heads produces similar results with the same parameter count.
  • 5.4 Ablation Study: A sparse pre-computed upsampling matrix improves regression accuracy over learnable upsampling layers while requiring far fewer parameters.The matrix has about 25K non-zero elements; removing progressive dimensionality reduction yields similar results but requires more parameters.
  • 5.5 Generalizability on FreiHAND: FastMETRO-L-H64 achieves competitive results on FreiHAND, supporting generalizability beyond human-body mesh recovery.The model uses hand-mesh topology to construct adjacency masking and joint and vertex tokens.

6 Conclusion

FastMETRO addresses the token-design bottleneck of encoder-based transformers through an encoder-decoder architecture with human-mesh priors. The resulting model improves the accuracy-efficiency trade-off, supports robustness to occlusions, and outperforms image-based methods on Human3.6M and 3DPW, although substantial training data remains necessary.

  • 6 Conclusion: FastMETRO improves the Pareto-front of accuracy and efficiency by using an encoder-decoder architecture instead of encoder-based token interactions.The architecture reduces parameters and inference time while addressing the identified token-design bottleneck.
  • 6 Conclusion: Human-mesh priors through masked self-attention and coarse-to-fine upsampling reduce optimization difficulty and lead to faster convergence with higher accuracy.The masking restricts attention between non-adjacent vertices, while upsampling uses the mesh structure.
  • 6 Conclusion: FastMETRO captures non-local relations among body joints and mesh vertices, achieving robustness to occlusions and outperforming image-based methods on Human3.6M and 3DPW.The conclusion attributes occlusion robustness to the learned non-local relations.
  • 6 Conclusion: A substantial number of training samples are required, as with the encoder-based transformers.This is the stated limitation of the method.

A Implementation Details

The implementation uses CNN image features, fixed positional encodings, sparse mesh structures, AdamW optimization, and several FastMETRO configuration settings. Training uses multi-dataset supervision, gradient clipping, and substantially fewer epochs and GPUs than the compared encoder-based methods.

  • A Implementation Details: The model uses ResNet-50 or HRNet-W64 backbones, 7×7×2048 image features, 512-dimensional tokens, 14 joint tokens, and 431 vertex tokens.The fine mesh contains 6,890 SMPL vertices, and multi-head attention uses 8 heads.
  • A Implementation Details: Fixed sine positional encodings retain spatial information after flattening image features, while sparse matrices implement mesh upsampling and attention masking.The upsampling matrix has about 25K non-zero elements and the adjacency matrix about 3K.
  • A Implementation Details: FastMETRO is trained with AdamW at learning rate 10^-4, batch size 16, for 60 epochs using gradient clipping with maximal norm 0.3.Training takes about 4 days on 4 NVIDIA V100 GPUs, compared with about 5 days on 8 GPUs for METRO and Mesh Graphormer.
  • A Implementation Details: The implementation evaluates an ablation model on Human3.6M with the default configuration FastMETRO-S-R50.The ablation study varies the method's components to assess their effects.

B Quantitative Evaluations

Additional ablations examine token construction and camera prediction choices in FastMETRO. The results indicate that camera prediction from transformer-estimated mesh coordinates can reduce regression accuracy, whereas a camera token performs similarly.

  • B Quantitative Evaluations: The ablation uses a naive encoder-decoder baseline with mesh-based input tokens, learnable upsampling, and no attention masking.Joint and vertex coordinates are linearly projected to construct the tokens.
  • B Quantitative Evaluations: Camera-token prediction in the transformer decoder produces similar results to the compared camera-prediction alternatives.The ablation evaluates different methods for estimating camera parameters.
  • B Quantitative Evaluations: Predicting camera parameters from the 3D mesh estimated by transformer layers reduces regression accuracy.This result is reported in the third and fourth rows of Table B1.

C Qualitative Evaluations

FastMETRO-L-H64 achieves competitive qualitative results with substantially fewer transformer parameters than encoder-based methods, particularly for detailed knee and ankle poses.

  • Comparison with Encoder-Based Transformers: FastMETRO-L-H64 achieves competitive qualitative results using about 25% of the transformer parameters required by encoder-based transformers.The comparison is shown on 3DPW qualitative results.
  • Comparison with Encoder-Based Transformers: FastMETRO captures more detailed body pose, especially around the knees and ankles, than the compared encoder-based transformers.This qualitative advantage is highlighted in the 3DPW comparison.
  • Comparison with Encoder-Based Transformers: The qualitative evaluations compare input images, METRO, Mesh Graphormer, and FastMETRO outputs.
  • SMPL Parameters from Estimated Mesh: Estimated mesh vertices can optionally be used to regress SMPL pose and shape parameters through a separate parameter regressor.The pipeline first estimates 3D mesh vertices and then predicts SMPL coefficients from them.
Loading 2207.13820v1…