PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphMLP: A Graph MLP-Like Architecture for 3D Human Pose Estimation

Wenhao Li, Mengyuan Liu, Hong Liu, Tianyu Guo, Ti Wang, Hao Tang, Nicu Sebe

arXiv:2206.06420v5cs.CVcs.AIcs.LG

TL;DR

Existing MLPs lack human-configuration priors and are less effective at modeling local joint interactions, while video methods can incur redundant computation and struggle with long sequences. GraphMLP injects GCNs into spatial and channel MLP blocks to encode skeleton connectivity and model local and global interactions, with a feature-channel representation for video sequences. GraphMLP improves over MGCN by 1.4 mm in MPJPE on Human3.6M; on MPI-INF-3DHP, it improves over MLP and GCN models by 6.6 mm and 6.9 mm, respectively.

  • Problem

    Existing MLPs lack human-configuration priors and are less effective at modeling local joint interactions, while video methods can incur redundant computation and struggle with long sequences.

    Source: §1 · pp. 1–2

  • Method

    GraphMLP injects GCNs into spatial and channel MLP blocks to encode skeleton connectivity and model local and global interactions, with a feature-channel representation for video sequences.

    Source: §1 · pp. 2–3

  • Results

    GraphMLP improves over MGCN by 1.4 mm in MPJPE on Human3.6M; on MPI-INF-3DHP, it improves over MLP and GCN models by 6.6 mm and 6.9 mm, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper concludes that GraphMLP is a simple, effective baseline for single-frame and video pose estimation, modeling temporal dynamics without explicit temporal modeling.

    Source: §6 · p. 10

    Human3.6M lacks sufficient pose, environment, and activity variation, limiting generalization to real-world scenarios.

    Source: §6 · p. 10

Abstract

from arXiv · show

Modern multi-layer perceptron (MLP) models have shown competitive results in learning visual representations without self-attention. However, existing MLP models are not good at capturing local details and lack prior knowledge of human body configurations, which limits their modeling power for skeletal representation learning. To address these issues, we propose a simple yet effective graph-reinforced MLP-Like architecture, named GraphMLP, that combines MLPs and graph convolutional networks (GCNs) in a global-local-graphical unified architecture for 3D human pose estimation. GraphMLP incorporates the graph structure of human bodies into an MLP model to meet the domain-specific demand of the 3D human pose, while allowing for both local and global spatial interactions. Furthermore, we propose to flexibly and efficiently extend the GraphMLP to the video domain and show that complex temporal dynamics can be effectively modeled in a simple way with negligible computational cost gains in the sequence length. To the best of our knowledge, this is the first MLP-Like architecture for 3D human pose estimation in a single frame and a video sequence. Extensive experiments show that the proposed GraphMLP achieves state-of-the-art performance on two datasets, i.e., Human3.6M and MPI-INF-3DHP. Code and models are available at https://github.com/Vegetebird/GraphMLP.

1. Introduction

GraphMLP combines MLPs and GCNs to model skeletal structure while capturing local and global joint interactions, and extends this design to video. It reports improved 3D pose estimation results on Human3.6M and MPI-INF-3DHP.

1. Introduction

  • GCNs aggregate neighboring joints but often have limited receptive fields; stacking layers can expand coverage while causing over-smoothing.Distant-joint relations, such as hand position relative to the foot, can indicate overall posture.Source: §1 · pp. 1–2
  • Vision MLPs can model distant-joint interactions but lack skeleton-specific priors and careful designs for adjacent-joint details.Without human-configuration priors, they may learn spurious dependencies associated with physically implausible poses.Source: §1 · p. 2
  • GraphMLP injects GCNs into spatial and channel MLP blocks, combining human-skeleton connectivity with local and global joint interactions.Its GraphMLP layers comprise spatial graph MLP (SG-MLP) and channel graph MLP (CG-MLP) blocks.Source: §1 · pp. 2–3
  • Video methods that model frames or time separately can incur redundant computation and may struggle to process long sequences.GraphMLP mixes temporal information in feature channels while keeping each joint as a token, enabling arbitrary-length sequences with negligible computational cost gains in sequence length.Source: §1 · pp. 2–3
  • 1.4 mm lower MPJPE than MGCN on Human3.6M; on MPI-INF-3DHP, GraphMLP improves MPJPE by 6.6 mm over MLP-Mixer and 6.9 mm over GCN.The figure compares MPJPE, for which lower values are better.Source: Figure 1 · p. 1, §1 · p. 3
  • The paper presents GraphMLP as the first MLP-like architecture for 3D human pose estimation, covering single-frame and video settings.The authors describe experiments on Human3.6M and MPI-INF-3DHP as demonstrating effectiveness and generalization ability.Source: §1 · p. 3

2. Related Work

Related work covers direct image-to-3D regression and 2D-to-3D pose lifting, alongside graph-based and Transformer approaches. GraphMLP differs by combining modern MLPs with GCNs in parallel and mixing features across video frames while retaining joints as tokens.

2. Related Work

  • Pose estimation methods either regress 3D joints directly from RGB images or lift detected 2D poses into 3D.The lifting approach is described as a mainstream method valued for efficiency and effectiveness.Source: §2 · p. 3
  • GCN-based lifting methods encode skeleton topology, while Transformer work includes spatial-temporal video models and graph-based designs.The passage names LCN, SemGCN, and MGCN among graph approaches, and PoseFormer, Strided Transformer, HDFormer, and RTPCA among Transformer approaches.Source: §2 · p. 3
  • GraphMLP combines modern MLPs and GCNs in parallel to model local and global information together, unlike serial GCN-attention designs such as Mesh Graphormer.The authors emphasize the efficiency of modern MLPs and their parallel combination with GCNs.Source: §2 · p. 3
  • Unlike prior video methods that treat frames as tokens, GraphMLP mixes features across frames while keeping each joint as a token.The authors describe this representation as economical and easy to train; they also note modern MLP models had not yet been applied to 3D human pose estimation.Source: §2 · pp. 3–4

3. Proposed GraphMLP

GraphMLP combines MLPs and GCNs to model global and local joint interactions while incorporating human-skeleton structure. Its layers build on spatial and channel MLPs, with graph convolution added for local feature communication.

3.2. Network Architecture

  • The model takes 2D joint locations as input and predicts 3D poses through a skeleton embedding module, GraphMLP layers, and a prediction head.Source: §3 · p. 4

3.1.1 Graph Convolutional Networks

  • GCN blocks use human joints as graph nodes and bone connections as edges, with different kernel weights for different connection types.The adjacency matrix represents bone connections for node information passing.Source: §3.1.1 · p. 4, Figure 3 · p. 4

3.1.2 Modern MLPs

  • The MLP-Mixer baseline uses spatial and channel MLPs, but lacks fine-grained local detail and prior knowledge of human configurations.Source: §3.1.2 · pp. 4–5

3.2.1 Skeleton Embedding

  • Each joint is an input token; a linear layer projects the tokens into high-dimensional features for the GraphMLP layers.Source: §3.2.1 · p. 5

3.2.2 GraphMLP Layer

  • GraphMLP layers pair global spatial or channel MLP processing with GCN blocks that aggregate local information between neighboring joints.The spatial and channel graph MLP blocks process their respective features in parallel with GCNs.Source: §3.2.2 · p. 5

3.2.3 Prediction Head

  • The prediction head applies a linear regression layer to the final features to predict the 3D pose.Source: §3.2.3 · p. 5

3.2.4 Loss Function

  • GraphMLP is trained end-to-end using an L2-norm loss between predicted and ground-truth 3D joint coordinates.Source: §3.2.4 · p. 5

3.3. Extension in the Video Domain

GraphMLP extends to video by combining each joint’s coordinate features across frames into a token, then predicting the center-frame pose. The representation supports arbitrary sequence lengths, with added computation confined to a linear layer whose cost grows linearly with sequence length.

3.3. Extension in the Video Domain

  • For video, GraphMLP concatenates each joint’s x-y coordinates across frames, maps the resulting token through a linear layer, and predicts the center-frame 3D pose.Source: §3.3 · pp. 5–6
  • The representation handles arbitrary-length sequences, including T = 1, 27, 81, or 243, while added computation comes from a linear layer with cost linear in sequence length.GraphMLP layers and the prediction head have the same computational overhead and parameter count across input sequence lengths.Source: §3.3 · p. 6
  • A single linear layer can encode temporal information competitively without treating each frame as a separate token.Source: §3.3 · p. 6

4. Experiments

Experiments evaluate GraphMLP on Human3.6M and MPI-INF-3DHP, including single-frame, video, and ablation comparisons. The results report strong benchmark performance, cross-dataset generalization, and lower computational cost than several video methods.

4.1. Datasets and Evaluation Metrics

  • The Human3.6M evaluation uses detected 2D keypoints and reports MPJPE and PA-MPJPE, while MPI-INF-3DHP uses PCK and AUC for cross-dataset testing.The implementation uses three GraphMLP layers with hidden size 512 and CPN detections for Human3.6M.Source: §4.1 · p. 6, §4.2 · p. 6

4.3. Comparison with State-of-the-Art Methods

  • 48.0 mm MPJPE for GraphMLP with refinement is 1.4 mm lower than MGCN on Human3.6M with detected 2D keypoints.Without refinement, GraphMLP reaches 49.2 mm MPJPE and 38.6 mm PA-MPJPE.Source: §4.3 · p. 6, Table 1 · p. 7
  • GraphMLP trained on Human3.6M obtains the best results across all scenes and metrics on MPI-INF-3DHP, supporting generalization to unseen scenarios.The evaluation uses the Human3.6M-trained model directly on MPI-INF-3DHP.Source: Table 3 · p. 7, §4.3 · p. 7
  • 30.3 mm MPJPE with ground-truth inputs is reported for GraphMLP, versus 31.3 mm for Poseformer; GraphMLP also uses 356M FLOPs versus PoseFormer’s 1625M.Against the MLP-Mixer baseline, GraphMLP improves MPJPE by 4.6 mm with CPN inputs and 4.9 mm with ground-truth inputs.Source: Table 5 · p. 8, §4.3 · p. 8

4.4. Ablation Study

  • L = 3 and hidden size C = 512 perform best among the tested configurations; increasing C from 128 to 512 reduces MPJPE from 50.2 mm to 49.2 mm as parameters rise from 0.60M to 9.49M.Performance saturates above C = 512, and the authors suggest Human3.6M may not be diverse enough to train a larger GraphMLP.Source: §4.4 · p. 8, Table 6 · p. 8
  • GraphMLP reaches 49.2 mm MPJPE versus 51.3 mm for the GCN-based model, and outperforms the tested Transformer and Transformer-GCN architectures.The authors suggest self-attention may be less effective for long-range dependencies with few tokens, such as 17 joints.Source: Table 8 · p. 9, §4.4 · p. 9
  • Parallel placement of GCN and MLP performs best among the tested design options; adding SG-MLP and CG-MLP together improves MPJPE by 2.8 mm over MLP-Mixer.Adding SG-MLP or CG-MLP individually yields 50.6 mm and 50.5 mm, respectively.Source: Table 9 · p. 9, §4.4 · pp. 9–10, Table 10 · p. 9

5. Qualitative Results

Qualitative comparisons show GraphMLP producing more accurate and plausible poses on Human3.6M and challenging in-the-wild images, including actions rare or absent from Human3.6M training. Failures remain under difficult input and pose conditions.

5. Qualitative Results

  • GraphMLP produces more accurate and plausible poses than the compared methods on challenging in-the-wild images, including actions rare or absent from Human3.6M training.Reported failure cases involve large 2D detection errors, half-body views, rare poses, and heavy occlusion.Source: §5 · p. 10

6. Conclusion and Future Works

GraphMLP is presented as the first modern MLP architecture dedicated to 3D human pose estimation, combining MLPs and GCNs. The authors report state-of-the-art single-frame and competitive video results, while noting dataset-quality limits.

6. Conclusion and Future Works

  • GraphMLP is presented as the first modern MLP architecture dedicated to 3D human pose estimation.The authors describe it as a simple yet effective baseline for single-frame and video pose estimation.Source: §6 · p. 10
  • Figure 8 presents challenging scenarios in which GraphMLP fails to produce accurate 3D human poses.Source: Figure 8 · p. 10
  • GraphMLP combines MLPs and GCNs in a global-local-graphical architecture without self-attention, and the authors report state-of-the-art single-frame performance and competitive video results with fewer computational costs.They also report modeling complex temporal dynamics without explicit temporal modeling.Source: §6 · p. 10
  • The authors identify dataset quality as a performance limit, noting that Human3.6M lacks pose, environment, and activity diversity that affects real-world generalization.They suggest domain adaptation and synthetic data, as well as more powerful MLPs or GCNs, as future directions.Source: §6 · p. 10

Supplementary Material

The supplementary material points readers to additional MLP descriptions, implementation details, ablations, and qualitative results.

Supplementary Material

A. Multi-Layer Perceptrons

This appendix details the MLP-Mixer layer and its spatial and channel MLPs. For the video version, it specifies additional layer normalization in the spatial MLP.

A. Multi-Layer Perceptrons

  • The MLP-Mixer layer applies spatial and channel MLP blocks with layer normalization, as expressed in its residual updates.Source: A. Multi-Layer Perceptrons · p. 13
  • Each spatial and channel MLP uses two linear layers, with GELU as the activation function.The appendix specifies separate weight matrices for the two linear layers in each MLP.Source: A. Multi-Layer Perceptrons · p. 13
  • For GraphMLP in video, the spatial MLP applies layer normalization after every fully connected layer, modifying its equation.Source: A. Multi-Layer Perceptrons · p. 13

B. Additional Implementation Details

The appendix describes baseline architectures and GraphMLP design variants used in the implementation and ablation studies. It highlights a parallel spatial GCN-and-MLP design and points to a transposition ablation.

B. Additional Implementation Details

  • The implementation defines FCN, GCN, Transformer, MLP-Mixer, Mesh Graphormer, Graph-Mixer, and Transformer-GCN comparison architectures.For example, Graph-Mixer replaces MLP-Mixer linear layers with GCN layers, while Transformer-GCN replaces GraphMLP spatial MLPs with self-attention.Source: B. Additional Implementation Details · p. 13
  • The selected GraphMLP design places spatial GCN and MLP blocks in parallel within SG-MLP, with the GCN processing tokens spatially.Source: B. Additional Implementation Details · p. 13
  • The appendix illustrates model-component variants and identifies a separate ablation study on transposition design options.The supplied table caption identifies the ablation's subject but does not report its results.Source: B. Additional Implementation Details · p. 13, Table 11 · p. 13

C. Additional Ablation Studies

The ablation compares transposition before versus after layer normalization (LN) in the spatial MLP. Transposition before LN performs better in both MLP-Mixer and GraphMLP.

C. Additional Ablation Studies

  • In the before-LN option, LN normalizes tokens along the spatial dimension before the spatial MLP and GCN operations.The formulation applies transposed inputs to LN, then transposes the spatial-MLP output back.Source: C. Additional Ablation Studies · pp. 13–14
  • The after-LN option applies LN before transposing the spatial-MLP input, while the GCN receives the normalized untransposed input.The two options change where transposition occurs relative to LN.Source: C. Additional Ablation Studies · p. 14
  • Transposition before LN brings more benefits in both MLP-Mixer and GraphMLP than transposition after LN.This differs from the original MLP-Mixer implementation, which transposes after LN.Source: C. Additional Ablation Studies · p. 14

D. Additional Qualitative Results

The qualitative results include reconstructions from Human3.6M, MPI-INF-3DHP, and challenging in-the-wild images. The authors report reliable and plausible poses in these challenging cases.

D. Additional Qualitative Results

Related papers

Loading 2206.06420v5…