Source-linked AI summary
Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation
Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, Pichao Wang, Wenming Yang
TL;DR
Video-based 3D human pose estimation must resolve depth ambiguity while exploiting redundant 2D pose sequences to generate a target 3D pose. The Strided Transformer uses VTE, STE, and full-to-single supervision to aggregate sequence information, enforce smoothness, and achieve state-of-the-art results with fewer parameters on two benchmarks.
Problem
Lifting 2D poses to 3D remains ill-posed because multiple 3D interpretations can project to the same 2D pose, while redundant sequences are not fully exploited for learning a single-pose representation.
Method
The Strided Transformer combines a Vanilla Transformer Encoder for long-range modeling, a Strided Transformer Encoder using strided convolutions for hierarchical aggregation, and full-to-single supervision at sequence and target-frame scales.
Results
The method achieves state-of-the-art results with fewer parameters on Human3.6M and HumanEva-I.
Takeaways & Limitations
Strided convolutions produce a single-vector pose representation while reducing computation cost, and full-to-single supervision enhances temporal smoothness and refines the target-frame representation.
Takeaways & Limitations
The method still has greater computational complexity and runtime cost than temporal convolutional networks, and the paper targets improving only the Transformer feed-forward network.
Abstract
from arXiv · showhide
Despite the great progress in 3D human pose estimation from videos, it is still an open problem to take full advantage of a redundant 2D pose sequence to learn representative representations for generating one 3D pose. To this end, we propose an improved Transformer-based architecture, called Strided Transformer, which simply and effectively lifts a long sequence of 2D joint locations to a single 3D pose. Specifically, a Vanilla Transformer Encoder (VTE) is adopted to model long-range dependencies of 2D pose sequences. To reduce the redundancy of the sequence, fully-connected layers in the feed-forward network of VTE are replaced with strided convolutions to progressively shrink the sequence length and aggregate information from local contexts. The modified VTE is termed as Strided Transformer Encoder (STE), which is built upon the outputs of VTE. STE not only effectively aggregates long-range information to a single-vector representation in a hierarchical global and local fashion, but also significantly reduces the computation cost. Furthermore, a full-to-single supervision scheme is designed at both full sequence and single target frame scales applied to the outputs of VTE and STE, respectively. This scheme imposes extra temporal smoothness constraints in conjunction with the single target frame supervision and hence helps produce smoother and more accurate 3D poses. The proposed Strided Transformer is evaluated on two challenging benchmark datasets, Human3.6M and HumanEva-I, and achieves state-of-the-art results with fewer parameters. Code and models are available at \url{https://github.com/Vegetebird/StridedTransformer-Pose3D}.
I. INTRODUCTION
Video-based 3D pose lifting must resolve depth ambiguity while exploiting redundant temporal context without losing long-range dependencies. Strided Transformer addresses this with hierarchical sequence aggregation and full-to-single supervision, achieving strong benchmark performance with fewer parameters.
- 2D-to-3D lifting remains ill-posed because multiple 3D poses can project to the same 2D pose.
- Existing temporal methods use past and future frames but often have limited receptive fields and struggle with long-range dependencies.
- Vanilla Transformer encoders model global dependencies, but redundant full-length sequences increase computation and memory costs.
- Strided Transformer replaces FFN layers with STE to reduce temporal redundancy and aggregate long-range information into a single pose representation hierarchically.
- Full-to-single supervision combines full-sequence temporal smoothness constraints with target-frame refinement to produce smoother and more accurate 3D poses.
- Experiments on Human3.6M and HumanEva-I report state-of-the-art performance with fewer parameters.
II. RELATED WORK
Prior work includes one-stage image-to-3D methods, two-stage 2D-to-3D lifting, temporal models, and visual Transformers. This paper uses a Transformer for 2D-to-3D lifting and adds strided convolutions to compress redundant video pose sequences.
- One-stage RGB-to-3D methods use sophisticated architectures with high computation costs that are impractical for realistic applications.
- Two-stage methods detect 2D keypoints before lifting them to 3D and have outperformed one-stage approaches in recent work.
- Video pose methods exploit temporal information through LSTMs, dilated temporal convolutions, graph models, and refinement networks.
- Unlike image-focused Transformers such as DETR, ViT, and METRO, this work maps 2D keypoints to 3D poses and incorporates strided convolutions for temporal redundancy.
- The proposed framework applies VTE to model long-range information and STE to aggregate it into one target-pose representation under full-to-single supervision.
- STE progressively compresses hidden-state sequences while modeling global and local information through multi-head self-attention and convolutional feed-forward networks.
2) Convolutional feed-forward network:
The paper replaces the Vanilla Transformer Encoder’s fully connected feed-forward layers with strided convolutions to reduce redundant temporal representations while preserving global and local context modeling.
- Motivation: The VTE maintains full-length hidden sequences, creating redundancy because nearby video poses are similar and increasing computation cost.The method therefore selectively aggregates useful information from the entire pose sequence.
- Convolutional feed-forward network: Strided convolutions replace the VTE feed-forward network’s fully connected layers and define the Strided Transformer Encoder.The modified encoder uses convolutional feed-forward networks within the Transformer architecture.
- Hierarchical aggregation: Self-attention models global context, while strided convolution captures local contexts by progressively merging nearby poses into shorter representations.This forms a hierarchical global-and-local architecture.
- Residual matching: Max pooling is applied to residuals so their temporal dimensions match those produced by strided convolutions.The architecture defines kernel size, strided factor, feature dimension, and hidden-unit count as its principal hyperparameters.
C. Network Architecture
The network embeds a sequence of 2D joint poses, processes the resulting tokens with a Vanilla Transformer Encoder, and supplies its output to the subsequent architecture for 3D pose estimation.
- Network components: The proposed network comprises pose embedding, Vanilla Transformer Encoder, Strided Transformer Encoder, and regression-head components.These components together map a sequence of 2D poses toward 3D pose predictions.
- Pose embedding: Pose embedding concatenates each frame’s J joint coordinates into tokens and maps them to dm-dimensional features using a 1D convolutional layer.Batch normalization, dropout, and ReLU follow the convolutional embedding.
- Vanilla Transformer Encoder: A learnable position embedding E1 is added before the first VTE layer to encode temporal token positions.The VTE contains N1 layers operating on the embedded sequence.
- Illustration: The architecture is illustrated with a 27-frame receptive field and J = 17 joints, while tensor sizes track frames and channels through the network.The figure depicts convolutional feed-forward networks and residual max pooling for temporal-shape matching.
3) Strided Transformer Encoder:
The STE is built on VTE outputs and uses strided layers with position embeddings to reduce sequence length, produce a single representation, and support target-frame 3D pose regression.
- STE construction: The STE takes the VTE output ZN1 as input and applies N2 STE layers with position embeddings adapted to the changing sequence lengths.The strided factor S determines the position-embedding sequence-length scale.
- Regression: The STE output has a single temporal position and is followed by batch normalization and a 1D convolutional regression layer.The corresponding VTE output is also passed through a regression head for sequence prediction.
- Outputs: The network produces both a 3D pose sequence ˜X and target-frame joint locations X.˜X has shape T × J × 3, whereas X has shape J × 3.
D. Full-to-Single Prediction
Full-to-single supervision trains sequence-scale and target-frame predictions together, adding temporal consistency while refining the final single-frame estimate. The paper also analyzes the resulting computational compression.
- Supervision scheme: The full-to-single scheme supervises outputs at both full-sequence and single-target-frame scales rather than relying on a single component.It is motivated by iterative refinement for producing more accurate estimations.
- Full sequence scale: The sequence loss Lf supervises VTE-derived 3D pose sequences against ground-truth sequences to enforce temporal consistency.The loss is intended to impose extra temporal smoothness constraints during training.
- Single target frame scale: The single-frame loss Ls supervises STE-derived target-frame poses by minimizing their distance from ground-truth joint annotations.This loss operates at the single target frame scale.
- Training: The entire network is trained end-to-end with supervision at both full-sequence and single-target-frame scales.The paper describes a total loss combining these supervision stages.
- Computational analysis: 1.35 is the compression ratio reported for the 27-frame Strided Transformer configuration.The configuration uses N1 = N2 = 3, strided factor S = 3, and kernel size K = 3.
A. Datasets and Evaluation
The method is evaluated on Human3.6M and HumanEva-I using standard pose-error, velocity-error, complexity, and speed measures. It achieves strong accuracy, smoothness, and parameter efficiency across the reported benchmarks.
- Datasets: Human3.6M uses 3.6 million images from four synchronized cameras, with five subjects for training and two for evaluation.HumanEva-I is smaller and uses all three subjects and all listed actions for training.
- Evaluation metrics: MPJPE measures average joint-position distance, P-MPJPE additionally aligns poses in translation, rotation, and scale, and MPJVE measures first-derivative pose error.MPJVE is used to assess sequence smoothness.
- Human3.6M results: 43.7 mm on protocol #1 and 35.2 mm on protocol #2 were achieved on Human3.6M, outperforming state-of-the-art methods under all reported metrics.The reported model uses a 351-frame receptive field with CPN input.
- Efficiency: The model remains lightweight as receptive fields increase, uses fewer total parameters than compared temporal convolutional networks, and retains acceptable real-time FPS.Table II reports computational complexity, MPJPE, and FPS on a single GeForce GTX 2080 Ti GPU.
- Additional evaluation: 28.5 mm MPJPE is reported with ground-truth 2D inputs on Human3.6M, while HumanEva-I results are promising across all tested actions.The ground-truth-input result is described as the best among compared methods.
- Temporal smoothness: 15.4% lower MPJVE, from 2.6 mm to 2.2 mm, indicates smoother predictions under full-sequence supervision.The comparison is reported on Human3.6M against other state-of-the-art models.
D. Ablation Studies
The ablations examine sequence length, detector inputs, hyperparameters, prediction supervision, and architectural components. They show benefits from longer temporal context, gradual temporal reduction, both encoder modules, and full-to-single supervision.
- Input sequence length: Using more input frames generally improves MPJPE for both CPN and ground-truth 2D inputs, although the error saturates beyond a certain sequence length.The study evaluates sequence lengths from 1 to 351 frames.
- Model hyperparameters: The N1 = 3, N2 = 3 configuration achieves 46.9 mm MPJPE and 0.128G FLOPs versus 47.9 mm and 0.17G for N1 = 6.Both configurations use dm = 256 and df = 512.
- Strided factor: A small strided factor and gradual temporal dimensionality reduction improve performance compared with alternative strided-factor settings.The receptive-field ablation uses T = 3 × 3 × 3 = 27.
- Prediction scheme: The full-to-single prediction scheme performs best, whereas schemes using only one prediction manner or matching supervision scales degrade performance.Full-to-single combines full-sequence and single-target-frame supervision.
- Model components: Selecting intermediate VTE predictions increases MPJPE from 46.9 mm to 48.1 mm, while replacing STE with pooling increases error by 0.4 mm.These comparisons support intermediate supervision and the proposed STE design.
- Model components: Removing VTE increases MPJPE by 1.1 mm, while removing STE raises MPJPE to 47.6 mm, supporting complementary module roles.VTE primarily models long-range information, whereas STE hierarchically aggregates global and local information.
E. Qualitative Results
The method’s attention maps show frame-dependent temporal weighting, while qualitative comparisons report plausible 3D predictions on challenging videos and identify failure under large 2D detection errors.
- Attention visualization: Attention maps assign different temporal weights across frames, with VTE emphasizing nearby input frames and STE emphasizing the center frame.Brighter regions indicate higher attention scores, and sparse heads reflect different temporal patterns or semantics.
- Attention visualization: Using past and future data improves target-frame representation by supporting inference from multiple temporal patterns.
- 3D reconstruction visualization: On challenging in-the-wild videos, the proposed method produces realistic and structurally plausible 3D predictions that outperform previous works.The comparison uses the same cascaded pyramid network 2D detector for all models.
- 3D reconstruction visualization: The method is reported as robust to partial occlusions and tolerant to depth ambiguity, but a large 2D detection error causes failure.
- Method summary: The architecture aggregates long-range information hierarchically into a single-pose representation and reduces Transformer computation cost.Its full-to-single supervision scheme further refines the target-frame representation and enhances temporal smoothness.
- Limitations: The method remains computationally more expensive than temporal convolutional networks despite reducing Transformer computation cost.
VI. APPENDIX
The appendix presents visual results on Human3.6M and HumanEva-I, plus qualitative results on challenging wild videos indexed by input-video frame number.
- Visual results: Figure 10 shows visual results on Human3.6M in its first three rows and HumanEva-I in its last two rows.
- Wild-video results: Figure 11 presents qualitative results on challenging wild videos, with numbers indicating input-video frame indices.