PaperlayerView exact revision on arXiv

Source-linked AI summary

PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation

Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, Chen Chen

arXiv:2303.17472v1cs.CV

TL;DR

Long input sequences make transformer-based pose lifting computationally expensive, while noisy 2D detections reduce performance. PoseFormerV2 uses low-frequency representations with time-frequency feature fusion, and experiments show improved speed-accuracy trade-offs and robustness across two benchmarks.

  • Problem

    Transformer-based pose estimators are limited by the computational cost of long input sequences and sensitivity to noisy 2D joint detections.

  • Method

    PoseFormerV2 uses low-frequency coefficients from complete skeleton sequences, central-frame spatial encoding, and a Time-Frequency Feature Fusion module.

  • Results

    PoseFormerV2 significantly outperforms PoseFormerV1 and other transformer-based variants in speed-accuracy trade-off and robustness on Human3.6M and MPI-INF-3DHP.

  • Takeaways & Limitations

    A compact frequency representation can simultaneously enlarge the receptive field for long sequences and improve robustness to noisy joint detection.

  • Takeaways & Limitations

    The method is based on the scarcely investigated Discrete Cosine Transform and is presented as an initial attempt intended to inspire further research.

Abstract

from arXiv · show

Recently, transformer-based methods have gained significant success in sequential 2D-to-3D lifting human pose estimation. As a pioneering work, PoseFormer captures spatial relations of human joints in each video frame and human dynamics across frames with cascaded transformer layers and has achieved impressive performance. However, in real scenarios, the performance of PoseFormer and its follow-ups is limited by two factors: (a) The length of the input joint sequence; (b) The quality of 2D joint detection. Existing methods typically apply self-attention to all frames of the input sequence, causing a huge computational burden when the frame number is increased to obtain advanced estimation accuracy, and they are not robust to noise naturally brought by the limited capability of 2D joint detectors. In this paper, we propose PoseFormerV2, which exploits a compact representation of lengthy skeleton sequences in the frequency domain to efficiently scale up the receptive field and boost robustness to noisy 2D joint detection. With minimum modifications to PoseFormer, the proposed method effectively fuses features both in the time domain and frequency domain, enjoying a better speed-accuracy trade-off than its precursor. Extensive experiments on two benchmark datasets (i.e., Human3.6M and MPI-INF-3DHP) demonstrate that the proposed approach significantly outperforms the original PoseFormer and other transformer-based variants. Code is released at \url{https://github.com/QitaoZhao/PoseFormerV2}.

1. Introduction

Transformer-based 2D-to-3D lifting methods model spatial and temporal dependencies effectively, but long sequences make self-attention costly and noisy 2D detections reduce robustness. PoseFormerV2 addresses both issues by combining central-frame features with low-frequency sequence representations.

  • Background: 2D-to-3D lifting methods use lightweight 2D joint sequences to estimate 3D human pose from monocular videos.Lifting-based methods have become dominant because 2D skeletons are memory-friendly and support long temporal inputs.
  • PoseFormer: PoseFormer factorizes feature extraction into spatial joint modeling within frames and temporal human-motion modeling across frames using transformer encoders.The spatial encoder models inter-joint dependencies, while frame tokens are passed to the temporal encoder.
  • Challenges: 81-frame PoseFormer training costs about 1.5 hours per epoch on an RTX 3090, compared with about 5 minutes for its 3-frame version.Dense self-attention becomes increasingly expensive as sequence length grows; PoseFormer uses 81 frames, while later methods use even longer sequences.
  • Challenges: Replacing ground-truth 2D detections with CPN detections increases PoseFormer’s Human3.6M MPJPE from 31.3mm to 44.3mm.The change illustrates the effect of detector noise on lifting accuracy.
  • Research Questions: Prior efficiency and robustness techniques address the two challenges separately, and multi-hypothesis modeling can improve robustness while adding computation.The paper identifies a need for one solution that handles long-sequence efficiency and unreliable detections simultaneously.
  • PoseFormerV2: PoseFormerV2 encodes long skeleton sequences with low-frequency coefficients, restricting spatial processing to central frames while fusing time- and frequency-domain features.Low-frequency components represent global trajectory characteristics and filter high-frequency jitters and outliers.

2. Related Work

Related work spans transformer architectures for 3D pose estimation, efficiency methods exploiting temporal redundancy, and frequency-domain representations in vision and motion prediction. Frequency-domain representations had not yet been explored for 2D joint sequences in lifting-based 3D pose estimation.

  • Transformer-based 3D Human Pose Estimation: PoseFormer introduced vision transformers to lifting-based 3D pose estimation, while later methods refined joint-specific modeling, uncertainty handling, and self-supervised learning.Alternating spatial-temporal layers, multiple hypotheses, and masked joint modeling represent successive transformer-based directions.
  • Efficient Transformer Methods: StridedTransformer and related methods improve efficiency by exploiting temporal redundancy in 2D joint sequences and reducing computation around long inputs.StridedTransformer replaces parameter-heavy fully connected layers with strided convolutions, while another approach downsamples video frames before detection and lifting.
  • Frequency-domain Vision Methods: Image-compression and image-classification studies allocate or retain more informative low-frequency DCT components to reduce memory or computation.JPEG, JPEG 2000, and adaptive DCT-channel removal motivate frequency-domain efficiency techniques in vision.
  • Frequency-domain Skeleton Methods: Motion-prediction studies transform skeleton sequences into DCT coefficients and report that discarding some high-frequency coefficients can preserve performance while smoothing predicted motions.These studies encode human dynamics rather than static joint coordinates.

3. Method

PoseFormerV2 retains PoseFormer’s spatial-temporal design while replacing dense long-sequence processing with central-frame features and low-frequency DCT representations. It fuses time- and frequency-domain features to reduce computation and improve resistance to noisy 2D detections.

  • PoseFormerV1 limitations: PoseFormerV1 models within-frame joint dependencies spatially and cross-frame motion temporally, but dense self-attention becomes costly for long sequences and is sensitive to noisy detections.The temporal encoder uses one token per frame, while self-attention has quadratic computation growth with token number.
  • Spatial encoder: PoseFormerV2 samples only a few central frames for fine-grained spatial encoding instead of processing every frame in the full input sequence.For an input sequence such as F=81, the sampled frame count F′ is typically much smaller than F.
  • Frequency representation: The full skeleton sequence is transformed with DCT, retaining the first N low-frequency coefficients as a compact representation that preserves temporal structure while removing high-frequency noise.Low-frequency coefficients capture rough contours, whereas high-frequency components encode details such as jitters or sharp changes.
  • Feature fusion: Time-domain frame features and frequency-domain coefficient embeddings are concatenated and passed to a Time-Frequency Feature Fusion module.The frequency features are linearly projected, combined with frequency positional embeddings, and concatenated with time-domain features.
  • Feature fusion: The fusion transformer shares self-attention across domains but uses separate feed-forward networks, including FreqMLP for frequency-domain processing.These modifications narrow the gap between time- and frequency-domain features while retaining transformer-based temporal dependency modeling.
  • Regression head: The regression head follows PoseFormerV1, using a 1D convolution and linear projection to predict the central frame’s 3D pose, trained with MPJPE loss.The output pose representation has shape y ∈ R1×(J·3).

4. Experiments

Experiments evaluate PoseFormerV2 on Human3.6M and MPI-INF-3DHP, measuring efficiency, accuracy, noise robustness, ablations, and transfer to other transformer models. The method generally improves speed-accuracy trade-offs and robustness while using compact frequency-domain inputs.

  • Comparisons with State-of-the-art Methods: PoseFormerV2 shows a stronger speed-accuracy trade-off than other transformer-based methods, with larger receptive-field expansion ratios producing larger improvements.Figure 5 summarizes comparisons on Human3.6M in millimeters; methods with additional pre-training and computationally heavy MixSTE are excluded.
  • Comparisons with State-of-the-art Methods: 47.6mm MPJPE is achieved with 77.2 MFLOPs at an expanding ratio of 81, compared with MHFormer’s 47.8mm MPJPE at 342.9 MFLOPs.This corresponds to MHFormer being 4.4× slower.
  • Comparisons with State-of-the-art Methods: 46.0mm MPJPE is obtained at around 350 MFLOPs and sequence length 81, while StridedTransformer obtains 47.5mm MPJPE under the same budget.The reported improvement is 3.2%.
  • Comparisons with State-of-the-art Methods: As Gaussian-noise standard deviation increases, PoseFormerV2 experiences less performance degradation and a more stable trend than comparison methods, including MHFormer.The robustness experiment adds zero-mean Gaussian noise to ground-truth 2D detections with all methods using 27 frames and PoseFormerV2 configured with f = n = 3.
  • Comparisons with State-of-the-art Methods: PoseFormerV2 outperforms competing approaches on MPI-INF-3DHP, including P-STMO with masked joint pre-training.The experiment uses 9 central frames and the first 9 DCT coefficients from an 81-frame sequence.
  • Comparisons with State-of-the-art Methods: Under highly deviated 2D detections, PoseFormerV2 produces reliable 3D poses while using 0.35 GFLOPs, versus 3.12 GFLOPs for MHFormer and 1.36 GFLOPs for PoseFormerV1.All methods use 81-frame inputs in this qualitative comparison.
  • Ablation Study: Increasing the retained DCT coefficients reduces error more than increasing spatial-encoder frames, with 2.4 reduction for n = 3 versus 1.0 reduction for f = 3.Matched f = n = 3 with expanding ratio 9 provides a satisfactory speed-accuracy trade-off.
  • Generalization Ability: Low-frequency DCT coefficients improve robustness and can also be incorporated into MixSTE and MHFormer.The paper reports that the frequency-domain approach generalizes to these transformer-based temporal models.

5. Conclusion

PoseFormerV2 addresses efficient long-sequence processing and robustness to noisy joint detection through a compact frequency-domain representation. Experiments on Human3.6M and MPI-INF-3DHP report improvements over previous transformer-based methods.

  • 5. Conclusion: PoseFormerV2 uses compact frequency representations of long 2D joint sequences to enlarge receptive fields efficiently and improve robustness.The approach targets efficiency and noise robustness simultaneously.
  • 5. Conclusion: Experimental results on Human3.6M and MPI-INF-3DHP show that PoseFormerV2 outperforms previous transformer-based methods.The conclusion reports this outcome across both benchmarks.

A. Overview

The supplementary material organizes formal DCT background, datasets and metrics, implementation details, frequency-only baselines, generalization experiments, visual analyses, and broader impacts and limitations.

  • A. Overview: The supplementary material includes sections on DCT, datasets and evaluation metrics, implementation, frequency-only baselines, model generalization, visualizations, and broader impacts and limitations.These topics are listed as Sections B through H.

B. Discrete Cosine Transform

The paper applies the Discrete Cosine Transform independently to each joint-coordinate trajectory. Keeping only low-frequency coefficients gives a smoother, slightly lossy representation that preserves broad temporal structure while discarding high-frequency details.

  • DCT Representation: Each x- or y-coordinate trajectory of each joint is treated as a one-dimensional time series for DCT transformation.For an F × J × 2 sequence, the transform is applied independently to J × 2 trajectories.
  • DCT Coefficients: The DCT coefficient for trajectory ˆx_j is computed by summing its time samples against cosine basis functions indexed by i.The displayed expression is Equation (5).
  • DCT Representation: Low-frequency coefficients encode the rough contour of a trajectory, whereas high-frequency coefficients encode details such as jitters or sharp changes.Each time step corresponds to one DCT coefficient, indexed from 1 to F.
  • DCT Reconstruction: The original trajectory is reconstructed by summing DCT coefficients against the corresponding cosine basis functions over coefficient indices.The displayed reconstruction expression is Equation (6).
  • DCT Representation: DCT is lossless when all coefficients are retained, while using a few low-frequency coefficients reconstructs the sequence approximately and more smoothly.The omitted high-frequency coefficients are set to zero during recovery.

C. Datasets and Evaluation Metrics

The paper evaluates on Human3.6M and MPI-INF-3DHP, using standard pose-estimation metrics tailored to each benchmark.

  • Datasets: Human3.6M contains over 3.6 million indoor video frames from 4 cameras, with 11 subjects performing 15 actions.The model trains on subjects S1, S5, S6, S7, S8 and tests on S9 and S11.
  • Datasets: MPI-INF-3DHP includes controlled indoor and challenging outdoor environments, with varied subjects, actions, and camera views.
  • Evaluation Metrics: Human3.6M evaluation reports MPJPE and P-MPJPE, with P-MPJPE applying rigid alignment before computing pose distance.
  • Evaluation Metrics: MPI-INF-3DHP evaluation reports MPJPE, PCK within 150mm, and AUC.

D. More Implementation Details

Implementation retains PoseFormerV1 settings while reformulating its temporal encoder for time-frequency feature fusion and combining temporal and frequency-domain information.

  • Model Setup: PoseFormerV2 adopts PoseFormerV1’s optimal model-architecture hyperparameters to provide fair comparisons, while acknowledging further tuning may help.
  • Model Setup: The spatial transformer uses feature dimension c=32 and four layers each for the spatial and feature-fusion transformers, with positional embeddings inherited from PoseFormerV1.
  • Training Details: Experiments use AdamW for 80 epochs, weight decay 0.1, initial learning rate 8e-4, decay factor 0.99, and horizontal-flip augmentation.CPN detections are used on Human3.6M, while ground-truth 2D detections are used on MPI-INF-3DHP.
  • Architecture: The PoseFormerV1 temporal encoder is reformulated as a Time-Frequency Feature Fusion module.
  • Architecture: The method combines time-domain features with low-frequency DCT representations rather than relying exclusively on frequency-domain features.Time-domain encoding preserves fine-grained central-frame motion, while low-frequency coefficients suppress high-frequency noise but may smooth informative details.

F. Generalization to More Models

The frequency-domain approach extends to MixSTE and MHFormer with limited architectural changes, improving efficiency and robustness across these transformer-based models.

  • Generalization: The proposed frequency-domain modifications generalize to MixSTE and MHFormer because both apply self-attention across all frames.
  • MixSTE: The MixSTE comparison evaluates expanded receptive fields, robustness to Gaussian noise, and computational cost using marker size.
  • MixSTE: MixSTE combines central-frame time-domain features with low-frequency coefficients from the complete sequence in its temporal encoders.
  • MHFormer: MHFormer’s improved version is evaluated for noise robustness, with marker size indicating computational cost.
  • MHFormer: The method improves robustness even for MHFormer, which already models uncertainty in noisy joint detection, demonstrating compatibility with other approaches.
  • Scope: The approach has so far been generalized to two transformer-based methods, while extension to CNN- and GNN-based architectures remains future work.

G. Visualizations and Analysis

Qualitative examples and analysis attribute PoseFormerV2’s robustness to low-frequency DCT representations that provide global sequence context and reduce the influence of frame-level noise.

  • Visualizations: In-the-wild examples cover occlusions, missed joints, and switched joints detected by HRNet.
  • Analysis: Low-frequency DCT coefficients provide a global view of the sequence, causing noise in individual frames to be dwarfed.
  • Broader Impacts: The broader-impact discussion frames the work around processing long sequences efficiently and handling noisy 2D detections in real-world 3D HPE.
  • Visualizations: PoseFormerV2 still infers correct poses despite imperfect 2D joint inputs in the reported challenging cases.
  • Broader Impacts: DCT is presented as a relatively underexplored signal-processing technique whose combination with deep learning may motivate further research.
  • Limitations: The method’s sampled-frame and retained-coefficient counts are selected through experiments or human experience rather than learned automatically.
Loading 2303.17472v1…