PaperlayerView exact revision on arXiv

Source-linked AI summary

PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling

Fabien Baradel, Romain Brégier, Thibault Groueix, Philippe Weinzaepfel, Yannis Kalantidis, Grégory Rogez

arXiv:2208.10211v2cs.CV

TL;DR

In-the-wild 3D annotations are difficult and expensive to obtain, and video pose methods have used limited weak or pseudo-3D labels. PoseBERT learns temporal pose modeling from MoCap via masked modeling and can be added to image-based estimators; experiments report improved performance across body and hand modeling.

  • Problem

    In-the-wild 3D pose annotations are cumbersome to collect, while video pose methods rely on weak or pseudo-3D labels from small datasets.

    Source: §1 · pp. 1–2

  • Method

    PoseBERT is a transformer trained on MoCap pose sequences via masked modeling that can be plugged on top of image-based pose estimators.

    Source: §1 · p. 2, §6 · p. 14

  • Results

    Experiments show that PoseBERT improves the performance of image-based models by leveraging temporal context for body and hand 3D modeling.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    The task-independent module can be applied without fine-tuning to pose refinement, future prediction, and motion completion.

    Source: §1 · p. 2

    Fast motion can be over-smoothed, and long occlusions can be worsened when neighboring image-based predictions provide misleading temporal context.

    Source: §5 · pp. 13–14

Abstract

from arXiv · show

Training state-of-the-art models for human pose estimation in videos requires datasets with annotations that are really hard and expensive to obtain. Although transformers have been recently utilized for body pose sequence modeling, related methods rely on pseudo-ground truth to augment the currently limited training data available for learning such models. In this paper, we introduce PoseBERT, a transformer module that is fully trained on 3D Motion Capture (MoCap) data via masked modeling. It is simple, generic and versatile, as it can be plugged on top of any image-based model to transform it in a video-based model leveraging temporal information. We showcase variants of PoseBERT with different inputs varying from 3D skeleton keypoints to rotations of a 3D parametric model for either the full body (SMPL) or just the hands (MANO). Since PoseBERT training is task agnostic, the model can be applied to several tasks such as pose refinement, future pose prediction or motion completion without finetuning. Our experimental results validate that adding PoseBERT on top of various state-of-the-art pose estimation methods consistently improves their performances, while its low computational cost allows us to use it in a real-time demo for smoothly animating a robotic hand via a webcam. Test code and models are available at https://github.com/naver/posebert.

1 INTRODUCTION

In-the-wild 3D annotations are difficult to collect, and video-based pose modeling has relied on limited weak or pseudo-3D labels. PoseBERT uses MoCap-trained masked modeling to provide a task-independent temporal module for body and hand pose, with reported gains across refinement and future prediction tasks.

1.1 Contributions

  • PoseBERT learns pose sequences through masked modeling on MoCap data, without requiring RGB images or frame-pose annotations for training.The contribution describes MoCap as relatively easy to acquire and notes that large-scale datasets are already available.Source: §1.1 · p. 2
  • PoseBERT supports human body and hand modeling and can be plugged on top of different image-based pose estimation methods.The introduction describes variants for SMPL body models and MANO hand models.Source: §1.1 · p. 2, §1 · p. 2
  • The task-independent module works out of the box for denoising, missing-pose recovery, 3D refinement, motion completion, and future prediction, regardless of the frame-based pose extractor.It handles frames with missing predictions and is evaluated with inputs ranging from 3D skeleton keypoints to SMPL and MANO model rotations.Source: §1.1 · p. 2, §1 · p. 2
  • PoseBERT improves pose refinement by 1.0 to 10.3 points in PA-MPJPE and future prediction by 10% to 50% versus strong baselines across 5 to 30-frame horizons.The paper also reports plausible future pose prediction up to 1 second ahead.Source: §1.1 · p. 2
  • A forward pass takes approximately 5 ms at 30 fps, adding 10% FLOP overhead; the authors use the model for real-time webcam-based robotic hand teleoperation.The real-world application transfers hand poses estimated from a monocular RGB webcam stream to a robotic hand gripper.Source: §1.1 · p. 2

2 RELATED WORK

Related work uses temporal information to improve video-based pose estimates, generates pose sequences under varied conditioning, and exploits MoCap data through several different strategies. PoseBERT applies a transformer directly to image-based pose predictions and trains it on MoCap data.

3D human pose estimation in videos with parametric models

  • Video pose methods use per-frame estimates or features with temporal models: HMMR uses convolution, VIBE, TCMR, and MEVA use GRUs, and Pavlakos et al. use a transformer on features,,,, [45].PoseBERT instead applies its transformer directly to image-based pose predictions, allowing training on MoCap data and use on top of different image-based methods.Source: §2 · p. 3

Pose sequence generation

  • Prior pose-sequence generation methods condition on sequence beginnings, action labels, music, or scene context, while PoseBERT can predict near-future poses or complete missing frames without conditioning,,,.The authors state that its deterministic output cannot handle long-term future prediction.Source: §2 · p. 3

Use of MoCap data

  • MoCap has supported synthetic image generation, but domain shift has often limited such data to pretraining or finetuning; other methods use proxy appearance or motion representations,,,,.Some MoCap-based methods generate pseudo-ground-truth by matching 2D annotations to reprojected MoCap data or lifting 2D poses,.Source: §2 · p. 3
  • MoCap-trained pose discriminators encourage realistic predictions but do not improve pose diversity, while an optimization-based motion-prior approach is described as very slow,,.The authors also train PoseBERT on MoCap and describe it as extremely fast and real-time.Source: §2 · p. 3

3 POSEBERT

PoseBERT is a temporal transformer module that turns noisy or incomplete pose sequences into coherent mesh sequences. It uses masked modeling and denoising and supports different body and hand pose representations.

3.1 Overview

  • PoseBERT takes pose sequences from frame-based estimators and outputs temporally coherent mesh sequences, inferring missing predictions when needed.Inputs can use different high-dimensional pose representations.Source: §3.1 · pp. 3–4

3.2 Architecture of the PoseBERT module

  • L transformer blocks contextualize pose embeddings, while a shared regressor iteratively updates pose estimates initialized at the model mean.The regressor is unrolled across layers rather than iterated within each layer.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3 Learning the parameters of PoseBERT

  • Training masks random timesteps and corrupts poses with Gaussian noise or same-batch poses; reconstruction against clean sequences teaches missing-pose recovery and denoising.Per-block masking was important for robust, smooth sequences and encourages interpolation rather than copying the last observed pose.Source: §3.3 · p. 5

3.4.1 Input representations

  • Inputs include 6D parametric-model rotations or normalized, aligned 3D keypoints, optionally joined with normalized 2D poses to cue scene location.The explored parametric models are SMPL for bodies and MANO for hands; predicted pose parameters are mapped to meshes.Source: §3.4.1 · pp. 5–6

4 EXPERIMENTS

The experiments cover SMPL body and MANO hand modeling, multiple pose tasks, architecture and training ablations, and a robotic teleoperation application.

4 EXPERIMENTS

  • Experiments cover SMPL body modeling with or without ground-truth person boxes, MANO hand modeling, several pose tasks, and robotic teleoperation.The study also includes transformer architecture and training-strategy ablations.Source: §4 · p. 6
  • PoseBERT was trained from scratch with Adam at a 1e-3 learning rate for 2 million iterations on one V100 GPU, taking 2 days.The framework was implemented in PyTorch.Source: §4 · p. 6
  • The authors report no training saturation with a sufficiently large network, strong input perturbations, and data augmentation.Source: §4 · p. 6

4.1 PoseBERT for human body mesh modeling

PoseBERT is evaluated on body-mesh modeling using MoCap-trained sequences and standard video datasets, then added to image-based pose estimators. Across four video datasets, adding it to SPIN consistently improves PA-MPJPE.

4.1.1 Datasets, training and metrics

  • PoseBERT is trained on 11,000 AMASS sequences, downsampled to 30 fps, and evaluated on four video datasets using pose-error and temporal-smoothness metrics.The evaluation includes MPJPE, PA-MPJPE, acceleration error, and, where applicable, MPVPE and PCK3D variants.Source: §4.1.1 · p. 6

4.1.2 Impact on image-based models

  • Because it takes pose sequences rather than image features, PoseBERT can be plugged on top of image-based models such as SPIN to make them video-based.Unlike VIBE and TCMR, it does not require image-based features as input.Source: §4.1.2 · p. 6
  • In a MuPoTS sequence with several occluded frames, PoseBERT produces plausible human meshes when the image-based model misses the person.The qualitative example uses no 2D bounding boxes as input.Source: Figure 4 · p. 7
  • Adding PoseBERT to SPIN improves PA-MPJPE by 2.3 mm on 3DPW, 3.7 mm on MPI-INF-3DHP, 2.1 mm on MuPoTS-3D, and 1.6 mm on AIST.These results also outperform VIBE on MPI-INF-3DHP, MuPoTS-3D, and AIST.Source: §4.1.2 · p. 6, Table 1 · p. 7
  • Adding PoseBERT also consistently improves results with MoCap-SPIN and with the video-based method VIBE, indicating gains beyond the SPIN image-based baseline.The authors describe the VIBE improvement as complementary to its use of temporal feature consistency.Source: §4.1.2 · p. 7

4.1.3 Impact of masking and noise perturbation for training

The study varies masking and noise during MoCap training and tests other training strategies. Partial masking and modest Gaussian noise yield smoother or slightly better predictions, while longer temporal spans do not help consistently.

4.1.3 Impact of masking and noise perturbation for training

  • Masking 12.5% of the input poses—two frames out of 16—produces smoother predictions while keeping PA-MPJPE low, and Gaussian noise with standard deviation 0.05 adds a small performance boost and smoother predictions.Increasing the noise standard deviation brings no benefit.Source: §4.1.3 · p. 7

4.1.4 Additional ablation study of the training strategy

  • PoseBERT gains 0.4–1.4 mm over the baseline when compared with using an iterative regressor on top of the temporal module.The passage reports the range without assigning values to individual datasets or metrics.Source: §4.1.4 · p. 7
  • Increasing the temporal span by lowering the frame rate does not significantly improve results and can decrease performance.The sequence length is held fixed in this comparison.Source: §4.1.4 · p. 7
  • Random-pose and random-joint noise each bring a small improvement over Gaussian noise, but are omitted from the best model's training scheme for simplicity.The study also considers these alternatives alongside masking and Gaussian noise.Source: §4.1.4 · p. 7

4.1.5 Impact of hyperparameters

The hyperparameter study examines architecture choices, sequence length, and computational cost. Its selected model uses positional encoding, a shared regressor, and a modest transformer configuration.

4.1.5 Impact of hyperparameters

  • PoseBERT provides a consistent 1–3 mm gain across the three datasets in the architecture ablation, while removing positional encoding leads to suboptimal performance.The authors identify incorporating temporal information within the network as a key design choice.Source: §4.1.5 · p. 8
  • Sharing the regressor reduces learnable parameters and improves predictions, while iterating it more than once after each layer does not improve performance and can slightly decrease it.The ablation uses 12.5% masked input poses.Source: §4.1.5 · p. 8
  • The selected configuration uses depth L = 4, embedding dimension Dt = 512, and T = 16 frames; it adds 7.3M parameters and 0.13 GFLOP per sequence.Increasing model complexity or training sequence length yields minimal or no further improvement; at T = 64, overhead is 0.53 GFLOP.Source: §4.1.5 · p. 8

4.1.6 Beyond the use of ground-truth 2D person bounding boxes.

The experiments test PoseBERT without ground-truth person boxes and with off-the-shelf detections. It improves pose estimates in these settings, including when detections are missing or noisy.

4.1.6 Beyond the use of ground-truth 2D person bounding boxes.

  • For the no-ground-truth-box experiment, LCR-Net++ detects and estimates poses from full RGB video, assigns detections across frames, and finds poses for 94% of annotated people in MuPoTS.PoseBERT is trained on 128-frame sequences for this noisier, more incomplete input setting.Source: §4.1.6 · p. 9
  • Compared with LCR-Net++ alone, PoseBERT improves MPJPE and PA-MPJPE by more than 20 mm; versus Savitzky–Golay filtering, it gains 11.92 mm on MPJPE and 3.97 on PA-MPJPE.The smoothing baseline only marginally improves results and does not correct implausible initial poses.Source: §4.1.6 · p. 9
  • Qualitative examples show plausible body meshes through several occluded frames and plausible hand-pose recovery from a motion-blurred first-frame estimate.The examples use LCR-Net++ for body poses and LCR-Net Hand expert for hand poses.Source: §4.1.6 · p. 9

4.1.7 Impact of detection quality

  • With ground-truth boxes, PoseBERT improves all reported metrics on 3DPW and MuPoTS-3D; the matched-capacity biGRU baseline performs worse than the image-based model.The authors suggest recurrent networks may not generalize when test-time masking differs from training masking.Source: §4.1.7 · p. 9
  • With Faster-RCNN detections, PoseBERT improves MPJPE by 21.5 points on 3DPW and 40.5 points on MuPoTS-3D, bringing performance close to the ground-truth-box setting.Only 72.3% of 3DPW people and 75% of MuPoTS-3D people meet the study's detection criterion.Source: §4.1.7 · p. 9

4.1.8 Impact of the temporal context for missing timesteps

Masking tests show PoseBERT uses surrounding temporal context to estimate missing poses, even when multiple neighboring frames are unavailable. Its framewise benefit is less consistent when the image-based estimate is already good.

4.1.8 Impact of the temporal context for missing timesteps

  • Masking only the target timestep raises MPJPE from 76.6 to 76.9 on 3DPW and from 71.4 to 71.6 on MuPots-3D.The authors interpret the marginal drop as evidence that PoseBERT uses surrounding timesteps.Source: §4.1.8 · p. 10
  • Even with 10 frames masked around a target timestep, PoseBERT outperforms the image-based model, showing it can use substantial temporal context to fill missing detections.Increasing the masking duration causes a larger performance drop, but PoseBERT still outperforms the image-based model.Source: §4.1.8 · p. 10

4.1.9 Finer analysis of PoseBERT improvements

  • PoseBERT beats HybrIK on 47.3% and 46.7% of frames with ground-truth boxes on 3DPW and MuPots-3D, rising to 52.4% and 50.2% with Faster-RCNN boxes.It can recover from large image-based errors, but sometimes slightly worsens already-good estimates.Source: §4.1.9 · p. 10

4.2 PoseBERT hand mesh modeling

The hand-modeling evaluation tests PoseBERT on DexYCB sequences, adding it to image-based hand estimators and assessing training strategies, robustness to missing detections, future prediction, and robotic teleoperation. Across these settings, PoseBERT improves hand estimates and supports real-time use.

4.2.1 Datasets, training and metrics

  • The DexYCB benchmark uses 1,000 grasping sequences averaging 60 frames, with the s0 split and MANO fits for training supervision.Metrics are computed from the MANO model's 21 keypoints.Source: §4.2.1 · p. 10

4.2.2 Impact on top of image-based models

  • LCR-Net Hand expert + PoseBERT reduced MPJPE by 36.92% and PA-MPJPE by 57%; HR-Net + PoseBERT lowered MPJPE by 3.29 mm and PA-MPJPE by 2.79 mm.For HR-Net, Accel fell from 12.77 to 3.62; LCR-Net Hand expert detects hands in 81.5% of images, leaving an average 18.5% of poses missing per video.Source: §4.2.2 · p. 10, Table 7 · p. 12

4.2.3 Ablation study for the training strategy

  • Longer input sequences improve all metrics; stronger training-time masking, sufficiently large Gaussian noise, and random-pose replacement each improve inference, with the perturbations complementary.Noise and random-pose replacement particularly affect Accel, the smoothness metric.Source: §4.2.3 · pp. 11–12

4.2.4 Robustness to miss detections

  • With HR-Net, PoseBERT's relative MPJPE gain over the image-based model ranges from 20% with no dropped frames to 80% with 90% missing poses, and exceeds the smoothing baseline throughout.With LCR-Net Hand expert, PoseBERT also beats the smoothing baseline by a larger margin, consistent with its noisier, out-of-dataset inputs.Source: §4.2.4 · p. 12

4.2.5 Future frame prediction

  • PoseBERT outperforms No-velocity and Velocity propagation at all tested future horizons and across image-based models, except LCR-Net Hand expert at Δt = 5.Constant-velocity prediction performs well at Δt = 5 but quickly deteriorates; PoseBERT is repurposed for future prediction despite not being trained for it.Source: §4.2.5 · pp. 12–13

4.2.6 Application example: robotic teleoperation

  • In the robotic-gripper application, PoseBERT smooths target commands and produced qualitatively reasonable predictions up to 15 frames ahead, making the system more reactive.The setup processes webcam detections at 30Hz and retargets predicted MANO poses to an Allegro hand.Source: §4.2.6 · p. 13

5 LIMITATIONS OF POSEBERT

PoseBERT can degrade initial estimates, particularly when fast motion is oversmoothed or long occlusions leave misleading temporal context. Its deterministic outputs and evaluation metrics also limit uncertainty representation and assessment of image-space accuracy.

5 LIMITATIONS OF POSEBERT

  • Fast motion can be oversmoothed, while poor neighboring estimates during long occlusions can provide misleading context and degrade PoseBERT's output.The authors suggest more diverse motion speeds, confidence information, or a more representative noise model as possible mitigations.Source: §5 · pp. 13–14
  • PoseBERT returns one deterministic pose per timestep, so it does not represent monocular pose uncertainty; predictions more than a few seconds ahead may be highly multimodal.The paper identifies pose distributions or variational approaches as possible directions.Source: §5 · p. 14
  • MPJPE-style 3D metrics do not measure image reprojection, and PoseBERT can produce inaccurate 2D joints despite good 3D results.The authors identify methods effective for both 3D and 2D pose estimation as future work.Source: §5 · p. 14

6 CONCLUSION

PoseBERT uses masked modeling on MoCap data to regress parametric 3D-model parameters from noisy or incomplete image-based estimates. The paper reports that it can be applied to body and hand models and plugged onto image-based models to use temporal context and improve performance.

6 CONCLUSION

  • PoseBERT is a transformer module for video-based human and hand 3D mesh recovery, trained purely on MoCap data through masked modeling.Source: §6 · p. 14
  • The module regresses parametric 3D-model parameters from noisy and/or incomplete image-based estimates.Source: §6 · p. 14
  • Experiments show PoseBERT can be trained for body or hand 3D models and plugged onto image-based models to use temporal context and improve performance.Source: §6 · p. 14

Related papers

Loading 2208.10211v2…