Source-linked AI summary
3D human pose estimation in video with temporal convolutions and semi-supervised training
Dario Pavllo, Christoph Feichtenhofer, David Grangier, Michael Auli
TL;DR
3D video pose estimation is ambiguous when 2D keypoints are lifted to 3D, especially when labeled data is scarce. The paper uses dilated temporal convolutions over 2D keypoints and back-projection from unlabeled video, outperforming prior methods in supervised and semi-supervised settings. Its semi-supervised method requires only camera intrinsic parameters.
Problem
Lifting 2D keypoints to 3D is inherently ambiguous, while collecting labeled 3D poses requires expensive motion-capture setups and lengthy recording sessions.
Method
The paper combines a fully convolutional temporal model with dilated convolutions and a back-projection loss that reconstructs unlabeled 2D keypoints from predicted 3D poses.
Results
6 mm lower average joint error, an 11% relative reduction, was achieved on Human3.6M versus the previous best result, with improvements also on HumanEva-I and in semi-supervised settings.
Takeaways & Limitations
The approach provides an efficient alternative to RNN-based models and uses unlabeled video when labeled 3D training data is scarce.
Takeaways & Limitations
The semi-supervised method requires camera intrinsic parameters, and the reported camera-space evaluation generally excludes global trajectory except in the semi-supervised setting.
Abstract
from arXiv · showhide
In this work, we demonstrate that 3D poses in video can be effectively estimated with a fully convolutional model based on dilated temporal convolutions over 2D keypoints. We also introduce back-projection, a simple and effective semi-supervised training method that leverages unlabeled video data. We start with predicted 2D keypoints for unlabeled video, then estimate 3D poses and finally back-project to the input 2D keypoints. In the supervised setting, our fully-convolutional model outperforms the previous best result from the literature by 6 mm mean per-joint position error on Human3.6M, corresponding to an error reduction of 11%, and the model also shows significant improvements on HumanEva-I. Moreover, experiments with back-projection show that it comfortably outperforms previous state-of-the-art results in semi-supervised settings where labeled data is scarce. Code and models are available at https://github.com/facebookresearch/VideoPose3D
1. Introduction
The paper addresses ambiguity in lifting 2D keypoints to 3D poses by using temporal convolutions, and extends the approach to semi-supervised learning with unlabeled video. It reports higher accuracy than prior methods in supervised and semi-supervised settings.
- The 2D-to-3D formulation is inherently ambiguous because multiple 3D poses can map to identical 2D keypoints.
- The proposed fully convolutional model applies dilated temporal convolutions to 2D keypoints, handling large contexts with greater accuracy, simplicity, and efficiency than RNN-based approaches.
- Unlabeled video is used for semi-supervised training through a scheme designed for settings with scarce labeled 3D pose data.
- The semi-supervised approach requires camera intrinsic parameters rather than ground-truth 2D annotations or multi-view imagery with extrinsic parameters.
- The approach outperforms previous best-performing methods in both supervised and semi-supervised settings, including supervised models trained with extra labeled data.
2. Related work
Prior work progressed from direct RGB-to-3D estimation toward lifting 2D keypoint trajectories, with recurrent models and several semi-supervised strategies addressing temporal modeling and limited labels. The paper places its approach within trajectory-based 3D pose estimation while contrasting limitations of existing methods.
- Two-step pose estimation: Two-step estimators first predict image-space 2D joints and then lift them to 3D, while successful approaches learn from 2D keypoint trajectories.
- Two-step pose estimation: LSTM sequence-to-sequence models encode 2D pose sequences into fixed-size vectors before decoding 3D pose sequences of the same length.
- Two-step pose estimation: Earlier sequence-to-sequence experiments found that output poses drift over lengthy sequences, while reinitializing the encoder every five frames sacrifices temporal consistency.
- Semi-supervised training: Prior semi-supervised approaches used multitask learning, transferred 2D features, unlabeled multi-view recordings, or adversarial discrimination of realistic poses.
- The architecture uses a receptive field of 243 frames with four blocks and 17 joints, and its residual connections are sliced to match valid-convolution outputs.
3. Temporal dilated convolutional model
The model is a residual fully convolutional network that transforms sequences of 2D poses through temporal convolutions. Dilations expand temporal context efficiently, while convolutional processing enables parallelization and fixed gradient paths.
- The residual fully convolutional architecture transforms 2D pose sequences through temporal convolutions and uses dilations to model long-term dependencies efficiently.
- Convolutional models parallelize across batch and time, unlike RNNs, and maintain a fixed input-output gradient path regardless of sequence length.
- Each residual block increases the receptive field exponentially by W while increasing parameter count only linearly.
- The final layer predicts 3D poses for all frames in the input sequence using temporal information from past and future frames.
- An example architecture uses a 243-frame receptive field, four blocks, kernel size W = 3, 1024 output channels, and dropout rate p = 0.25.
4. Semi-supervised approach
The semi-supervised method combines supervised 3D pose learning with an autoencoding back-projection loss on unlabeled video. It also models global trajectory and encourages plausible bone lengths, while requiring only camera intrinsics.
- Unlabeled video and an off-the-shelf 2D detector extend the supervised objective with a back-projection loss that checks reconstructed 2D poses.
- The supervised and unsupervised objectives are optimized jointly, with labeled examples using ground-truth 3D targets and unlabeled examples using projection consistency.
- Trajectory model: The method regresses global 3D trajectory because perspective projection depends on both root position and the pose relative to that root.
- Bone length L2 loss: A soft bone-length constraint matches mean bone lengths in unlabeled batches approximately to those in labeled batches, encouraging plausible 3D poses.
- The approach requires only camera intrinsic parameters and can be applied to any 3D pose detector that takes 2D keypoints as input.
5. Experimental setup
The evaluation uses Human3.6M and HumanEva-I, multiple pose-error protocols, and varied 2D keypoint detectors. Training and implementation choices address detector initialization, camera-space evaluation, temporal sampling, optimization, and sequence-specific issues.
- Datasets and evaluation: Human3.6M uses 17 joints, five training subjects, two test subjects, four synchronized cameras, and 50 Hz recordings across 15 actions.
- Datasets and evaluation: HumanEva-I uses 15 joints, three actions, three camera views at 60 Hz, and separate single-action or multi-action models.
- Datasets and evaluation: Protocol 1 measures MPJPE, Protocol 2 measures rigidly aligned P-MPJPE, and Protocol 3 measures scale-aligned N-MPJPE for semi-supervised experiments.
- 2D detectors: The study compares stacked hourglass, Mask R-CNN, and CPN 2D detectors, with Mask R-CNN and CPN pretrained on COCO and fine-tuned on Human3.6M.
- Training and implementation: Experiments train in camera space using Amsgrad, 80 epochs, and an exponentially decaying learning rate on Human3.6M.
- Training and implementation: Temporal models sample clips from different video segments to reduce batch-normalization bias caused by correlated pose sequences.
6. Results
The temporal convolutional model improves reconstruction accuracy, smoothness, efficiency, and generalization, while back-projection provides larger gains when labeled data is scarce. Results also show that detector quality and the bone-length constraint affect semi-supervised performance.
- Temporal dilated convolutional model: 6 mm lower average error than the previous best Human3.6M result corresponds to an 11% error reduction under Protocol 1.
- Temporal dilated convolutional model: The model achieves lower average error than competing approaches under both Human3.6M evaluation protocols without additional data.
- Temporal dilated convolutional model: 76% lower MPJVE than the single-frame baseline demonstrates substantially smoother predicted poses.
- Generalization and efficiency: The model outperforms the previous state of the art on HumanEva-I and achieves lower error even when computations are halved.
- Generalization and efficiency: About 150k FPS is measured on one NVIDIA GP100 GPU for a single long sequence when 2D poses are already available.
- Semi-supervised approach: 9–10.4 mm lower N-MPJPE is achieved over the supervised baseline with fewer than 5K labeled frames.
- Semi-supervised approach: Up to 14.7 mm MPJPE improvement is obtained over the supervised baseline on the full-frame-rate semi-supervised setup.
- Semi-supervised approach: Removing the bone-length term increases error from 78.1 mm to 91.3 mm for 1% of S1, compared with 100.7 mm for the supervised baseline.
7. Conclusion
The paper combines a fully convolutional temporal model with back-projection to estimate 3D poses from video and exploit unlabeled data. It reports improved supervised accuracy and practical semi-supervised gains with only camera intrinsics required.
- Dilated temporal convolutions over 2D keypoint trajectories provide the fully convolutional 3D pose estimator.
- Back-projection improves performance when labeled 3D data is scarce by using unlabeled video and requiring only intrinsic camera parameters.
- 6 mm lower average joint error on Human3.6M corresponds to an 11% relative reduction, with additional improvements on HumanEva-I.
- Back-projection improves accuracy by about 10 mm N-MPJPE and 15 mm MPJPE over a strong baseline with 5K or fewer annotated frames.
A.1. Dilated convolutions and information flow
Dilated convolutions expand the temporal receptive field while retaining sparse computation, and the model’s information flow can use symmetric or causal temporal context.
- Dilated convolutions space kernel points uniformly, inserting zeros between them according to the dilation factor.For example, dilation factors 2 and 3 expand a three-point filter by inserting one or two zeros between adjacent kernel points.
- The sparse structure enables implementations to skip computations over zero points.This preserves roughly the computational cost of regular convolutions with the same number of non-zero entries while increasing the receptive field.
- Symmetric convolutions use both past and future frames, whereas causal convolutions access only past frames.Causal convolutions are considered for real-time scenarios.
- The model’s information flow is illustrated from input to output, with dashed lines denoting skip-connections.The accompanying illustration distinguishes symmetric convolutions from causal convolutions.
A.2. Computational cost analysis
The computational analysis counts matrix-multiplication operations for comparable implementations and derives the convolutional cost per frame, totaling 17.089 MFLOPs for the illustrated model.
- The analysis counts matrix multiplications while treating biases, batch normalization, and activations as negligible to final complexity.The reference implementation of [16] was profiled in TensorFlow using amortized cost per predicted frame.
- The analyzed model has a 27-frame receptive field and two residual blocks, with convolution-only matrix multiplications enabling hand computation.The model’s sample architecture is shown in Figure 7.
- A 1D convolution with Cin input channels, Cout output channels, and width W costs 2 N W Cin Cout FLOPs, or 2 W Cin Cout FLOPs per frame.
- 17.089 MFLOPs per frame is the total reported cost for the 27-frame model.
A.3. Ablation of receptive field and channel size
The ablations examine receptive field, channel size, augmentation, convolution type, training step size, and implementation strategy, identifying saturation, efficiency trade-offs, and optimization benefits.
- Receptive field and channel size: Receptive fields of 1, 9, 27, 82, and 243 frames were evaluated, with error saturating quickly as the receptive field increased.The model did not appear to overfit, and the authors generally adopted 243 frames.
- Receptive field and channel size: Channel sizes from 128 to 2048 showed similar non-overfitting behavior, while error saturated beyond a certain point.Because computational complexity grows quadratically with channel size, the authors adopted C = 1024.
- Augmentation: Removing test-time augmentation increased error from 46.8 mm to 47.7 mm, while also removing train-time augmentation raised it to 49.2 mm.
- Convolution type: Replacing dilated with dense convolutions increased error from 48.8 mm to 50.4 mm (+1.6 mm) and increased parameters and computations by approximately 3.5×.The result is reported for a 27-frame model with fine-tuned CPN detections.
- Training step size: Larger training step sizes reduced speed and impaired performance, whereas single-frame optimization achieved approximately 2× speedup and enabled training with 243 frames.The speedup reached approximately 4× for models with 81-frame receptive fields.
- Implementation strategy: For short sequences, layer-by-layer computation wastes boundary states, so an optimized implementation avoids discarded intermediate states; inference uses layer-by-layer computation for multi-frame predictions.
- Qualitative evaluation: Supplementary videos compare temporal-convolution predictions with single-frame predictions and ground-truth poses, including their temporal smoothness.