Source-linked AI summary
Flowing ConvNets for Human Pose Estimation in Videos
Tomas Pfister, James Charles, Andrew Zisserman
TL;DR
The paper addresses the challenge of estimating human pose in videos by exploiting temporal information across multiple frames. It proposes a deeper heatmap-based ConvNet with spatial fusion, optical-flow alignment, and learned pooling, and reports state-of-the-art performance on three video pose datasets. The authors also identify failure modes and discuss broader applications of optical-flow-mediated information combination.
Problem
Human pose estimation in videos remains challenging, despite the useful temporal dependencies between poses in nearby frames.
Method
A deeper ConvNet predicts joint heatmaps for neighboring frames, aligns them with dense optical flow, and learns spatial fusion and parametric pooling.
Results
The ConvNet outperforms the state of the art on BBC Pose, ChaLearn, and Poses in the Wild, with pooling from neighboring frames providing a further significant boost.
Takeaways & Limitations
Heatmap regression combined with optical flow and spatial fusion improves video pose estimation, while the same flow-mediated combination could extend to classification and segmentation.
Takeaways & Limitations
Further improvements may require additional spatial-ConvNet inputs, although prior work had shown little benefit from multiple RGB frames or optical flow.
Abstract
from arXiv · showhide
The objective of this work is human pose estimation in videos, where multiple frames are available. We investigate a ConvNet architecture that is able to benefit from temporal context by combining information across the multiple frames using optical flow. To this end we propose a network architecture with the following novelties: (i) a deeper network than previously investigated for regressing heatmaps; (ii) spatial fusion layers that learn an implicit spatial model; (iii) optical flow is used to align heatmap predictions from neighbouring frames; and (iv) a final parametric pooling layer which learns to combine the aligned heatmaps into a pooled confidence map. We show that this architecture outperforms a number of others, including one that uses optical flow solely at the input layers, one that regresses joint coordinates directly, and one that predicts heatmaps without spatial fusion. The new architecture outperforms the state of the art by a large margin on three video pose estimation datasets, including the very challenging Poses in the Wild dataset, and outperforms other deep methods that don't use a graphical model on the single-image FLIC benchmark (and also Chen & Yuille and Tompson et al. in the high precision region).
1. Introduction
Video pose estimation remains challenging, but temporal dependencies offer useful information beyond still images. This work proposes a ConvNet that aligns and pools heatmap predictions across frames while learning spatial dependencies between body parts, outperforming prior methods and the state of the art on video benchmarks.
- Video pose estimation is challenging, while nearby frames provide an important cue because pose positions have strong temporal dependencies.
- Additional spatial fusion layers learn dependencies between body parts through an implicit spatial model.The authors report that these layers remove pose failures that are kinematically impossible.
- The approach differs from earlier temporal ConvNets that input flow or RGB from nearby frames and predict positions in the current frame.
- The proposed ConvNet predicts joint heatmaps for neighboring frames, warps them with optical flow, and combines them through learned pooling.This uses dense flow to align position confidences to the target frame rather than using flow only as an input feature.
- The ConvNet outperforms the state of the art on BBC Pose, ChaLearn, and Poses in the Wild, with optical-flow pooling providing a further significant boost.
2. Temporal Pose Estimation Networks
The architecture estimates pose by regressing per-joint heatmaps, modeling spatial dependencies, aligning neighboring-frame predictions with optical flow, and learning to pool them into a composite map.
- Spatial ConvNet: Each input frame produces a separate heatmap for every joint, from which the pose is estimated at maximum confidence.The heatmap volume is fixed-size, and neighboring-frame outputs are later combined at the target frame.
- Spatial ConvNet: Gaussian heatmaps centered on ground-truth joints replace direct coordinate regression and support multimodal predictions during learning.The model uses an l2 loss between predicted and synthesized target heatmaps; incorrect modes can be suppressed progressively.
- Architecture: The network preserves heatmap resolution through minimal pooling, unit strides, and a fully convolutional design with 1 × 1 convolutions replacing fully connected layers.Sample activations progress from edge responses in early layers toward body-part responses in deeper layers.
- Spatial ConvNet: Deeper spatial fusion layers learn dependencies between body-part locations and remove kinematically impossible pose failures.These layers combine pre-heatmap activations with a skip-layer representation through additional convolutions.
- Optical flow alignment: Dense optical flow warps confidence maps from neighboring frames onto frame t by shifting confidences along pixel-wise temporal tracks.The architecture uses frames within a temporal neighborhood and aligns their predictions before pooling.
- Confidence pooling: A parametric cross-channel pooling layer learns per-heatmap weights to combine aligned predictions into one composite heatmap for each joint.Frames farther from the target may receive lower weight because their optical-flow errors can be larger; the final pose uses maximum composite confidence.
3. Implementation Details
The experiments use standard GPU-based training and a specified optical-flow implementation, with real-time inference reported for the overall method.
- Inputs use a temporal neighbourhood of n = 15 in the parametric pooling layer.
- Training uses four NVIDIA GTX Titan GPUs with a modified Caffe framework; SpatialNet takes 3 days on FLIC and SpatialNet Fusion takes 7 days.
- Optical flow is computed using FastDeep-Flow with Middlebury parameters.
4. Datasets
The study evaluates video pose estimation across four datasets spanning broadcast video, Kinect gesture data, and movie footage, with varied annotation quality and visual difficulty.
- Experiments use four large video pose estimation datasets, including two from signed TV broadcasts, one Italian gesture dataset, and one Hollywood-movie dataset.
- BBC Pose contains 20 BBC videos, with semi-automatically assigned training poses and 1,000 manually annotated test frames.
- Extended BBC Pose adds 72 training videos, producing 85 total training videos, but its automatically assigned annotations are noisier.
- ChaLearn contains 23 hours of Kinect data from 27 people with RGB, depth, foreground segmentation, skeletons, and noisy training and testing labels.
- Poses in the Wild contains 30 Hollywood sequences totaling 830 frames, featuring clutter, severe camera motion, and occlusions; FLIC supplies about 4.5K annotated training images.
5. Experiments
Experiments compare the proposed heatmap-based networks with coordinate regression and alternative pooling, then evaluate them against prior methods across video and still-image benchmarks. The method benefits from deeper spatial processing, additional data, optical-flow pooling, and learned pooling weights, while achieving strong state-of-the-art results.
- Evaluation protocol: Evaluation plots accuracy against pixel distance from ground truth, counting a joint correct when it falls within d pixels of the marked joint centre.
- Alternative architectures: The CoordinateNet baseline directly regresses joint positions, whereas the proposed networks regress heatmaps and incorporate spatial and temporal processing.
- Component evaluation: 79.6% to 86.1% at d = 6: training SpatialNet on Extended BBC Pose adds 6.6%, whereas CoordinateNet cannot effectively use the noisier training data.
- Component evaluation: Parametric pooling performs best and improves with larger neighbourhoods, while sum- and max-pooling deteriorate because they cannot down-weight temporally distant, error-prone predictions.
- Component evaluation: Spatial fusion resolves vanilla heatmap-network failures involving selection of incorrect modes that produce kinematically impossible poses.
- Comparison to state of the art: BBC Pose improves over prior work by 10% for wrists at d = 6, while ChaLearn gains 3.5% without flow, 5.5% with flow, and 13% from a deeper network at d = 6.
- Comparison to state of the art: On Poses in the Wild, the method exceeds the state of the art by 30% for wrists and 24% for elbows at d = 8; optical flow adds 10% and 13%, respectively.
6. Conclusion
The proposed ConvNet combines multiple-frame appearance with optical flow and spatial fusion to improve video pose estimation. It outperforms the state of the art on three large video datasets and performs strongly on FLIC.
- Conclusion: The architecture improves heatmap regression by combining optical flow with spatial fusion layers.Optical flow aligns estimates across frames, while spatial fusion models dependencies between body parts.
- Conclusion: The method outperforms the state of the art on three large video pose estimation datasets.The evaluated datasets are BBC Pose, ChaLearn, and Poses in the Wild.
- Conclusion: 20% is added at d = 0.05 on FLIC against competing methods without a graphical model.Figure 15 compares deep models with and without graphical models in the high-precision region.
- Conclusion: The benefits of optical-flow alignment are complementary to architectures with explicit spatial MRF and refinement layers.This suggests the alignment mechanism can coexist with graphical-model-style spatial refinement.
- Conclusion: The optical-flow-mediated combination of information may also apply to video classification and segmentation.The paper presents this as a possible extension beyond human pose estimation.