DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimation

• The paper identifies temporal dependencies and attention to body-joint regions as two unresolved challenges for adapting diffusion models to video pose estimation.Source: §1 · p. 2
• STRL extracts spatial information within frames, integrates cross-frame knowledge with cascaded Transformers, and supplies its features as a fixed condition at each denoising step.Source: §1 · p. 2
• LMSFI uses noisy heatmaps to retrieve local joint features and models their multi-scale correlations with global context to focus representations on keypoint regions.The introduction reports that its representations converge consistently around keypoint regions.Source: §1 · p. 2
• Multiple sampled pose estimates can be combined for greater robustness, while the number of refinement steps can change without retraining.The introduction highlights challenging joints such as wrists and ankles for ensemble gains.Source: §1 · p. 2
• DiffPose achieves state-of-the-art results on PoseTrack2017, PoseTrack2018, and PoseTrack21.Source: §1 · p. 2
• DiffPose is presented as the first diffusion model for multi-frame human pose estimation, formulated as conditional generation of keypoint heatmaps.It progressively adds Gaussian noise to ground-truth heatmaps and denoises them with a Pose-Decoder.Source: §1 · p. 2

• Heatmap-based keypoint representations have gained popularity because they are associated with faster optimization convergence.Source: §2 · p. 2
• Earlier video pose methods merge consecutive-frame heatmaps [37] or use deformable convolutions for implicit motion compensation [38].Source: §2 · p. 2
• These video methods typically predict deterministic poses and may lack recalibration, leaving challenging joints vulnerable to localized detection failures.Source: §2 · p. 3
• Diffusion models have been applied to image segmentation [1], object detection [10], and instance segmentation [24], but the paper reports no prior successful adaptation to multi-frame human pose estimation.Source: §2 · p. 3

3.1. Preliminaries

• DiffPose trains a Pose-Decoder to recover ground-truth keypoint heatmaps from noisy heatmaps, conditioned on spatiotemporal features extracted from a cropped video sequence.The sequence contains crops of the same person across consecutive frames around a keyframe.Source: §3.1 · p. 3

3.2. Spatiotemporal Representation Learner

• STRL extracts spatial features per frame, embeds and combines frame tokens, then uses cascaded Transformers and an MLP to produce spatiotemporal features.These features provide temporal context to the Pose-Decoder.Source: §3.2 · pp. 3–4

3.3. Pose-Decoder

• LMSFI uses noisy heatmaps to look up local joint features, models their interaction with global context across scales, and produces representations focused on joint regions.The decoder uses Transformer interactions at low resolution and convolutions at higher resolutions; the resulting scale features are integrated to predict a heatmap.Source: §3.3 · pp. 4–5

3.4. Overall Training and Inference Algorithms

• Training corrupts ground-truth heatmaps and supervises denoising with mean-square error; inference progressively refines noisy heatmaps using DDIM updates.At each sampling step, the decoder takes a random noisy heatmap or the preceding step's predicted heatmap as input.Source: §3.4 · p. 5

3.5. Discussion

• DiffPose can average predictions from multiple noisy initializations and adjust inference sampling steps without retraining; additional steps yield more delicate representations that foster accurate pose estimation.The authors report that complementary pose ensembling improves performance, especially for challenging joints.Source: §3.5 · pp. 5–6

4.1. Experimental Settings

• Experiments use PoseTrack2017, PoseTrack2018, and PoseTrack21, evaluate mean average precision, and run inference with 10 noise groups and 4 denoising steps.The implementation uses 256×192 inputs and terminates training within 60 epochs.Source: §4.1 · p. 6

4.2. Comparison with State-of-the-art Approaches

• 86.4 mAP on PoseTrack2017, 83.0 mAP on PoseTrack2018, and 82.9 (↑1.7) mAP over FAMI-Pose on PoseTrack21.On PoseTrack2017, DiffPose gains 1.6 mAP over FAMI-Pose; the PoseTrack2018 results are reported as outperforming all other approaches.Source: §4.2 · pp. 6–7

4.3. Ablation Study

• LMSFI raises Direct-single from 52.5 to 82.4 mAP, and adding spatiotemporal cues improves performance by a further 4.0 mAP.The component ablation attributes the first increase to LMSFI and the further gain to the full DiffPose setting.Source: §4.3 · p. 7, Table 4 · p. 7
• Multi-scale fusion yields 86.4 (↑2.0) mAP, while concatenation and aggregation yields 86.2 mAP.The comparison tests designs within LMSFI; the reported change between the two fusion approaches is marginal.Source: §4.3 · p. 7, Table 5 · p. 7
• Ensembling predictions from 1, 5, and 10 initial noises raises mAP from 80.4 to 84.0 to 86.4, with wrist and ankle gains of 8.9 and 6.1 mAP.The authors attribute the improvement to averaging plausible pose predictions from different noise inputs.Source: §4.3 · pp. 7–8, Table 6 · p. 8
• More denoising iterations improve performance as features are progressively refined toward keypoint regions.The study compares 1, 2, and 4 steps; the supplied passage reports the trend but not the individual scores.Source: §4.3 · p. 8, Table 7 · p. 8

4.4. Qualitative Analyses on DiffPose

• Visualizations contrast DiffPose activations and pose estimates with baselines, and report more robust predictions in challenging scenes.The examples include fast motion and mutual occlusion, with inaccurate detections highlighted in red circles.Source: Figure 3 · p. 7, §4.4 · p. 8, Figure 4 · p. 8

Conclusion

• DiffPose combines STRL for temporal cues and LMSFI for keypoint-region attention in conditional heatmap generation.The conclusion describes flexible pose ensembling and iterative refinement as improving performance without retraining.Source: §5 · p. 8
• Evaluations on PoseTrack2017, PoseTrack2018, and PoseTrack21 show state-of-the-art performance.The authors identify 3D human pose estimation, pose tracking, and accelerated inference as future directions.Source: §5 · p. 8

Training

• Training corrupts ground-truth heatmaps with Gaussian noise and trains the Pose-Decoder to reverse the diffusion process.The training algorithm takes an image sequence and ground-truth heatmap as inputs.Source: Training and Inference Algorithms · p. 13

Inference

• At inference, the Pose-Decoder maps random noisy or previously predicted heatmaps to current-step estimates, and DDIM updates the heatmaps for the next step.The number of iterative denoising steps is controlled by the parameter steps.Source: Training and Inference Algorithms · p. 13
• The inference procedure takes an image sequence, steps, and T, initializes Gaussian noise, and returns a predicted heatmap.The algorithm initializes spatiotemporal features with STRL before sampling.Source: Training and Inference Algorithms · p. 13

Study on temporal interval δ

• 82.4 mAP at δ = 0 rises to 84.5 at δ = 1 and 86.4 at δ = 2; the authors choose δ = 2 to balance performance and computation.The ablation varies the number of supporting frames in STRL; the table caption identifies the comparison as a temporal-interval ablation.Source: Table 8 · p. 13, Supplementary Experiments · p. 13

Visualized results

• Figures 7–9 show DiffPose predictions on PoseTrack2017, PoseTrack2018, and PoseTrack21 in challenging scenes such as fast motion and occlusion.The authors describe the estimates as accurate and robust across these scenes.Source: Supplementary Experiments · p. 13, Figure 7 · p. 14, Figure 8 · p. 15
• Figure 6 presents video comparisons, pose ensembling, and gradual denoising, with inaccurate estimates highlighted in its comparison panel.“DiffPose-M.” denotes multiple pose predictions in pose ensembling.Source: Supplementary Experiments · p. 13, Figure 6 · p. 14