PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Knowledge Distillation for Efficient Pose Estimation

Zheng Li, Jingwen Ye, Mingli Song, Ying Huang, Zhigeng Pan

arXiv:2108.02092v2cs.CV

TL;DR

Accurate pose estimation can require heavy models, and prior pose distillation relies on pre-trained teachers and two-stage training. OKDHP uses one-stage online distillation, with an FAU assembling branch heatmaps as shared targets. The authors report effectiveness on MPII and COCO, while noting inherited model limitations in occlusion and insufficient light.

  • Problem

    Pose estimation’s computationally heavy models and prior pose distillation’s reliance on pre-trained teachers and two-stage training motivate a more efficient approach.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Method

    OKDHP trains a multi-branch network in one stage, using an FAU to ensemble branch heatmaps as targets and pixel-wise KL divergence to supervise branches.

    Source: §1 · p. 2

  • Results

    Experiments validate OKDHP’s effectiveness on the MPII and COCO benchmark datasets.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    OKDHP supports different target compression rates, and removing auxiliary branches adds no test-time cost.

    Source: §5 · p. 8

    Predictions remain biased under body occlusion or insufficient ambient light, reflecting limitations inherited from the underlying model.

    Source: Failure Case Analysis · p. 11

Abstract

from arXiv · show

Existing state-of-the-art human pose estimation methods require heavy computational resources for accurate predictions. One promising technique to obtain an accurate yet lightweight pose estimator is knowledge distillation, which distills the pose knowledge from a powerful teacher model to a less-parameterized student model. However, existing pose distillation works rely on a heavy pre-trained estimator to perform knowledge transfer and require a complex two-stage learning procedure. In this work, we investigate a novel Online Knowledge Distillation framework by distilling Human Pose structure knowledge in a one-stage manner to guarantee the distillation efficiency, termed OKDHP. Specifically, OKDHP trains a single multi-branch network and acquires the predicted heatmaps from each, which are then assembled by a Feature Aggregation Unit (FAU) as the target heatmaps to teach each branch in reverse. Instead of simply averaging the heatmaps, FAU which consists of multiple parallel transformations with different receptive fields, leverages the multi-scale information, thus obtains target heatmaps with higher-quality. Specifically, the pixel-wise Kullback-Leibler (KL) divergence is utilized to minimize the discrepancy between the target heatmaps and the predicted ones, which enables the student network to learn the implicit keypoint relationship. Besides, an unbalanced OKDHP scheme is introduced to customize the student networks with different compression rates. The effectiveness of our approach is demonstrated by extensive experiments on two common benchmark datasets, MPII and COCO.

1. Introduction

Accurate pose estimation often relies on computationally heavy models, while prior pose distillation requires a pre-trained teacher and two-stage training. OKDHP introduces one-stage online pose distillation, using a multi-branch network and FAU to transfer pose structure and accommodate different compression rates.

1. Introduction

  • Heavy pose-estimation models can limit deployment on resource-intensive mobile devices, motivating lightweight approaches.Prior lightweight networks target reduced computation cost and improved efficiency.Source: §1 · p. 1
  • Prior pose distillation uses a large pre-trained teacher in a two-stage process, while online classification methods do not address pixel-level structural knowledge.The paper identifies this as a gap for pose estimation.Source: §1 · pp. 1–2
  • Existing pixel-level distillation uses MSE, which the authors say weakly transfers knowledge and conflicts with the ground-truth supervision objective.MSE does not effectively measure relative entropy between the teacher and student heatmap distributions.Source: §1 · p. 2
  • OKDHP trains multiple student branches together, and an FAU combines their predictions into a teacher that supervises them in reverse.The FAU uses multi-scale information to produce higher-quality target heatmaps.Source: §1 · p. 2
  • Pixel-wise KL divergence matches predicted and target heatmaps, and removing auxiliary branches yields a single-branch model without added test-time cost.The framework offers balanced and unbalanced versions to customize target networks for different compression rates.Source: §1 · p. 2
  • Experiments on MPII and COCO validate the effectiveness of OKDHP.Source: §1 · p. 2

2. Related Work

Related work covers pose-estimation architectures and knowledge distillation, from traditional teacher–student transfer to online branch-based methods. Prior pose-distillation approaches remain dependent on heavy pre-trained teachers and complex two-stage training.

Human Pose Estimation

  • CNN pose estimators include direct coordinate regression, heatmap-based pose matching, stacked hourglass refinement, and attention or pyramid-residual approaches.Source: §2 · p. 2
  • Efficient pose-estimation networks reduce computation for real applications, with examples using part-affinity fields, keypoint assignment, or lightweight backbones.MultiPoseNet is reported to achieve similar accuracy to Mask-RCNN while being at least 4x faster.Source: §2 · p. 2

Knowledge Distillation

  • Knowledge distillation transfers a large model’s soft predictions to a computationally efficient student, with methods also transferring features, attention, or flow matrices.Source: §2 · pp. 2–3
  • Online distillation removes the need for a pre-trained teacher, but earlier pose-distillation work uses heavy teachers and complex two-stage learning.Prior pose methods include FPD, which transfers knowledge from an 8-Stack hourglass network to a 4-Stack network.Source: §2 · p. 3

3. Methodology

OKDHP jointly trains multiple pose-estimator branches and uses FAU-assembled predictions as online teacher targets, avoiding a separately pre-trained teacher. Branches receive task supervision and pixel-wise distillation, while unequal branch depths support different deployment sizes.

3.1. Teacher-Student Learning

  • Vanilla distillation first trains and fixes a cumbersome teacher, then transfers its knowledge to a compact student, increasing training complexity.Source: §3.1 · p. 3

3.2. Online Human Pose Distillation

  • OKDHP uses a multi-branch network and an FAU module to build the teacher online for efficient pose estimation.The architecture uses Hourglass networks as its basic building block.Source: §3.2 · p. 4

3.2.1 The Main Network

  • The balanced main network shares initial Hourglass modules across T branches, supervises each branch's heatmaps against ground truth with MSE, and also supervises intermediate outputs.The branches have the same configuration in the balanced version.Source: §3.2.1 · p. 4

3.2.2 The FAU Module

  • FAU combines pooled and parallel 3×3, 5×5, and 7×7 convolution paths, then learns channel-wise branch weights to sum predicted heatmaps into targets.The paths capture multi-scale information across local and global fields.Source: §3.2.2 · p. 4

3.2.3 Pixel-wise Distillation

  • Each branch learns from FAU target heatmaps through pixel-wise KL, while the full objective combines this distillation loss with pose-task MSE.The target heatmaps play the teacher role, and the KL loss minimizes divergence between teacher and student heatmaps.Source: §3.2.3 · p. 5

3.3. Unbalanced Architecture

  • Unbalanced branches use different Hourglass depths, letting a larger branch provide stronger teaching and yielding three differently sized networks with comparable performances in one training run.The configuration can be customized to actual needs.Source: §3.3 · pp. 5–6

4. Experiments

Experiments evaluate OKDHP on MPII and COCO and ablate its training choices. MPII results show gains over conventional training and prior distillation, while COCO and the ablations support the method’s broader effectiveness and design choices.

4.1. Implementation Details

  • Experiments use MPII and COCO with their respective PCKh@0.5 and OKS metrics; MPII uses 16 joints and COCO uses 17.The default OKDHP setup uses three branches, and a 4-Stack network has two shared and two individual HG modules per branch.Source: §4.1 · p. 6

4.2. Results on MPII dataset

  • OKDHP improves MPII baseline networks by 0.3 to 0.8 PCKh points, and a 2-Stack HG network reaches performance similar to the original 4-Stack network with half the HG modules.The authors report that these gains are significant relative to prior work’s typical 0.1%–0.3% PCKh improvements.Source: Table 2 · p. 6, §4.2 · p. 7
  • Both balanced and unbalanced OKDHP outperform FPD on MPII; unbalanced gains 0.2% over balanced with 36% more FLOPS, while balanced has the lowest training cost.The FPD teacher is an 8-Stack HG network scoring 90.2 PCKh@0.5.Source: Table 3 · p. 6, §4.2 · p. 7

4.3. Results on COCO dataset

  • On COCO val2017, OKDHP yields a more generalizable model than independent learning in the reported two-stage top-down evaluation.The evaluation uses a person detector with detection AP 56.4.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. Ablation Study

  • Pixel-wise KL divergence performs better than MSE as the distillation loss on MPII, supporting transfer of pose structural knowledge.MSE is also the task-specific supervised term used by FPD.Source: §4.4 · pp. 7–8, Table 6 · p. 7
  • OKDHP scales with branch size, and a 2-Stack network can improve further when trained with more branches; the n=1 case is omitted because it cannot form an ensemble..Source: §4.4 · p. 8, Table 7 · p. 8
  • For a 4-Stack network, very few shared HG modules reduce performance; the authors use equal numbers of shared and individual modules for accuracy-efficiency trade-offs.The proposed explanation connects reduced individual modules with less branch diversity and weaker within-group knowledge transfer.Source: §4.4 · p. 8, Table 8 · p. 8
  • Performance remains robust for β values from 0.5 to 5, and FAU outperforms mean aggregation and Gate by 0.3% and 0.2%, respectively.FAU is evaluated on a 4-Stack HG network on MPII validation set.Source: §4.4 · p. 8, Table 10 · p. 8, Table 9 · p. 8

5. Conclusion

OKDHP performs one-stage pose-structure distillation by integrating multiple student branches into a teacher through FAU, which then teaches the branches in reverse. It also supports target networks with different compression rates and is validated on MPII and COCO.

5. Conclusion

  • OKDHP uses a one-stage multi-branch network in which FAU integrates student branches into a teacher that optimizes them in reverse.The framework distills human-pose structure knowledge.Source: §5 · p. 8
  • The unbalanced OKDHP scheme enables customization of target networks with different compression rates.Source: §5 · p. 8
  • Experiments on MPII and COCO validate the effectiveness of OKDHP.Source: §5 · p. 8

Supplementary Material

The supplementary experiments report benefits from OKDHP across three MobilePose backbones, with larger gains for smaller backbones. Failure cases remain under occlusion and insufficient ambient light.

Additional Experiments

  • MobilePose shares its encoder across parallel decoder branches, while retaining the same FAU module.The tested backbones are ResNet-18, MobileNet-V2, and ShuffleNet-V2.Source: Additional Experiments · p. 11
  • All networks benefit from the proposed OKDHP, particularly small-backbone networks, which achieve larger performance gains; Table 11 reports PCKh@0.5 scores on the MPII validation set.Source: Additional Experiments · p. 11, Table 11 · p. 11

Failure Case Analysis

Loading 2108.02092v2…