PaperlayerView exact revision on arXiv

Source-linked AI summary

Peer Collaborative Learning for Online Knowledge Distillation

Guile Wu, Shaogang Gong

arXiv:2006.04147v2cs.CV

TL;DR

Traditional distillation relies on a pre-trained teacher, while online methods have not jointly solved high-capacity teacher construction and branch collaboration. PCL builds feature-based peer ensemble and temporal mean teachers in one-stage training, improving generalisation across CIFAR-10, CIFAR-100, and ImageNet and outperforming state-of-the-art methods.

  • Problem

    Traditional knowledge distillation depends on pre-training a high-capacity teacher before transferring knowledge to a compact student.

  • Method

    PCL integrates online ensembling and network collaboration through peer ensemble and temporal mean teachers in a one-stage framework.

  • Results

    PCL improves generalisation across CIFAR-10, CIFAR-100, and ImageNet and outperforms state-of-the-art online distillation methods.

  • Takeaways & Limitations

    The unified peer teachers enable online distillation without pre-training a high-capacity teacher model.

  • Takeaways & Limitations

    The peer ensemble teacher adds computational cost mainly through training its additional classifier.

Abstract

from arXiv · show

Traditional knowledge distillation uses a two-stage training strategy to transfer knowledge from a high-capacity teacher model to a compact student model, which relies heavily on the pre-trained teacher. Recent online knowledge distillation alleviates this limitation by collaborative learning, mutual learning and online ensembling, following a one-stage end-to-end training fashion. However, collaborative learning and mutual learning fail to construct an online high-capacity teacher, whilst online ensembling ignores the collaboration among branches and its logit summation impedes the further optimisation of the ensemble teacher. In this work, we propose a novel Peer Collaborative Learning method for online knowledge distillation, which integrates online ensembling and network collaboration into a unified framework. Specifically, given a target network, we construct a multi-branch network for training, in which each branch is called a peer. We perform random augmentation multiple times on the inputs to peers and assemble feature representations outputted from peers with an additional classifier as the peer ensemble teacher. This helps to transfer knowledge from a high-capacity teacher to peers, and in turn further optimises the ensemble teacher. Meanwhile, we employ the temporal mean model of each peer as the peer mean teacher to collaboratively transfer knowledge among peers, which helps each peer to learn richer knowledge and facilitates to optimise a more stable model with better generalisation. Extensive experiments on CIFAR-10, CIFAR-100 and ImageNet show that the proposed method significantly improves the generalisation of various backbone networks and outperforms the state-of-the-art methods.

Introduction

Traditional knowledge distillation depends on a pre-trained high-capacity teacher, while online methods train end-to-end but leave gaps in teacher construction and branch collaboration. PCL addresses these gaps by combining online ensembling with peer collaboration and reports improved generalisation across benchmarks.

  • Motivation: Traditional knowledge distillation uses two-stage training to transfer knowledge from a high-capacity teacher to a smaller student.The student reduces model complexity while retaining competitive accuracy.
  • Motivation: Online knowledge distillation removes teacher pre-training through one-stage end-to-end optimisation using collaborative, mutual, or ensemble-based architectures.Collaborative and mutual learning use hierarchical or parallel networks, while online ensembling uses a multi-branch network.
  • Proposed method: PCL integrates online ensembling and network collaboration by constructing a multi-branch network whose branches are treated as peers.The framework adds auxiliary high-level layers to a target network and designs two online teachers.
  • Proposed method: The peer ensemble teacher assembles peer feature representations after multiple random input augmentations using an additional classifier.This design supports knowledge transfer from an online high-capacity teacher while further optimising the ensemble teacher.
  • Results: PCL combines the peer ensemble teacher with temporal mean peer teachers to improve peer learning and model generalisation.The contributions include unified peer collaboration and experiments across CIFAR-10, CIFAR-100, and ImageNet.

Related Work

Prior work studies knowledge distillation and neural-network ensembling, including online methods that aggregate predictions or exploit temporal models. PCL instead uses peer features with an additional classifier and combines this ensemble teacher with peer mean teachers.

  • Knowledge distillation: Traditional knowledge distillation transfers soft predictions or intermediate representations from a high-capacity teacher to a compact student.The objective is model compression with competitive performance for deployment.
  • Online distillation: Online distillation methods use parallel networks or aggregate branch logits, but logit aggregation can impede ensemble-teacher optimisation and omit branch collaboration.These limitations motivate a feature-based collaborative framework.
  • Neural-network ensembling: Neural-network ensembling improves generalisation but traditionally requires multiple networks, increasing training time and computational cost.Temporal ensembling instead uses models or predictions from different training phases.
  • PCL distinction: PCL assembles peer feature representations from a multi-branch network with an additional classifier rather than using predictions from different phases or separately generated networks.Its peer mean teacher retains the collaborative role of Mean Teacher within online distillation.

Peer Collaborative Learning

PCL trains multiple peers in one shared network and uses two online teachers: a feature-based peer ensemble teacher and temporal mean teachers for peer collaboration. Random augmentation diversifies peer inputs, while the combined objective supports classification, ensemble distillation, and peer-mean distillation without extra inference cost for a single-peer deployment.

  • PCL shares low-level layers across peers and separates their high-level layers to reduce training cost and improve collaboration.
  • The training objective combines peer and ensemble classification losses with peer ensemble teacher distillation and peer mean teacher distillation.
  • A single peer mean model is deployed as the target model without extra inference cost, while peer mean teachers plus the additional classifier can form PCL-E for higher-capacity deployment.
  • Input Augmentation for Peers: Random augmentation generates multiple counterparts of each input, assigning one to each peer to reduce homogenisation and assemble discriminative features.
  • Online Ensembling: The peer ensemble teacher assembles peer feature representations with an additional classifier, enabling online distillation without additional convolutional operations or multiple networks.
  • Peer Mean Teacher: Temporal mean models serve as peer mean teachers, stabilising soft predictions and supporting collaborative distillation among peers.

Experiment

Experiments evaluate PCL across datasets, backbones, online distillation baselines, components, ensembles, and training choices. Results show improved generalisation, stronger ensembles, stable optimisation, and competitive performance with limited deployment overhead.

  • Experimental Setup: PCL is evaluated on CIFAR-10, CIFAR-100, and ImageNet using multiple backbone networks and online knowledge-distillation competitors.The evaluation includes backbone baselines and six online KD state-of-the-art methods under comparable branch settings.
  • Comparison with State-of-the-Art: PCL improves state-of-the-art top-1 error rates by approximately 0.1–0.3% on CIFAR-10 and 0.3–1.1% on CIFAR-100 across ResNet-32 and ResNet-110.On ImageNet, PCL improves the ResNet-18 baseline by approximately 0.9% and achieves about 29.6% top-1 error against alternative methods.
  • Component Effectiveness Evaluation: The peer ensemble teacher improves the backbone by about 2.6%, while adding the peer mean teacher yields a further improvement of about 1.1%.Replacing either proposed teacher with contemporary variants degrades performance, and the full model achieves the best component-evaluation result.
  • Component Effectiveness Evaluation: PCL’s full model reaches better generalisation with rapidly decreasing test error early in training, whereas removing the peer mean teacher causes stronger fluctuations.The reported comparison uses ResNet-110 on CIFAR-100.
  • Ensemble Effectiveness Evaluation: PCL-E improves competing online KD ensembles by about 0.3% on CIFAR-10 and 0.6% on CIFAR-100, adding only 0.01M and 0.08M parameters over ONE-E.The comparison includes ONE-E, FFL, and OKDDip-E with ResNet-110.
  • Further Analysis and Discussion: PCL outperforms traditional two-stage distillation without pre-training a high-capacity teacher, while four branches remain competitive and removing input augmentation reduces performance by approximately 0.7%.The four-branch setting reports 19.8% versus OKDDip’s reported 21.1%.

Conclusion

PCL integrates online ensembling and network collaboration for one-stage, end-to-end online knowledge distillation. Experiments across backbone networks show superiority over state-of-the-art methods on CIFAR-10, CIFAR-100, and ImageNet.

  • PCL combines peer ensemble and peer mean teachers to improve online knowledge distillation quality in a one-stage, end-to-end trainable framework.
  • Experiments with various backbone networks show PCL outperforms state-of-the-art methods on CIFAR-10, CIFAR-100, and ImageNet.
Loading 2006.04147v2…