Source-linked AI summary
Deep Mutual Learning
Ying Zhang, Tao Xiang, Timothy M. Hospedales, Huchuan Lu
TL;DR
Large neural networks can be difficult to deploy because they require substantial memory or execution time, motivating compact yet accurate models. The paper proposes deep mutual learning, in which untrained peer students jointly combine supervised and peer-mimicry losses, and reports improved performance over independent learning and static-teacher distillation. The results support mutual learning across architectures and cohort settings without requiring a prior powerful teacher.
Problem
Large neural networks may be too slow or memory-intensive for some applications, while conventional distillation depends on a powerful pre-trained teacher to train smaller students.
Method
Deep mutual learning jointly trains untrained students with supervised losses and mimicry losses aligning each student’s class posterior with those of other cohort members.
Results
Deep mutual learning improves students over independent training and conventional distillation, with benefits across network architectures, cohort sizes, CIFAR-100, and Market-1501 settings.
Takeaways & Limitations
A prior powerful teacher is unnecessary: peer-trained compact networks can outperform students distilled from a static teacher, while mutual learning also benefits large networks and cohort ensembles.
Abstract
from arXiv · showhide
Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, that is better suited to low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy where, rather than one way transfer between a static pre-defined teacher and a student, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on CIFAR-100 recognition and Market-1501 person re-identification benchmarks. Surprisingly, it is revealed that no prior powerful teacher network is necessary -- mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher.
1 Introduction
Deep mutual learning replaces one-way distillation from a fixed powerful teacher with collaborative training among untrained peer students. The approach improves students over independent learning and conventional distillation, while extending to varied cohort sizes and architectures.
- Motivation: The method addresses the deployment problem of large networks, whose depth, width, and parameter counts can make them slow or memory-intensive.This motivates research into compact, accurate models through architecture design, compression, pruning, binarisation, and distillation.
- Approach: Deep mutual learning trains a pool of untrained students simultaneously, combining supervised learning with mimicry losses that align each student’s class posterior with those of its peers.Unlike conventional distillation, knowledge transfer is bidirectional and occurs throughout training rather than from a pre-trained static teacher.
- Approach: Deep mutual learning improves each student over conventional supervised learning and can outperform students distilled from a larger pre-trained teacher.The comparison directly challenges the assumption that a more powerful static teacher is required.
- Mechanism: Mutual learning’s additional information comes from peer networks’ differing probability estimates for secondary classes, while supervised loss anchors predictions to true labels.Different initial conditions create varied secondary-class probabilities that provide distillation-like supervision without a pre-existing teacher.
- Scope: The approach improves generalisation across cohort sizes and architectures, benefits heterogeneous cohorts, and also helps large networks compared with independent training.The efficacy increases with the number of networks, and the full cohort can additionally serve as an ensemble.
- Positioning: Mutual learning dispenses with the powerful teacher traditionally used in distillation by allowing an ensemble of students to teach one another.The paper positions this as a general-classification alternative to related interactive learning approaches restricted to special translation settings.
2 Deep Mutual Learning
Deep Mutual Learning trains a cohort of networks jointly, combining supervised classification with peer-matching losses and extending naturally beyond two networks. Each model is updated from its peers’ predictions throughout training, while ensemble-teacher averaging can reduce performance by producing overly peaked posteriors.
- Formulation: The supervised objective for multi-class classification is the cross-entropy error between predicted values and correct labels.
- Formulation: DML trains each network with both a supervised loss for true labels and a KL-based mimicry loss matching a peer’s probability estimates.The peer posterior supplies additional training experience alongside conventional supervised learning.
- Optimization: At every mini-batch update, both networks exchange predictions and iteratively update their parameters until convergence.The procedure starts from differently initialized models and repeats prediction, gradient computation, and parameter updates.
- Larger Student Cohorts: With K networks, each model uses the other K −1 networks as teachers, with the mimicry term averaged by 1/(K−1).The multi-network extension can distribute models across devices while passing small probability vectors between them.
- Larger Student Cohorts: An alternative ensemble-teacher variant performs worse because averaging peers makes posterior probabilities more peaked at the true class and reduces posterior entropy.This conflicts with DML’s objective of producing robust solutions with high posterior entropy.
3 Experiments
Experiments evaluate DML across datasets, architectures, cohort sizes, ensembles, and comparisons with distillation. Results show consistent gains for collaborative training and suggest more robust solutions.
- Datasets and Settings: Experiments use CIFAR-100 Top-1 accuracy and Market-1501 person re-identification, evaluated with classification and feature-matching protocols.CIFAR-100 contains 100 classes; Market-1501 uses 751-way classification and nearest-neighbour matching from final-layer features.
- Results on CIFAR-100: All tested CIFAR-100 network combinations improve over independent learning, with smaller networks generally benefiting more from DML.WRN-28-10 also benefits when trained with a smaller peer, and cohorts of large networks remain beneficial.
- Results on Market-1501: DML greatly improves MobileNet performance on Market-1501, with and without ImageNet pre-training, and two-MobileNet DML surpasses prior deep-learning methods.The reported comparison uses mAP and rank-1 accuracy for two-network cohorts.
- DML with Larger Student Cohorts: On Market-1501, the average individual-network mAP increases as the DML cohort grows, and ensemble predictions benefit from cohort training as well.Ensemble predictions outperform individual predictions, while DML improves independent ensembles with minimal additional cost when ensembles are already used.
- How and Why does DML Work?: DML models reach training accuracy of 100% yet show greater resistance to parameter noise, suggesting wider minima associated with better test generalisation.For MobileNet on Market-1501, perturbing parameters raises independent-model loss more than DML-model loss; DML also aggregates secondary class probabilities.
4 Conclusion
The paper proposes peer-based mutual distillation as a simple, generally applicable way to improve deep networks. It yields compact networks that outperform those distilled from a strong static teacher and can also benefit large networks and ensembles.
- Peer-based mutual distillation improves deep neural network performance through collaborative cohort training.
- Compact networks trained with peers can outperform networks distilled from a strong static teacher.
- The approach also improves large powerful networks, and the trained cohort can be combined as an ensemble for further gains.