Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition

1 INTRODUCTION

• Existing online KD methods focus on peer class probabilities and often ignore feature information valuable for distillation.Feature embeddings are described as containing structured knowledge that can capture dependencies among networks.Source: §1 · p. 1
• MCL transfers contrastive distributions among networks, with interactive contrastive learning aggregating cross-network embeddings and maximizing a lower bound on mutual information.Unlike vanilla contrastive learning, ICL forms contrastive similarity distributions across different networks’ embedding spaces.Source: §1 · p. 2
• Soft ICL labels aggregate cross-network embeddings and are described as more informative than soft VCL labels formed within one network.The paper characterizes soft contrastive distributions as labels that can supervise peer networks.Source: §1 · p. 2
• Layer-wise MCL reports consistent gains over baseline methods and outperformance of other online KD methods on CIFAR-100, ImageNet, and COCO transfer tasks.Collaborative learning occurs during training; any cohort network can be kept for inference without additional inference cost over the original network.Source: §1 · pp. 2–3
• Layer-wise MCL extends contrastive learning to intermediate features and uses meta-optimization to learn adaptive layer matching.The paper also describes a gated module that aggregates intermediate logit distributions into a virtual peer teacher.Source: §1 · p. 2
• The paper evaluates layer-wise MCL across architectures and transfer settings, including object detection and instance segmentation.It also reports additional ablation studies and analyses of the method’s components.Source: §1 · pp. 2–3

2.1 Traditional Offline Knowledge Distillation

• Offline KD transfers teacher knowledge such as predictions, feature maps, attention maps, and structural relations, but requires pretraining a high-capacity teacher before distillation.The two-stage pipeline is described as time-consuming.Source: §2.1 · p. 3

2.2 Online Knowledge Distillation

• Prior online KD methods commonly distill logits using mutual learning or online teacher aggregation, while MCL adds collaborative representation learning and can be combined with logit-based methods.The cited approaches include peer mutual learning and weighted ensemble teacher strategies.Source: §2.2 · p. 3

2.3 Contrastive Learning

• Contrastive learning pulls positive pairs together and pushes negative pairs apart; MCL applies this representation-learning approach to multi-peer interaction in online KD.The related work also describes supervised and self-supervised ways of forming positive and negative pairs.Source: §2.3 · p. 3

2.4 Embedding-based Relational Distillation

• Prior embedding-based contrastive distributions are often constructed within one network, whereas MCL aggregates cross-network embeddings to model interaction and maximize mutual information between networks.Fig. 2 is identified as an overview of MCL between two networks.Source: §2.4 · pp. 3–4, Figure 2 · p. 4

3.1 Architecture Formulation

• Each network gains a projection module that maps extracted features to latent embeddings for contrastive learning; at inference, the modules are discarded and one network is retained.The retained network has the original architecture, so the method introduces no extra inference cost.Source: §3.1 · p. 4

3.2 Vanilla Contrastive Learning (VCL)

• VCL forms a softmax distribution over similarities between an anchor embedding and one positive plus K negative embeddings, then uses cross-entropy to pull the positive close and push negatives away.In supervised learning, the positive is from the anchor’s class and negatives are from different classes; the resulting loss is equivalent to a (K + 1)-way softmax classification loss.Source: §3.2 · p. 4

3.3 Interactive Contrastive Learning (ICL)

• ICL forms contrastive distributions using embeddings from another network rather than the anchor network’s own embedding space, modeling cross-network dependencies.The method performs ICL for pairs of networks and extends it across the cohort.Source: §3.3 · pp. 4–5
• ICL’s contrastive objective is attributed to maximizing a lower bound on mutual information between networks, allowing each to gain contrastive knowledge from others.The passage states that this mutual information increases with K, indicating more common knowledge can be learned between the networks.Source: §3.3 · p. 5

3.4 Soft Contrastive Learning with Online Mutual Mimicry

• Online mutual mimicry treats contrastive distributions as soft labels and uses KL divergence to align each network’s distributions with peer-provided labels.The paper applies this alignment to distributions from both VCL and ICL.Source: §3.4 · p. 5

3.4.1 Soft Vanilla Contrastive Learning (Soft VCL)

• Soft VCL uses the other networks’ contrastive distributions as detached soft labels to align each network’s own VCL distribution.Detaching the soft labels is specified as a stability measure.Source: §3.4.1 · p. 5

3.4.2 Soft Interactive Contrastive Learning (Soft ICL)

• Soft ICL aligns the interactive distributions in both directions between network pairs, using the peer distribution as a detached soft label for stability.The alignment is extended across every pair in the cohort.Source: §3.4.2 · p. 5

3.4.3 Discussion with Soft VCL and Soft ICL

• Unlike Soft VCL’s single-network-space distributions, Soft ICL’s interactive distributions aggregate cross-network embeddings and are described as more informative.The paper reports that Soft ICL empirically yields greater performance gains than Soft VCL.Source: §3.4.3 · p. 5

4.2 Layer-wise MCL

• Adaptive weighted layer association addresses semantic-level differences between peer networks, particularly across distinct architectures.Rather than matching only same-staged layers one-to-one, a feature layer can weight its associations with layers from another network.Source: §4.2 · p. 6, Figure 3 · p. 6
• L-MCL extends mutual contrastive learning from final embeddings to intermediate and final embeddings in a cross-layer manner.The original MCL uses final-layer embeddings, while L-MCL also incorporates intermediate representations.Source: §4 · p. 5, §4.2 · p. 6

4.3 Training Meta-Networks

• The total feature-level online KD loss combines the basic task loss with the layer-wise MCL loss.The task loss supplies task-aware supervision alongside feature-level mutual contrastive learning.Source: §4.3 · p. 6
• Meta-optimization alternates network-parameter updates with a three-stage bilevel procedure that updates the meta-network using task loss.The stages optimize the networks with L-MCL, adapt them once using task loss, then update the meta-network based on the adapted task loss.Source: §4.3 · p. 7
• A meta-network learns matching weights by projecting and normalizing embeddings, then applying dot-product similarity and a sigmoid.The resulting weights parameterize the layer associations used for layer-wise MCL.Source: §4.2 · p. 6, §4.3 · p. 7

4.4 Logit-level Online Knowledge Distillation

• A learnable gate aggregates each network’s multi-branch logits, which are then mutually distilled among networks using KL-divergence.The gate derives branch weights from concatenated feature embeddings; other networks’ ensemble logits supervise each network.Source: §4.4 · p. 7

4.5.1 Overall loss

• The overall loss combines task loss with embedding-based layer-wise MCL, without extra weights for cross-entropy and KL-divergence.The authors state that these entropy-based losses often have the same magnitude.Source: §4.5.1 · p. 8

4.5.2 Contrastive sample mining

• Batch-based mining uses paired samples from each of B/2 classes, while memory-based mining retrieves past embeddings to expand the contrastive sample pool.The paper uses batch-based mining for CIFAR-100 and memory-based mining for ImageNet.Source: §4.5.2 · p. 8

4.5.3 Training complexity

• The overall computational complexity is O(M^2(L^2Kd + C)), with extra computational costs arising mainly from feature-level contrastive learning.Here, M is the number of networks, L the number of network layers, d the embedding dimension, K the number of negative samples, and C the number of classes.Source: §4.5.3 · p. 8
• 20 min/epoch and 15.8 GB/GPU for L-MCL, versus 14 min/epoch and 12 GB/GPU for independently trained baseline networks.The reported comparison uses two ResNet-50 networks trained for ImageNet classification.Source: §4.5.3 · p. 8

5.1 Results on CIFAR-100 Classification

• On CIFAR-100, L-MCL beats PCL by 1.28%, 1.31%, 1.77%, and 0.86% on ResNets, WRNs, ShuffleNet, and HCGNet, respectively.Against independent training, its average gains are 4.49%, 4.41%, 5.77%, and 4.19% for those families. With three networks, it beats PCL by 1.36%, 1.52%, and 1.34% on ResNet, WRN, and ShuffleNet; few-shot margins over PCL are 1.73%, 1.18%, and 1.26% with 25%, 50%, and 75% of training samples. Gains increase only 0.35% on average across six architectures when moving from two to three networks.Source: §5.1 · pp. 9–10, Table 1 · p. 9, Table 3 · p. 10
• For different-architecture CIFAR-100 pairs, L-MCL beats PCL by average accuracy margins of 0.92% on Net1 and 0.89% on higher-capacity Net2.Higher-capacity networks still benefit from lower-capacity partners; for example, ResNet-110 gains 4.42% with ResNet-32, and L-MCL can pair a target network with a lightweight partner.Source: §5.1 · pp. 9–10, Table 2 · p. 9

5.2 Results of Transfer Learning to Image Classification on STL-10 and TinyImageNet

• On STL-10 and TinyImageNet transfer, L-MCL exceeds the baseline by average margins of 4.86% and 4.89%, respectively, while other online KD methods do not effectively improve downstream classification.The evaluation freezes CIFAR-100-pretrained features and trains a linear classifier on each target dataset.Source: Table 4 · p. 10, §5.2 · p. 11

5.3 Results on ImageNet Classification

• On ImageNet, L-MCL beats CKD-MKT by an average 0.78% and improves over the baseline by 1.43%, 1.44%, 2.07%, and 1.66% on ResNet-18, ResNet-34, ResNet-50, and ShuffleV2.For different-architecture pairs, it reaches 66.44% on ShuffleNetV2 and 71.69% on ResNet-18, beating CKD-MKT by 1.01% and 0.67%; on Swin-Tiny and Swin-Small it gains 0.86% and 0.54%, versus DML's roughly 0.1%–0.2% gains. L-MCL also improves over prior MCL by an average 0.80%.Source: §5.3 · pp. 11–12, Table 5 · p. 11, Table 6 · p. 11

5.4 Results of Transfer Learning to Object Detection and Instance Segmentation on COCO-2017

• On COCO-2017 transfer, L-MCL improves average mAP over the baseline by 1.7 and 1.8 points for detection and segmentation, and over CKD-MKT by 0.8 and 0.9 points.The pretrained ImageNet ResNets serve as Mask-RCNN backbones.Source: §5.4 · p. 12, Table 8 · p. 12

5.5 Ablation Study and Parameter Analysis

• On CIFAR-100, ICL gains 3.05% and 2.40% over baseline on ResNet-110 and WRN-40-2, versus VCL's 2.06% and 0.77%; combining feature L-MCL and logit KD yields gains of 4.19% and 4.07%.ICL with its soft labels exceeds VCL with its soft labels by 1.20% and 1.21%; feature-based L-MCL alone gains 3.76% and 2.95%, compared with 1.84% and 1.75% from logit-level KD. Removing gated aggregation drops performance by 0.67% and 0.79%.Source: §5.5 · pp. 12–13, Table 9 · p. 12, Table 10 · p. 12
• Weighted-all-to-all layer matching, whose weights are learned by meta-optimization, performs best among the three tested matching mechanisms and adaptively associates layers across architectures.The learned weights favor same-level layers while assigning moderate weights to others, supporting data-driven matching that does not require architecture-specific manual pairing.Source: §5.5 · p. 13, Table 11 · p. 13
• Under the same K, mini-batch mining generally outperforms memory-bank mining, and its upper bound is higher on CIFAR-100; on ImageNet, memory-bank mining with more negatives breaks the mini-batch upper bound.K saturates at a capacity for the memory bank; the study uses K = 126 for CIFAR-100 and K = 8192 for ImageNet. Accuracy rises from embedding dimension 32, plateaus at 128, and saturates at 256; α ∈ [0.05, 0.4] works well, with performance declining above 0.4.Source: §5.5 · pp. 13–14
• The selected settings are α = 0.1 and β = 1.0, with KL divergence-based β working well toward a weight of 1.0.Source: §5.5 · p. 14
• MCL makes two ResNet-32 embedding distributions more similar and more discriminative than independent training, while improving accuracy.The embedding spaces are visualized in Fig. 8.Source: §5.5 · p. 14, Figure 8 · p. 14

6 CONCLUSION

• Layer-wise MCL applies contrastive learning to intermediate and final feature layers to improve knowledge interaction among networks and maximize mutual information between them.The paper reports that its experiments across image classification and object detection show better features and performance than popular logit-level online KD approaches.Source: §6 · p. 14

A.1 Proof of Maximizing the Lower bound of the Mutual Information

• The distribution q uses indicator C to distinguish pairs drawn from the joint distribution from pairs drawn from the product of marginals.Source: §A.1 · p. 17
• With one positive pair per K negative pairs, the prior probabilities are q(C = 1) = 1/(1 + K) and q(C = 0) = K/(1 + K).Source: §A.1 · p. 17
• The proof derives the positive-pair class posterior by Bayes’ rule and connects expectations over its log posterior to mutual information.The ICL loss is described as the negative log class posterior of the positive pair.Source: §A.1 · p. 17

B.1 Algorithm of on CIFAR-100

• For CIFAR-100-like small-scale datasets, MCL performs batch-based mining.Source: §B.1 · p. 17

B.2 Algorithm of on ImageNet

• For ImageNet-100-like large-scale datasets, MCL performs memory-based mining.Source: §B.2 · p. 17

• The overview depicts feature-based layer-wise MCL and logit-based distillation over two networks, f1 and f2.Source: Appendix C · p. 17, Figure 9 · p. 18