Collaborative Learning for Deep Neural Networks

Guocong Song; Wei Chai

Source-linked AI summary

Collaborative Learning for Deep Neural Networks

Guocong Song, Wei Chai

arXiv:1805.11761v2May 30, 2018stat.MLcs.CVcs.LG

TL;DR

Deep neural network training faces difficult nonconvex optimization, while conventional ensembles improve predictions at excessive inference cost. The paper proposes collaborative learning, simultaneously training multiple heads with consensus, ILR sharing, and backpropagation rescaling. Across CIFAR and ImageNet, it reports improved generalization and label-noise robustness without extra inference cost.

Problem
Deep networks face nonconvex optimization challenges, and ensembles that improve predictions are too computationally expensive at inference time.
Method
Collaborative learning simultaneously trains multiple classifier heads on shared data, using peer consensus, ILR sharing, and backpropagation rescaling while preserving individual-learning hyperparameters.
Results
Collaborative learning significantly reduces generalization error and improves robustness to label noise across CIFAR and ImageNet, without extra inference cost.
Takeaways & Limitations
The framework provides a flexible end-to-end training approach that improves target-network performance while keeping the inference graph unchanged.
Takeaways & Limitations
The experiments fix β and T, and more extensive hyperparameter searches may further improve performance on specific datasets.

Abstract

from arXiv · show

We introduce collaborative learning in which multiple classifier heads of the same network are simultaneously trained on the same training data to improve generalization and robustness to label noise with no extra inference cost. It acquires the strengths from auxiliary training, multi-task learning and knowledge distillation. There are two important mechanisms involved in collaborative learning. First, the consensus of multiple views from different classifier heads on the same example provides supplementary information as well as regularization to each classifier, thereby improving generalization. Second, intermediate-level representation (ILR) sharing with backpropagation rescaling aggregates the gradient flows from all heads, which not only reduces training computational complexity, but also facilitates supervision to the shared layers. The empirical results on CIFAR and ImageNet datasets demonstrate that deep neural networks learned as a group in a collaborative way significantly reduce the generalization error and increase the robustness to label noise.

1 Introduction

Collaborative learning trains multiple classifier heads within one network to improve accuracy and generalization without changing inference cost. Its consensus and shared-representation mechanisms also support robustness and efficient training.

Collaborative learning trains several classifier heads of the same network simultaneously on the same training data.
The framework combines auxiliary training, multi-task learning, and knowledge distillation while keeping the inference graph unchanged.
Consensus among classifier heads supplies supplementary information and regularization that improves each classifier’s generalization.
ILR sharing and backpropagation rescaling aggregate gradient flows, reduce training cost, and facilitate supervision of shared layers.
Collaborative learning is reported to improve accuracy without extra inference cost, require no additional model architecture, and remain robust to label noise.

2 Related work

Prior approaches improve training or prediction through auxiliary classifiers, distillation, label smoothing, or temporal consensus, but may add architectural, training-time, or inference costs. The paper instead emphasizes improving a target network without changing its inference graph while retaining training efficiency.

Auxiliary training adds classifiers at intermediate layers but requires specific additional network designs.
Label smoothing reduces vulnerability to noisy labels by replacing hard one-hot targets with smoothed values and relaxing label confidence.
Temporal ensembling forms consensus predictions across epochs but is difficult to scale because it requires memorizing smoothed labels.
Two-way distillation alternately optimizes networks, while the paper notes that these algorithms are far from optimized.
Self-distillation can require repeated training and increase inference time when multiple student generations are ensembled.
The paper’s goal is to improve target-network accuracy without changing its inference graph while emphasizing training efficiency.

3 Collaborative learning

Collaborative learning trains multiple classifier heads within one network, using shared intermediate representations and peer predictions while retaining the original inference graph. Simultaneous optimization and backpropagation rescaling address efficiency and stability, while diverse head predictions support robustness to label noise.

The framework generates a population of classifier heads in the training graph, formulates a joint objective, and optimizes the classifiers collaboratively.
3.1 Generation of training graph: Heads may be independent copies, share low-level intermediate representations, or share multiple hierarchical representations.At inference, one head and its dependent nodes are retained while the other training-time components are discarded.
3.1 Generation of training graph: ILR sharing reduces training memory consumption and speeds training compared with multiple unshared network instances.The training graph's parameter count is proportional to the number of heads without sharing, whereas shared representations reduce this cost proportionally.
3.2 Learning objectives: Each head learns from ground-truth labels and a consensus soft label formed from the other heads' predictions.The soft-label objective measures the discrepancy between the population's average prediction and each head's prediction, regularizing training.
3.3 Optimization for a group of classifier heads: The method applies simultaneous SGD to update all head parameters using the summed head losses and regularization.This avoids the repeated prediction recalculation associated with alternative one-head-at-a-time updates and retains individual-learning hyperparameters.
3.3 Optimization for a group of classifier heads: Without rescaling, shared-layer backpropagation sums gradients from all heads, causing variance to grow with the number of heads and making the flow asymptotically unstable.Backpropagation rescaling inserts an operation between shared and head-specific subnets to normalize the shared-layer flow while preserving the single-classifier flow in the head-specific subnet.
3.4 Robustness to label noise: Diverse predictions across heads make collaborative learning more robust to incorrect labels and difficult data augmentations than individual learning.The paper states that this robustness is validated experimentally in Section 4.1.

4 Experiments

Experiments on CIFAR and ImageNet show that collaborative learning improves classification while reducing training resource demands and preserving inference cost. The results also support simultaneous optimization, backpropagation rescaling, robustness to noisy labels, and ILR-sharing mechanisms.

CIFAR Datasets: Collaborative learning lowers CIFAR generalization error as the number of classifier heads increases, while ILR sharing also reduces GPU memory consumption and training time.The CIFAR experiments use ResNet-32, ResNet-110, and DenseNet-40-12.
CIFAR Datasets: Simultaneous optimization provides an additional 1%+ accuracy gain over alternative optimization on CIFAR-100, with T = 2 producing another 1% boost.The comparison uses two ResNet-32 instances with T = 1 and β = 0.5, then evaluates T = 2.
CIFAR Datasets: Backpropagation rescaling outperforms no scaling and loss scaling for four-head ResNet-32 training with shared low layers.No scaling produces overly large shared-layer gradients, whereas loss scaling uses a 1/4 loss factor and can make independent-layer updates too small.
CIFAR Datasets: Collaborative learning substantially lowers CIFAR-10 test error relative to the baseline under label noise, with larger gains at higher noise levels.Noisy labels are randomly regenerated every epoch, and four hierarchical heads can become worse at high noise levels than configurations retaining greater prediction diversity.
ImageNet Dataset: On ImageNet, validation top-1 error falls from 23.47% for the baseline to 22.70% with two simple-ILR-sharing heads and 22.29% with four hierarchical heads.The ImageNet experiment uses ResNet-50; collaborative learning also reduces training memory and time through shared layers while leaving inference cost unchanged.
ImageNet Dataset: ILR sharing reduces near-zero dead weights relative to distillation, while consensus across views centralizes weight distributions and provides additional regularization.The reported weight-distribution analysis concerns trained ResNet-50 models and attributes the regularization to consensus from multiple views of the same data.

5 Conclusion

Collaborative learning trains multiple generated classifiers together, using consensus and shared gradient flows to improve generalization while lowering training cost. The authors report better performance and suggest extensions to noisy-label and other learning settings.

Consensus among classifier heads supplies supplementary information and regularization, improving generalization.
ILR sharing with backpropagation rescaling aggregates gradient flows, lowering training cost and supervising shared layers.
Empirical results validate simultaneous optimization and backpropagation rescaling in group learning.
Collaborative learning is presented as a flexible end-to-end approach for improving deep neural network performance.
The authors suggest potential benefits for semi-supervised learning and regression, based on group collaboration and noisy-label resistance.

Supplementary Material of Collective Training for

The supplementary material addresses the finiteness of gradient variance under backpropagation rescaling for an arbitrary number of classifier heads. The argument invokes the Cauchy–Schwarz inequality and finite per-head variances.

The supplementary material proves that gradient variance with backpropagation rescaling remains finite for an arbitrary number of heads H.
The proof establishes an intermediate variance equivalence before applying a bound.
Inequality (3) follows from the Cauchy–Schwarz inequality.
The finiteness conclusion assumes Var(X_h) < ∞ for every head h.

2 Training setup for CIFAR

CIFAR experiments use standard augmentation and evaluate ResNet-32, ResNet-110, and DenseNet-40-12 with SGD-based training schedules. The networks use weight decay, Nesterov momentum, and scheduled learning-rate reductions.

CIFAR training uses standard data augmentation and evaluates ResNet-32, ResNet-110, and DenseNet-40-12.
All CIFAR networks use weight decay of 10^-4 and Nesterov momentum of 0.9 with SGD.
ResNet-32 and ResNet-110 use mini-batches of 128 and train for up to 200 epochs.
Their learning rate starts at 0.1 and is divided by 10 at epochs 100, 150, and 192.

3 Training setup for ImageNet

ImageNet training uses augmented 224x224 crops, horizontal flips, per-color normalization, and a ResNet-50 trained with momentum SGD and scheduled learning-rate reductions.

Each ImageNet input is a 224x224 random crop from an augmented image or its horizontal flip.
ResNet-50 is trained for up to 100 epochs with Nesterov momentum of 0.9 and weight decay of 10^-4.
The learning rate starts at 0.1 and is divided by 10 at epochs 30, 60, and 90.
Validation uses a single central crop with size 224x224.

4 Training time of distillation

Distillation training time is the sum of teacher training, student training, and teacher forward-passing time. For a ResNet-50 distilled from two ResNet-50 teachers, this totals roughly 3.4 times individual learning.

Training-time decomposition: Distillation training time combines teacher training, student training, and teacher forward-passing time.The paper expresses this relationship as Ttrain = Tt + Ts + Ttf.
Example estimate: 3.4x total training time results when distilling a ResNet-50 from an ensemble of two ResNet-50s.In this example, teacher training takes twice the student training time and teacher forward passing takes approximately 0.4 times the student training time.
Example estimate: Teacher forward passing contributes approximately 0.4 times the student training time in the ResNet-50 example.

5 Details of ResNet-50 weight distribution

The paper examines per-layer weight distributions and standard deviations in trained ResNet-50 models across training approaches. Reducing weight decay in early layers reduces the near-zero weight spike.

Weight-decay experiment: Reducing weight decay to 0.5 · 10^-4 in early ResNet-50 layers reduces the near-zero spike in weight distributions.The reduced value applies to conv1, conv2_x, and conv3_x, while other layers retain 1 · 10^-4.
Weight-decay experiment: The experiment targets near-zero “dead” weights that may result when vanished gradients leave weight decay as the dominant influence.

6 Impact of hyperparameters on accuracy on CIFAR-10

The experiments examine how accuracy changes with β and T and how split-point location affects ResNet-32 performance on CIFAR-10. Error is not sensitive to the tested β and T settings.

β and T: CIFAR-10 error is not sensitive to the tested β and T values.The reported settings are β = 0.5 and T = 2.
β and T: Careful tuning of β and T may improve results beyond the current settings, but the expected improvement is small.
Split-point location: The study also evaluates different split-point locations in ResNet-32 with two-head simple ILR sharing on CIFAR-10.Results are summarized in Table 2.