PaperlayerView exact revision on arXiv

Source-linked AI summary

MEAL: Multi-Model Ensemble via Adversarial Learning

Zhiqiang Shen, Zhankui He, Xiangyang Xue

arXiv:1812.02425v2cs.CVcs.AIcs.LG

TL;DR

Traditional ensembles improve neural-network performance but require storing and executing multiple models, limiting their use in large-scale or resource-constrained settings. MEAL distills diverse teacher knowledge into one student through soft-label supervision, intermediate losses, and adversarial learning. Across several benchmarks, it reports improved accuracy with single-network inference, including 21.79% Top-1 and 5.99% Top-5 ImageNet validation error.

  • Problem

    Traditional ensembles can be accurate and robust but require redundant models, extra computation, and substantial storage, limiting large-scale and resource-constrained deployment.

  • Method

    MEAL transfers diverse outputs from multiple teacher networks into a student using soft-label supervision, intermediate feature alignment, and adversarial discriminators.

  • Results

    MEAL consistently improves accuracy across CIFAR-10/100, SVHN, and ImageNet while using a single inference network; its ImageNet result is 21.79% Top-1 and 5.99% Top-5 validation error.

  • Takeaways & Limitations

    The method compresses multi-model knowledge into one network while retaining ensemble benefits without additional testing cost.

Abstract

from arXiv · show

Often the best performing deep neural models are ensembles of multiple base-level networks. Unfortunately, the space required to store these many networks, and the time required to execute them at test-time, prohibits their use in applications where test sets are large (e.g., ImageNet). In this paper, we present a method for compressing large, complex trained ensembles into a single network, where knowledge from a variety of trained deep neural networks (DNNs) is distilled and transferred to a single DNN. In order to distill diverse knowledge from different trained (teacher) models, we propose to use adversarial-based learning strategy where we define a block-wise training loss to guide and optimize the predefined student network to recover the knowledge in teacher models, and to promote the discriminator network to distinguish teacher vs. student features simultaneously. The proposed ensemble method (MEAL) of transferring distilled knowledge with adversarial learning exhibits three important advantages: (1) the student network that learns the distilled knowledge with discriminators is optimized better than the original model; (2) fast inference is realized by a single forward pass, while the performance is even better than traditional ensembles from multi-original models; (3) the student network can learn the distilled knowledge from a teacher model that has arbitrary structures. Extensive experiments on CIFAR-10/100, SVHN and ImageNet datasets demonstrate the effectiveness of our MEAL method. On ImageNet, our ResNet-50 based MEAL achieves top-1/5 21.79%/5.99% val error, which outperforms the original model by 2.06%/1.14%. Code and models are available at: https://github.com/AaronHeee/MEAL

1. Introduction

MEAL compresses ensemble knowledge into one student network by using diverse teacher outputs, soft labels, and adversarial learning, avoiding the inference cost of traditional ensembles. Experiments report improved accuracy across datasets and architectures.

  • Approach: MEAL forms ensemble supervision from outputs of different pretrained networks while testing with a single network.The method replaces traditional one-hot labels with soft labels that capture visually related objects and scenes.
  • Motivation: Traditional ensembles improve robustness and accuracy but require redundant models, extra computation, and substantial storage.These costs limit use on mobile, embedded, real-time, and other resource-constrained applications.
  • Approach: Soft labels provide image-specific distributions that encode additional intra- and inter-category relations.For example, visually different tobacco-shop images can receive distributions reflecting similarity to libraries or grocery stores.
  • Results: Adversarial learning forces student outputs to resemble teacher outputs and improves robustness across architectures and datasets.The reported experiments cover CIFAR-10/100, SVHN, and ImageNet.
  • Contributions: MEAL targets an end-to-end teacher-student framework that ensembles multiple networks without additional testing cost.The stated contributions also include improved accuracy across CIFAR-10/100, SVHN, and ImageNet for varied architectures.

2. Related Work

Prior work includes explicit and implicit neural-network ensembles, knowledge distillation, and adversarial learning. MEAL focuses on transferring ensemble knowledge while reducing the testing cost of explicit ensembles.

  • Explicit ensembles: Earlier ensemble studies generally focused on improving individual-network generalization, while Snapshot Ensembles addressed training cost.MEAL instead targets the testing cost of ensembles.
  • Implicit ensembles: Implicit ensemble methods such as Dropout, Drop-Connection, Stochastic Depth, and Swapout create many shared-weight networks during training and ensemble them at test time.MEAL differs by transferring knowledge through differences in labels from trained networks.
  • Knowledge transfer: Knowledge-distillation methods transfer information between teacher and student networks, including through intermediate representations or refined labels.The cited approaches include parameter-based intermediate-layer transfer and iterative ground-truth-label updates.

3. Overview

MEAL uses a siamese-like teacher–student framework with alignment layers, similarity losses, and discriminators. Teacher and student branches may have identical or different structures, provided they have the same number of blocks for intermediate-output alignment.

  • The framework contains teacher and student branches, alignment layers, similarity-loss layers, and discriminators.
  • Teacher and student structures may differ, but they must have the same number of blocks to use intermediate outputs.
  • Intermediate outputs are aligned through adaptive pooling, then discriminators adversarially train the student to generate similar teacher features.

4. Adversarial Learning (AL) for Knowledge Distillation

The method distills teacher predictions and intermediate features into a student through similarity losses, adaptive pooling, and stacked adversarial discriminators. The student is trained against frozen teacher outputs, with multi-stage adversarial losses combined end-to-end.

  • 4.1 Similarity Measurement: The student learns from teacher-generated soft labels on the same images while the pretrained teacher parameters remain frozen.
  • 4.1 Similarity Measurement: The method evaluates ℓ1, ℓ2, and KL-divergence distances between student and teacher output probabilities.
  • 4.2 Intermediate Alignment: Adaptive pooling maps teacher and student intermediate outputs to a predefined common length, allowing different teacher networks to be aligned with the student.
  • 4.2 Intermediate Alignment: Similarity losses from selected intermediate layers are summed, using the last layer of each network block in the experiments.
  • 4.3 Adversarial Learning: Stacked discriminators distinguish teacher from student outputs, while the student learns to generate outputs that fool them.
  • 4.3 Adversarial Learning: The final framework combines similarity and adversarial losses in an end-to-end objective with trade-off weights and block-balancing coefficients.

5. Multi-Model Ensemble via Adversarial Learning (MEAL)

MEAL compresses knowledge from different network architectures into one student by training on their soft outputs, achieving ensemble behavior without additional testing cost. Its learning procedure pretrains a teacher model zoo and repeatedly trains the student against randomly selected teachers.

  • 5. Multi-Model Ensemble via Adversarial Learning: MEAL uses soft outputs from different architectures to compress ensemble knowledge into a single network without additional testing cost.
  • 5.1 Learning Procedure: The framework supports a student with the same structure as its teacher or a randomly selected teacher structure at each iteration.
  • 5.1 Learning Procedure: Training first builds and pretrains a teacher model zoo, then minimizes the MEAL loss to match teacher and student outputs.
  • 5.1 Learning Procedure: At each iteration, the procedure randomly selects a teacher and updates the student through adversarial learning.

6. Experiments and Analysis

Experiments across CIFAR-10/100, SVHN, and ImageNet evaluate MEAL's architecture choices, adversarial learning, ensemble comparisons, training settings, and learned feature structure. MEAL generally outperforms single and traditional ensembles while retaining single-network inference.

  • Experimental Setup: MEAL is evaluated on CIFAR-10/100, SVHN, and ImageNet with VGGNet, ResNet, DenseNet, MobileNet, and shake-shake architectures.The experiments use multiple teacher architectures and datasets to assess the method broadly.
  • Ablation Studies: Cross-entropy, intermediate-layer losses, adversarial learning, and average adaptive pooling produce the best ablation result.The ablation reports that adding intermediate outputs improves performance and that max pooling reaches 6.32% accuracy, much worse than average pooling.
  • Comparison with Traditional Ensemble: Cross-dataset comparisons show MEAL consistently outperforms single and traditional ensemble methods on error rate.Figure 6 compares base models, adversarially trained base models, traditional ensembles, and MEAL results.
  • Results: On ImageNet, MEAL Plus achieves 21.79% Top-1 and 5.99% Top-5 validation error, versus 23.85%/7.13% for the original ResNet-50 and 22.76%/6.49% for the traditional ensemble.The MEAL student uses ResNet-50, while VGG-19 with batch normalization and ResNet-50 serve as teachers; inference is more efficient than the traditional ensemble.
  • Analysis: An ensemble of four models is usually best, while too many training models can reduce VGGNet and DenseNet accuracy; training beyond 400 epochs appears optimal.The model fully converges at about 500 epochs on CIFAR datasets.
  • Analysis: Different architectures provide diverse soft-label distributions, and MEAL produces better ImageNet feature embeddings than the single model.Weaker network pairs show greater prediction diversity, while t-SNE visualizations compare single-model and ensemble-model embeddings.

7. Conclusion

The paper presents MEAL as a learning-based ensemble method that compresses multi-model knowledge into one network with adversarial learning. Experiments on CIFAR-10/100, SVHN, and ImageNet support its effectiveness across several architectures, while future work targets cross-domain ensemble adaptation.

  • Conclusion: MEAL compresses multi-model knowledge into a single network using adversarial learning.The method is framed as a learning-based ensemble approach.
  • Conclusion: Experiments on CIFAR-10/100, SVHN, and ImageNet achieve state-of-the-art accuracy across a variety of network architectures.The conclusion summarizes the reported experimental scope and outcome.
  • Conclusion: Future work will investigate MEAL for cross-domain ensemble and adaptation.This is identified as the paper's stated next research direction.
Loading 1812.02425v2…