Source-linked AI summary
Data-Free Learning of Student Networks
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, Qi Tian
TL;DR
Training compact networks is difficult when original data and detailed architectures are unavailable. DAFL uses a fixed teacher network to guide a generator that synthesizes training samples, then distills the teacher’s knowledge into a portable student network; on CIFAR-10, the student achieved 92.22% accuracy using only synthetic data.
Problem
Original training data and detailed network architecture are often unavailable, while conventional compression methods require the training dataset for effective fine-tuning.
Method
DAFL treats the pretrained teacher network as a fixed discriminator, trains a generator to synthesize samples, and uses knowledge distillation to train a smaller student network.
Results
92.22% accuracy was achieved on CIFAR-10 by a student network trained with only synthetic data.
Takeaways & Limitations
DAFL can learn portable deep neural networks without the original training dataset.
Takeaways & Limitations
Vanilla GAN training cannot train its discriminator without real images, so the proposed approach relies on using an already trained network as a fixed discriminator.
Abstract
from arXiv · showhide
Learning portable neural networks is very essential for computer vision for the purpose that pre-trained heavy deep models can be well applied on edge devices such as mobile phones and micro sensors. Most existing deep neural network compression and speed-up methods are very effective for training compact deep models, when we can directly access the training dataset. However, training data for the given deep network are often unavailable due to some practice problems (e.g. privacy, legal issue, and transmission), and the architecture of the given network are also unknown except some interfaces. To this end, we propose a novel framework for training efficient deep neural networks by exploiting generative adversarial networks (GANs). To be specific, the pre-trained teacher networks are regarded as a fixed discriminator and the generator is utilized for derivating training samples which can obtain the maximum response on the discriminator. Then, an efficient network with smaller model size and computational complexity is trained using the generated data and the teacher network, simultaneously. Efficient student networks learned using the proposed Data-Free Learning (DAFL) method achieve 92.22% and 74.47% accuracies using ResNet-18 without any training data on the CIFAR-10 and CIFAR-100 datasets, respectively. Meanwhile, our student network obtains an 80.56% accuracy on the CelebA benchmark.
1. Introduction
Deep CNNs are powerful but often too computationally and memory intensive for edge devices, while conventional compression methods generally require unavailable training data. DAFL addresses this setting by generating approximate training images from a given network and using them to train a portable student network.
- Motivation: VGGNet requires over 500MB of memory and over 10^10 multiplications per image, illustrating the deployment burden of heavy CNNs on edge devices.The passage identifies autonomous cars and micro robots as example edge platforms.
- Existing approaches: Existing compression methods use techniques including vector quantization, low-rank decomposition, hashing, pruning, quantization, and Huffman coding to reduce model cost.These approaches are described as effective when applied to pre-trained deep models with accessible training data.
- Problem setting: Most network compression and speed-up algorithms assume access to the original training samples, but privacy and transmission limitations often make those data unavailable.The text also notes that network parameters and architecture information may be unavailable beyond limited interfaces.
- Proposed approach: DAFL treats the given heavy neural network as a fixed discriminator and trains a generative network to approximate the original training set for learning smaller networks.The generated data are then used with the teacher network to train a portable student network.
2. Related Works
Prior portable-network methods are commonly data-driven, whereas data-free compression methods address settings where original training data are unavailable. Existing data-free approaches have architectural or performance limitations, motivating methods that learn efficient CNNs with comparable performance.
- Categories: Portable network learning methods are divided into data-driven and data-free categories according to their assumptions and applications.This distinction frames whether original training data are available during learning.
- Data-driven methods: Data-driven compression methods reduce redundancy through weight representation, matrix decomposition, pruning, and related transformations, but require the original training dataset.The cited approaches include vector quantization, singular value decomposition, and pruning.
- Teacher-student and related methods: Knowledge distillation transfers information from a heavy teacher network to a smaller student network, while feature-based and binarization methods provide related efficiency strategies.These methods aim to transfer useful information or reduce computation in portable networks.
- Data-free methods: Existing data-free methods merge fully connected neurons or reconstruct data from activation metadata, but have architectural restrictions or produce worse performance than baseline models.Neuron merging cannot be applied to convolutional layers, and activation metadata are not provided for most well-trained CNNs.
3. Data-free Student Network learning
The framework learns portable student networks without original training data by using a generator with a fixed teacher network as discriminator. Generated samples are shaped by classification, activation, and entropy objectives, then used for knowledge-distillation training.
- Framework: The proposed framework embeds a generator into the teacher-student paradigm to compress deep networks without the original training data.The teacher network remains available while the generator produces training samples for student learning.
- Teacher-Student Interactions: Knowledge distillation transfers output information from a heavy teacher network to a smaller student using only network interfaces.The teacher and student outputs are compared through a distillation loss, without requiring the teacher’s parameters or architecture.
- GAN for Generating Training Samples: The teacher network is treated as a fixed discriminator, allowing the generator to be optimized directly without jointly training a discriminator.Unlike vanilla GANs, the teacher classifies generated images into concepts rather than predicting whether they are real or fake.
- GAN for Generating Training Samples: The one-hot loss encourages teacher outputs for generated images to approach one-hot class vectors, making synthetic samples exclusively compatible with teacher categories.Generated images are assigned pseudo-labels from the teacher’s predicted classes.
- GAN for Generating Training Samples: The activation loss encourages generated inputs to produce strong responses in teacher convolutional feature maps associated with intrinsic training-data patterns.The feature objective uses the conventional l1 norm.
- GAN for Generating Training Samples: Minimizing information entropy encourages generated images to be distributed across classes with roughly equal probability.The combined objective balances one-hot, activation, and entropy terms through hyperparameters α and β.
4. Experiments
Experiments evaluate data-free student learning across MNIST, CIFAR, and CelebA, including component ablations and visualizations. The proposed method generates useful samples and transfers teacher knowledge to compact students with strong accuracy.
- MNIST: MNIST experiments use LeNet-5 and a half-channel student, plus fully connected teacher-student architectures with fewer student parameters.The generator is trained for 200 epochs using Adam, with batch normalization added at its output.
- MNIST: 98.20% accuracy is achieved on MNIST using GAN-generated data, approaching the 98.65% student upper bound and exceeding random, USPS, and meta-data alternatives.The alternatives achieve 88.01%, 94.56%, and 92.47%, respectively.
- MNIST: 97.91% accuracy is achieved for the fully connected MNIST student without original data, compared with 87.58% for random noise, 91.24% for meta-data, and 93.99% for USPS.The teacher and standard student accuracies are 98.39% and 98.11%, respectively.
- Ablation Experiments: 98.20% accuracy is obtained when all generator losses are combined, while one-hot and entropy-related combinations outperform randomly generated samples in ablation tests.Combining one-hot or feature-map activation loss with information entropy yields 97.25% or 95.53%, respectively.
- CIFAR: 92.22% accuracy is achieved on CIFAR-10 with only synthetic data, while the method reaches 74.47% on CIFAR-100 without real-world training data.The teacher accuracies are 95.58% on CIFAR-10 and 77.84% on CIFAR-100.
- CelebA and Visualization Results: On CelebA, the proposed student reaches 80.03% accuracy versus 81.59% for the teacher, and generated-data training produces filters similar to those learned from original data.MNIST visualizations also show generated images with patterns similar to training images.
- Extended Experiments: Same-architecture students trained from generated data obtain 98.91% on LeNet-5 and 98.39% on HintonNet, close to their teacher accuracies.The results are reported as evidence that teacher information can approximate the original training dataset.
5. Conclusion
The paper presents DAFL, a framework that trains portable student networks without the original training data or detailed teacher architecture information. It uses a generator to approximate the original dataset and knowledge distillation to train the student network.
- DAFL trains portable deep neural networks without access to the original training data.
- The framework trains a generator to approximate the original dataset without using the training data itself.
- Knowledge distillation then enables effective learning of portable networks from the generated data.