PaperlayerView exact revision on arXiv

Source-linked AI summary

Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion

Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose M. Alvarez, Arun Mallya, Derek Hoiem, Niraj K. Jha, Jan Kautz

arXiv:1912.08795v2cs.LGcs.CVstat.ML

TL;DR

The paper addresses knowledge transfer when the original training data cannot be accessed. It proposes DeepInversion to synthesize class-conditional images from a trained teacher using batch-normalization statistics, and Adaptive DeepInversion to increase diversity through teacher–student disagreement. The synthesized images achieve high fidelity and realism and support data-free pruning, knowledge transfer, and continual learning.

  • Problem

    Knowledge-transfer methods generally assume access to the previously used training dataset, which can be impractical because of dataset scale and data-privacy concerns.

  • Method

    DeepInversion optimizes class-conditional inputs from noise while regularizing intermediate feature distributions with batch-normalization statistics, and Adaptive DeepInversion maximizes teacher–student Jensen-Shannon divergence for diversity.

  • Results

    The synthesized images show high fidelity and realism and support data-free pruning, knowledge transfer, and continual learning across CIFAR-10 and ImageNet demonstrations.

  • Takeaways & Limitations

    The method enables applications that use neither natural images nor labeled data, including network pruning, knowledge transfer, and continual learning.

  • Takeaways & Limitations

    Synthesizing 215K 224x224 ImageNet images for ResNet-50 requires 2.8K NVIDIA V100 GPU-hours, though multi-resolution synthesis reduces this by 10.7x.

Abstract

from arXiv · show

We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network. We 'invert' a trained network (teacher) to synthesize class-conditional input images starting from random noise, without using any additional information about the training dataset. Keeping the teacher fixed, our method optimizes the input while regularizing the distribution of intermediate feature maps using information stored in the batch normalization layers of the teacher. Further, we improve the diversity of synthesized images using Adaptive DeepInversion, which maximizes the Jensen-Shannon divergence between the teacher and student network logits. The resulting synthesized images from networks trained on the CIFAR-10 and ImageNet datasets demonstrate high fidelity and degree of realism, and help enable a new breed of data-free applications - ones that do not require any real images or labeled data. We demonstrate the applicability of our proposed method to three tasks of immense practical importance -- (i) data-free network pruning, (ii) data-free knowledge transfer, and (iii) data-free continual learning. Code is available at https://github.com/NVlabs/DeepInversion

1. Introduction

DeepInversion addresses data-free knowledge transfer by synthesizing class-conditional images from a trained CNN, using batch-normalization statistics to approximate the training distribution. Adaptive DeepInversion increases diversity through teacher–student disagreement, supporting pruning, knowledge transfer, and continual learning.

  • Motivation: Existing knowledge-transfer methods generally require access to the prior training dataset or other training information.This constraint complicates transfer from very large datasets and raises privacy, security, proprietary, and competitive concerns.
  • DeepInversion: Batch-normalization running means and variances encode information about previously seen data at multiple representation levels.Modeling intermediate activations with these Gaussian statistics produces images closer to the training distribution than prior visualization methods.
  • DeepInversion: DeepInversion synthesizes class-conditional images from a trained CNN without training data or metadata.It optimizes inputs while regularizing intermediate activations using statistics available in the trained model’s batch-normalization layers.
  • Adaptive DeepInversion: Adaptive DeepInversion improves synthesized-image diversity by exploiting disagreements between pretrained teacher and in-training student networks.It maximizes the Jensen-Shannon divergence between their responses to expand training-set coverage.
  • Applications: The synthesized data supports data-free network pruning, knowledge transfer, and continual learning.The authors report comparable performance to dataset-dependent or oracle methods in the demonstrated applications.

2. Related Work

Prior work studies knowledge distillation, generative image synthesis, model inversion, and network visualization, but often relies on training data, auxiliary priors, or limited network settings.

  • Knowledge distillation: Knowledge distillation trains a compact student to mimic the output distributions of expert teacher models.Related neural-network methods include approaches by Bucilua et al., Ba and Caruana, and Hinton et al.
  • Image synthesis: Generative adversarial networks can capture image distributions and produce high-fidelity images, but training their generators requires access to data.The cited discussion notes difficulty scaling some alternative synthesis methods to ImageNet classification.
  • Model inversion: Model-inversion methods synthesize class images from a CNN through input optimization, but prior demonstrations used shallow networks or extra intermediate features.These methods arise in a security-oriented line of work.
  • Network visualization: Network-visualization methods synthesize natural pre-images or improve inversion with GAN generators, latent priors, or training-data cluster centroids.The cited approaches target network understanding or image quality and diversity through additional priors or data-derived information.

3. Method

The method first frames data-free distillation as model inversion followed by application-specific distillation, then synthesizes images by matching classifier and feature-distribution statistics. Adaptive DeepInversion adds student-driven competition to broaden image coverage.

  • Background and framework: The data-free knowledge-distillation framework consists of model inversion followed by application-specific knowledge distillation.Standard distillation trains a student to match a teacher’s temperature-scaled soft targets on images from the same domain.
  • 3.1. Background: DeepDream optimizes a random-noise or natural-image input for a target class using classification loss and image regularization.Total-variation and ℓ2 penalties stabilize convergence, but the resulting images can remain far from natural or training images.
  • 3.2. DeepInversion: DeepInversion adds feature-distribution regularization to match synthetic and training-image feature statistics across network layers.The method models batch feature statistics with means and variances, using batch-normalization running statistics as estimates when training data are unavailable.
  • 3.2. DeepInversion: The feature-distribution regularization substantially improves the quality of generated images and yields a generic inversion method for trained CNN classifiers.The regularizer extends the image prior with statistics-based guidance for synthetic inputs.
  • 3.3. Adaptive DeepInversion: Adaptive DeepInversion introduces iterative competition between image generation and the student network to improve diversity.Its competition term favors new images that the student cannot easily classify while the teacher can, expanding distributional coverage.
  • 3.4. DeepInversion vs. Adaptive DeepInversion: DeepInversion can synthesize a one-time image set for knowledge distillation, whereas Adaptive DeepInversion keeps a student in the loop to enhance diversity.The interactive process favors evolving students that force new image features to emerge.

4. Experiments

Experiments show that DeepInversion synthesizes realistic, class-conditional images and supports data-free knowledge transfer, pruning, and continual learning across CIFAR-10 and ImageNet.

  • CIFAR-10 knowledge transfer: Adaptive DeepInversion further improves accuracy by 1%-10%, bringing the student within 2% of a teacher trained on real CIFAR-10 images.The method introduces harder images by exploiting teacher-student disagreements.
  • ImageNet synthesis: DeepInversion produces high-fidelity, high-resolution ImageNet samples with detailed features and contextually relevant textures and backgrounds.Examples include clouds around balloons, water around catamarans, and forests below volcanoes.
  • ImageNet synthesis: On ImageNet, DeepInversion improves Inception Score over DeepDream by 54.2 and exceeds multiple GAN baselines without sophisticated training.Table 3 reports Inception Score comparisons across synthesis methods.
  • Data-free pruning: Data-free pruning achieves performance comparable to prior dataset-dependent methods, with accuracy loss reduced from 2.8% to 2.1% when hardware awareness is combined with ADI.The joint ADI+HA setup also provides faster inference.
  • Data-free continual learning: Synthesized images enable data-free continual learning, with results close to oracle performance and generally better than LwF.MC and DeepDream.The evaluation uses ImageNet and reports results in Table 7.

5. Discussion

The discussion extends the data-free paradigm to quantization and identifies synthesis cost and image similarity as limitations, with continual learning sensitivity to class similarity.

  • Data-free quantization: Data-free model inversion can also support network quantization alongside pruning, knowledge transfer, and continual learning.The discussion cites independently studied data-free quantization of several network architectures.
  • Limitations: 2.8K NVIDIA V100 GPU-hours are required to generate 215K ImageNet samples at 224ˆ224 resolution for ResNet-50.The reported time is 22 hours on 128 GPUs and scales linearly with the number of synthesized images.
  • Limitations: 10.7ˆ is the synthesis-time reduction achieved by the multi-resolution scheme, lowering the cost to 0.26K GPU-hours.This corresponds to 4 hours on 64 GPUs for the same ImageNet synthesis setting.
  • Limitations: Synthesized images show color and background similarity, while style varies with optimization hyperparameters.The discussion attributes this to discriminative representations and the default Gaussian initialization, while noting hyperparameter-dependent style variation.
  • Limitations: Continual-learning performance on iCIFAR and iILSVRC is statistically equivalent or slightly worse than LwF.MC.The authors suspect synthesized images are more effective when old and new classes differ than when they are similar.

Conclusions

The paper concludes that DeepInversion synthesizes high-resolution, high-fidelity training images from trained CNNs, while Adaptive DeepInversion improves diversity and supports data-free applications.

  • Conclusions: DeepInversion synthesizes high-resolution, high-fidelity training images using only a trained CNN.The conclusion presents this as the core proposed capability.
  • Conclusions: Adaptive DeepInversion improves image diversity by using a student network during inversion.The conclusion identifies the student-in-the-loop design as the mechanism for improving diversity.
  • Conclusions: The experiments support the methods’ generalizability and effectiveness across data-free tasks of practical significance.The supplementary material identifies pruning, knowledge transfer, and continual learning as the evaluated task areas.

A.1. Implementation Details

The implementation details describe the distillation schedule, BN-based feature-statistics approximation, and an alternative using a small image subset.

  • A.1. Implementation Details: Knowledge-distillation experiments run for 250 epochs with learning rate 0.1, decayed by 0.1 every 100 epochs.One epoch contains 195 gradient updates, and image generation is interleaved with network updates.
  • A.1. Implementation Details: DeepInversion approximates feature statistics in Rfeature using batch-normalization parameters.The BN parameters provide the mean and variance information used for feature-statistics regularization.
  • A.1. Implementation Details: 100 images suffice for feature statistics in Adaptive DeepInversion to achieve almost the same accuracy as BN statistics.Statistics estimated from randomly sampled original images also support DeepInversion and Adaptive DeepInversion, including when BNs are absent.
  • A.1. Implementation Details: Table 8 compares CIFAR-10 ablations using statistics from randomly sampled original images versus BN running mean and variance.The teacher is VGG-11-BN at 92.34% accuracy, and the student is a freshly initialized VGG-11-BN.

B.1. DeepInversion Implementation

The implementation describes image regularization, filter-importance estimation, and hardware-aware latency modeling for data-free pruning.

  • B.1. DeepInversion Implementation: Total variance regularization uses ℓ2 distances between the base image and one-pixel shifted variants.The shifts include two diagonals, one vertical, and one horizontal; inputs also receive random flipping and jitter.
  • B.1. DeepInversion Implementation: The pruning criterion ranks neurons using both network-error change and target-hardware latency change.The least important neurons are removed iteratively according to ISpWq.
  • B.1. DeepInversion Implementation: The error importance estimate uses a first-order Taylor approximation of individual filter contributions.The implementation realizes this approximation with gate layers.
  • B.1. DeepInversion Implementation: IS,lat measures the latency difference after zeroing a filter group from the network parameters.LAT denotes the latency of the resulting intermediate pruned model on target hardware.
  • B.1. DeepInversion Implementation: Overall latency is approximated as the sum of convolutional-operator runtimes.This assumption is stated for mobile GPU, DSP, and server GPU inference platforms.
  • B.1. DeepInversion Implementation: A hardware-benchmarked lookup table estimates latency for intermediate models from their remaining filters.The lookup table focuses on convolutional operations at reduced input and output filter counts.

C.2. Implementation Details

The pruning experiments profile convolution configurations on a V100 GPU and use the resulting latency estimates to guide hardware-aware pruning.

  • Latency keys encode Conv2D input channels, output channels, kernel size, stride, and input feature-map dimension.Parameters marked with superscript ˚ retain their teacher-model defaults.
  • Each configuration is profiled individually on a V100 GPU with batch size 1 across eight computation kernels.The stored latency is the fastest kernel’s mean over 1000 computations, using CUDA 10.1 and cuDNN 7.6.
  • The latency estimator has high linear correlation with measured latency, with R2 = 0.994.The pruning procedure uses η = 0.01, removes 32 filters per step in groups of 16, and prunes every 30 mini-batches until the threshold is met.

C.3. Hardware-aware Loss Evaluation

The hardware-aware pruning loss improves accuracy under matched latency constraints, while Adaptive DeepInversion expands synthesized-image coverage and diversity.

  • Hardware-aware Loss Evaluation: 0.5%-14.8% top-1 accuracy improvement is achieved by the hardware-aware loss over the pruning baseline at the same latency constraints.This ablation uses ground-truth data for comparison with prior work.
  • Hardware-aware Loss Evaluation: KL-divergence-based pruning changes absolute top-1 accuracy by only ´0.7% to `0.1% relative to the original cross-entropy approach.The comparison uses original data and no fine-tuning.
  • Adaptive DeepInversion: Adaptive DeepInversion gradually expands feature-space distribution coverage across two sequential competition rounds after initial DeepInversion.Figure 7 projects ResNet-50-avgpool feature vectors for ImageNet ‘hand-held computer’ training images and additional competition-round images onto the first two principal components.
  • Adaptive DeepInversion: Competition discovers teacher-scope hand features, and synthesized images differ from nearest neighbors, supporting distribution capture rather than input memorization.Figure 8 compares handheld-computer images generated without and with the competition scheme.

D. Data-free Knowledge Transfer Appendix

The appendix specifies synthesis and continual-learning training settings for ResNet-18 and VGG-16-BN, including image counts, optimization schedules, and batch construction.

  • The synthesized images are sampled equally between the multi-resolution scheme and the Section 4.2 scheme using 2k iterations.The stated setup also clips synthesized images using ImageNet normalization parameters m and s.
  • For ResNet-18 and VGG-16-BN, DeepInversion generates 250 and 64 images respectively per existing class at 224 × 224 resolution.Images are regenerated after each new dataset is added.
  • Synthesis uses learning rates of 0.2 and 0.5, 10k gradient updates, and learning-rate decay every 1.5k steps by a 0.3 multiplier.Both ℓ2 and ℓ1 norms are used for total variance regularization.
  • Each training iteration combines a batch of DeepInversion data with a batch of real data from newly added classes.Both batches use size 128 for ResNet-18 and 64 for VGG-16-BN.
  • Continual-learning training uses SGD with momentum 0.9, clips gradient ℓ2 magnitude to 0.1, and freezes batch-normalization updates.These settings substantially improve the proposed methods and oracles while changing the LwF.MC baseline by about 2%.

E.2. VGG-16-BN Results

On VGG-16-BN, the proposed data-free continual-learning method substantially outperforms prior art while approaching an oracle trained with real images.

  • Up to 42.6% absolute top-1 accuracy improvement over prior art is obtained for data-free continual learning.Results are reported for incrementally extending the softmax output space with classes from a new dataset.
  • The proposal remains within 2% combined error of the best-performing oracle in this experimental setting.The comparison supports replacing ImageNet images for the continual-learning task.
  • DeepInversion-based continual learning operates on any regularly trained model given widespread batch-normalization usage.It does not require model-provider collaboration to support continual-learning adaptation.
Loading 1912.08795v2…