PaperlayerView exact revision on arXiv

Source-linked AI summary

The Knowledge Within: Methods for Data-Free Model Compression

Matan Haroush, Itay Hubara, Elad Hoffer, Daniel Soudry

arXiv:1912.01274v2cs.LGcs.CVstat.ML

TL;DR

High-compression model quantization often needs training data for calibration or fine-tuning, which is a problem when the data is sensitive or unavailable. This paper generates synthetic samples from trained models and uses them for data-free compression; its best-performing method achieves accuracy results comparable with real samples under identical optimization conditions and at large scale.

  • Problem

    High-compression quantization commonly requires training data for low-precision calibration or fine-tuning, despite cases where that data is sensitive or unavailable.

    Source: §1 · p. 1

  • Method

    The paper proposes data-free synthetic-sample generation and uses those samples for quantization calibration and knowledge-distillation fine-tuning, including a method that matches stored batch-normalization statistics.

    Source: §1 · pp. 1–2, §3 · p. 2

  • Results

    The data-free distillation approach achieved final accuracy results comparable with real samples under identical optimization conditions and at large scale.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The authors conclude that the approach could reduce real-data exposure during deployment and that BN statistics may help evaluate correspondence between reference data and alternative samples.

    Source: §5 · p. 7

    Generating samples through backpropagation can be computationally costly, and cross-model and cross-dataset applications remain future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recently, an extensive amount of research has been focused on compressing and accelerating Deep Neural Networks (DNN). So far, high compression rate algorithms require part of the training dataset for a low precision calibration, or a fine-tuning process. However, this requirement is unacceptable when the data is unavailable or contains sensitive information, as in medical and biometric use-cases. We present three methods for generating synthetic samples from trained models. Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process. Our best performing method has a negligible accuracy degradation compared to the original training set. This method, which leverages intrinsic batch normalization layers' statistics of the trained model, can be used to evaluate data similarity. Our approach opens a path towards genuine data-free model compression, alleviating the need for training data during model deployment.

1. Introduction

High-compression quantization often depends on training data for calibration or fine-tuning, creating a problem when that data is sensitive or unavailable. The paper investigates synthetic samples for data-free compression and proposes using trained-model statistics to generate samples and assess their similarity to a reference dataset.

1. Introduction

  • High-compression quantization commonly uses training samples for calibration and may require the full training set for fine-tuning, even when that data is sensitive or unavailable.The section frames synthetic data as a possible substitute for real deployment-time data.Source: §1 · p. 1
  • Training a dedicated generative model can produce high-resolution synthetic images, but requires access to the real training data the paper aims to avoid.The paper also revisits model-based feature visualization, which adjusts an arbitrary input by back-propagation to maximize target-feature responses under input priors.Source: §1 · pp. 1–2
  • The study examines synthetic samples for reduced-precision calibration and knowledge-distillation fine-tuning, with a full-precision model or its parts serving as teacher to a low-precision student.The stated focus is classification tasks, including CIFAR and ImageNet.Source: §1 · p. 1
  • The authors propose generating synthetic samples from trained models and report compression results comparable to the original training data with minimal accuracy degradation.They also propose measuring sample similarity through divergence between intermediate-layer statistics, using measurements stored in Batch Normalization layers.Source: §1 · p. 2

2. Related work

Prior data-free compression work used stored batch-normalization statistics or model-output metadata, but existing approaches had limitations in fine-tuning, scale, or dataset performance. This paper instead compares synthetic-sample statistics directly with stored statistics from a trained model.

2. Related work

  • A prior data-free quantization method used batch-normalization measurements to set layer ranges, but required layer statistics or analytical estimates and could not handle cases where calibration caused extreme degradation requiring fine-tuning.The cited method is attributed to Nagel et al. [23].Source: §2 · p. 2
  • Earlier data-free distillation methods sampled random layer targets from collected metadata, but capturing inter-layer relations via graph spectral analysis was computationally impractical for large models or inputs.Lopes et al. [19] described the metadata approach and its inter-layer limitation.Source: §2 · p. 2
  • Bhardwaj et al. [3] collected classifier-adjacent metadata from training samples and reported CIFAR10 experiments with relatively large degradation versus real samples.Their approach clustered layer outputs and used PCA to construct optimization targets.Source: §2 · p. 2
  • The proposed approach uses low-order statistics already stored in batch-normalization layers and optimizes synthetic samples to match their divergence, preserving inter-layer relations through the model structure.This contrasts with prior approaches [19] [3] that sampled activations generated by real data.Source: §2 · p. 2

3. Methods for data-free distillation

The paper describes three data-free generation schemes for distillation and calibration, then combines two of them. Their designs range from random Gaussian inputs to optimizing model outputs or matching internal batch-normalization statistics.

3. Methods for data-free distillation

  • The data-free setup assumes a trained model without its training dataset and offers Gaussian, Inception, and BN-Statistics schemes for synthetic data generation.The schemes respectively sample Gaussian inputs, maximize model logits, or optimize an internal-statistics divergence measure.Source: §3 · p. 2

3.1. Gaussian Scheme

  • The Gaussian scheme matches the original input's first and second moments and may suffice for mild-quantization calibration, but may fail when extreme compression requires parameter adjustment.For distillation, the authors suggest keeping BN running estimates fixed during evaluation to mitigate the problem of altered statistics.Source: §3.1 · pp. 2–3

3.2. Inception Scheme

  • The Inception scheme maximizes one final-output neuron while smoothing the input, using an arbitrary label to inject class information into the optimization.The objective combines a Gaussian-smoothing prior with a class-logit loss; the scheme can yield high-confidence outputs but is sensitive to hyperparameters.Source: §3.2 · p. 3

3.3. BN-Statistics Scheme

  • BN-Statistics generates samples by matching their batch-normalization activation statistics to stored reference means and variances, minimizing mean KL divergence across layers.The method assumes access to predetermined empirical measurements in the model's BN layers and adjusts random input batches by optimization.Source: §3.3 · p. 3

3.4. Combining BNS + I

  • BNS + I combines BN-Statistics and Inception objectives to add class constraints, but introduces additional loss-scaling parameters.The authors report that simple aggregation improves results in some cases, without exhaustively investigating the best combination.Source: §3.4 · p. 3

4. Experiments

Experiments evaluate synthetic samples for quantization calibration and distillation, reporting results near real-data performance and using internal-statistics divergence to assess data correspondence. On ImageNet, the authors report successful quantized-model fine-tuning without real data at this scale.

4.1. Generating data samples

  • BNS + I generation combines statistics matching with Inception and image-prior losses; BNS and Inception are special cases formed by scaling loss components.The statistics loss JKL compares reference batch-normalization statistics with those observed for generated samples.Source: §4.1 · pp. 3–4
  • In-batch augmentations duplicate each sample with random transformations to smooth gradients per sample, while input trimming clips values to [0, 1]..Source: §4.1 · p. 4

4.2. Data-free model compression

  • Quantized distillation uses teacher predictions without labels, and the added Intermediate Quantization loss compares teacher and student intermediate features to support stable convergence under extreme quantization.Input MixUp is also applied without mixing teacher outputs.Source: §4.2 · pp. 4–5
  • BNS-generated samples outperform other generation schemes and achieve accuracy comparable to real data on CIFAR, including recovery from extreme post-calibration degradation.Synthetic data may still yield slightly lower final accuracy because of its optimization process and the limited sample space captured by the model weights.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • ImageNet results report successful quantized-model fine-tuning using no real data beyond the trained model, with BNS results on par with real data.The table covers multiple architectures and compression settings.Source: §4.2 · p. 5, Table 3 · p. 6

4.3. Analysis of Data Generation Schemes

  • In the Inception scheme, internal-statistics divergence increases after several optimization steps despite smoothing, while minimizing JKL also lowers smoothness loss.The comparison monitors JKL during Inception generation and smoothness loss during BNS generation.Source: §4.3 · pp. 6–7
  • With BNS samples, lower statistic loss corresponds to better validation accuracy, while results remain close to real-data results with slight degradation under identical settings..Source: §4.3 · p. 7

4.4. Data Correspondence: BNS Across Datasets

  • JKL preserves its relative relationship to the training data across datasets and models, while FGSM-perturbed samples can show increased internal-statistics divergence.The authors suggest exploring internal-statistics monitoring for adversarial outlier detection.Source: §4.4 · p. 7

5. Discussion and Future Work

The discussion reports that data-free distillation achieved accuracy comparable to real samples at large scale under identical optimization conditions. It highlights BNS as a strong sample-generation method while noting practical costs and remaining limitations.

5. Discussion and Future Work

  • BNS generates examples by matching batch-normalization inputs’ per-channel means and variances through JKL optimization, producing smooth input features.The authors identify a connection between BN-Stats loss and local input structure, and call for further study beyond images.Source: §5 · p. 7
  • Random data can replace real data for calibration under mild 8-bit quantization, while Inception samples have high prediction confidence but greater internal-statistics divergence than BNS.The authors also consider BNS divergence as a possible measure of correspondence between training data and alternative samples.Source: §5 · p. 7
  • Data-free distillation achieved accuracy comparable to real samples at large scale under identical optimization conditions.The authors describe this as the first such result to their knowledge.Source: §5 · p. 7
  • With more synthetic samples, final accuracy improves in KD on quantized ResNet-44 (2w4a, CIFAR10) using BNS data.Figure 3 varies the number of data points per class and relates dataset size and JKL to model accuracy.Source: Figure 3 · p. 8
  • The authors contrast their low-precision deployment focus with and report low-precision KD results and recovery from extreme accuracy degradation not explored by.They note that covers broader data-free applications with limited analysis.Source: §5 · pp. 7–8
  • Sample generation through backpropagation may impede continuous deployment with large models, and BNS + I showed no dramatic improvement amid suspected objective tension.The authors suggest shared internal-statistics behavior may avoid regenerating samples when new training data does not change significantly.Source: §5 · p. 8

Appendices

The appendix gives generation settings and a ResNet-44-CIFAR10 configuration table.

A.1. Generation

  • Generation experiments use ADAM with learning rate 0.1 and betas (0.9,0.999).When image-prior scale is non-zero, a N(0,1) smoothing kernel of size 5 is used; batch size generally fits available memory, with statistics accumulated across devices.Source: §A.1 · p. 10
  • The ResNet-44-CIFAR10 BNS configuration table lists batch size, in-batch steps, learning-rate drops, and BNS, CLS, and prior scales.The passage provides the table headings but no configuration values.Source: §A.1 · p. 10

A.2. Hyper-Parameters sensitivity for calibration

The appendix examines how the Gaussian smoothing-kernel variance affects Inception-related generation and reports selected calibration results for quantized ResNet44.

A.2. Hyper-Parameters sensitivity for calibration

  • The appendix tests how changing Gaussian smoothing-kernel variance, used as the image prior, affects the usability of generated data.Calibration results for quantized ResNet44 are compared in Table 5.Source: §A.2 · p. 10
  • Table 5 reports post-calibration validation accuracy with a 5x5 kernel and inception loss scale 0.001.The passage does not provide the table’s accuracy values.Source: Table 5 · p. 10
  • Results for I and BNS + I are sensitive to hyper-parameter adjustments, and the reported results are best results without an exhaustive search.The stated fixed settings are a 5x5 kernel and inception loss scale 0.001.Source: Table 5 · p. 10

A.3. Distillation

The distillation appendix describes a KD framework with IQ and input-mixup tweaks, and specifies its training settings and ImageNet comparison scope.

A.3. Distillation

  • Algorithm-2 describes a KD framework incorporating IQ and input-mixup tweaks, which tend to improve convergence when the available dataset is limited.The tweaks are not strictly required when the full dataset is available.Source: §A.3 · p. 10
  • Table 6 compares dataset size and fine-tune objective effects on ImageNet validation accuracy for ResNet-18 quantized to 4 bits.The caption says the starred regime follows [20].Source: Table 6 · p. 11
  • The experiments use standard KL divergence for KD, smoothed-L1 for the IQ term, and SGD with learning rate 0.1 and batch size 256.Sampling is with replacement for a fixed number of steps per epoch; additional shared settings are α=1, β=0.01, and mix rate 0.5.Source: §A.3 · p. 11

B. Additional results for ImageNet dataset

Even with the full dataset, distilled variants have an accuracy gap relative to the fine-tuned model. The authors attribute this partly to distillation relying on teacher predictions without ground-truth labels.

B. Additional results for ImageNet dataset

  • Distilled variants remain less accurate than the fine-tuned model even when the full dataset is available.The authors suggest that without ground-truth loss, student accuracy depends on teacher predictions, including their mistakes.Source: B. Additional results for ImageNet… · p. 11

C. Generating CIFAR10 and ImageNet

The generated CIFAR10 samples share visual features across schemes, while BNS + I samples appear smoother and clearer. ImageNet samples show improved detail and diversity over naive inception generation.

C. Generating CIFAR10 and ImageNet

D. BN-Stats measurement

The appendix reports JKL measurements on selected datasets and shows additional synthetic samples, including BNS + I samples generated without an image prior.

D. BN-Stats measurement

  • Additional DenseNet-121 synthetic samples are shown.Figure 6 presents the samples.Source: Figure 6 · p. 13
  • The BNS + I samples in panel (b) were generated without an image prior.This panel description specifies the generation condition.Source: D. BN-Stats measurement · p. 13
  • Table 7 reports JKL values computed over the entire training dataset and all batch-normalization layers in the reference model.Its FGSM samples use a small perturbation ratio of ϵ = 0.1; measured loss generally grows with epsilon.Source: D. BN-Stats measurement · p. 12, Table 7 · p. 13

E. Perceived dataset bias

The validation set is balanced, yet ResNet-18 predictions show class bias; the authors hypothesize this may impede KD. BNS + I improves mean tail degradation despite slightly worse overall accuracy.

E. Perceived dataset bias

  • ResNet-18 predictions are somewhat biased over the balanced validation set, and the authors hypothesize this may impede KD performance.Figure 7 plots mean soft outputs and hard class-index choices across the validation set.Source: E. Perceived dataset bias · pp. 13–14, Figure 7 · p. 14
  • BNS samples favor certain classes, whereas BNS + I samples appear balanced; this perceived balance did not yield a significant KD validation-accuracy benefit over BNS.Each synthetic dataset contains 100K samples.Source: E. Perceived dataset bias · p. 14

F. Mean tail degradation

  • BNS + I improves mean tail degradation over BNS despite slightly worse overall accuracy, considering only classes with observed accuracy degradation.Classes are ordered by the reference model’s mean hard prediction over BNS samples; classes with improved accuracy are excluded.Source: F. Mean tail degradation · p. 15

Related papers

Loading 1912.01274v2…