Adversarial Network Compression

1 Introduction

• Deep ConvNets’ complexity and hardware requirements make real-time inference difficult and generally prevent deployment on resource-constrained devices such as mobile phones.Source: §1 · p. 1
• A discriminator distinguishes teacher from student outputs, while adversarial training pushes their distributions closer and an L2 output loss also guides the student.The teacher is trained in advance; student training does not require labels.Source: §1 · pp. 1–2
• Across five datasets, the student has a small accuracy drop, beats related knowledge-transfer approaches and the same network trained with labels, and outperforms other compression approaches.The evaluation uses three teacher and three student architectures to support generalization.Source: §1 · p. 2
• The approach adds a regularization scheme to prevent a trivially strong discriminator without reducing network capacity.Source: §1 · p. 2

2 Related Work

• Quantization and binarization reduce weight precision, while pruning removes connections to reduce model size and can prevent over-fitting.Source: §2 · p. 2
• Factorization uses low-rank filter bases, while efficient architectures such as ResNet reduce parameters while maintaining or improving performance.ResNet is selected as the standard teacher-student model because the approach applies to its variations and other architectures.Source: §2 · p. 3
• Unlike methods that reduce parameters, change network structure, or design efficient components, this approach transfers knowledge to a simpler network without architectural intervention.Source: §2 · p. 3
• Distillation transfers teacher knowledge through logits or soft targets, and the paper studies this approach with recent ConvNet architectures.Source: §2 · p. 3
• The paper introduces adversarial learning for teacher-student compression, aiming to make student outputs match teacher outputs without requiring labels during student training.The authors report results consistently better than network distillation.Source: §2 · p. 4

3.1 Knowledge Transfer

• The teacher is pretrained on labeled data, while the smaller student is trained to match its outputs using logits and features from earlier layers.The described feature representations are taken from the networks’ last layers before the logits.Source: §3.1 · p. 4

3.2 Generative Adversarial Networks

• GANs use a generator–discriminator game to align real and generated data distributions; the discriminator classifies inputs as real or generated.The discriminator objective uses binary cross-entropy, with the discriminator maximizing and generator minimizing the objective.Source: §3.2 · p. 5

3.3 Adversarial Compression

• The student and discriminator play an adversarial game over teacher and student features, with the student’s labels inverted to guide it toward teacher-like predictions.The teacher is fixed, and dropout is applied only during student updates for more stable results.Source: §3.3 · pp. 5–6
• A separate data objective minimizes the difference between teacher and student outputs, updating all student parameters and supporting convergence toward output matching.The paper says the adversarial objective alone does not force the student to match the teacher’s output as closely as possible.Source: §3.3 · pp. 6–7

3.4 Network Architectures

• Experiments use ResNet-164 or NiN teachers, ResNet-18 or ResNet-20 students, and LeNet-4 students for small-scale experiments; the discriminator is a three-layer fully connected network.The discriminator uses 128–256–128 units, ReLU intermediate activations, and a sigmoid binary output.Source: §3.4 · p. 7

3.5 Discriminator Regularization

• Because the discriminator can readily distinguish teacher from student samples, the method tests L1, L2, and adversarial-sample regularization to prevent it from dominating while retaining capacity.Adversarial-sample regularization updates the discriminator using student samples labeled as teacher.Source: §3.5 · pp. 7–8
• Regularization is required for good results, and the introduced student-conditioned regularization has the strongest influence on fooling the discriminator.The paper motivates the approach as prolonging the game so the student receives more valuable gradient updates before the discriminator wins.Source: §3.5 · p. 8

3.6 Learning & Optimization

• Training proceeds in two phases: first train the teacher on labeled data, then train the randomly initialized student and discriminator together using SGD with momentum.The experiments use dropout of 0.5 for the adversarial-sample input and equal weighting of the two objective terms as a compromise across evaluations.Source: §3.6 · p. 9

4.1 Discriminator Model

• The selected CIFAR-100 discriminator is a three-layer fully connected model with depths 128-256-128; convolutional discriminators perform worse.The selected discriminator is used in the remaining evaluations.Source: §4.1 · p. 10, Table 1 · p. 10
• The proposed adversarial-sample regularization performs better than the other tested regularizers, while no regularization performs worse than supervised training.The comparison uses the complete model with features input to the discriminator and dropout on the student.Source: §4.1 · p. 10, Table 2 · p. 11

4.2 Component Evaluation

• On CIFAR-100, adding student dropout with features input to the discriminator reduces error by 1%.The passage describes this as the largest improvement after adding dropout.Source: §4.2 · p. 12
• On SVHN, the student outperforms both the supervised student and the teacher; on Fashion-MNIST, the complete model has 8.61% error versus 8.77% for the supervised student.For Fashion-MNIST, features input to the discriminator and dropout are important to the best result among the reported baselines.Source: §4.2 · p. 12
• Across evaluations, feature input to the discriminator yields the lowest error; students outperform supervised counterparts and generalize across teacher–student architectures.On CIFAR-10, compression baselines are about 1.5% behind the teacher, and the complete feature-input model with student dropout performs best.Source: §4.2 · pp. 11, 13
• On CIFAR-10 and CIFAR-100, the student achieves the lowest error and uses the fewest parameters among compared methods; it also consistently outperforms other approaches on SVHN and Fashion-MNIST.The comparisons use the same student architecture for each dataset.Source: Table 7 · p. 12, §4.2 · p. 13, Table 8 · p. 13

4.3 ImageNet Evaluation

• On ImageNet, the features-input and dropout model performs best and beats the supervised student, while MobileNets have fewer parameters.The ResNet-50 student has around 2x fewer parameters than ResNet-152.Source: §4.3 · p. 13, Table 10 · p. 14

• The method achieves state-of-the-art performance without labels across five datasets and three teacher and three student architectures.Source: §5 · p. 14
• Regularization empirically helps the student compete with the discriminator.Source: §5 · p. 14
• Adversarial network compression transfers knowledge from a large model to a smaller model with limited capacity.Source: §5 · p. 14
• Future work will explore more discriminators using intermediate features and transfer the approach to object detection and segmentation.Source: §5 · p. 14