Source-linked AI summary
Knowledge Distillation via Route Constrained Optimization
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, Xiaolin Hu
TL;DR
Small students must approximate powerful teachers despite capacity and deployment constraints, while converged-teacher representations can impose overly strong supervision. RCO uses anchor points from the teacher’s optimization route as an easy-to-hard curriculum, and experiments report stronger results across distillation and hint-learning settings. Its main practical limitation is added training time, while curriculum-sequence selection remains imperfect.
Problem
Converged-teacher representations can be too strong a constraint for small students, producing a high lower bound of congruence loss despite the goal of efficient knowledge transfer.
Method
RCO supervises the student with selected anchor points from the teacher’s optimization route, forming an easy-to-hard learning sequence.
Results
RCO outperforms KD and other knowledge-transfer methods across CIFAR-100, ImageNet-1K, and large-scale face recognition, and can boost previous transfer methods.
Takeaways & Limitations
Intermediate teacher states are valuable knowledge for easing student learning and narrowing the performance gap within the evaluated tasks.
Takeaways & Limitations
RCO adds substantial training time, while the simple EEI strategy can ignore differences in anchor-point hardness and produce an improper curriculum sequence.
Abstract
from arXiv · showhide
Distillation-based learning boosts the performance of the miniaturized neural network based on the hypothesis that the representation of a teacher model can be used as structured and relatively weak supervision, and thus would be easily learned by a miniaturized model. However, we find that the representation of a converged heavy model is still a strong constraint for training a small student model, which leads to a high lower bound of congruence loss. In this work, inspired by curriculum learning we consider the knowledge distillation from the perspective of curriculum learning by routing. Instead of supervising the student model with a converged teacher model, we supervised it with some anchor points selected from the route in parameter space that the teacher model passed by, as we called route constrained optimization (RCO). We experimentally demonstrate this simple operation greatly reduces the lower bound of congruence loss for knowledge distillation, hint and mimicking learning. On close-set classification tasks like CIFAR100 and ImageNet, RCO improves knowledge distillation by 2.14% and 1.5% respectively. For the sake of evaluating the generalization, we also test RCO on the open-set face recognition task MegaFace.
1. Introduction
Knowledge transfer addresses the deployment gap between powerful heavy networks and resource-constrained miniaturized models. RCO reframes distillation as curriculum learning over the teacher’s optimization route, using intermediate states to narrow the student–teacher performance gap.
- Heavy CNNs improve performance through deeper and wider architectures but are difficult to deploy on processors with limited energy and memory.
- Knowledge distillation transfers softened teacher predictions or hidden-layer activations so students receive supervision beyond one-hot labels.
- Converged-teacher supervision can leave students stuck approximating teacher performance, whereas earlier teacher states produce smaller performance gaps.
- RCO constrains student training with intermediate points along the teacher’s optimization route, creating an easy-to-hard learning sequence.
- RCO generalizes to knowledge distillation and hint learning and outperforms KD under the same data and computational cost on CIFAR, ImageNet, and Megaface.
2. Related Work
Related work transfers knowledge from large teachers to small models through final predictions, softened logits, or intermediate representations. RCO differs by using the teacher’s training trajectory rather than only its converged model.
- Neural-network miniaturization combines small-architecture design with knowledge-transfer methods that improve compact models.
- Knowledge distillation commonly targets close-set classification by transferring richer teacher logits instead of one-hot labels.
- Hint-based learning guides students with intermediate teacher feature maps and is often used for open-set tasks such as face recognition.
- Existing knowledge-transfer methods supervise students with converged teachers and therefore do not capture knowledge from the teacher’s training process.
- RCO instead transfers knowledge from the teacher’s training trajectory, using intermediate training states as part of the supervision framework.
3. Route Constrained Optimization
Route Constrained Optimization (RCO) trains a student through teacher checkpoints rather than only the converged teacher, using an easy-to-hard sequence to reduce optimization difficulty. The method selects anchor points along the teacher’s trajectory and can use equal-interval or greedy strategies for efficient sequencing.
- 3.2. Difficulty in Optimizing Student: A larger teacher is difficult for a smaller student to mimic because of their capacity gap, and converged targets are harder to approach.Experiments compare student training with teacher checkpoints from epochs 10, 40, 120, and 240.
- 3.3. RCO: RCO uses intermediate teacher checkpoints as anchor points that provide an easy-to-hard learning sequence for student optimization.The student is trained sequentially against outputs from anchor points along the teacher’s optimization route.
- 3.3. RCO: At each step, the student switches its learning target to the output of the next anchor point and carries forward the preceding student parameters.The final student weights are obtained after sequentially optimizing all selected anchors.
- 3.4. Rationale for RCO: RCO changes target-label hardness gradually, whereas conventional curriculum learning changes the hardness of data lessons.Both approaches use easy-to-hard training to move parameters toward a desirable minimum.
- 3.5. Strategy for Selecting Anchor Points: Equal Epoch Interval sampling reduces the cost of selecting anchors, but it ignores differences in target hardness and can produce an improper curriculum sequence.A more desirable sequence should be efficient and smooth in hardness.
- 3.5. Strategy for Selecting Anchor Points: The greedy strategy measures KL divergence on validation data to identify anchor points near the boundary of what the current student can learn.The threshold δ determines whether a later teacher epoch is considered too hard for the student trained on an earlier epoch.
4. Experiments
Experiments evaluate RCO across CIFAR-100, ImageNet-1K, MegaFace, and limited-training settings. RCO improves knowledge transfer results, with route-based supervision and anchor-point strategies central to its performance and cost.
- Common Settings: RCO uses ResNet-50 teachers and compact MobileNetV2 students, with expansion ratio and width multiplier controlling student complexity.The default MobileNetV2 configuration uses expansion ratio 6 and width multiplier 0.5.
- CIFAR-100: 2.1% top-1 accuracy improvement over KD is reported on CIFAR-100.The comparison uses CIFAR-100 top-1 accuracy.
- CIFAR-100: RCO is evaluated on 16 MobileNetV2 configurations spanning expansion ratios 4, 6, 8, and 10 and width multipliers 0.35, 0.5, 0.75, and 1.0.The study varies both parameters to investigate smaller and faster student networks.
- CIFAR-100: The proposed method gains more promotion with smaller student networks in CIFAR-100 MobileNetV2 experiments.Figure 4 reports CIFAR-100 top-1 accuracy across width multipliers and expansion ratios.
- ImageNet: 1.5%/0.7% improvement over KD is reported for ImageNet top-1/top-5 accuracy, respectively.These results support applying RCO to large-scale classification.
- Face Recognition: RCO largely boosts MobileNetV2 performance over original hint-based learning on the MegaFace face-recognition task.MegaFace is evaluated using training data from MS-Celeb-1M and IMDb-Face.
- Ablation Studies: With four anchor points, sequential RCO training requires four times the training epochs of KD or Softmax.The limited-training analysis addresses this additional training cost through one-stage EEI.
- Ablation Studies: Under limited training epochs, RCO with Mgap 10 improves CIFAR-100 results by 4.2% over KD and 3.8% over DML.One-stage EEI trains the student for 240 epochs while changing teacher anchor supervision across stages.
5. Conclusion
RCO boosts small student networks by constructing easy-to-hard learning-target sequences and offers two strategies for selecting anchor points.
- RCO uses an easy-to-hard sequence of learning targets to improve small student-network performance over other knowledge-transfer methods.
- The method provides two strategies for constructing the sequence of anchor points.
- Automatically designing the learning sequence remains an identified direction for future work.