PaperlayerView exact revision on arXiv

Source-linked AI summary

Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net

Guorui Zhou, Ying Fan, Runpeng Cui, Weijie Bian, Xiaoqiang Zhu, Kun Gai

arXiv:1708.04106v3stat.MLcs.LG

TL;DR

Real-time prediction applications require accuracy under strict inference-time limits, making complex deep models unsuitable for some settings. Rocket launching jointly trains a lightweight target and a cumbersome booster, which guides the target throughout learning. The paper reports state-of-the-art results on public benchmarks and industrial data, with performance above other teacher–student approaches.

  • Problem

    Real-time applications such as CTR prediction require high accuracy and fast responses, limiting the suitability of complex deep models.

    Source: Introduction · p. 1

  • Method

    Rocket launching simultaneously trains a lightweight inference model and a deeper booster, using hint loss to guide the light model throughout training.

    Source: Introduction · p. 1

  • Results

    The method reports state-of-the-art results on public benchmarks and industrial data, outperforming other teacher–student approaches.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The authors report that simultaneous training shortens total training time, supporting more efficient updates of inference models used in industrial applications.

    Source: Simultaneous training · p. 3

    When the light model’s softmax output is near zero, the gradient can vanish, so final-softmax MSE may fail to learn differences from the booster.

    Source: Hint loss functions · p. 4

Abstract

from arXiv · show

Models applied on real time response task, like click-through rate (CTR) prediction model, require high accuracy and rigorous response time. Therefore, top-performing deep models of high depth and complexity are not well suited for these applications with the limitations on the inference time. In order to further improve the neural networks' performance given the time and computational limitations, we propose an approach that exploits a cumbersome net to help train the lightweight net for prediction. We dub the whole process rocket launching, where the cumbersome booster net is used to guide the learning of the target light net throughout the whole training process. We analyze different loss functions aiming at pushing the light net to behave similarly to the booster net, and adopt the loss with best performance in our experiments. We use one technique called gradient block to improve the performance of the light net and booster net further. Experiments on benchmark datasets and real-life industrial advertisement data present that our light model can get performance only previously achievable with more complex models.

Introduction

Real-time applications need both accuracy and low inference latency, constraining the use of complex networks. Rocket launching trains a lightweight inference model with guidance from a simultaneously trained booster.

Introduction

  • Complex networks can achieve stronger performance but incur inference times unsuitable for industrial settings requiring hundreds of ad predictions per user within milliseconds.The paper frames accuracy and latency as joint requirements for these applications.Source: Introduction · p. 1
  • Rocket launching trains the light target and a deeper, more complex booster simultaneously, using a hint loss to guide the light model throughout training.Only the trained light network is used at inference; the two models also share lower layers.Source: Introduction · p. 1
  • The method reports state-of-the-art results on public benchmarks and industrial data, outperforming other teacher–student approaches.Combining it with other teacher–student approaches further improves performance.Source: Introduction · p. 2
  • The framework analyzes hint-loss functions and uses gradient block to prevent hint-loss back-propagation from constraining booster updates.The paper presents gradient block as a way to improve both networks.Source: Introduction · p. 2

Related work

Prior lightweight-model work either simplifies computation directly or transfers knowledge from a larger model. Rocket launching differs by jointly training the light model and booster with shared parameters, applying simultaneous training and parameter sharing to model compression.

Related work

  • The paper situates complex, deep networks as costly at inference, motivating lightweight models for applications with inference-time limits.Its background cites increasing depth, parameter counts, and complexity as a trend in network design.Source: Related work · p. 2
  • Direct-compression approaches simplify or prune network operations, including SVD approximation, lightweight architectures, and filter pruning.Examples include MobileNets, ShuffleNet, and ThiNet.Source: Related work · p. 2
  • Earlier teacher–student approaches transfer knowledge from a larger model, including ensemble-generated labels, logit regression, and soft-target distillation.Knowledge distillation uses class probabilities from a cumbersome model as soft targets for a small model.Source: Related work · p. 2
  • Other teacher–student methods transfer intermediate representations or attention maps in addition to, or instead of, final outputs.FitNets use intermediate representations as hints, while attention transfer trains a student to mimic a teacher’s attention maps.Source: Related work · p. 2
  • Rocket launching uses a jointly trained booster to guide the light model throughout learning, rather than relying only on a pretrained teacher’s stationary outputs.The framework also shares some parameters between the light model and booster.Source: Related work · p. 2
  • The authors characterize simultaneous training and parameter sharing for small-net compression as a previously untried combination and report experimental effectiveness.They distinguish these techniques from prior uses in multi-input, semi-supervised, and multi-task settings.Source: Related work · p. 2

Our approach

Rocket launching trains a light network alongside a more capable booster, sharing lower-level layers and using a hint loss to transfer the booster’s knowledge. The method also compares hint losses and uses gradient blocking to protect booster learning while it supervises the light network.

The sketch of our method

  • The light and booster networks share lower-level layers, use separate task-specific layers, and are trained on the same task.The objective uses cross-entropy for both networks and an MSE hint loss between their logits, weighted by λ.Source: The sketch of our method · p. 3

Parameter sharing

  • Parameter sharing gives the light network shared low-level representations; the authors assume tasks can build on such representations.They note that reusing embedding layers from other tasks can help a new CTR task converge more easily and perform better.Source: Parameter sharing · p. 3

Simultaneous training

  • Unlike teacher-student methods with a pretrained, fixed teacher, rocket launching trains both networks simultaneously so the booster guides the light network throughout learning.The authors say this shortens training time and supports more frequent updates of the inference model.Source: Simultaneous training · p. 3

Hint loss functions

  • Final-softmax MSE can have vanishing gradients when a light-network logit is very negative, even if its output differs greatly from the booster’s.Logit MSE directly reduces the difference between pre-softmax logits.Source: Hint loss functions · p. 4
  • Logit-MSE hint loss achieves the best performance among the tested formulations and avoids the vanishing-gradient problem of final-softmax MSE.The paper also reports that optimizing all logit differences outperforms knowledge distillation in this framework.Source: Hint loss functions · p. 4
  • Knowledge distillation uses temperature to soften class probabilities, but the authors report that matching all logits with Lmimic outperforms LKD in their framework.They suggest very negative logits may carry useful knowledge; their discussion also notes that the high-temperature approximation can yield a negligible gradient.Source: Hint loss functions · p. 4

Gradient block

  • Gradient block prevents the booster’s specific parameters from receiving hint-loss gradients, leaving them to learn from labels while their probabilities supervise the light network.The authors motivate this by noting that hint-loss updates can hinder booster learning and, in turn, diminish the light network’s learning potential.Source: Gradient block · p. 4, Figure 2 · p. 4

Experiments

Experiments show rocket launching improves light-network performance on CIFAR-10, SVHN, CIFAR-100, and real advertising data, while framework ablations and visualizations examine its components and learned representations.

Experiments on CIFAR-10

  • On CIFAR-10, rocket launching uses shared lower WRN layers and distinct light-net and booster structures; the experiments also compare an alternative sharing arrangement.The booster-specific layers are removed at inference.Source: Experiments on CIFAR-10 · p. 5, Figure 3 · p. 6
  • On CIFAR-10, rocket launching consistently improves on the base light net across settings and outperforms other teacher-student methods; the first reported setting improves by 0.9%.Combining rocket launching with KD further improves performance.Source: Experiments on CIFAR-10 · p. 5, Table 1 · p. 5
  • For different hint losses on CIFAR-10, matching logits performs best, while matching probabilities performs worst.The authors attribute the probability-loss result to gradient vanishing.Source: Experiments on CIFAR-10 · p. 5, Table 3 · p. 6

Performance of each part of our framework

  • Gradient block, parameter sharing, and simultaneous training each contribute to CIFAR-10 improvements; gradient block adds 0.63% and sharing adds 0.19% over their respective ablations.Training the light net alone after initializing it from the booster performs worse than rocket launching.Source: Performance of each part of our… · p. 5, Table 2 · p. 6

Performance with different depths

  • Across light-net depths, rocket launching performs better than the base and KD models, while its improvement over the base does not scale proportionally with depth.The authors suggest the pattern may reflect a balance between light-net learning ability and information from the booster.Source: Performance with different depths · p. 6, Figure 4 · p. 6

Visualization of rocket launching and attention transfer

  • Visualizations show similar lower-group feature maps for rocket launching and attention transfer, indicating that parameter sharing and attention have similar effects on lower layers.The compared groups show booster-net and light-net results side by side.Source: Visualization of rocket launching and… · p. 6, Figure 5 · p. 7

Experiments on SVHN and CIFAR-100

Experiments on real Advertisement Dataset

  • On the real advertising dataset, the light net improves GAUC by 0.3% at the base model’s latency, while the booster requires 23.2 ms per inference request.The booster has the best offline metric, but its inference time is unacceptable for the online system.Source: Experiments on real Advertisement… · p. 7, Table 5 · p. 7

Conclusion

Rocket launching trains a light model with a cumbersome booster to produce an efficient, well-performing model. The conclusion highlights hint-loss transfer, shared parameters, gradient blocking, and a future focus on reducing inference costs further.

Conclusion

  • Rocket launching trains a light model and a cumbersome booster on the same task, using hint loss to guide the light model throughout training.The framework is intended to produce an efficient, well-performing light model.Source: Conclusion · p. 7
  • Sharing parameters lets the light model receive low-level representations directly from the booster.Source: Conclusion · p. 7
  • Gradient block is designed to prevent booster deterioration while the paper analyzes alternative hint losses for knowledge transfer.Source: Conclusion · p. 7
  • Future work will explore reducing both network depth and the number of neurons per layer to improve inference efficiency.Source: Conclusion · p. 7

Related papers

Loading 1708.04106v3…