PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Deep Nets Really Need to be Deep?

Lei Jimmy Ba, Rich Caruana

arXiv:1312.6184v7cs.LGcs.NE

TL;DR

The paper asks whether deep networks are necessary for learning functions that achieve strong speech and vision performance. It trains shallow networks to mimic deep models through model compression and finds that shallow models can perform similarly on TIMIT and CIFAR-10, though the approach depends on suitable unlabeled data or highly accurate teachers.

  • Problem

    The paper examines why deep networks outperform shallow networks and whether shallow models can learn equally accurate functions with similar parameter counts.

  • Method

    The authors train shallow fully connected networks to mimic deep models using model compression, including teacher-generated targets and a bottleneck linear layer for large models.

  • Results

    Shallow mimic networks perform similarly to deep convolutional architectures on TIMIT and CIFAR-10, including comparable performance with no convolutional or pooling layers on TIMIT.

  • Takeaways & Limitations

    For a fixed parameter budget, depth may make learning easier but may not always be essential, suggesting better algorithms might train accurate shallow networks directly.

  • Takeaways & Limitations

    The mimic-learning method depends on either a large unlabeled dataset, a highly accurate teacher, or both, and TIMIT compression reused labeled training inputs as unlabeled data.

Abstract

from arXiv · show

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this extended abstract, we show that shallow feed-forward networks can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using a total number of parameters similar to the original deep model. We evaluate our method on the TIMIT phoneme recognition task and are able to train shallow fully-connected nets that perform similarly to complex, well-engineered, deep convolutional architectures. Our success in training shallow neural nets to mimic deeper models suggests that there probably exist better algorithms for training shallow feed-forward nets than those currently available.

1 Introduction

The paper asks why deep networks outperform shallow ones and tests whether shallow networks can learn the same functions. Its approach trains shallow models to mimic trained deep models, sometimes with comparable parameter counts.

  • 1 Introduction: The paper investigates whether deep networks outperform shallow networks because of parameter count, representational capacity, inductive bias, convolution, optimization, or combinations of these factors.The motivating example reports 86% test accuracy for a shallow network versus 91% for a deeper network on the same 1M-point training set.
  • 1 Introduction: Earlier theory showed that sufficiently wide single-hidden-layer sigmoid networks can approximate any decision boundary, but empirical work found such networks difficult to train.
  • 1 Introduction: The authors first train a state-of-the-art deep model, then train a shallow model to mimic its outputs using model compression.
  • 1 Introduction: Model compression lets shallow networks match some deep-model accuracies even when direct training on the original labeled data does not reach those accuracies.
  • 1 Introduction: Matching a deep model with a similarly sized shallow network suggests that the learned function need not require a deep representation.

2 Training Shallow Nets to Mimic Deep Nets

The method trains shallow networks on outputs from larger deep models rather than original labels, using logit regression and a factorized bottleneck to make optimization practical. These techniques preserve the teacher’s learned function while addressing slow convergence and memory costs in large shallow models.

  • 2.1 Model Compression: Model compression trains a compact mimic network on synthetic labels produced by a larger, more accurate model rather than on the original labels.Unlabeled data are passed through the teacher to collect prediction scores used as training targets.
  • 2.2 Mimic Learning via Regressing Logit with L2 Loss: The mimic objective treats teacher logits as regression targets, allowing the shallow model to learn relationships among class predictions that probability targets can obscure.The shallow model is trained on the log probabilities before the teacher’s softmax activation.
  • 2.2 Mimic Learning via Regressing Logit with L2 Loss: Logits regression outperforms KL-divergence and probability-space L2 losses, while logit normalization provides only a slight additional improvement.
  • 2.3 Speeding-up Mimic Learning by Introducing a Linear Layer: Large shallow networks train slowly because matching deep-model parameter counts requires many hidden units and an O(HD) input-to-hidden weight matrix with correlated parameters.The authors report that convergence can take multiple weeks even with a GPU.
  • 2.3 Speeding-up Mimic Learning by Introducing a Linear Layer: A k-unit linear bottleneck factorizes W into U and V, accelerating convergence and reducing memory from O(HD) to O(k(H + D)).The factorization uses k << D, H and is learned by back-propagation through the linear layer.
  • 2.3 Speeding-up Mimic Learning by Introducing a Linear Layer: The bottleneck reduces representational power but can be absorbed into a single weight matrix, while its reparameterization makes previously infeasible large shallow models trainable.

3 TIMIT Phoneme Recognition

On TIMIT, shallow nets trained on original labels trail deep models, but model compression enables shallow mimic nets to approach deep-model accuracy without convolution or pooling.

  • Data and models: TIMIT uses 1845-dimensional normalized filter-bank inputs and 183-dimensional tri-state phoneme targets derived from 61 labels.The corpus has 462 training speakers, with separate development and test speakers.
  • Data and models: The evaluated deep models are a three-layer ReLU DNN, a convolutional/max-pooling CNN with three hidden layers, and a nine-model CNN ensemble.The ensemble is formed from CNNs trained with varied training sets and convolutional kernel sizes.
  • Baseline results: Original-data shallow nets remain 1.4%–2% less accurate than the DNN, 3.5%–4.1% less accurate than the CNN, and 4.5%–5.1% less accurate than the ECNN.These shallow models use up to 10X as many parameters as the deep models.
  • Compression results: Compression trains 8k- and 400k-unit shallow mimic nets on ECNN relabeled data, allowing a shallow model to match a similarly sized DNN and approach CNN accuracy.The 400k-unit mimic has no convolutional or pooling layers, while both mimic models use 250 linear units before the nonlinear hidden layer.
  • Implication: Shallow nets can reach DNN-level accuracy from original training data, supporting the possibility of better learning algorithms for accurate shallow models.The paper notes this occurs without increasing the number of training examples.
  • Parameter scaling: At about 1 million parameters, shallow, deep, and mimic models have similar accuracy; with larger models, mimic accuracy continues rising toward CNN-level performance.The DNN plateaus around 10M parameters, whereas the mimic model plateaus around 100M and does not reach the ECNN accuracy.

4 Object Recognition: CIFAR-10

On CIFAR-10, shallow mimic networks achieved accuracy comparable to much deeper convolutional networks, extending the TIMIT findings to object recognition. Their performance improved further when mimicking an ensemble of deep CNNs.

  • Evaluation setup: CIFAR-10 experiments were designed to test whether the TIMIT results generalize across learning problems and task domains.CIFAR-10 contains 50,000 training and 10,000 test images across 10 object classes.
  • Model compression: The experiments used an ensemble of deep CNNs to label images, then trained a shallow neural network on the teachers’ logit predictions.The CIFAR-10 inputs have 3072 dimensions, compared with 1845 for TIMIT, while the dataset is much smaller.
  • Model design: Non-convolutional networks performed poorly on CIFAR-10 when trained on raw pixels, motivating the mimic-model approach for this task.The authors note that prior shallow models instead used SIFT features, while related work studied architectures with convolution and pooling layers.
  • Results: A shallow mimic net trained from a teacher CNN achieved accuracy comparable to CNNs with multiple convolutional and pooling layers.The deeper convolutional models had fewer parameters because of weight sharing, despite having more hidden units.
  • Results: 0.9% additional accuracy was obtained by training the shallow model to mimic an ensemble of CNNs rather than a single teacher CNN.The mimic models achieved accuracies previously unseen on CIFAR-10 for models with so few layers.

5 Discussion

The discussion attributes mimic-model gains to softer teacher targets and effective regularization, while Figure 3 indicates that shallow-model accuracy continues to track teacher accuracy. Shallow models also offered faster training and potentially lower-latency inference.

  • Why mimic models help: Teacher predictions can outperform original labels because they may filter label errors, simplify difficult target regions, and represent uncertainty across multiple outputs.These effects are presented as possible reasons that prediction targets from other models can produce more accurate students than original labels.
  • Why mimic models help: Model compression acts as regularization that reduces the shallow–deep performance gap, whereas shallow models trained on original targets are more prone to overfitting.The paper states that shallow models can begin overfitting before learning the accurate functions learned by deeper models, even with dropout.
  • Capacity and representation: Figure 3 compares two shallow mimic models across teacher accuracies, with teacher accuracy on the x-axis and mimic accuracy on the y-axis.Parallel-to-diagonal trends indicate similar increases in student and teacher accuracy; the larger SNN-MIMIC-160K model consistently outperformed SNN-MIMIC-8K on the same targets.
  • Capacity and representation: 10 times more parameters gave SNN-MIMIC-160K a consistent advantage over SNN-MIMIC-8K, although the smaller model eventually matched the larger model when learning from a better teacher.The results suggest that mimic models had not yet exhausted their capacity as teacher accuracy improved.
  • Computational trade-offs: 1–2 hours versus 8–12 hours: shallow mimic models reached similar accuracies much faster than deep models in the reported GPU experiments.The paper also describes shallow inference as potentially requiring only 2 or 3 parallel cycles, unlike depth-proportional sequential inference.

6 Future Work

Future work considers labeling the 80M-image Tiny Images dataset with a teacher to train shallow models without convolution or pooling. The approach could also trade accuracy against computational cost, but currently depends on strong teachers or abundant unlabeled data.

  • Future experiments: The authors are investigating whether labeling all 80M Tiny Images with a teacher can train shallow models without convolutional or pooling layers.This proposed extension targets shallow models trained to mimic deep convolutional models.
  • Practical applications: Student models of small-to-medium size and depth could mimic very large deep models and deep ensembles while reducing runtime cost.The paper frames this as a practical application of its model-compression results.
  • Practical applications: Model compression allows a flexible trade-off between accuracy and computational cost.The supported scope is model deployment after training, where student size and depth can be adjusted.
  • Open limitation: The current mimic-training algorithm depends on either a large unlabeled dataset, a highly accurate teacher, or both.Developing a direct training algorithm for accurate shallow models from original data is identified as significant future work.

7 Conclusions

The paper demonstrates that shallow neural networks can match performances previously associated with deep models on speech and vision tasks. Its results suggest that depth may ease learning for a fixed parameter budget without always being essential.

  • Conclusions: Shallow neural nets achieved performances previously achievable only by deep models on TIMIT phoneme recognition and CIFAR-10 image recognition.Single-layer fully connected feed-forward nets trained to mimic deep models performed similarly to complex deep convolutional architectures.
  • Conclusions: The findings suggest that deep learning’s strength may partly reflect a match between deep architectures and current training procedures.The authors therefore propose that better algorithms might train more accurate shallow feed-forward networks.
  • Conclusions: For a given number of parameters, depth may make learning easier but may not always be essential.This conclusion is explicitly framed as a possibility supported by the empirical results.
Loading 1312.6184v7…