PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

Gaurav Menghani

arXiv:2106.08962v2cs.LG

TL;DR

Growing model footprints make efficiency important alongside predictive quality, particularly for practical training and deployment. The paper surveys five efficiency areas, provides an experiment-based guide with code, and demonstrates strategies that improve quality–footprint tradeoffs. It also identifies limits in current evidence, including uncertain real-world footprint gains from unstructured pruning and unverified claims for extreme quantization.

  • Problem

    Growing model parameters, training resources, latency, and deployment costs create a need to evaluate deep learning models beyond task quality.

  • Method

    The paper surveys five core efficiency areas across modeling, infrastructure, and tools, and supplies an experiment-based practitioner guide with code.

  • Results

    13.8× smaller and 2.94× faster at 84.18% accuracy, the distilled and augmented model approaches the 84.42% target while reducing size and latency.

  • Takeaways & Limitations

    Efficiency techniques can improve Pareto frontiers beyond manual tuning, while the survey gives practitioners actionable tradeoffs for meeting quality and footprint constraints.

  • Takeaways & Limitations

    Unstructured pruning often reports theoretical parameter or FLOP reductions, but its translation into actual footprint reductions is not currently clear.

Abstract

from arXiv · show

Deep Learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, resources required to train, etc. have all have increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code, for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. Our hope is that this survey would provide the reader with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

1 INTRODUCTION

Deep learning models have achieved strong benchmark results while their parameters, training costs, deployment resources, and latency have grown substantially. The paper frames efficiency as finding models on the Pareto frontier of quality and footprint, supported by coordinated algorithms, techniques, tools, and infrastructure.

  • Motivation: GPT-3 has 175 billion parameters and costs millions of dollars to train for one iteration, excluding experimentation costs.The broader growth in model footprint is also associated with increased training resources and prediction latency.
  • Motivation: High-performing models may still be unsuitable for real-world deployment because their efficiency does not meet practical constraints.The paper highlights deployment challenges involving cost, server-side resources, carbon footprint, and on-device requirements.
  • Efficiency Dimensions: Inference efficiency concerns parameters, disk size, RAM consumption, and latency, while training efficiency concerns duration, device count, memory fit, and data needs.These metrics correspond to the questions practitioners face when deploying or training models.
  • Efficiency Dimensions: For equally accurate models, practitioners may prefer the one with better inference or training efficiency, especially under constrained or costly resources.Inference efficiency is particularly relevant for mobile, embedded, and cloud deployment, while training efficiency matters when resources are limited or expensive.
  • Pareto Optimality: Pareto-optimal models achieve the best quality–footprint tradeoffs, forming a frontier that efficiency work should discover and improve.A Pareto-optimal model has no alternative with better accuracy at the same latency, or lower latency at the same accuracy.
  • Approach: The paper proposes combining algorithms, techniques, tools, and infrastructure to train and deploy models that are Pareto-optimal for quality and footprint.This frames efficiency as a coordinated system spanning model quality and resource-related metrics.

2 A MENTAL MODEL

The paper organizes efficient deep learning into five areas: compression, learning, automation, efficient architectures, and infrastructure. The first four target modeling, while infrastructure and tools enable deployment of the resulting efficiency gains.

  • Framework: The framework divides efficient deep learning into five areas, with four focused on modeling and one on infrastructure and tools.The five areas are compression techniques, learning techniques, automation, efficient architectures, and infrastructure.
  • Compression Techniques: Compression techniques reduce layer representation size, such as replacing 32-bit floating-point weights with 8-bit unsigned integers through quantization.The goal is to reduce precision with minimal quality loss.
  • Learning Techniques: Learning techniques change training to reduce errors, data needs, or convergence time, with distillation improving a smaller model by mimicking a larger one.Improved quality can be exchanged for a smaller or more efficient model by trimming parameters.
  • Automation: Automation includes hyper-parameter optimization and architecture search for improving accuracy, latency, model size, or related metrics.Architecture search tunes the architecture itself to optimize loss or accuracy together with another footprint metric.
  • Efficient Architectures: Efficient architectures replace baseline building blocks with more efficient alternatives, including parameter-sharing convolutional layers for image classification.The paper contrasts convolutional layers with fully connected layers and attention with recurrent neural networks.
  • Infrastructure: Infrastructure includes training frameworks and deployment tools, whose platform support is required to realize gains such as quantized-model size and latency improvements.Inference platforms must support common neural-network layers in quantized mode.

3 LANDSCAPE OF EFFICIENT DEEP LEARNING

Efficient deep learning combines generic compression and learning techniques with hardware- and representation-aware designs to reduce footprint metrics while preserving quality. The survey highlights pruning and quantization as practical efficiency approaches, but notes that theoretical savings do not always translate into deployment gains.

  • Efficiency techniques: Compression techniques reduce model size, inference latency, or training time while aiming to minimize quality loss.They may also improve generalization when models are over-parameterized.
  • Pruning: Pruning removes a minimal subset of parameters while keeping model quality above a desired threshold; sparsity is the fraction of parameters removed.Higher sparsity means fewer non-zero parameters remain.
  • Pruning: Classical pruning iteratively removes parameters with low saliency, fine-tunes the remaining network, and repeats until the target pruning level is reached.OBD estimates saliency using diagonal Hessian elements rather than the full Hessian, which would be unwieldy for large networks.
  • Pruning: Structured pruning can improve practical latency: sparse convolutional networks achieved 1.3–2.4× speedups with ≈66% of parameters while retaining Top-1 accuracy.The approach uses a BCSR representation and fast kernels on ARM devices and WebAssembly, with constraints on which sparse networks can be accelerated.
  • Quantization: Quantization reduces 32-bit floating-point weights and activations to lower-precision values, enabling smaller models and potentially lower inference latency.Latency improvements require fixed-point activations so operations throughout the quantized graph use fixed-point arithmetic.
  • Quantization: 8-bit quantization-aware training produced a model close to the floating-point baseline while using 4× less disk space and running 1.64× faster.Post-training quantization also reduced model size 4× but had slightly higher latency because weights required dequantization during inference.

Discussion:

The paper surveys compression and learning techniques that reduce model footprint while preserving or improving quality, with quantization and distillation as prominent examples. These methods trade training complexity or numerical precision for smaller, faster deployable models.

  • Quantization: 8-bit quantization often reduces model size by 4× while also lowering inference latency.Activation quantization additionally reduces working memory for intermediate computations, which is important on low-memory devices.
  • Quantization: Quantization-aware training generally achieves better accuracy than post-training quantization, although representative data can reduce post-training losses.Fixed-point operations and folding common operations such as batch normalization and activations can further improve performance.
  • Other compression techniques: Other active compression techniques include low-rank matrix factorization, K-means clustering, and weight sharing for further compressing model hotspots.
  • Learning techniques: A smaller neural network can mimic large ensembles while being 1000× smaller and faster, transferring ensemble knowledge into one model.The approach uses synthetic data labeled by large ensembles before training the smaller network.
  • Distillation: Distillation trains a student with both hard ground-truth labels and softened teacher outputs, while the pre-trained teacher remains frozen.The combined loss weights the ground-truth and distillation objectives through λ1 and λ2.
  • Distillation: Distillation has been empirically shown to improve student accuracy and convergence speed across domains, enabling smaller models suitable for deployment.Reported applications include matching a 10-model speech-recognition ensemble and improving shallow students on CIFAR-10.

Discussion:

Learning techniques improve model quality or data efficiency through methods such as distillation, augmentation, and self-supervision. These approaches can reduce labeling or training demands while preserving or improving task performance.

  • Distillation: Distillation can improve student predictions using teacher outputs, including intermediate representations and pseudo-labels from unlabeled data.Intermediate-layer distillation adds losses matching semantically corresponding teacher and student outputs.
  • Data Augmentation: Data augmentation synthetically expands labeled datasets with transformations whose labels remain unchanged or can be inferred cheaply.Augmentations include label-invariant and label-mixing transformations.
  • Data Augmentation: 2048 × effective dataset expansion helped AlexNet win the 2012 ImageNet competition, while augmentation has since become common in image classification.The cited transformations have also been demonstrated to improve convolutional-network accuracy.
  • Data Augmentation: AutoAugment improves accuracy through learned augmentation policies, but its ImageNet search required as many as 15000 GPU hours.The approach uses reinforcement-learning-based search over transformations and their hyper-parameters.
  • Self-Supervised Learning: SimCLR achieved 73.9% Top-1 ImageNet accuracy with only 1% labels and outperformed the supervised ResNet-50 baseline with 10% labels.It learns representations by maximizing similarity between augmented views of the same input and minimizing similarity for dissimilar inputs.
  • Self-Supervised Learning: Self-supervised learning uses unlabeled data and pretext tasks before fine-tuning, improving convergence and enabling high quality with fewer labeled examples.This pattern has been demonstrated across NLP, vision, and speech.

3.3 Automation

Automation searches training configurations and architectures to improve model quality or efficiency with less manual effort. Search methods trade search cost, parallelism, and guidance from prior trials.

  • Hyper-Parameter Optimization (HPO): Hyper-Parameter Optimization searches values such as learning rate and weight decay that affect convergence and network architecture.The search approximates an intractable hyper-parameter space with a finite set of trials.
  • Hyper-Parameter Optimization (HPO): Random Search can outperform Grid Search when the search space has low effective dimensionality because it samples values independently across trials.Grid Search evaluates all specified combinations and suffers from the curse of dimensionality.
  • Hyper-Parameter Optimization (HPO): Bayesian Optimization sequentially selects trials using a surrogate objective model and acquisition function informed by completed trials.It can require fewer trials than random search, but sequential guidance can waste work when trials run in parallel.
  • Hyper-Parameter Optimization (HPO): Early stopping terminates trials whose intermediate performance is not promising, reducing training resources spent on poor configurations.Vizier’s Median Stopping Rule compares a trial’s performance at time t with the median of trials run so far.
  • Other Search Algorithms: Population Based Training combines parallel search and training by periodically replacing weaker trials with stronger weights and perturbing their hyper-parameters.It applies to adaptive hyper-parameters but not changes to model structure.
  • Neural Architecture Search (NAS): Neural Architecture Search can generate architectures with reinforcement learning, but one cited controller-training process required 22400 GPU hours.The controller generates architectural hyper-parameters such as filters, strides, and filter sizes.
  • Discussion: Automation can improve quality and reduce manual involvement, while Bayesian Optimization, Hyper-Band, and early stopping address high HPO costs.The survey identifies Tune, Vizier, and NNI as ready-to-use software packages.

3.4 Efficient Architectures

Efficient architectures redesign layers or models to reduce computation, parameters, memory, or latency while maintaining task quality. Examples span depth-separable convolutions, attention, and projection-based language models.

  • Computer Vision: Convolutional layers share filters spatially, preserving locality while using far fewer parameters than fully connected layers for image inputs.Pooling further reduces feature-map dimensions and processing cost.
  • Computer Vision: Depth-separable convolution decomposes convolution into point-wise and spatial operations, producing the same output shape with much fewer parameters.Its parameter count is (1 × 1 × input_channels × output_channels)+(s_x×s_y×output_channels), versus s_x×s_y×input_channels×output_channels for regular convolution.
  • Computer Vision: Xception reached ImageNet convergence sooner and higher accuracy than the cited baseline while keeping the number of parameters the same.It used depth-wise separable convolutions within the Inception architecture.
  • Natural Language Understanding: Self-Attention processes all input tokens together, enabling pairwise relationships in O(1) rather than the RNN sequence dependency of O(n).This parallelism makes optimized GPUs and TPUs easier to leverage.
  • Natural Language Understanding: Projection-based models replace embedding tables with lower-dimensional mappings, reducing model size for deployment.Locality Sensitive Hashing implementations preserve similarity in the lower-dimensional space.
  • Natural Language Understanding: Projection-based models reported comparable or strong NLU quality with substantially smaller models, including 100× and 350× reductions.PRADO used 20–40 ms inference on a Nexus 5X, while another cited model achieved 97.1% of BERT-like quality at 350× smaller size.
  • Natural Language Understanding: ProFormer reached ≈97.2% of BERT-base’s performance using 13% of its memory and 14.4 million versus 110 million parameters.It combines projection with localized attention in a Local Projected Attention layer.

3.5 Infrastructure

Efficient training and inference require coordinated software and hardware infrastructure. The survey covers graph compilers, mobile runtimes, low-level libraries, reduced precision, and specialized accelerators.

  • Infrastructure Overview: The infrastructure stack spans model training and inference on servers or devices such as mobile phones, IoT systems, and edge devices.The survey presents software and hardware components as interacting foundations for efficiency.
  • On-Device Inference: TensorFlow Lite supplies an interpreter and operation kernels optimized primarily for inference in low-resource ARM-based environments.Its tooling supports graph preparation, operation fusion, metadata, and post-training quantization.
  • Server-Side Acceleration: XLA compiles model graphs into customized kernels and can fuse operations into composite operations to optimize linear algebra computations.Graph-specific kernels avoid repeated intermediate work in fused operations.
  • PyTorch: PyTorch supports mobile runtimes, post-training quantization, graph optimizations, and JIT compilation through TorchScript.Its tuning guidance also includes mixed precision, operation fusion, and buffer checkpointing.
  • Hardware Libraries: ARM Neon SIMD instructions accelerate vectorized linear algebra, supported by libraries such as QNNPACK and XNNPACK.Accelerate and NNAPI abstract hardware acceleration decisions for higher-level frameworks.
  • GPU Acceleration: Reduced-precision Tensor Core MAC operations multiply fp16, bfloat16, or TensorFloat32 values while accumulating in fp32.The speedup comes from performing expensive matrix multiplication in lower precision.
  • Precision Formats: bfloat16 preserves fp32’s range while trading away precision, whereas fp16 reduces both precision and range.The format comparison distinguishes range and mantissa precision as separate floating-point properties.
  • Specialized Accelerators: TPUs are application-specific integrated circuits tuned for parallelizing and accelerating deep-learning linear algebra operations.Their specialization excludes the need to support non-machine-learning applications.

4 A PRACTITIONER’S GUIDE TO EFFICIENCY

This guide frames efficiency as navigating the quality–footprint trade-off toward Pareto-optimal models. It presents Shrink-and-Improve and Grow-Improve-and-Shrink strategies, then demonstrates their practical trade-offs with model scaling, distillation, data augmentation, and quantization.

  • Practical framing: Pareto-optimal models offer the best quality–footprint trade-offs, forming a Pareto frontier against which efficiency techniques can be evaluated.The guide considers quality and footprint dimensions such as accuracy, model size, and latency.
  • Strategies: Shrink-and-Improve reduces model capacity or footprint first, then recovers quality through learning techniques such as distillation and data augmentation.The strategy targets footprint-sensitive deployments while attempting to preserve quality.
  • Strategies: Grow-Improve-and-Shrink first increases capacity, improves quality, and then compresses the model back toward the original footprint.This strategy targets quality-sensitive deployments with constrained footprint metrics.
  • Experiments: 91.4× parameter growth and 80.2× model-size growth occur across the width-multiplier range, while latency increases 3.5–10× depending on device.The models vary width multipliers from 0.05 to 1.0; data augmentation and distillation improve quality, while quantization performs the shrink phase.
  • Experiments: 47.3× smaller size and 5.31× faster latency are achieved after quantizing the w = 0.25 model while retaining 84.18% accuracy.Before quantization, the model is 13.8× smaller and 2.94× faster than the target w = 1.0 model at comparable accuracy.
  • Experiments: A quality-sensitive example raises accuracy from an 80.6% baseline to 87.86% while keeping model size approximately comparable and making latency 1.6× faster.The final quantized model is 359.31 KB and has 0.58 ms latency on Galaxy S10.

5 CONCLUSION

The conclusion presents the survey as a mental model for the broad landscape of deep learning efficiency. It combines surveys of optimization techniques, infrastructure and hardware coverage, and actionable practitioner guidance with code.

  • Conclusion: The paper frames model efficiency as necessary because deep learning models and their training and deployment footprints have grown rapidly.The conclusion emphasizes that practitioners must make explicit or implicit efficiency decisions.
  • Conclusion: The survey organizes model efficiency into multiple focus areas and covers techniques, software libraries, and hardware supporting efficient training and inference.This organization is intended to help readers understand and apply the state of the art.
  • Conclusion: Actionable insights supplemented by code provide practitioners with guidance on optimizing models under quality and footprint constraints.The paper aims to help readers move from a sub-optimal model to one meeting both types of constraints.
Loading 2106.08962v2…