PaperlayerView exact revision on arXiv

Source-linked AI summary

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size

Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, Kurt Keutzer

arXiv:1602.07360v4cs.CVcs.AI

TL;DR

CNN research has emphasized accuracy, while smaller architectures could offer practical advantages at equivalent accuracy. This paper introduces SqueezeNet, achieving AlexNet-level ImageNet accuracy with 50× fewer parameters and compressing it to less than 0.5MB.

  • Problem

    CNN research has emphasized accuracy, leaving a need to identify architectures with fewer parameters but equivalent accuracy and practical deployment advantages.

  • Method

    The paper proposes SqueezeNet through disciplined CNN design-space exploration using Fire modules and controlled architectural metaparameters.

  • Results

    50× fewer parameters than AlexNet are achieved at AlexNet-level ImageNet accuracy, while compression reduces model size by 510× to less than 0.5MB.

  • Takeaways & Limitations

    A SqueezeNet-like model can store its parameters entirely within an FPGA, eliminating off-chip memory accesses for model parameters.

  • Takeaways & Limitations

    Deep Compression uses codebooks and 6- or 8-bit quantization, so achieving a speedup is not trivial on most commodity processors.

Abstract

from arXiv · show

Recent research on deep neural networks has focused primarily on improving accuracy. For a given accuracy level, it is typically possible to identify multiple DNN architectures that achieve that accuracy level. With equivalent accuracy, smaller DNN architectures offer at least three advantages: (1) Smaller DNNs require less communication across servers during distributed training. (2) Smaller DNNs require less bandwidth to export a new model from the cloud to an autonomous car. (3) Smaller DNNs are more feasible to deploy on FPGAs and other hardware with limited memory. To provide all of these advantages, we propose a small DNN architecture called SqueezeNet. SqueezeNet achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters. Additionally, with model compression techniques we are able to compress SqueezeNet to less than 0.5MB (510x smaller than AlexNet). The SqueezeNet architecture is available for download here: https://github.com/DeepScale/SqueezeNet

1 INTRODUCTION AND MOTIVATION

The introduction motivates designing CNNs with fewer parameters while maintaining equivalent accuracy, citing efficiency benefits in training, model export, and hardware deployment. It presents SqueezeNet as such an architecture and introduces a disciplined exploration of CNN architecture design choices.

  • Motivation: Equivalent accuracy can be achieved by multiple CNN architectures, motivating the search for models with fewer parameters.The paper frames parameter reduction as a design objective rather than pursuing accuracy alone.
  • Motivation: Fewer parameters reduce communication overhead in distributed CNN training, enabling small models to train faster.For distributed data-parallel training, communication overhead is directly proportional to the number of model parameters.
  • Motivation: Smaller CNNs also reduce overhead when exporting new models to autonomous-driving clients.The passage gives periodic model copying for autonomous driving as an example.
  • Motivation: Limited-memory hardware further favors smaller CNN architectures, such as the Xilinx Vertex-7 FPGA with 8.5 MBytes of on-chip memory and no off-chip memory.The FPGA constraint illustrates why model size affects deployment feasibility.
  • Contribution: The paper introduces SqueezeNet as a CNN with fewer parameters but equivalent accuracy and presents a disciplined approach to searching for novel architectures.Later sections describe and evaluate SqueezeNet and explore its design space.

2 RELATED WORK

Prior work pursues smaller CNNs through lossy model compression and explores CNN microarchitectures, macroarchitectures, and automated design-space exploration. Related macroarchitecture studies examine depth and bypass connections, while this work evaluates SqueezeNet and its design choices.

  • Model compression: Model compression research seeks to reduce CNN parameters while preserving accuracy, including lossy compression of pretrained models using singular value decomposition.Denton et al. applied singular value decomposition to pretrained CNN models.
  • CNN microarchitecture: CNN microarchitecture research studies convolutional layers and higher-level modules that organize multiple convolution layers with fixed filter configurations.Examples include Inception modules using filters such as 1x1, 3x3, and sometimes 5x5.
  • CNN macroarchitecture: CNN macroarchitecture concerns the system-level organization of multiple modules into an end-to-end architecture.The paper distinguishes macroarchitecture from individual layers and modules, which comprise the microarchitecture.
  • CNN macroarchitecture: Recent macroarchitecture work studies depth and bypass connections, reporting higher ImageNet accuracy from deeper networks and using connections that skip multiple layers.VGG models span 12 to 19 layers, later CNNs reach up to 30 layers, and ResNet and Highway Networks use bypass connections.
  • Design-space exploration: Neural-network design-space exploration uses automated approaches to find architectures that deliver higher accuracy across microarchitectures, macroarchitectures, solvers, and hyperparameters.The paper then evaluates SqueezeNet and explores microarchitecture and macroarchitecture choices for SqueezeNet-like CNNs.

3 SQUEEZENET: PRESERVING ACCURACY WITH FEW PARAMETERS

SqueezeNet is designed to use few parameters while maintaining competitive accuracy through three CNN design strategies and a Fire-module-based architecture. It consists of a standalone convolution layer, eight Fire modules, and a final convolution layer, with pooling placed relatively late in the network.

  • 3.1 Design Strategies: SqueezeNet targets few parameters while maintaining competitive accuracy through three CNN architecture design strategies.The strategies reduce parameter quantity while attempting to preserve accuracy and maximize accuracy within a limited parameter budget.
  • 3.1 Design Strategies: 9X fewer parameters make 1x1 filters preferable to 3x3 filters for the majority of convolution filters.This is Strategy 1, applied under a fixed budget for the number of convolution filters.
  • 3.1 Design Strategies: Reducing the number of input channels to 3x3 filters further decreases a convolution layer’s parameter count.For an all-3x3 convolution layer, the parameter quantity is (number of input channels) * (number of filters) * (3*3).
  • 3.1 Design Strategies: Downsampling late keeps convolution layers’ activation maps large, supporting higher accuracy under a limited parameter budget.Strategy 3 concentrates strides greater than 1 later in the network rather than using large early strides.
  • 3.2 Fire Module: The Fire module combines a 1x1-only squeeze convolution layer with an expand layer mixing 1x1 and 3x3 filters.Its tunable dimensions are s1x1, e1x1, and e3x3.
  • 3.3 SqueezeNet Architecture: SqueezeNet contains conv1, eight Fire modules (fire2-9), and conv10, with stride-2 max-pooling after conv1, fire4, fire8, and conv10.The relatively late pooling placements implement Strategy 3.

4 EVALUATION OF SQUEEZENET

SqueezeNet is evaluated against AlexNet and prior model-compression methods on ImageNet. Compression reduces SqueezeNet to 0.47MB while preserving equivalent accuracy, though codebook quantization may limit commodity-processor speedups.

  • Evaluation basis: SqueezeNet uses AlexNet and associated compression results as the evaluation basis.The baseline is bvlc alexnet from the Caffe codebase.
  • Comparison with prior compression: 1.4× smaller model size than the best model-compression efforts is achieved with uncompressed 32-bit values while maintaining or exceeding baseline accuracy.
  • Compressed SqueezeNet: 0.66 MB yields a 363× smaller model than 32-bit AlexNet with equivalent accuracy using 33% sparsity and 8-bit quantization.
  • Compressed SqueezeNet: 0.47MB yields a 510× smaller model than 32-bit AlexNet with equivalent accuracy using 33% sparsity and 6-bit quantization.
  • Compressed SqueezeNet: 10× compression of SqueezeNet preserves baseline accuracy, showing that Deep Compression also applies to compact fully convolutional architectures.
  • Inference considerations: 4x and 5.3x are the nominal speedups from 8-bit and 6-bit quantization, respectively, but codebook quantization is not trivial to accelerate on most commodity processors.

5 CNN MICROARCHITECTURE DESIGN SPACE EXPLORATION

This section explores the microarchitectural design space of SqueezeNet-like CNNs, focusing on how architectural choices affect model size and accuracy rather than maximizing accuracy in every experiment. It uses higher-level metaparameters to sweep Fire-module dimensions, including squeeze ratio and the proportion of 1x1 versus 3x3 expand filters.

  • Section objective: The experiments aim to provide intuition about the microarchitectural design space and the effects of CNN architectural choices on model size and accuracy.The stated goal is analysis of design-space shape, not maximum accuracy in every experiment.
  • Metaparameters: 24 dimensional hyperparameters across 8 Fire modules are controlled through higher-level metaparameters for broad sweeps of SqueezeNet-like architectures.Each Fire module has squeeze and expand dimensions, while the metaparameters control dimensions across the CNN.
  • Squeeze ratio: The squeeze ratio defines squeeze-layer filters relative to expand-layer filters, with one shared SR value across all Fire modules.The metaparameter is specified in the range [0, 1].
  • Squeeze ratio: The squeeze-ratio experiment trains independent models from scratch with SR values from 0.125 to 1.0, using basee = 128, incre = 128, pct3x3 = 0.5, and freq = 2.SqueezeNet corresponds to the SR=0.125 point in this experiment.
  • Filter composition: The filter-composition experiment fixes basee = incre = 128, freq = 2, and SR = 0.500 while varying pct3x3 from 1% to 99%.This sweeps each expand layer from mostly 1x1 filters to mostly 3x3 filters across 8 Fire modules.

6 CNN MACROARCHITECTURE DESIGN SPACE EXPLORATION

The macroarchitecture exploration compared vanilla SqueezeNet with simple and complex bypass connections among Fire modules. Simple bypass connections produced the strongest reported improvement, increasing top-1 and top-5 accuracy without increasing model size.

  • Macroarchitecture variants: The study explored three macroarchitectures: vanilla SqueezeNet, SqueezeNet with simple bypass connections, and SqueezeNet with complex bypass connections.These designs changed high-level connections among Fire modules while holding the microarchitecture fixed to the SqueezeNet configuration.
  • Bypass design: The bypass connections add residual pathways around selected Fire modules, including Fire3, Fire5, Fire7, and Fire9, using elementwise addition for simple bypasses.For example, the input to Fire4 equals the output of Fire2 plus the output of Fire3.
  • Bypass design: Only half of the Fire modules can use simple bypass connections because straightforward bypasses require equal input and output channel counts.When channel counts differ, the architecture uses a complex bypass connection that includes a 1x1 convolution.
  • Motivation: With a squeeze ratio of 0.125, bypass connections were intended to alleviate the representational bottleneck caused by squeeze layers having 8x fewer output channels than expand layers.The dimensionality reduction limits how much information can pass through squeeze layers.
  • Results: 2.9 percentage-points in top-1 accuracy and 2.2 percentage-points in top-5 accuracy were achieved by simple bypass connections without increasing model size.Simple bypass connections yielded a higher accuracy improvement than complex bypass connections.

7 CONCLUSIONS

The paper presents SqueezeNet as a disciplined CNN design-space exploration outcome, achieving AlexNet-level ImageNet accuracy with 50× fewer parameters and a compressed size below 0.5MB. Its compactness also supports FPGA deployment and motivates broader, systematic exploration of CNN architectures.

  • 7 CONCLUSIONS: 50× fewer parameters than AlexNet while maintaining AlexNet-level accuracy on ImageNet.SqueezeNet was proposed as part of a more disciplined approach to convolutional neural network design-space exploration.
  • 7 CONCLUSIONS: Less than 0.5MB after compression, making SqueezeNet 510× smaller than AlexNet without compression.The paper reports this compressed model size as an additional reduction beyond the parameter-count comparison.
  • 7 CONCLUSIONS: An FPGA implementation stored a SqueezeNet-like model’s parameters entirely on-chip and eliminated off-chip memory accesses for loading them.Gschwend developed and implemented a SqueezeNet variant on an FPGA after the model’s release.
  • 7 CONCLUSIONS: ImageNet-trained CNN representations have been applied to fine-grained object recognition, logo identification, image caption generation, and autonomous-driving applications.The paper focused on ImageNet but notes these representations’ use across multiple application areas.
  • 7 CONCLUSIONS: SqueezeNet is one of several CNNs discovered through broad design-space exploration and is intended to encourage more systematic exploration of CNN architecture possibilities.The authors explicitly hope readers will explore the design space’s broad range of possibilities.
Loading 1602.07360v4…