Source-linked AI summary
Rethinking Differentiable Search for Mixed-Precision Neural Networks
Zhaowei Cai, Nuno Vasconcelos
TL;DR
Uniform low-precision networks overlook filter-specific bit-width sensitivity, motivating mixed-precision search. EdMIPS uses differentiable, complexity-aware optimization and efficient composite convolution to search large models without proxy tasks, with mixed-precision models substantially outperforming uniform counterparts.
Problem
Uniform bit allocation ignores differing filter sensitivities, while mixed-precision search has a combinatorial N^L configuration space and loss-only optimization favors the highest bit-width.
Method
EdMIPS combines differentiable architecture search with a complexity-budgeted Lagrangian, joint gradient updates, and composite convolution based on weighted sums of parallel weights.
Results
EdMIPS searches Inception-V3 directly on ImageNet in 8 GPU days and substantially outperforms uniform low-precision counterparts across multiple popular networks.
Takeaways & Limitations
Mixed-precision search can produce strong, practical bit allocations for large networks without proxy tasks.
Takeaways & Limitations
Differentiable search ordinarily scales linearly in computation and memory with search-space dimension, making large-network search impractical without the proposed efficiency measures.
Abstract
from arXiv · showhide
Low-precision networks, with weights and activations quantized to low bit-width, are widely used to accelerate inference on edge devices. However, current solutions are uniform, using identical bit-width for all filters. This fails to account for the different sensitivities of different filters and is suboptimal. Mixed-precision networks address this problem, by tuning the bit-width to individual filter requirements. In this work, the problem of optimal mixed-precision network search (MPS) is considered. To circumvent its difficulties of discrete search space and combinatorial optimization, a new differentiable search architecture is proposed, with several novel contributions to advance the efficiency by leveraging the unique properties of the MPS problem. The resulting Efficient differentiable MIxed-Precision network Search (EdMIPS) method is effective at finding the optimal bit allocation for multiple popular networks, and can search a large model, e.g. Inception-V3, directly on ImageNet without proxy task in a reasonable amount of time. The learned mixed-precision networks significantly outperform their uniform counterparts.
1. Introduction
Low-precision networks reduce deployment costs, but uniform bit-widths ignore filter-specific sensitivities and can be suboptimal. EdMIPS performs differentiable, complexity-constrained mixed-precision search efficiently, including on large networks without proxy tasks.
- Mixed-precision allocation addresses uniform quantization’s failure to account for different filter properties and sensitivities.Uniform low-precision networks use the same bit-width everywhere, although filters differ in location, structure, and parameter cardinality.
- The search space grows as N^L for L layers and N candidate bit-widths, making manual optimization impractical.MPS also differs from generic NAS because proxy tasks can be ineffective and loss-only optimization favors the highest bit-width.
- EdMIPS searches mixed-precision allocations with a differentiable architecture, complexity-budgeted Lagrangian optimization, simplified joint updates, and efficient composite convolution.The composite convolution exploits homogeneous candidate filters so training complexity remains independent of search-space size.
- EdMIPS searches Inception-V3 directly on ImageNet in 8 GPU days and substantially outperforms uniform low-precision solutions across AlexNet, ResNet, GoogLeNet, and Inception-V3.The reported evaluations include networks of varied sizes and properties, with uniform solutions serving as baselines.
2. Related Work
Prior low-precision methods largely use uniform quantization, while bit-allocation research explores sensitivity-based, analytical, and sequential approaches to assigning precision.
- Uniform low-precision methods quantize weights and activations consistently, improving efficiency but historically suffering accuracy losses relative to full precision.Later methods, including HWGQ-Net, LQ-Net, and PACT, improved accuracy or learned quantization parameters while remaining uniform.
- Mixed-precision research assigns bit-widths across network components using filter-sensitivity checks, SQNR optimization, and sequential allocation strategies.These approaches address precision allocation rather than applying one bit-width throughout the network.
3. Low-Precision Neural Network
The paper models neural networks as collections of filtering operators and uses low-precision quantization, while motivating mixed precision through filter-specific sensitivity differences. In an Inception module, raising precision for selected filters can improve accuracy at different computational costs.
- 3.1. Deep Neural Network: A neural network is represented by learned filters that map inputs through weighted convolution or matrix-vector operations and nonlinear activations.The filters are learned by minimizing classification risk defined by cross-entropy on training data.
- 3.2. Low-Precision Neural Network: Mixed-precision networks allow arbitrary weight and activation bit-widths, using HWGQ-Net quantization as the paper’s starting point.The framework uses uniform quantizers for hardware compatibility.
- 3.3. Filter Sensitivity: Uniform bit allocation can be suboptimal because filters differ in their sensitivity to bit-width.This motivates allocating precision at a finer granularity than the whole network or all layers.
- 3.3. Filter Sensitivity: More than 2% accuracy improvement accompanies increasing “ch1x1” from 2-bit to 4-bit, with only a 25% computation increase.The sensitivity experiment changes one Inception-module filter at a time relative to a uniform 2-bit GoogLeNet baseline.
- 3.3. Filter Sensitivity: The “ch3x3” filter raises computation to 125% for only a 1.3% accuracy gain, whereas “ch5x5red” changes accuracy and computation only slightly.These differing trade-offs support the potential advantage of mixed-precision networks over uniform networks.
4. Mixed-Precision Network
Mixed-precision search assigns bit-widths across filters and activations while optimizing accuracy under a user-defined complexity constraint. EdMIPS makes this search differentiable and efficient by exploiting homogeneous candidate filters, weight sharing, and direct gradient-based optimization.
- Mixed-Precision Network: Mixed-precision search targets an optimal bit-width for each filter, but its discrete search space grows as N^L for L layers and N candidates.This makes manually crafting the optimal configuration infeasible.
- Complexity-Aware Learning: EdMIPS optimizes bit-width configurations by minimizing classification risk under a complexity constraint, avoiding the trivial all-highest-bit solution.The complexity term can represent computation, memory, model size, energy, or running speed.
- Efficient Composite Convolution: Homogeneous candidate filters let EdMIPS replace multiple parallel convolutions with one convolution using a composite filter.The candidates are replicas of the same filter with different bit-widths, unlike heterogeneous NAS operators.
- Efficient Composite Convolution: The composite-convolution design keeps training time constant with respect to the number of candidate bit-widths per filter, enabling large-network training.This avoids applying every candidate convolution separately to the same input.
- Efficient Composite Convolution: Weight sharing accumulates gradients from all branches into one universal weight tensor, preventing low-probability branches from receiving too few updates.A branch with probability 0.1 receives only 10% of the overall gradients when weights are not shared.
- Learning: EdMIPS learns architecture and weight parameters by gradient descent, using vanilla end-to-end backpropagation by default or a more expensive alternating strategy.Both approaches avoid the impracticality of bilevel optimization for large models and datasets without proxy tasks.
- Architecture Discretization: After continuous search, winner-take-all discretization selects the highest-probability branch, while sampling produces random architectures with high variance.The default winner-take-all strategy yields a deterministic architecture determined by relative selector rankings.
5. Experiments
EdMIPS searches mixed-precision allocations across diverse ImageNet networks and substantially outperforms uniform low-precision baselines. Experiments also show how complexity constraints, search design choices, and efficient composite convolutions affect the resulting models and search cost.
- Effect of Complexity Constraint: Stronger complexity constraints favor lower bit-widths and can change final layer decisions from 4-bit to 2-bit, showing that the selected architecture depends on the constraint.The comparison uses η = 0.001 and η = 0.002 during ResNet-18 search.
- Comparison to Uniform Bit Allocation: EdMIPS improves uniform 2-bit HWGQ-Net accuracy by about 0.9 point on AlexNet, 0.8 on ResNet-18, 2.8 on GoogLeNet, 1.5 on ResNet-50, and 1.7 on Inception-V3.The comparison covers five popular networks and reports substantial gains between the weakest and strongest search-space endpoints.
- Learned Optimal Bit Allocation: Learned allocations vary by architecture and layer sensitivity: residual connections often receive high bit-widths, while expensive GoogLeNet 3x3 filters frequently receive lower bit-widths.AlexNet input/output-adjacent layers and GoogLeNet filters with higher sensitivity receive relatively higher allocations.
- Ablation Studies: Ablations find little cost from discretization, no accuracy difference from alternating optimization, and lower variance with multinomial than categorical sampling.The default shared-weight, single-pass procedure is therefore retained in the reported EdMIPS configuration.
- Ablation Studies: Efficient composite convolution reduces computation by 30-50% and memory by 20-40%, cuts ResNet-18 search time from 35 to 18 hours on 2 GPUs, and limits full search overhead to about 45%.Weight sharing reduces model size by almost four, although practical savings are smaller than theoretical predictions because of additional bottlenecks.
6. Conclusion
EdMIPS is an efficient differentiable framework for mixed-precision network search that scales to large ImageNet models. Its learned models substantially outperform uniform low-precision counterparts and establish baselines for future MPS research.
- EdMIPS searches large models directly on ImageNet at affordable cost while substantially outperforming uniform low-precision counterparts.
- EdMIPS provides a differentiable framework for mixed-precision network search with multiple contributions tailored to the MPS problem.
- The results establish a solid set of baselines for future mixed-precision search research.