PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Differentiable Search for Mixed-Precision Neural Networks

Zhaowei Cai, Nuno Vasconcelos

arXiv:2004.05795v1cs.LGcs.CVcs.NE

TL;DR

Low-precision networks reduce computation and storage, but uniform bit allocation assigns the same bit-width across filters despite their differing properties. EdMIPS applies differentiable, complexity-aware search to learn mixed-precision networks, which substantially outperform uniform counterparts across multiple popular networks.

  • Problem

    Low-precision networks reduce computation and storage, but uniform bit allocation assigns the same bit-width across filters despite their differing properties.

    Source: §1 · p. 1

  • Method

    EdMIPS uses differentiable architecture search with a complexity budget to optimize the accuracy–complexity trade-off, updating architecture and network parameters in one forward-backward pass.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Across multiple popular networks, learned mixed-precision models substantially outperform uniform low-precision counterparts.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The learned mixed-precision models establish baselines for future mixed-precision network search research.

    Source: §6 · p. 9

    Differentiable architecture search has computation and memory costs that grow linearly with search-space dimension, making searches of large networks impractical and typically prompting proxy tasks.

    Source: §4.4 · p. 4

Abstract

from arXiv · show

Low-precision networks, with weights and activations quantized to low bit-width, are widely used to accelerate inference on edge devices. However, current solutions are uniform, using identical bit-width for all filters. This fails to account for the different sensitivities of different filters and is suboptimal. Mixed-precision networks address this problem, by tuning the bit-width to individual filter requirements. In this work, the problem of optimal mixed-precision network search (MPS) is considered. To circumvent its difficulties of discrete search space and combinatorial optimization, a new differentiable search architecture is proposed, with several novel contributions to advance the efficiency by leveraging the unique properties of the MPS problem. The resulting Efficient differentiable MIxed-Precision network Search (EdMIPS) method is effective at finding the optimal bit allocation for multiple popular networks, and can search a large model, e.g. Inception-V3, directly on ImageNet without proxy task in a reasonable amount of time. The learned mixed-precision networks significantly outperform their uniform counterparts.

1. Introduction

Uniform low-bit quantization reduces deployment costs but ignores filter-specific requirements, while mixed-precision search must handle a huge, task-sensitive allocation space. EdMIPS adapts differentiable search to these constraints and reports results without proxy tasks.

1. Introduction

  • Uniform low-precision networks use the same bit-widths [14] [23] [35] [4] [37] [34], but mixed-precision networks [18] [17] [33] [32] [31] [9] optimize bit-widths per filter to avoid suboptimal allocations.Mixed-precision representations are becoming practically relevant as hardware support emerges, including NVIDIA AMP1's choice among floating-point representations during training.Source: §1 · p. 1
  • With L layers and N candidate bit-widths, MPS has N^L configurations, making manual optimal allocation infeasible.Proxy tasks transfer poorly because layer importance and optimal allocations can differ between datasets such as CIFAR-10 and ImageNet; classification loss alone can favor the highest bit-width.Source: §1 · pp. 1–2
  • EdMIPS uses a differentiable search architecture to avoid proxies and constrains learning with a complexity budget, optimizing the accuracy-complexity trade-off through a Lagrangian.The architecture is motivated by DARTS, while the budget addresses the trivial tendency to select the highest bit-width.Source: §1 · p. 2
  • EdMIPS updates architecture and network parameters in one forward-backward pass and uses a weighted-sum composite convolution to keep training complexity constant as the search space grows.The simpler update avoids DARTS's expensive second-order bi-level optimization; the composite convolution exploits convolution linearity and enables training large networks.Source: §1 · p. 2
  • In 8 GPU days, EdMIPS searched the optimal mixed-precision Inception-V3 with 93 filters on ImageNet and outperformed uniform low-precision solutions by a large margin.Evaluations on AlexNet, ResNet, GoogLeNet, and Inception-V3 establish baselines for mixed-precision networks.Source: §1 · p. 2

2. Related Work

Prior work developed uniform low-precision networks, layerwise bit-allocation methods, and neural architecture search; recent mixed-precision search methods inherit limitations in efficiency and applicability.

Uniform low-precision

  • Uniform low-precision methods progressed from joint weight-and-activation binarization [14] [23] to improved accuracy methods [35] [4], learned step sizes [34], and clipping [7].The cited methods remain uniform low-precision networks.Source: §2 · p. 2

Bit allocation

  • Earlier bit-allocation methods examined filter sensitivities by layer, optimized layerwise SQNR [18], or allocated weight precision sequentially until a bit budget ran out [17].The paper characterizes these methods as predating NAS and as less powerful, less applicable to practical network design, or both.Source: §2 · p. 2

Mixed Precision

  • Earlier mixed-precision search methods cast MPS as NAS, but RL-based approaches are inefficient and a DARTS-based approach requires a proxy task and samples dozens of architectures.The paper positions EdMIPS as simpler, more efficient, effective, and applicable to a broader set of networks than these approaches.Source: §2 · p. 2

Neural architecture search

  • NAS methods include reinforcement learning [38], supernets [3] [10], and DARTS [20], while the RL search for CIFAR-10 required 1,000 GPU days.DARTS relaxes the discrete search space to enable gradient-descent optimization; the paper notes NAS’s large, discrete search spaces make it expensive.Source: §2 · p. 2

3. Low-Precision Neural Network

Low-bit quantization can reduce model size and computation, but uniform bit allocation may be suboptimal because filters have different sensitivities. The section describes the HWGQ quantization approach and illustrates why mixed-precision allocation may outperform uniform allocation.

3.1. Deep Neural Network

  • Quantizing weights and activations to low bit-width replaces expensive floating-point operations with efficient bit operations, reducing model size and computation [23] [4].The network’s filters are learned by minimizing classification risk defined by cross-entropy loss on the training set.Source: §3.1 · p. 3

3.2. HWGQ Network

  • HWGQ-Net [4] supports higher-precision weights through activation quantization, with Gaussian quantizers precomputed by Lloyd’s algorithm [21, 22], rescaled for weight variance, and made uniform for hardware.The work starts from HWGQ-Net, which uses binary weights; its activation quantization technique is used to produce weights of higher precision.Source: §3.2 · p. 3

3.3. Filter Sensitivity

  • In GoogLeNet [29], changing ch1x1 from 2 to 4 bits improved accuracy by more than 2% with only a 25% computation increase over the uniform 2-bit baseline.Source: §3.3 · p. 3

4. Mixed-Precision Network

EdMIPS searches filter-specific weight and activation bit-widths by balancing classification risk against user-defined complexity. It relaxes the discrete search into a differentiable architecture and uses efficient convolution, learning, and discretization strategies to make mixed-precision search practical.

4. Mixed-Precision Network

  • The search assigns each filter its own weight and activation bit-width from candidate pools, but the large discrete configuration space makes hand-designing an optimum infeasible.The goal is an optimal bit-width configuration for the whole network.Source: §4 · p. 3

4.1. Complexity-Aware Learning

  • Complexity-aware learning minimizes classification risk subject to a complexity bound, avoiding the trivial choice of the highest bit-width everywhere.The complexity target is user-defined and may reflect computation, memory, model size, energy, or running speed.Source: §4.1 · p. 4

4.2. Model Complexity

  • Within that user-defined complexity, model cost can be measured by FLOPs or BitOps and normalized relative to the first layer.BitOps accounts for the bit-widths of both weights and activations.Source: §4.1 · p. 4, §4.2 · p. 4

4.3. Relaxed Mixed-Precision Network

  • Relaxing the binary search space into continuous architecture parameters enables gradient descent instead of combinatorial optimization over bit-width configurations.The relaxation follows the approach suggested by [20].Source: §4.3 · p. 4

4.4. Efficient Composite Convolution

  • Composite convolution combines homogeneous bit-width branches into one convolution, making training time independent of the number of candidates.Sharing a universal weight tensor prevents low-probability branches from being under-trained and keeps search-model size constant; the method assumes the shared weights represent multiple quantizers, justified by similar branch representations and the goal of optimizing bit allocation.Source: §4.4 · pp. 4–5

4.5. Search Space

  • EdMIPS decouples weight and activation branches, retaining the full Cartesian search space while reducing computation and memory; it uses Bα = {1, 2, 3, 4} and Bβ = {2, 3, 4}.Earlier coupled-pair approaches manually pruned the possible pairs, which the decoupled design avoids.Source: §4.5 · p. 5

4.6. Learning

  • The default learning strategy jointly updates architecture parameters and weights by end-to-end backpropagation; alternating updates are also considered but have twice its complexity.Both strategies optimize the Lagrangian by gradient descent, and the paper reports them as efficient and effective for MPS.Source: §4.6 · p. 5

4.7. Architecture Discretization

  • After learning, the default winner-take-all discretization selects the highest-probability branch; single-trial sampling has high variance, while multiple trials approach winner-take-all.The passage gives n = 50 as an example of multiple trials; as n →∞, the sampled architecture converges to winner-take-all.Source: §4.7 · p. 5

5. Experiments

Experiments evaluate EdMIPS on ImageNet and extend mixed-precision search beyond simple networks to complex ResNet-50, GoogLeNet, and Inception-V3, establishing broad baselines for future MPS studies.

5.2. Architecture Evolution

  • In ResNet-18’s {2, 4}-bit search, lower bits dominate early, then 4-bit choices emerge as network weights improve; weights and activations can select different bit-widths.The complexity penalty avoids always choosing the highest bit-width; residual connections may receive 4 bits early because their complexity penalty is smaller than that of 3×3 convolutions.Source: Figure 4 · p. 6, §5.2 · p. 6

5.1. Implementation Details

  • Search used 25 epochs before winner-take-all discretization, then classifiers trained for 50 epochs for exploration or 95 for final comparison.All models were trained from scratch; auxiliary losses were removed for GoogLeNet and Inception-V3.Source: §5.1 · p. 6

5.3. Effect of Complexity Constraint

  • Increasing η reinforces low-bit preferences, delays switches to higher bits, and can change final choices from 4 to 2 bits, showing that the selected architecture depends on the complexity constraint.For example, choices switched to 2 bits for weights in layers 6/8/11 and activations in layers 18–19.Source: §5.3 · p. 6

5.4. Comparison to Uniform Bit Allocation

  • About 0.9, 0.8, 2.8, 1.5, and 1.7 points: EdMIPS improves uniform 2-bit HWGQ-Net on AlexNet, ResNet-18, GoogLeNet, ResNet-50, and Inception-V3, respectively.Source: Figure 5 · p. 7, §5.4 · p. 7

5.5. Learned Optimal Bit Allocation

  • At roughly the uniform 2-bit models’ BitOps, learned allocations vary by layer: cheap residual and 1×1 filters often receive more bits, while expensive GoogLeNet 3×3 filters often receive fewer.ResNet-50 allocations are closer to uniform because bottleneck filters have similar computation; Inception-V3’s module-D takes the lowest width and many module-E filters receive higher widths.Source: Figure 6 · p. 7, §5.5 · pp. 7–8

5.6. Ablation Studies

  • Search complexity usually changes only slightly after epoch 15, and classifiers at epoch 15 are comparable to those after 25; a coarser search space yields no gain over uniform models.For the search space of (17), efficient composite convolution saves 30–50% computation and 20–40% memory versus vanilla parallel convolutions, reducing ResNet-18 search time on 2 GPUs from 35 to 18 hours; Inception-V3 search can run with 8 GPUs and 12GB memory.Source: §5.6 · p. 8
  • Discretization causes little accuracy loss, shared candidate weights work as well as unshared weights, multinomial sampling is better and less variable than categorical sampling, and single-pass optimization matches alternating optimization.Source: §5.6 · p. 8

5.7. Comparison to state-of-the-art

  • 3 points: EdMIPS gains over uniform HWGQ-Net on GoogLeNet; at similar BitOps to 2-bit uniform models, it achieves top performance across base networks, with AlexNet matching full precision.The paper could not compare directly with prior MPS works [31] and [32] because of their limited network or bit-width scope and unavailable reproduction code.Source: Table 1 · p. 9, §5.7 · p. 9

6. Conclusion

EdMIPS is a differentiable framework for mixed-precision network search that can search large models directly on ImageNet at affordable cost. Its learned models substantially outperform uniform low-precision counterparts and provide baselines for future MPS research.

6. Conclusion

  • EdMIPS’s learned mixed-precision models substantially outperform their uniform low-precision counterparts and establish baselines for future MPS research.The conclusion reports results across multiple popular networks.Source: §6 · p. 9
  • EdMIPS searches large models, including ResNet-50 and Inception-V3, directly on ImageNet at affordable cost.The framework is based on a differentiable architecture for mixed-precision network search.Source: §6 · p. 9

Related papers

Loading 2004.05795v1…