Source-linked AI summary
SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture Search
Sharath Nittur Sridhar, Maciej Szankin, Fang Chen, Sairam Sundaresan, Anthony Sarah
TL;DR
Joint architecture and quantization-policy search is under-explored for Transformer and foundation-model settings. The paper combines elastic super-networks, lightweight predictor-based multi-objective search, and static PTQ, reporting improved efficiency across Transformer and convolutional models without accuracy degradation against INT8 baselines.
Problem
Quantization-policy search is well studied for CNNs but remains under-explored when jointly searching architecture and precision for Transformers and foundation models.
Source: §1 · pp. 1–2Method
Results
The approach improves latency, model size, and accuracy objectives across BERT, BEiT-3, ViT, and ResNet models without compromising accuracy.
Source: §4 · pp. 4–5, §5 · p. 5Takeaways & Limitations
Abstract
from arXiv · showhide
Recent one-shot Neural Architecture Search algorithms rely on training a hardware-agnostic super-network tailored to a specific task and then extracting efficient sub-networks for different hardware platforms. Popular approaches separate the training of super-networks from the search for sub-networks, often employing predictors to alleviate the computational overhead associated with search. Additionally, certain methods also incorporate the quantization policy within the search space. However, while the quantization policy search for convolutional neural networks is well studied, the extension of these methods to transformers and especially foundation models remains under-explored. In this paper, we demonstrate that by using multi-objective search algorithms paired with lightly trained predictors, we can efficiently search for both the sub-network architecture and the corresponding quantization policy and outperform their respective baselines across different performance objectives such as accuracy, model size, and latency. Specifically, we demonstrate that our approach performs well across both uni-modal (ViT and BERT) and multi-modal (BEiT-3) transformer-based architectures as well as convolutional architectures (ResNet). For certain networks, we demonstrate an improvement of up to $4.80x$ and $3.44x$ for latency and model size respectively, without degradation in accuracy compared to the fully quantized INT8 baselines.
1 Introduction
NAS improves deployment efficiency by searching specialized architectures, while quantization adds another efficiency dimension. Joint architecture–quantization search remains challenging because it expands computational demands and search spaces, motivating the proposed two-step approach.
- NAS systematically explores architectures for task-specific efficiency, while quantization addresses deployment constraints involving computation and energy use.Source: §1 · p. 1
- Jointly optimizing architecture and numerical precision expands both computational demand and the already extensive search space.Source: §1 · p. 2
- OFA uses dynamically adjustable super-networks and weight sharing to separate training from architecture search, reducing sub-network evaluation overhead.Source: §1 · p. 2
- OFA super-networks support changes in depth, width, kernel size, and resolution without retraining, but their training is complex and resource-intensive.Source: §1 · p. 2
- APQ extends joint search to quantization but remains costly, requires extensive predictor data, and was designed for CNNs rather than Transformers.Source: §1 · p. 2
- The paper addresses super-network cost with InstaTune-generated models and search inefficiency with the LINAS algorithm.Source: §1 · p. 2
2 Related Work
Joint quantization–architecture search methods use either QAT or PTQ, but both face efficiency or scope limitations. Existing Transformer PTQ methods mainly optimize quantization policy alone, motivating joint search over architecture and precision.
- Joint search methods are broadly categorized into quantization-aware training and post-training quantization.Source: §2 · p. 2
- QAT methods incorporate precision choices during training through meta-learning, reinforcement learning, differentiable super-networks, or adaptive statistics.Source: §2 · p. 2
- Training QAT-based super-networks can be extremely time-consuming and computationally expensive.Source: §2 · p. 2
- PTQ methods decouple training from quantization and support efficient low-precision conversion using reinforcement learning, Hessian information, or calibration methods.Source: §2 · p. 2
- Transformer PTQ approaches mainly search for quantization policies rather than jointly optimizing network architecture and quantization.Source: §2 · p. 2
3 Methodology
The methodology combines pretrained-model super-networks with predictor-based multi-objective search over architecture and quantization policy. It applies static PTQ to Transformer and convolutional architectures using a constrained INT8/FP32 precision space.
Super-network construction
- The approach uses InstaTune or conventional OFA super-networks, with InstaTune forming elastic models during downstream fine-tuning and knowledge distillation.Source: §3 · p. 3
Search algorithm
- LINAS iteratively trains lightweight predictors to accelerate multi-objective search toward a near-optimal Pareto frontier.Source: §3 · p. 3
Joint search
- The joint search space combines network parameters with weight and activation bit widths, reaching sizes from 10^34 to 10^50 across models.Source: §3 · p. 3
Quantization
- Selected sub-networks receive static PTQ with mixed INT8 and FP32 policies using Intel Neural Compressor and a representative calibration dataset.Source: §3 · p. 3
Architecture spaces
- Transformer super-networks vary layers, attention heads, and intermediate MLP dimensions, while OFA ResNet architectures vary depth, expansion ratio, and width multiplier.Source: Table 2 · p. 3, §3 · pp. 3–4, Table 3 · p. 3
4 Experiments
Across BERT, BEiT-3, ViT, and ResNet super-networks, joint architecture–quantization search improves accuracy–efficiency trade-offs, while LINAS improves search progression over alternatives.
4 Experiments
- The experiments span image classification, sentiment analysis, and multimodal classification using BERT, BEiT-3, ViT, and ResNet super-networks with accuracy plus latency or model size objectives.Figure 1 presents Pareto fronts against baselines whose linear layers are fully quantized to INT8.Source: §4 · p. 4, Figure 1 · p. 4
- 2.20x faster BERT configurations at equal accuracy and a 1.83x model-size reduction demonstrate gains over the INT8 baseline.The highest-accuracy BERT network also provides a 1.06x size improvement.Source: §4 · p. 4
- BEiT-3 produces a 1.02x smaller equal-accuracy model and a 1.12x faster comparable-accuracy model, while its highest-accuracy model is 0.69% better but 1.06x slower.Source: §4 · p. 4
- ViT achieves 1.97x latency improvement without accuracy loss, while its size-focused search yields 42.40% improvement at comparable accuracy.The highest-accuracy ViT sub-network reaches 78.77%, a 2.30% absolute improvement, with 1.20x better latency.Source: §4 · p. 4
- 2x, 3.06x, and 3.44x model-size reductions and 2.30x, 4.36x, and 4.8x latency improvements were achieved without accuracy loss for ResNet50, ResNet101, and ResNet152 baselines.Source: §4 · p. 5
4.1 Ablations on Quantization only search
- LINAS search progresses toward the near-optimal Pareto front as evaluations increase, and its hypervolume results are compared with NSGA-2 and Random Search.The ablation also compares joint search against quantization-only search and reports improved model size at similar accuracy.Source: §4.1 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5
5 Conclusions
The paper concludes that jointly searching architecture and quantization improves efficiency across varied models without compromising accuracy. It identifies broader precisions, domains, and large language models as future directions.
5 Conclusions
- Joint architecture and quantization search improves latency, model size, and classification accuracy across BERT, BEiT3, ViT, and ResNet50 without compromising accuracy.Source: §5 · p. 5
- Future work includes broader numerical precisions such as BFloat16 and INT4, wider application domains, and current research frontiers such as large language models.Source: §5 · pp. 5–6