Source-linked AI summary
I-Parakeet: Integer-Only Conformer ASR on Mobile NPU
Taichi Nishimura
TL;DR
Large Conformer ASR models strain edge-device memory, latency, and power. I-Parakeet uses integer-only attention, Swish approximation, and activation-range remedies to run Parakeet-CTC on a smartphone NPU, reaching 4.97% WER on LibriSpeech test-other at an RTF of 0.048.
Problem
At 0.6B parameters, Parakeet-CTC makes on-device deployment challenging because floating-point inference strains edge-device memory, latency, and power.
Source: §1 · p. 1Method
I-Parakeet combines integer relative-positional attention, a minimax Swish approximation, and targeted activation-range remedies in an integer-only Parakeet-CTC implementation.
Source: §1 · p. 1, §2.2 · p. 2Results
4.97% WER on LibriSpeech test-other at an RTF of 0.048 on a smartphone NPU.
Source: §5 · p. 4Takeaways & Limitations
The result shows that modern Conformer ASR can run without floating-point inference on a smartphone NPU.
Source: §5 · p. 4
Abstract
from arXiv · showhide
In this paper, we propose I-Parakeet, an integer-only implementation of NVIDIA's Parakeet-CTC (0.6B parameters) that runs on a smartphone NPU without any floating-point operator or CPU fallback. Modern Conformer ASR models are hard to deploy on edge devices because of their size, and quantized models still fall back to floating point for numerically sensitive operations. This prevents them from fully exploiting integer accelerators such as mobile NPUs. To achieve this, our contributions are threefold. First, we derive an integer formulation of the relative-positional self-attention at the core of the Conformer. We fuse its two score branches with different quantization scales and the relative shift into integer-only operations. Second, we introduce a minimax-optimized Swish approximation that minimizes the maximum error of the Swish output. Third, a layer-wise range analysis of activations yields two targeted remedies: an INT16 grid for the BatchNorm output and percentile calibration for the heavy-tailed pre-encoder activations. I-Parakeet achieves 4.97% WER on LibriSpeech test-other, running on a Qualcomm NPU at a real-time factor of 0.048, 7.5x faster than a CPU baseline.
1. INTRODUCTION
Modern Conformer ASR models have grown difficult to deploy on edge devices, while quantized ASR often still relies on floating-point operations. I-Parakeet addresses the untested question of whether an integer-only Conformer can run on a mobile NPU.
1. INTRODUCTION
- Prior integer-only ASR hardware evidence was limited to GPUs, leaving mobile-NPU deployment of an integer-only Conformer unexplored.Earlier work demonstrated integer-only inference for CNN- and Conformer-based ASR models.Source: §1 · p. 1
- I-Parakeet runs NVIDIA’s Parakeet-CTC on a consumer smartphone NPU with no floating-point operator or CPU fallback.The paper calls this a deployment property of integer-only inference.Source: §1 · p. 1
- I-Parakeet fuses relative-positional attention’s differently scaled score branches with the relative shift in integer-only operations.The method addresses the scale mismatch between the two score branches and the zero-padding interleaved by the relative shift.Source: §1 · p. 1
- A layer-wise range analysis assigns INT16 to the BatchNorm output and percentile calibration to pre-encoder activations, while other tensors remain INT8.Using a wider grid for every tensor would harm NPU throughput.Source: §1 · p. 1
2. I-PARAKEET OVERVIEW
Parakeet-CTC uses a pre-encoder, 24 Conformer blocks, and a linear CTC head. I-Parakeet quantizes the network and addresses operations and activation ranges that require specialized integer handling.
2.1. Parakeet-CTC
- Parakeet-CTC comprises a convolutional pre-encoder, 24 Conformer blocks, and a linear CTC head.The pre-encoder reduces the input frame rate by a factor of eight, then projects features to model dimension 1024.Source: §2.1 · p. 2
2.2. Quantization of Parakeet-CTC
- I-Parakeet uses uniform symmetric quantization, with tensor clipping ranges calibrated offline and bit width set to 8 unless otherwise stated.Activations use one clipping range per tensor, while weights use a range per channel.Source: §2.2 · p. 2
- Linear layers accumulate integer products in INT32 and requantize using a precomputed fixed-point multiplier and right shift, without floating-point operations or division.Convolutions follow the same rule; BatchNorm is folded into the preceding depthwise convolution, and LayerNorm and Softmax use I-BERT integer kernels.Source: §2.2 · p. 2
- The design targets integer relative-positional attention, Swish approximation, and activation ranges that a single INT8 grid cannot cover.It uses INT16 for the BatchNorm output and 99.9th-percentile clipping for pre-encoder activations, while other tensors stay INT8.Source: §2.2 · p. 2
3. METHOD
The method makes relative-positional attention integer-only despite differently scaled score branches, then addresses Swish approximation error and activation-range outliers. It uses fused requantization, minimax fitting, and targeted changes to activation precision and calibration.
3.1. Integer Relative-Positional Self-Attention
- Integer attention combines content and position score branches with distinct calibrated scales through one requantization that fuses conversion, addition, and scaling.The relative shift can be applied directly to quantized position scores as a static index map.Source: §3.1 · p. 3
3.2. Minimax-Optimized Integer Swish
- The Swish approximation minimizes maximum output error by fitting Swish directly under L∞ rather than least-squares fitting tanh.The paper identifies the maximum error of the Swish output as the optimization target.Source: §3.2 · p. 3
3.3. Layer-wise Activation Range Analysis
- Range analysis assigns INT16 to BatchNorm outputs and 99.9th-percentile calibration to pre-encoder activations, while retaining min–max calibration in the encoder.BatchNorm scale spread exceeds four orders of magnitude in some layers; pre-encoder max-to-99.9th-percentile ratios reach 11–21×, compared with 2–3× inside the encoder.Source: §3.3 · p. 3
4. EXPERIMENTS
I-Parakeet runs Parakeet-CTC entirely on a smartphone NPU, outperforming the CPU baseline in speed and memory use at a WER cost. Simulation results show that its targeted range remedies recover accuracy over Naive INT8, while ablations evaluate Swish approximation, BatchNorm precision, and calibration choices.
4.1. On-Device Results
- On-device, I-Parakeet achieved 4.97% WER, running entirely on the NPU without CPU fallback, 7.5× faster than FP16 parakeet.cpp with 60% less peak memory at a 1.21-point WER cost.Both stock Qualcomm paths give 100% WER.Source: §4.1 · p. 4
4.2. Accuracy Analysis in Simulation
- 2.61% test-clean and 5.32% test-other WER are 0.74/1.56 points behind FP32; I-Parakeet recovers 0.40/1.06 points over Naive INT8, with the same ordering on Common Voice.Table 2 reports simulated WER on PyTorch; on-device results are in Table 1.Source: §4.2 · p. 4, Table 2 · p. 4
- The maximum-output-error ranking matches WER: L∞ beats L2, fitting Swish output beats fitting tanh, and both choices contribute independently.Hard-Swish degrades WER by one point.Source: §4.2 · p. 4, Table 3 · p. 4
- 5.91 WER results from per-tensor INT8 BatchNorm output; deployable per-tensor INT16 reaches 5.32, within 0.03 points of unsupported per-channel INT16 at 5.29.Table 3 reports ablations on LibriSpeech test-other WER.Source: §4.2 · p. 4, Table 3 · p. 4
- 8.03 WER with p99.9 calibration everywhere shows that clipping helps only where activations are heavy-tailed; hybrid scaling beats min–max everywhere.Table 3 reports ablations on LibriSpeech test-other WER.Source: §4.2 · p. 4, Table 3 · p. 4
5. CONCLUSION
I-Parakeet is an integer-only Parakeet-CTC-0.6B running on a smartphone NPU. It achieves 4.97% WER on LibriSpeech test-other at an RTF of 0.048, showing that modern Conformer ASR needs no floating-point inference.