16-bit Precision of Convolutional Neural Networks on Microcontroller Units for 8-bit Costs

1 Introduction

• MCUs often have less than 1 MB of memory and clock speeds of 40 to 500 MHz, while applications can demand real-time response.The cited applications include safety-critical health monitoring, industrial IoT, predictive maintenance, and autonomous vehicles.Source: §1 · p. 1
• Quantization reduces DNN memory use and enables integer arithmetic, but INT8 resolution can impair accuracy, especially in regression.The introduction motivates 16-bit quantization as a way to retain higher resolution without the expected increase in time and energy on ARMv7E-M.Source: §1 · p. 2
• W16A16 TCNs outperform mixed quantization and Cube.AI reference schemes while matching full FLOAT32 performance across classification and regression tasks.The authors also report computation counts, inference time, and energy consumption comparable to INT8 quantization.Source: §1 · p. 2
• The work analyzes ARMv7E-M instructions, implements W16A16 for TCNs as an open-source package, and provides the implementation online.The implementation is available at the cited project URL.Source: §1 · p. 2

2.1 Integer Quantization Foundations

• Quantization maps floating-point DNN weights and activations to integers, reducing memory use and accelerating computation; W16A16 uses 16-bit values for both.The notation specifies weight precision after W and activation precision after A.Source: §2.1 · pp. 2–3

2.1.1 Static v.s. Dynamic Quantization

• Static quantization fixes scale and zero-point parameters before inference, avoiding the computation overhead of dynamically selecting parameters on an MCU.Dynamic quantization can adjust parameters to incoming activations, but the passage says its selection overhead is infeasible on MCUs.Source: §2.1.1 · p. 3

2.1.2 Symmetric vs. Asymmetric Quantization

• W16A16 uses symmetric quantization for activations and weights: 16-bit activation range can fit the full range without major errors, avoiding asymmetric zero-point subtraction.Asymmetric activations require subtracting the zero point before each weight-activation multiplication; the passage says this overhead is unavoidable for 8-bit quantization without major errors.Source: §2.1.2 · p. 4

2.1.3 Per-layer vs. per-channel quantization

• The method uses per-layer quantization for activations and per-channel quantization for weights to account for output-channel variation.The passage describes per-channel weight quantization as standard for convolutional networks because of channel-wise variation.Source: §2.1.3 · p. 4

2.2 Multiply-and-accumulate operations over quantized data

• A convolution combines input and weight values through MAC operations, whose speed determines overall DNN execution speed on MCUs.Each MAC multiplies one input by one weight and adds the product to an accumulator; the resulting accumulator corresponds to an output channel.Source: §2.2 · pp. 4–5

2.2.1 Re-quantization

• After a layer's MACs, requantization reduces the wider accumulator to the activation bit width and converts its scale to the next layer's output scale.The accumulated value is multiplied by the requantization scaling factor and rounded back to 8 or 16 bits.Source: §2.2.1 · p. 5

2.2.2 Fixed-point approximation of float scaling

• W16A16 uses a Q31 fixed-point multiplier-and-shift method to approximate scaling, avoiding the large errors possible when scaling factors are restricted to powers of two.The passage gives a 25% error example for approximating a scale of 0.1 by the nearest power of two.Source: §2.2.2 · pp. 5–6

2.3.1 ARMv7E-M Dual MAC operations

• ARMv7E-M can divide a 32-bit register into two 16-bit sub-words, enabling dual INT16 MAC instructions such as SMLAD and SMLALD.SMLAD performs two INT16 × INT16 MACs in one clock cycle, but does not perform four INT8 × INT8 MACs.Source: §2.3.1 · p. 6
• Because most DNN execution operations are MACs, INT8 and INT16 quantized DNNs are similarly fast on this architecture.This follows from the passage's account of equal MAC throughput for INT8 and INT16 operations.Source: §2.3.1 · p. 6
• CMSIS-NN's INT8 convolution uses sign extension, zero-point addition, and data reordering instructions to prepare operands and outputs.The example names SXTAB16 for weight extension and zero-point addition, and PKHTB and PKHBT for output ordering.Source: §2.3.1 · p. 6
• INT16 data loads need no equivalent preprocessing, giving INT16 processing a speed advantage over INT8 processing.The comparison identifies preprocessing instructions as the source of the stated advantage.Source: §2.3.1 · p. 6

2.3.2 Single-issue vs. Dual-issue

• Cortex-M4 executes one instruction per clock cycle in order, whereas Cortex-M7 has a six-stage superscalar pipeline with dual-issue capability.Dual issue lets M7 execute two instructions within one clock cycle.Source: §2.3.2 · p. 6
• M7 dual issue requires instruction pairs without data dependencies such as Read After Write; the study evaluates both M4 and M7 architectures.The evaluation is intended to assess the quantization method across both micro-architectures.Source: §2.3.2 · p. 6

2.4 Limitations of Existing Inference Frameworks

• TFLM supports W8A8 and W8A16, but its SAME/VALID-only padding support prevents its use for the paper's TCN implementation on MCUs.The TCN triggers an error in TFLM because it requires padding beyond those supported options.Source: §2.4 · pp. 6–7
• Together, TFLM and X-CUBE-AI are described as infeasible for effective quantized TCN implementation.TFLM is limited by padding support, while X-CUBE-AI has the TCN fallback and precision limitations described above.Source: §2.4 · p. 7
• X-CUBE-AI falls back to floating-point computation for TCNs, causing performance drops and energy waste, and it supports only INT8 quantization.The paper nevertheless uses X-CUBE-AI as an experimental baseline.Source: §2.4 · p. 7

3.1 Temporal Convolutional Network Quantization

• TCNs provide the reference architecture for quantized time-series classification and regression, with dilated 1D convolution as their core operation.The described TCN stacks blocks of causal dilated convolutions and uses weight normalization, ReLU, and dropout; a residual 1D convolution may adjust feature-map size.Source: §3.1 · pp. 7–8, §3 · p. 7

3.2 Assembly Instruction Analysis

• W16A16 uses 6 instructions for four input-weight pairs, 50% fewer than W8A8, by removing sign extension and reordering.The scheme uses 64-bit accumulation to avoid potential overflow, with SMLALD reported to match SMLAD speed.Source: §3.2 · p. 9

3.3 Memory Layout and Convolutional operator design

• Seq2col flattens sliding windows into sequentially ordered 1D-convolution inputs, addressing discontinuous access caused by mismatched input and weight layouts.The method handles causal left padding and dilation; input channels are padded to an even number for aligned dual-word loads, and receptive-field data is concatenated into a continuous array.Source: §3.3 · pp. 10, 12

3.4 Requantization and Fixed-Point Approximation

• Requantization approximates floating-point scaling with integer multiplication and shifts, representing the 64-bit accumulator times a 32-bit scaling factor as a full-precision 96-bit intermediate.The method cascades three 32-bit registers because standard C lacks 96-bit integers and cross-register shifts; the shift converts the result to the next layer's quantization range.Source: §3.4 · pp. 12–14

3.5 Two-Stage Saturation

• Two-stage saturation first clamps extreme shifted values to 32-bit limits, then uses SSAT to bring the result into the next layer's 16-bit range.The first clamp prevents an out-of-range positive value from being misread as negative when higher-register sign bits are ignored; saturation can implement ReLU using USAT instead of SSAT.Source: §3.5 · p. 16

• The evaluation compares W8A8, W8A16, and proposed W16A16 on machine-learning tasks, execution time, and energy consumption.The passage identifies W16A16 as the proposed scheme.Source: §4 · p. 16
• The comparison covers both task performance and deployment costs, measuring execution time and energy consumption.Source: §4 · p. 16
• An implementation of the experiment is available as an online package.The passage provides a GitLab URL for the implementation.Source: §4 · p. 16

Hardware

• The experiments use an STM32H723ZG Cortex-M7 at up to 550 MHz and a GD32F450VET Cortex-M4 at up to 200 MHz.Both devices support Armv7E-M and DSP extension instruction sets.Source: §4.1 · p. 16

Quantization settings

• W8A8 uses PyTorch’s official quantization toolbox, while custom approaches implement W8A16 and W16A16.All models are trained in floating point and quantized after training.Source: §4.1 · p. 17
• Activation calibration scales maximum observed values by 2.0 to limit precision loss from unexpected outliers.Source: §4.1 · p. 17

Compiler settings

• Inference operators use Arm Compiler 6 with -Ofast and link-time optimization; memory barriers support precise cycle measurements.Source: §4.1 · p. 17

4.2 Regression

• Quantization error is measured against the float32 TCN output, isolating quantization noise from error due to model capacity.Source: §4.2 · p. 17

4.2.1 C-MAPSS Dataset

• C-MAPSS evaluates engine remaining-useful-life estimation using 14 input channels, 30-step windows, and a four-block TCN.The dataset has 100 training and 100 test engines; the TCN uses kernel size 5 and block dilations 2^i.Source: §4.2.1 · p. 17
• Against the float32 reference, C-MAPSS end-to-end quantization RMSE was 1.89373 for W8A8, 0.31093 for W8A16, and 0.01101 for W16A16.Source: Table 1 · p. 18, §4.2.1 · p. 18
• On C-MAPSS, W8A8 quantization RMSE rises from 0.01197 in the first layer to 0.02456 in the first block and 1.89373 end-to-end.The end-to-end error is about 14% of the model’s overall regression RMSE, which the authors consider too high for practical purposes.Source: §4.2.1 · p. 18, Table 1 · p. 18

4.2.2 NinaPro DB2 force regression

• NinaPro DB2 median quantization RMSE was 0.0015 for W16A16, 0.0161 for W8A16, and 0.1970 for W8A8 (Wilcoxon signed-rank p < 10^-10).The median summarizes results across 40 subjects.Source: Table 2 · p. 18, §4.2.2 · p. 19
• NinaPro DB2 evaluates mapping 12-channel forearm sEMG to six prosthesis-force outputs with a four-block TCN.The reported quantization RMSE is the median across 40 subjects.Source: §4.2.2 · pp. 18–19

4.3 Classification

• The NinaPro DB2 classification task recognizes 50 gestures from sEMG, including hand and wrist movements, grasping and functional movements, force patterns, and rest.Source: §4.3 · p. 19
• W16A16 and W8A16 accuracy is about 0.02% below float32, whereas W8A8 accuracy drops 11.66% (Wilcoxon signed-rank p < 10^-10).Source: Table 3 · p. 19, §4.3 · p. 19

4.4.1 Layer-level benchmarking

• At the layer level, W8A8 has the lowest latency, W16A16 follows, and W8A16 is slowest across configurations and platforms.GD32F4 needs roughly twice the execution cycles of STM32H7, attributed to its lack of I-Cache and D-Cache.Source: §4.4.1 · p. 19, Table 4 · p. 20, Table 5 · p. 20

4.4.2 End-to-end model level benchmarking

• W8A8 saves 2.29% of STM32H7 model cycles versus W16A16, while W16A16 reduces cycles by 17.6% versus W8A16; on GD32F4, the respective differences are 8.70% and 24.1%.Across the full TCN model, W8A8 is fastest, W16A16 next, and W8A16 slowest.Source: §4.4.2 · pp. 19–20

4.4 Inference speed

• Across layer and model benchmarks, W16A16 is faster than W8A16, while W8A8 is faster but has quantization errors the authors judge likely not worth its time savings.The benchmarks quantify performance in CPU execution cycles on STM32H7 and GD32F4.Source: §4.4 · p. 19, §4.4.2 · p. 20

4.4.2 End-to-end model level benchmarking

• Cube.AI’s INT8 and Float32 TCN versions run similarly on STM32H7 and are much slower than the other tested quantization approaches.Cube.AI leaves 77.5% of operations in floating point because it cannot convert some required TCN operations, including left asymmetric padding; GD32F4 does not support Cube.AI.Source: §4.4.2 · p. 20

4.5 Energy Consumption

• Inference energy is calculated as 3.3 V multiplied by accumulated charge, equivalently 3.3 V multiplied by average current and inference time.The paper also defines algorithmic execution energy by subtracting sleep-mode current before calculating energy.Source: §4.5 · pp. 21–22
• On STM32H7, W16A16 is 0.6 ms faster and uses 0.4 mJ less energy than W8A16, while W8A8 is 0.05 ms faster and uses 0.15 mJ less energy than W16A16.The reported ordering is W8A8, W16A16, then W8A16 for both time and energy.Source: §4.5 · p. 21, Table 6 · p. 21
• Cube.AI Float32 and INT8 are each about 5x slower and use about 4x the energy of W16A16 on STM32H7.Energy was measured with a PPK2 at 3.3 V; GPIO transitions mark the inference window for synchronization with the current waveform.Source: §4.5 · pp. 20–21
• On GD32F4, W8A16 takes 4.69 ms more and uses 2.11 mJ more than W16A16; W16A16 takes 1.44 ms more and uses 0.5 mJ more than W8A8.The same time-and-energy ordering appears on GD32F4, with larger differences than on STM32H7.Source: §4.5 · p. 21, Table 7 · p. 21

5 Conclusion

• W16A16 is about 17.6% faster than W8A16 on a full STM32H7 TCN and has higher accuracy than W8A8, whose regression quantization error is about 100x worse.The conclusion describes W8A8 as slightly faster and more energy-efficient than W16A16, but with significantly worse empirical classification and regression performance.Source: §5 · p. 22
• On Cortex-M, 8-bit and 16-bit integer MAC operations have the same maximum throughput, while 8-bit processing adds preprocessing instructions; the method also uses 96-bit cross-register shifting and two-stage saturation to avoid intermediate overflow.The authors modified CMSIS-NN’s 64-bit requantization.Source: §5 · p. 22

5.1 Limitations

• Whether architecture-specific resolution gains with little time or energy overhead translate to other MCUs or hardware architectures remains a direction for future work..Source: §5 · p. 22
• The cost advantage depends on ARMv7E-M’s equal speed for 8-bit and 16-bit operations and may not transfer to ARMv8.1-M or RISC-V, which can MAC four 8-bit elements in one instruction.The authors note that ARMv7E-M is widely applied in Cortex-M devices.Source: §5.1 · p. 22
• The experiments cover TCNs, three machine-learning tasks, and two datasets; results on other models, tasks, or datasets may differ.The authors remain fairly confident in the qualitative pattern observed across their metrics, datasets, and hardware.Source: §5.1 · p. 22