PaperlayerView exact revision on arXiv

Source-linked AI summary

ShiftQAT: Quantization-Aware Training for the Numerical Contract of a Fixed-Function Edge CNN Accelerator

Oguzhan Buyuksolak, Osman Erman Okman, Mehmet Gorkem Ulkar

arXiv:2610.21695v1cs.LG

TL;DR

Fixed-function accelerators run only networks that obey their numerical contract, a challenge for quantization methods that satisfy only parts of it. ShiftQAT constructs networks for the MAX7800x contract by resolving scale dependencies and jointly scaling weights and biases. It matches less constrained QAT’s W8A8 accuracy drop and stays within 2.9 pp of FP32 at W4A8.

  • Problem

    Fixed-function accelerators require networks to obey a precise numerical contract, yet prior compared methods satisfy only parts of the scaling constraints.

    Source: §1 · p. 1

  • Method

    ShiftQAT resolves topology dependencies for static scale propagation and trains weights and biases on a joint power-of-two grid that fits the 8-bit bias budget.

    Source: §1 · pp. 1–2, §4.5 · p. 6

  • Results

    At W4A8, ShiftQAT stays within 2.9 pp of FP32 and matches less constrained QAT methods’ ImageNet accuracy drop at W8A8.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ShiftQAT meets the full MAX7800x contract, and its trained networks run end-to-end on MAX78002 at millisecond latency and millijoule energy per inference.

    Source: §8 · p. 9

    Adopting learned power-of-two thresholds is a direct way to narrow ShiftQAT’s W4A8 gap to QAT methods that relax the contract.

    Source: §7 · p. 8

Abstract

from arXiv · show

Fixed-function CNN accelerators make edge inference fast and energy-efficient, but they execute only networks that obey a precise numerical contract. On the MAX7800x family, this contract requires power-of-two, per-tensor, symmetric scales, 8-bit activations and biases, integer-only arithmetic, and one statically configured shift per layer. We present ShiftQAT, the quantization-aware training (QAT) flow we developed for this family, which produces networks satisfying the contract by construction. It is the default training path of ai8x-training, the family's open-source training toolchain. ShiftQAT resolves residual, concatenation, and weight-sharing dependencies so that activation scales propagate statically and each layer's requantization reduces to one offline-configured shift. It also trains weights and biases on a joint power-of-two grid that fits the 8-bit bias budget. Among the compared methods, only ShiftQAT satisfies all of the contract's scaling constraints, including signed 8-bit activation storage. Even so, on ImageNet it matches the W8A8 accuracy drop of less constrained QAT methods and stays within 2.9 percentage points of FP32 at W4A8. On MAX78002 silicon, ShiftQAT-trained MobileNetV2 and ResNet-18 classify ImageNet images in 14.6 and 8.4 ms at 5.1 and 2.8 mJ per inference.

1 INTRODUCTION

Fixed-function accelerators impose a numerical contract that networks must satisfy to compile, while prior methods meet only parts of it. ShiftQAT builds networks for the full contract and retains competitive ImageNet accuracy.

1 INTRODUCTION

  • Quantization shrinks model memory footprints and enables fixed-point inference on resource-limited edge devices.Source: §1 · p. 1
  • The MAX7800x contract requires power-of-two, per-tensor symmetric scales, 8-bit activation and bias storage, integer-only inference, and one offline-configured shift per layer.Source: §1 · p. 1
  • A trained checkpoint maps deterministically to integer weights, 8-bit biases, and shift values without a separate calibration step.Source: §1 · p. 2
  • ShiftQAT resolves topology dependencies and propagates activation scales statically, while jointly scaling weights and biases on a power-of-two grid.The joint grid is designed to fit the 8-bit bias budget.Source: §1 · pp. 1–2
  • At W8A8, ShiftQAT matches less constrained QAT methods’ ImageNet accuracy drop and stays within 2.9 pp of FP32 at W4A8.The comparison is on MobileNetV2 and ResNet-18.Source: §1 · p. 2
  • The paper distills five lessons for fixed-function accelerator training flows, and ShiftQAT is the default training path in the open-source ai8x-training toolchain.Source: §1 · pp. 1–2

2 BACKGROUND AND RELATED WORK

Prior work provides integer-only inference, scale-moving, and hardware-friendly quantization techniques, but fixed-function accelerators constrain where and how rescaling can occur. ShiftQAT builds on these ideas while addressing the accelerator’s full contract.

2 BACKGROUND AND RELATED WORK

  • A fixed-function accelerator can apply only a statically configured power-of-two shift per layer, not an arbitrary-mantissa multiplication.Integer-only schemes may decompose the output multiplier into a fixed-point mantissa and shift; that structure is inexpensive on processors with a general integer multiplier.Source: §2 · p. 2
  • Positive homogeneity lets activation scales move through ReLU or Leaky ReLU into subsequent weights; ShiftQAT defers scales to the network output.ShiftQAT adds topology handling and bias treatment to preserve this approach under an 8-bit bias budget.Source: §2 · p. 2
  • Hardware-friendly methods learn clipping ranges or step sizes, but some strong post-training quantizers rely on per-channel scales unavailable on MAX7800x.ShiftQAT adopts HPTQ’s activation-threshold search; TQT’s learned scales remain powers of two.Source: §2 · p. 2
  • This paper focuses on the MAX78000/MAX78002 training flow and ImageNet-scale deployments, alongside prior latency and energy studies on smaller vision and audio workloads.Source: §2 · p. 2

3 THE ACCELERATOR’S NUMERICAL CONTRACT

The MAX7800x numerical contract consists of datapath constraints rather than optional design preferences. Those constraints jointly restrict the scaling choices and accuracy levers available to quantized networks.

3 THE ACCELERATOR’S NUMERICAL CONTRACT

  • Table 1 pairs the accelerator’s C1–C7 datapath constraints with the ShiftQAT mechanism each constraint requires.Source: §3 · p. 2, Table 1 · p. 3
  • Standard integer-only QAT typically violates non-power-of-two multiplier and 32-bit bias constraints, while other compared methods violate additional contract requirements.TQT is closest, but its 16-bit biases and unsigned post-ReLU activations violate constraints; per-channel PTQ also violates the scale-granularity constraint.Source: §3 · p. 3
  • Biases must remain 8-bit after absorbing activation scales propagated forward, so satisfying constraints independently may not satisfy them jointly.Source: §3 · p. 3
  • The contract rules out per-channel scales and leaves post-ReLU tensors with only 7 of 8 bits in signed storage.The hardware clips ReLU outputs to [0, 127], leaving half the code space unused for non-negative tensors.Source: §3 · p. 3

4 SHIFTQAT MECHANISMS REQUIRED BY THE CONTRACT

ShiftQAT prepares activation scales offline, reconciles graph dependencies, and propagates scales so intermediate layers need no runtime recovery. During QAT, weights and biases share a power-of-two scale, and each layer’s hardware shift is trained alongside the parameters.

4 SHIFTQAT MECHANISMS REQUIRED BY THE CONTRACT

  • ShiftQAT folds batch normalization, proposes activation scales, resolves dependencies, propagates scales, and then re-estimates weight/bias scales during QAT.All scales are powers of two with integer exponents.Source: §4 · p. 3

4.1 Batch-Normalization Folding

  • Batch normalization is folded into the preceding convolution or linear layer because the accelerator has no normalization stage.The affine transform is absorbed into weights and biases before quantization.Source: §4.1 · p. 3

4.2 Activation-Scale Proposal

  • Activation-scale proposal searches five power-of-two candidates, starting at the smallest non-clipping scale and selecting the one minimizing histogram-weighted quantization error.Activation statistics come from 2048-bin histograms collected in one training-set forward pass; outliers are removed at z-score 8.Source: §4.2 · pp. 3–4

4.3 Topology Dependency Resolution

  • Residual inputs adopt the addition’s scale, while concatenation and shared-weight inputs adopt the largest scale to reconcile branches before runtime.These choices can trade resolution on a narrower branch for a guarantee that no branch clips.Source: §4.3 · p. 4

4.4 Static Scale Propagation

  • Static scale propagation folds each activation scale into the next layer, leaving only one scale recovery at the network output.The derivation requires positively homogeneous activations; the output recovery can be omitted for argmax classification.Source: §4.4 · p. 5

4.5 Joint Weight/Bias Power-of-Two Scale

  • Weights and biases are quantized at their own bit-widths using one shared power-of-two scale, recomputed at every training iteration.Their pooled absolute values set a quantile-based range bound; q is 0.999 for 8-bit layers and 0.9 for 4-bit layers.Source: §4.5 · pp. 5–6

4.6 The PoT Scaler Is the Per-Layer Shift

  • The PoT scaler merges weight-quantization and propagated activation-scale shifts into the single per-layer shift programmed by the hardware.The hardware applies the shift to the 32-bit intermediate result and accepts total shifts from −15 to 15, so training clips k to that range.Source: §4.6 · p. 6

5 EVALUATION

The evaluation compares ImageNet accuracy at W8A8 and W4A8, then measures ShiftQAT-trained models on MAX78002 silicon. ShiftQAT matches comparable QAT accuracy at W8A8, remains within 2.9 pp of FP32 for W4A8 MobileNetV2, and runs reduced variants at millisecond latency and millijoule energy.

5.1 Setup

  • Accuracy drops, rather than absolute accuracies, are compared because methods use different FP32 references; differences below about 1 pp are treated as comparable.All methods in the tables use 32-bit biases, and baseline comparisons are accuracy-only because no baseline can be deployed without violating part of the contract.Source: §5.1 · p. 6

5.2 Accuracy Under the Contract’s Scaling Constraints

  • ShiftQAT loses 1.026 pp on MobileNetV2 and 0.558 pp on ResNet-18 at W8A8, matching comparable QAT drops while satisfying every property column.The table comparison uses 32-bit biases to isolate weight and activation constraints; deployed models use 8-bit biases.Source: §5.2 · p. 6, §5.1 · p. 6, Table 2 · p. 7
  • ShiftQAT loses 2.864 pp on MobileNetV2 and 1.904 pp on ResNet-18 at W4A8; only its scaling scheme in Table 3 is executable by the datapath.Per-tensor QAT with learned scales reports smaller drops, but uses asymmetric activations and non-power-of-two scales; differences also reflect distinct FP32 references and training regimes.Source: §5.2 · pp. 6–7, Table 3 · p. 7

5.3 End-to-End Deployment on MAX78002

  • 14.6 ms and 5.1 mJ per inference for MobileNetV2, and 8.4 ms and 2.8 mJ for ResNet-18, were measured on MAX78002.The deployed networks use reduced variants, 128×128 inputs, mixed 4/8-bit weights, and 8-bit biases; their accuracy is not directly comparable to full-size Section 5.2 results.Source: §5.3 · pp. 7–8
  • Table 4 demonstrates deployment feasibility rather than a controlled accuracy comparison, since the silicon models are narrower, use 128×128 inputs, and mix 4/8-bit weights.Their 8-bit biases differ from the 32-bit-bias accuracy-table setup.Source: §5.1 · p. 6, §5.3 · pp. 7–8

6 LESSONS FROM BUILDING AND DEPLOYING THE FLOW

The lessons emphasize specifying the accelerator’s full numerical contract, solving interacting constraints together, and resolving network topology for static scaling. They also connect on-chip memory limits to model design and favor training that emits the exact hardware configuration.

L1. The contract, not the bit-width, is the specifica-

  • “W4A8” alone does not specify a deployable network: symmetry, scale granularity, bias width, and rescaling behavior determine whether it compiles.Writing down the contract first made each training-flow mechanism necessary rather than optional.Source: L1. The contract, not the bit-width,… · p. 8

L2. Constraints interact, so satisfy them jointly.

  • Activation-scale folding moves scales into biases, so the 8-bit bias budget and shared power-of-two weight/bias grid must be handled jointly.The joint per-iteration scale exists because these constraints interact.Source: L1. The contract, not the bit-width,… · p. 8

L3. Topology, not arithmetic, is where static scaling

  • Static scaling requires resolving every branch merge and reused weight; this is necessary for compiling ResNet- or FPN-style graphs without dynamic cross-branch rescaling.The propagation identity is simple, but topology handling makes it valid on real networks.Source: L3. Topology, not arithmetic, is where… · p. 8

L4. On-chip weight memory shapes the deployed model.

  • On-chip weight memory shaped deployment width, resolution, and bit mix; wider or higher-resolution models require more memory or fewer average bits per weight.The authors identify accurate 4-bit interior layers as a practical way to fit larger networks.Source: L4. On-chip weight memory shapes the… · p. 8

L5. Make training emit the hardware configuration.

  • The PoT scaler computes each layer’s output-shift register value during training, making checkpoints map deterministically to integer parameters without post-training calibration.The authors value this because hardware does not run a configuration that training did not see.Source: L5. Make training emit the hardware… · p. 8

7 DISCUSSION AND SCOPE

ShiftQAT’s scale-propagation approach is described as applicable beyond MAX7800x to shift-based fixed-point datapaths, while its activation-function scope and W4A8 accuracy gap remain bounded. The deployed variants also illustrate the reported ImageNet top-1 results at 128×128 inputs.

7 DISCUSSION AND SCOPE

  • ShiftQAT’s mechanisms apply to shift-based fixed-point datapaths; on accelerators without a free per-layer shifter, static propagation would also remove per-layer rescaling.On MAX78002, the output shift is free, so static propagation’s value is making the network executable.Source: §7 · p. 8
  • Learned power-of-two thresholds, as in TQT, remain contract-compatible and offer a direct way to narrow the W4A8 gap to less constrained QAT methods.Source: §7 · p. 8
  • 62.812 ImageNet top-1 for MobileNetV2 (m=0.75) and 53.230 for ResNet-18 (m=0.5), both at 128×128 inputs.Table 4 covers end-to-end MAX78002 deployments.Source: §7 · p. 9, Table 4 · p. 9
  • Scale propagation requires positively homogeneous activations; networks with Sigmoid, Tanh, GELU, or SiLU would need scale recovery unavailable in the static datapath.Source: §7 · pp. 8–9

8 CONCLUSION

ShiftQAT meets the MAX7800x numerical contract, matching less constrained QAT methods’ W8A8 accuracy drop and remaining within 2.9 percentage points of FP32 at W4A8. The contract reflects the accelerator’s uniform, symmetric, power-of-two, per-tensor scaling requirements.

8 CONCLUSION

  • ShiftQAT meets the full MAX7800x contract, matches the W8A8 accuracy drop of less constrained QAT methods, and stays within 2.9 pp of FP32 at W4A8.Source: §8 · p. 9

A HARDWARE-FRIENDLY QUANTIZATION PROPERTIES AND THEIR MAX7800X MANDATE

B ADDITIONAL EXPERIMENTAL DETAILS

The additional experimental details specify distinct training schedules for MobileNetV2 and ResNet-18, followed by activation-scale estimation and bounds for weight/bias quantization and per-layer shifts.

B.1 Hyperparameters

  • MobileNetV2 uses SGD with momentum 0.9 and weight decay 4 × 10−5 for 200 epochs, with QAT starting at epoch 150 and cosine annealing from 0.05.Source: §B.1 · p. 12
  • ResNet-18 uses SGD with momentum 0.9 and weight decay 10−4 for 120 epochs, with QAT starting at epoch 90 and MultiStepLR milestones at 30, 60, 90, and 105.The learning-rate multiplier is γ = 0.1.Source: §B.1 · p. 12
  • Before QAT starts, activation statistics are collected as 2048-bin histograms in one forward pass over the training set, without backpropagation; outliers are removed using z-score threshold 8.0.Source: §B.1 · p. 12
  • Weight/bias range bounds use the 0.999 quantile for 8-bit layers and 0.9 for 4-bit layers; total per-layer shifts are clamped to [−15, 15].Source: §B.1 · p. 12

B.2 Hardware Implementation Details

The MAX78002 deployment changes adapt input resolution, model width, channel count, stride handling, and bias precision to the device’s memory and architectural limits.

B.2 Hardware Implementation Details

  • Deployment uses 128×128 inputs, width multipliers 0.75 for MobileNetV2 and 0.5 for ResNet-18, and caps layers at 1024 channels.Source: §B.2 · p. 12
  • Each stride-2 conv2d is replaced with 2×2 max-pooling followed by a stride-1 conv2d.Source: §B.2 · p. 12
  • Deployment quantizes biases to the required 8 bits, while accuracy comparisons in Tables 2 and 3 use 32-bit biases for all methods to match published baselines.Source: §B.2 · p. 12

B.3 Compute Resources

Training used two NVIDIA RTX 4090 GPUs, with runtimes of approximately 12 hours from a pretrained checkpoint and 48 hours from scratch.

B.3 Compute Resources

  • Training ran on two NVIDIA RTX 4090 GPUs, each with 24 GB of memory.Source: §B.3 · p. 12
  • Runs resumed from a pretrained checkpoint took approximately 12 hours.Source: §B.3 · p. 12
  • Runs started from scratch took approximately 48 hours.Source: §B.3 · p. 12

Related papers

Loading 2610.21695v1…