ShatterQuant: Breaking Uniform Precision with Block-Wise Mixed-Precision on a Systolic Transformer Hardware Accelerator

1 Introduction

• Transformer inference incurs substantial computation and memory traffic, amplified in diffusion models by repeated denoising steps.Prior cost-reduction approaches include architecture search, distillation, pruning, and state-space architectures.Source: §1 · pp. 1–2
• Aggressive sub-8-bit quantization can degrade accuracy or generation quality because transformer parameters differ in quantization sensitivity.Block-wise quantization independently scales smaller tensor groups, permitting more flexible precision reduction.Source: §1 · p. 2
• ShatterQuant combines hardware-aware post-training quantization with a custom accelerator because current GPUs struggle to efficiently support mixed-precision blocks.The stated challenges are non-uniform memory accesses and fixed-function compute datapaths.Source: §1 · p. 2
• The contributions span block-level mixed-precision quantization, a configurable accelerator, and implementation tradeoff evaluation on DeiT and PixelDiT.The accelerator supports 1-, 2-, 4-, and 8-bit weights with 8-bit activations and couples block size to PE configuration.Source: §1 · p. 2

2.1 Diffusion Transformers

• PixelDiT operates directly in the pixel domain without a latent autoencoder, while ShatterQuant targets core ViT-style linear projections, softmax, and nonlinearities.The approach also applies to PixelDiT’s attention and feed-forward structure; other operations are offloaded to external processors.Source: §2.1 · p. 2

2.2 Mixed Precision Quantization

• Prior quantization work spans weight PTQ, quantization-aware training, calibration methods, layer-sensitivity mixed precision, and channel-wise intra-tensor quantization.ShatterQuant instead enables grouped sub-channel quantization with dedicated hardware support.Source: §2.2 · p. 2

2.3 Hardware Accelerators

• Prior transformer accelerators optimize low-bit inference, dataflows, memory, and attention, but largely use coarse-grained precision or evaluate modern models only narrowly.Some exploit sparsity by skipping operations rather than varying weight precision.Source: §2.3 · p. 2

3 Proposed Approach

• ShatterQuant is presented as a four-part approach covering architecture, blockwise mixed precision, post-training quantization, and accelerator design.Source: §3 · p. 2
• Figure 2 places PixelDiT alongside the ViT architecture as a representative vision diffusion model.Source: Figure 2 · p. 3
• The linear-projection workflow proceeds through calibration and initial weight setting, quantization with weight-rounding correction, and hardware deployment.Source: Figure 2 · p. 3

3.1 Model Architecture

• An input image is divided into patches, projected into the embedding dimension, and combined with an optional class token and learned positional embedding.Source: §3.1 · p. 3
• Each transformer layer projects embeddings into queries, keys, and values, then computes scaled dot-product attention across heads.Each head has dimension d_k = D/H.Source: §3.1 · p. 3
• Concatenated head outputs are projected before a feed-forward network applies a nonlinear activation; residuals and LayerNorm complete the layer.The feed-forward network uses W1 and W2, with D_ff typically set to 4D.Source: §3.1 · p. 3
• PixelDiT adds patch-wise and pixel-wise pathways and conditioning-based AdaLN, while retaining linear projections and attention operations.Source: §3.1 · p. 3
• ShatterQuant targets conventional ViTs and PixelDiT, handling parameterized linear and attention operations on its accelerator.Initial and final linear operations, LayerNorm, RoPE, reshaping, and scaling are offloaded to external processors.Source: §3.1 · pp. 3–4

3.2 Intra-Tensor Blockwise Mixed Precision

• Weights are normalized by a tensor-wise AbsMax scale, then quantized using block-specific precision and scale.Block scales are constrained to powers of two and represented by a signed 6-bit exponent and 2-bit precision.Source: §3.2 · p. 4
• Weight quantization applies to the block containing each element, with supported precisions of 8, 4, 2, or 1 bit.Source: §3.2 · p. 4
• Lower weight precision packs two, four, or eight weights—and corresponding output columns—into each 8-bit representation, while activations use a tensor-wise scale.Source: §3.2 · p. 4

3.3 Post-Training Blockwise Quantization

• Superblocks with greater standard deviation receive fewer bits, while those with lower standard deviation receive higher bitwidths, according to CDF cutoffs.Each superblock occupies the same number of weights as a single 1-bit block.Source: §3.3 · p. 4
• LS-ViT reconstruction adapts AdaRound to blockwise weights and learns power-of-two block scales alongside per-weight rounding decisions.The two fully quantized exponent candidates are annealed with the same regularizer against the least-squares Hessian objective.Source: §3.3 · p. 4

3.3.1 Extension to PixelDiT.

• PixelDiT adaptation uses trajectory-output squared error for Hessian estimation, eight timestep-bin activation scales, and relative variance to assign precision.After reconstruction, activation scales across layers are further optimized to minimize trajectory MSE from the full-precision model.Source: §3.3.1 · p. 4

3.4 ShatterQuant Transformer Accelerator

• The accelerator combines weight, scale, bias, and feature buffers, an 8-bit activation and 1/2/4/8-bit systolic array, a 32-bit accumulator, and an integrated quantizer.AXI-Stream handles data streaming, while AXI-Lite handles configuration and initialization.Source: §3.4 · p. 4

3.4.1 ShatterQuant Processing Element.

• A barrel rotator lets each weight-stationary PE support 1-, 2-, 4-, and 8-bit weights using a unified 8-bit multiplier and 32-bit result.As precision falls, the PE reuses input features across cycles to generate two, four, or eight output columns.Source: §3.4.1 · p. 4

3.4.2 ShatterQuant PE Array.

• The p×p PE array propagates weights and biases downward, features and precision metadata across, and partial sums to the output accumulator.Its buffer and scale organization ties software block size to PE configuration; lower precision expands effective processing width.Source: §3.4.2 · pp. 4–5, Figure 3 · p. 5

3.4.3 ShatterQuant Output Accumulator.

• The output accumulator rescales partial sums at block boundaries rather than dequantizing each block, enabling integer-domain accumulation.Rescaling shifts by s_i−s_i+1: positive differences shift right, negative differences shift left.Source: §3.4.3 · p. 5

3.4.4 ShatterQuant Quantizer.

• The integrated quantizer performs requantization and nonlinear operations in integer arithmetic, using piecewise-linear approximations instead of impractical large lookup tables.For softmax, the 128 factor scales the bounded exponential to the int8 range, while s_O sets the output scale and following-layer input scale.Source: §3.4.4 · pp. 5–6
• Figure 4 shows per-layer effective bitwidths for DeiT and PixelDiT-XL under specified precision splits; Table 1 compares ImageNet-1K Top-1 accuracy and effective weight bitwidth for three DeiT variants.Both evaluations use a 4×4 PE configuration.Source: Figure 4 · p. 6, Table 1 · p. 6

4.1 Model Performance

• At max Δ3.3% accuracy difference, ShatterQuant spans 1–8-bit effective precision against coarse-grained per-tensor assignments; PixelDiT retains similar generation capability at gFID 14.4.The PixelDiT gFID increase is attributed to increased quantization noise.Source: §4.1 · pp. 6–7

4.2 Hardware Implementation

• The accelerator was synthesized in TSMC 16nm at 1 GHz, evaluating four PE arrays from 4×4 through 32×32 with memory capacity and workload held constant.Source: §4.2 · p. 7
• On-chip buffers occupy 86% of area at 4×4 and 79% at 32×32; at 32×32, the PE array accounts for 13% of area and 40% of power.Total area ranges from 1.60 to 1.92 mm2, while power ranges from 64 to 521 mW.Source: §4.2 · p. 7
• The accelerator reports 1.5 TOPS MAC throughput at 521 mW, with 760 GOPS/mm2 area efficiency and 2.8 TOPS/W energy efficiency.Source: §4.2 · p. 7

4.3 Hardware-Software Co-Analysis

• Larger PE arrays improve energy efficiency at the cost of model accuracy; at 32×32 with 20% 1-bit weights, DeiT-Base retains substantially higher accuracy than DeiT-Tiny.Source: §4.3 · p. 8, Figure 6 · p. 8
• For PixelDiT with a 20% 4-bit and 80% 8-bit split, gFiD rises to 50 at 16×16, versus 14 at 4×4.Source: §4.3 · p. 8, Figure 7 · p. 8

• ShatterQuant jointly optimizes quantization, integer-only execution, and a configurable systolic accelerator supporting dynamic 1/2/4/8-bit computation.Source: §5 · p. 8
• The TSMC 16nm implementation achieves 1.5 TOPS, 760 GOPS/mm2 area efficiency, and 2.8 TOPS/W energy efficiency, with DeiT accuracy within 3.3% of state-of-the-art mixed-precision methods at a 2-bit lower effective bitwidth and comparable PixelDiT generation quality.Source: §5 · p. 8
• Future work includes extending mixed-precision quantization to activations and autoregressive transformers, exploring multi-processor integration, and approximating advanced layers.The named layers include RoPE positional embeddings, mixture-of-experts, and state-based architectures.Source: §5 · p. 8