PaperlayerView exact revision on arXiv

Source-linked AI summary

MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference

Romain Facq, Sami Ben Ali, Olivier Sentieys

arXiv:2609.28358v1cs.ARcs.AI

TL;DR

Microscaling quantization is difficult to apply efficiently to convolution because tensors are quantized twice and im2col expands activations. MicroQonv reorders tensors to quantize each once, reporting reduced memory movement and quantization overhead, with accuracy gains in quantized latent replay.

  • Problem

    Microscaling for convolutional layers can require two quantizations per tensor and expanded activation storage or movement around im2col.

    Source: §I · p. 1, C. Limitations of related work · pp. 4–5

  • Method

    MicroQonv reorders tensors so each is quantized once, quantizing activations before a modified im2col transformation.

    Source: §I · p. 1, B. Efficient quantization directions · p. 5

  • Results

    MicroQonv reduces quantization cost by ×2 for weights and gradients and up to ×9 for activations, while reducing memory movement and storage by up to ×7.53.

    Source: §VIII · p. 11

  • Takeaways & Limitations

    With a quantized latent replay strategy, 4-bit MicroQonv improves accuracy by +5.7% to +11% for the same memory budget.

    Source: §VIII · p. 11

    The related square-quantization method discussed considers only fully-connected models, leaving its convolutional-layer application as an open question.

    Source: C. Limitations of related work · p. 5

Abstract

from arXiv · show

Microscaling quantization techniques are increasingly used to represent neural network parameters with 8 bits or fewer while preserving near-full precision accuracy. However, applying these methods efficiently in convolutional layers is not straightforward. A naive approach transfers full-precision weights and activations to processing units and quantizes each tensor twice, resulting in much more memory movement than expected. Additional overhead comes from the activation tensors, whose sizes grow substantially because of the im2col transformation applied before quantization. We propose MicroQonv, a way to combine microscaling with convolutional layers' forward and backward operations by quantizing each tensor only once and quantizing the activation tensor before applying a modified version of im2col: channel-batch-first im2col. MicroQonv reduces the quantization cost by a factor of $\times2$ for weights and gradients, and by up to $\times9$ for activations, at a negligible accuracy cost. It reduces memory movement and storage by up to $\times7.53$ compared to their full-precision counterparts. This way, MicroQonv reduces microscaling-quantized activation memory movement by $\times3.5$ for state-of-the-art object detection models YOLOV8nano and $\times2.2$ for YOLOV26nano. It also enables 4-bit microscaling in a quantized latent replay strategy for continual learning at the edge, improving accuracy by +5.7% to +11%.

I. INTRODUCTION

Microscaling can reduce the costs of quantized training, but convolutional layers make it inefficient because tensors are quantized twice and activations expand through im2col. MicroQonv quantizes each tensor once and quantizes activations before a modified im2col, targeting lower memory movement and quantization overhead.

I. INTRODUCTION

  • Backpropagation is energy-intensive and time-consuming, while local continual learning can adapt cloud-trained models to changing device environments and new tasks.Replay buffers let devices retain environmental data for training while encountering new data.Source: §I · p. 1
  • Microscaling tensors must be quantized twice because weights, activations, and gradients participate in two GeMMs with different computation directions.Im2col expands activations before quantization, and differing block distributions can require retaining either the expanded quantized tensor or the original full-precision tensor for backpropagation.Source: §I · p. 1
  • MicroQonv quantizes each tensor once and quantizes activations before channel-batch-first im2col to reduce memory movement, storage, and quantization overhead.The paper compares its method with other quantized training methods and reports lower computation with tiny accuracy loss or higher accuracy at the same quantization level.Source: §I · p. 1

1) Forward Pass:

  • Convolution computes output feature maps from a batch of input feature maps and convolutional weights.For the stated standard convolution setup, the output spatial dimensions are Hout = Hin−Kh+1 and Wout = Win−Kw+1.Source: 1) Forward Pass · p. 2

2) Backward Pass:

Backpropagation computes gradients for the input activations and weights; a learning-rate-scaled gradient then updates the weights.

2) Backward Pass:

  • Backpropagation computes an input-activation gradient by convolving the output gradient with transposed kernels.The transposed kernels are formed by swapping the input- and output-channel dimensions of W.Source: 2) Backward Pass · p. 2
  • The weight gradient is accumulated over the batch dimension.It is the gradient with respect to the convolution weights.Source: 2) Backward Pass · p. 2

3) Weight Update:

  • Weights are updated after gradient computation using a standard Stochastic Gradient Descent step with learning rate η.The update subtracts η times the weight gradient from W.Source: 3) Weight Update · p. 2

B. Compute Architecture Model

The paper models computation on data held in on-chip memory, with transfers to and from off-chip memory. It describes convolution as im2col-based matrix multiplication and explains the associated memory overhead.

B. Compute Architecture Model

  • The compute model transfers weights, activations, and gradients between off-chip memory and on-chip memory, where computations occur.It assumes optimal reuse of tiles and tensors once they are in on-chip memory.Source: B. Compute Architecture Model · p. 2, Figure 1 · p. 2

C. Im2col Transformation

  • Im2col rearranges local input patches into columns of a large matrix, mapping convolution to general matrix multiplication.Rows represent kernel positions across input channels, while columns represent sliding-window positions across batch samples.Source: C. Im2col Transformation · p. 3, 1) Input Rearrangement · p. 3

2) Kernel Flattening:

  • Each flattened kernel filter becomes a row of Wf, with length CinKhKw and channel-major layout.This produces the flattened kernel matrix used in the convolution formulation.Source: 2) Kernel Flattening · p. 3

3) Matrix Multiplication Formulation:

  • After reshaping, forward convolution is computed as Yf = Wf · X′, with Yf representing the flattened output tensor.The im2col and flattened formats are also used to compute weight and input gradients during backpropagation.Source: 3) Matrix Multiplication Formulation · p. 3

C. Im2col Transformation

  • The im2col feature matrix can use up to Kh × Kw times as much memory as the original feature map.The transformation enables optimized matrix-multiply libraries or dedicated hardware units.Source: 3) Matrix Multiplication Formulation · p. 3

4) Channel-First Im2col:

Channel-first storage allows im2col columns to be generated on the fly from contiguous channelwise loads, avoiding explicit storage of the expanded tensor. Its access pattern is not equally efficient for backward computation, motivating a layout that supports both convolution passes.

4) Channel-First Im2col:

  • Channel-first storage generates im2col columns on the fly from contiguous channelwise loads, avoiding explicit storage of the expanded tensor.The input feature maps are stored as a compact channel-first tensor rather than as the full im2col representation.Source: 4) Channel-First Im2col · p. 3
  • The channel-first optimization applies only to forward convolution; weight-gradient computation needs transposed activations and incurs discontinuous accesses across Win.These accesses limit the layout's efficiency for backward computation and restrict it to inference or forward-only execution paths.Source: 4) Channel-First Im2col · p. 3
  • On-demand im2col methods avoid expanded-tensor storage, but microscaling can assign repeated activation pixels different quantized values and require re-quantization.Retaining either full-precision activations or the expanded quantized tensor for backward propagation also reduces the memory gains.Source: 4) Channel-First Im2col · p. 3
  • MicroQonv proposes a storage layout that keeps memory access contiguous in both forward and backward passes without explicit off-chip im2col representations.The layout is designed for end-to-end training.Source: 4) Channel-First Im2col · p. 3

D. Quantization

Microscaling quantizes tensor blocks with shared scales to balance rounding and clipping error. The paper introduces this quantization background, motivates its use for CNN tensors, and describes the NVFP4-based quantization procedure used in its experiments.

D. Quantization

  • Per-block microscaling balances rounding and clipping errors by avoiding clipping a block's maximum value.The balance supports quantization of weights, activations, and gradients.Source: D. Quantization · pp. 3–4

III. MOTIVATIONS

  • The paper frames microscaling quantization for convolution as a problem prior work had not addressed.Source: §III · p. 4

A. Microscaling quantization

  • Each tensor is divided into B-element blocks, and values use both per-tensor and per-block scaling before quantization.The described NVFP4 setup uses FP4 E2M1 elements in blocks of 16, with a master FP32 weight copy for updates.Source: A. Microscaling quantization · p. 4, D. Quantization · p. 4
  • The quantization procedure rounds block scales to FP8 E4M3 and elements to FP4 E2M1, using nearest rounding for weights and activations and stochastic rounding for gradients.Fake quantization rescales the quantized values by the block and tensor scales.Source: A. Microscaling quantization · p. 4

B. Issues with dot products of microscaled blocks

Microscaling is efficient when its block direction aligns with the dot product, but transposed convolution computations can require a different direction. Existing square quantization addresses reuse for fully connected layers, while its convolutional extension can erase memory gains.

B. Issues with dot products of microscaled blocks

  • When microscaling blocks align with a dot-product direction, the product can be expressed using quantized elements and their block scales.The formulation partitions vectors into blocks of B elements and sums blockwise products.Source: B. Issues with dot products of… · p. 4
  • When the quantization direction differs from the dot product, scales must be applied after each low-precision multiplication, making whole-tensor requantization more efficient.Training uses tensors in two GeMM directions, including transposed weights and activations during backpropagation.Source: B. Issues with dot products of… · p. 4

C. Limitations of related work

  • Square quantization allows forward-quantized tensors to be reused in backward computation, but the cited work considered only fully connected layers.Extending it to convolution would require retaining square-quantized im2col-expanded activations, reducing or cancelling memory benefits.Source: C. Limitations of related work · pp. 4–5

IV. MICROQONV: RESHAPING MEMORY LAYOUT FOR EFFICIENT MICROSCALING TRAINING

MicroQonv reorders convolution tensors so each can be quantized once while retaining efficient computation. Its channel-batch-first layout supports contiguous access in both convolution passes, and square microscaling blocks align quantization with the computation directions.

IV. MICROQONV: RESHAPING MEMORY LAYOUT FOR EFFICIENT MICROSCALING TRAINING

  • MicroQonv reorders weights, activations, and gradients to support one-time quantization during training and efficient computation.The section also compares its quantization and memory-movement gains and accuracy with other methods.Source: §IV · p. 5

A. Reshaping memory layout with channel-batch-first im2col

  • Channel-batch-first storage makes activation accesses contiguous along channels for forward im2col and along the batch dimension for backward input gradients.It also allows contiguous activation blocks to be loaded once and reused on-chip, reducing memory movement.Source: A. Reshaping memory layout with… · p. 5

B. Efficient quantization directions

  • The method reshapes weights and output gradients into matrices for the three convolution GeMM computations, then reorders their outputs into gradient tensors.Source: B. Efficient quantization directions · p. 5
  • Square quantization blocks align with computation directions: activations use Cin and N, weights use Cin and Cout, and output gradients use Cout and N.The experiments use NVFP4 with B = 64 (8×8), while the approach is described as compatible with other microscaling block scales and element formats.Source: B. Efficient quantization directions · p. 5
  • Figure 3 depicts compact off-chip activation storage, quantization blocks across N and Cin, and reuse that reduces redundant memory reads.MicroQonv is described as hardware-agnostic and requiring no additional circuitry.Source: Figure 3 · p. 5, B. Efficient quantization directions · p. 6

C. Gains in memory movement and per-element quantization

MICROQONV reduces activation quantization and backward memory overhead while keeping accuracy close to full-precision baselines in classification and object detection. Its accuracy trade-off can be reduced with a higher-bit activation variant.

C. Gains in memory movement and per-element quantization

D. Accuracy analysis

  • All methods remain within 0.6% of FP32 accuracy in both training scenarios, while MICROQONV is slightly below Cuyckens et al., which is slightly below Dacapo.Quantization methods fall within a 0.25% σ window, described as roughly the margin of error.Source: D. Accuracy analysis · p. 6
  • MICROQONV+ matches the FP32 baseline by using NVFP6 activations, with roughly ×5.1 less memory movement than Dacapo and ×6 less than Cuyckens et al. extrapolation.The variant uses NVFP4 for weights and gradients and requires approximately 50% more activation memory movement than MICROQONV.Source: D. Accuracy analysis · p. 6

E. Viability in Object Detection

  • Object-detection evaluation uses NVFP8 for weights, activations, and gradients, with selected layers kept at full precision and YOLOV26nano depthwise layers kept in BF16.YOLOV8nano is finetuned on PascalVOC and YOLOV26nano on KITTI; both use pretrained models and 20-epoch finetuning.Source: E. Viability in Object Detection · p. 7
  • YOLOV8nano MICROQONV-8 is within 0.3 mAP of FP32 and less than 0.15 away from Dacapo-8 and Cuyckens et al.-8; YOLOV26nano MICROQONV-8 is above both alternatives.From-scratch runs are within 0.2 mAP of FP32 for YOLOV8nano; for YOLOV26nano they are .2 mAP away on mAP50 and less than 0.8 mAP away on mAP50-95.Source: E. Viability in Object Detection · p. 7

V. DRAM SIMULATION

The DRAM simulation estimates memory-transfer costs between off-chip DRAM and on-chip SRAM during forward and backward propagation. It evaluates four model and dataset configurations using an LPDDR4 memory model.

V. DRAM SIMULATION

  • The simulation counts DRAM–SRAM transfers during forward and backward passes, including quantized weights and activations saved for backward propagation.Input and output tensors for one layer are assumed to fit on-chip; other layer tensors are stored off-chip.Source: §V · p. 7
  • Only off-chip-to-on-chip movement is compared, because it is described as more costly than on-chip movement and computation.The methods use the same convolution operations, so internal on-chip transfer and computation costs are expected not to differ much.Source: §V · p. 7

A. Experimental Setup

  • Four configurations are simulated: ResNet32/CIFAR-100, ResNet18/ImageNet, YOLOV8nano/PascalVOC, and YOLOV26nano/KITTI, using LPDDR4.Batch sizes are 32 for the classification configurations and 8 for the object-detection configurations.Source: A. Experimental Setup · p. 7

B. Results

MICROQONV improves memory-transfer efficiency while maintaining competitive accuracy across image classification and object detection. Its gains vary with model and convolution structure, and the DRAM simulation excludes quantization cost.

B. Results

  • ×7.53, ×3.93, and ×3.4 are the reported gains in activation transfer and storage over full-precision activations for ResNet-32/18, YOLOV8n, and YOLOV26n, respectively.The YOLOV8n gain reflects FP8 quantization, while YOLOV26n's BF16 depthwise layers reduce its gain.Source: B. Results · p. 8
  • ×3.5 and ×2.2 are the reported activation memory, energy, and latency gains for YOLOV8n and YOLOV26n, respectively, over im2col expansion.The smaller YOLOV26n gain is linked to more 1×1 kernels and depthwise convolutions quantized in BF16; the simulation excludes quantization cost.Source: B. Results · p. 8
  • Table V reports whole-model tensor transfer size, DRAM transfer energy, and latency, with MICROQONV improving over both comparison methods.The authors report the same gains for these three measures under the same quantization method and model, across LPDDR4, DDR5, DDR6, and HBM models.Source: Table V · p. 8, B. Results · p. 8

VI. TRAINING PERFORMANCE

  • The image-classification and object-detection experiments use matched quantization-method setups, including learning rate and epoch count, for fair comparison.Object-detection experiments keep input and output layers in higher precision; YOLOV26nano depthwise layers use BF16.Source: §VI · p. 8, B. Object detection · p. 9

A. Image classification

  • MICROQONV beats LUQ† by more than 2% on ResNet-32 and 0.8% on ResNet-18, while QLR has slightly better accuracy at much higher training cost.The comparison reports mean and standard deviation over five runs; MICROQONV uses a unified format, unlike LUQ's different forward and backward formats.Source: A. Image classification · p. 8, Table VI · p. 9

B. Object detection

  • MICROQONV-6 stays within 0.3mAP of the FP32 baseline on YOLOV8nano and within 0.3mAP in YOLOV26nano fine-tuning, while outperforming per-channel quantization.For YOLOV26nano training from scratch, it is less than 1.3mAP away from the baseline; it also exceeds Dacapo-6 by 0.1mAP.Source: B. Object detection · p. 9, Table VII · p. 9

VII. MICROQONV IN A CONTINUAL LEARNING SETTING

The continual-learning setting addresses sequential learning with new classes, where limited data storage and training only on currently available data contribute to catastrophic forgetting. The section situates MICROQONV within replay-based approaches and quantized latent replay.

VII. MICROQONV IN A CONTINUAL LEARNING SETTING

  • Continual learning targets catastrophic forgetting as models learn sequential tasks with new classes and cannot retain the complete history of data.The section focuses on class-incremental learning with latent replay.Source: §VII · pp. 9–10

A. Related Work

  • Latent replay stores activations from a frozen feature extractor's early layers instead of original images, reducing backpropagation cost.Early layers tend to learn generic features, which limits accuracy degradation compared with whole-model retraining.Source: A. Related Work · p. 10
  • Prior quantized latent replay work includes QLR's 8-bit inference quantization with FP32 training and LifeLearner's 8-bit quantization with a meta-learned feature extractor and product quantization.LifeLearner requires pre-training on the test data distribution and is described as sensitive to distribution shifts; QLR uses no compression beyond quantization.Source: A. Related Work · p. 10
  • MICROQONV applies its microscaling format to latent replay to improve on the cited quantized latent replay results.The paper frames this as its aim in the continual-learning setting.Source: A. Related Work · p. 10

B. Experimental Results

The experiments evaluate MICROQONV in quantized latent replay across replay-buffer configurations and datasets. They report higher accuracy than QLR and, on CIFAR-100 at a fixed 2,000-sample budget, accuracy less than 1% below the FP32 baseline.

B. Experimental Results

  • MICROQONV accuracy is systematically and significantly above QLR across tested latent-replay layers and replay-buffer sizes.The paper links the accuracy difference to storing approximately ×2.5 more samples because MICROQONV saves 3.125 bits per activation pixel instead of 8.Source: B. Experimental Results · p. 10, Figure 7 · p. 10
  • MICROQONV exceeds QLR by +5.7% to +11% on CIFAR-100, +4.8% to +9.4% on CORE50, and +9.6% to +16.7% on CUB-200.The accuracy gap tends to be higher with fewer samples.Source: Table IX · p. 11, B. Experimental Results · p. 11
  • At a fixed 2,000-sample CIFAR-100 budget, accuracy is 48.16 ± .66% for MICROQONV, 48.77 ± .23% for QLR, and 48.83 ± .36% for FP32.MICROQONV is reported as less than 1% from the baseline.Source: B. Experimental Results · p. 11

VIII. CONCLUSION

  • MICROQONV reduces quantization overhead by ×2 for weights and gradients and up to ×9 for activations, while reducing activation memory movement and storage by up to ×7.53.The conclusion reports activation movement gains of ×3.5 for YOLOV8nano and ×2.2 for YOLOV26nano, and +5.7% to +11% accuracy improvement with 4-bit activations in latent replay at the same memory budget.Source: §VIII · p. 11

Related papers

Loading 2609.28358v1…