PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionMX: Unlocking Microscaling Post-Training Quantization for Vision Models

Elad Dror Cohen, Ofir Gordon, Lior Dikstein, Idan Achituve, Hai Victor Habi

arXiv:2610.03218v1cs.CV

TL;DR

Vision-specific evidence on post-training microscaling quantization remains limited and mixed, leaving open when it works and what adaptations are needed. VisionMX combines bounded-range weight rounding with foldable affine activation correction, and outperforms established PTQ baselines across broad vision tasks and MX formats.

  • Problem

    Vision-specific evidence on MX quantization is limited and mixed, leaving open when it is effective for vision inference and which PTQ adaptations are needed.

    Source: §1 · p. 1

  • Method

    VisionMX combines bounded-range adaptive weight rounding with an affine activation correction that shifts and scales activations before quantization and folds the inverse transformation into the subsequent operation.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    VisionMX outperforms established PTQ baselines on MXFP4-UE8M0, MXFP4-UE5M3, and NVFP4 across a broad range of vision tasks.

    Source: §7 · p. 9

  • Takeaways & Limitations

    VisionMX achieves state-of-the-art performance in most benchmarked models, and the results indicate that MX data formats are well suited for vision applications.

    Source: §7 · p. 9

    The bounded candidate-window guarantee applies under its stated assumptions, including the theorem’s conditions on the weights and task-loss approximation.

    Source: §5.2 · p. 5

Abstract

from arXiv · show

Microscaling (MX) formats are emerging as a hardware-supported approach to efficient training and inference. They combine low-precision elements with shared block scales, but their impact on vision models remains underexplored. We systematically investigate post-training MX quantization across vision models and tasks. An analysis of direct conversion identifies three sources of error: block-scale representation, the poor alignment of some small convolutional weight tensors with nonuniform element grids, and the underuse of signed codes by nonnegative activations. These findings motivate VisionMX, a post-training MX quantization method that optimizes bounded weight rounding and applies a foldable affine correction to activations. We evaluate VisionMX across image classification, object detection, semantic segmentation, and low-light image enhancement using several MX-style formats. It improves on direct conversion and the evaluated post-training quantization baselines, with the largest performance recoveries in architectures most sensitive to MX conversion

1. Introduction

The paper asks when microscaling quantization works for vision inference and what post-training adaptations are needed when direct conversion falls short. It analyzes architecture and representation sensitivity, then introduces VisionMX and evaluates it across vision models and tasks.

Motivation

  • Vision-specific evidence on MX quantization is limited and mixed, motivating study of when it works and which PTQ adaptations are needed.The introduction notes substantial MXFP4 degradation in some architectures and settings.Source: §1 · p. 1

Study scope

  • The study examines how architecture, element and block-scale formats, and weight versus activation quantization affect vision-model performance.It spans image classification, object detection, semantic segmentation, and low-light enhancement.Source: §1 · p. 1

Analysis findings

  • Evaluated transformers are relatively robust, while compact convolutional and hybrid models can degrade severely; nonuniform grids and signed codes for nonnegative activations are representation mismatches.The paper links these findings to block-scale effects and limitations of adaptive rounding on nonuniform grids.Source: §1 · pp. 1–2

VisionMX

  • VisionMX combines RangeRound, which optimizes weight assignments over a local range of levels, with affine activation correction that shifts and scales activations before quantization.The inverse activation transformation is folded into the following linear operation.Source: §1 · p. 2

Evaluation

  • VisionMX improves over established PTQ baselines in most evaluated settings under joint 4-bit weight and activation quantization.The evaluation covers convolutional, transformer, and hybrid architectures; ablations assess each component and the rounding range.Source: §1 · p. 2

2. Background and Notation

The background introduces the paper’s notation and describes microscaling formats as low-precision elements paired with shared block scales. It distinguishes configurations by their element and scale formats.

2. Background and Notation

  • Vectors and their scalar components are denoted by x and x_i, while matrices and tensors are denoted by A.Source: §2 · p. 2

2.1. MX Block-Scaled Quantization

  • MX quantization groups values into blocks that share a scale, with element and scale quantizers mapping to their respective representable sets.The paper considers MXFP4 with E2M1 elements and either UE8M0 power-of-two scales or UE5M3 scales with finer granularity.Source: §2.1 · p. 2

3. Related Work

Prior work has developed MX methods mainly for language models, while vision-specific MX evidence remains limited. VisionMX addresses this gap by combining learned weight rounding with affine activation correction for post-training MX quantization.

MX quantization

  • Most subsequent MX research targets language models, exploring mixed formats, feature transformations, reconstruction, and block-scale choices [9] [13] [23] [33].FAAR adapts learned rounding to NVFP4’s nonuniform grid [17], while [10] proposes UE5M3 scales.Source: §3 · p. 3

Vision MX quantization

  • Vision studies report MXFP4 degradation, particularly for compact convolutional networks, while TetraJet studies oscillation-reduced MXFP4 training for vision transformers rather than PTQ [36] [7].A separate study investigates NVFP4 for image classification [39].Source: §3 · p. 3

Vision PTQ

  • Integer vision PTQ includes learned rounding and reconstruction methods such as AdaRound [31] and BRECQ [18], but the discussed methods target integer quantization.VisionMX explores post-training MX quantization by combining learned weight rounding with affine activation correction.Source: §3 · p. 3

4. Analysis

The analysis tests architecture-dependent MX sensitivity and examines weight rounding on nonuniform grids and signed-code use by nonnegative activations. It finds that sensitivity varies across models and motivates bounded-range rounding and activation affine correction.

4.1. Architecture Behavior under MX Quantization

  • Direct MXFP4-UE8M0 conversion produces widely varying losses: MobileNetV2 loses 97.40% of reference accuracy under weight-only and 52.30% under activation-only quantization.DeiT-B, Swin-T, ViT-S/16, and ConvNeXt-Tiny remain within 5% relative loss in both modes; EfficientViT-L2 approaches complete accuracy loss.Source: §4.1 · p. 3, Figure 1 · p. 4
  • At MXFP8 precision, performance remains close to the floating-point reference for most evaluated models.Source: §4.1 · p. 3

4.2. Analysis of Weight MX Quantization

  • Under the stated task-Hessian conditions, the unique optimal INT4 assignment lies in the rounding up–down set, and the FP4 inequality follows from candidate inclusion.This result concerns the quadratic loss approximation; extension to exact task loss requires control of the Taylor remainder.Source: §4.2 · p. 4
  • For FP4 E2M1, adjacent element spacings are 0.50, 1, and 2 before block scaling, so nearest up–down rounding can miss a lower-loss assignment.In the quadratic example, assignment (2, 4) has loss 0.315 versus 0.365 for (1.50, 4), despite greater squared weight error.Source: §4.2 · p. 4
  • In weight-only sweeps for ResNet-18 and ConvNeXt-Tiny, UE5M3 achieves the highest ImageNet top-1 accuracy among the evaluated scale formats.This is consistent with prior work recommending UE5M3 scales [10].Source: §4.2 · pp. 4–5

4.3. MX Quantization after Nonlinearities

  • Nonnegative ReLU outputs cannot use negative codes in signed MX grids, leaving roughly half of nonzero values unused for an approximately symmetric grid.Changing the shared scale changes the range but cannot recenter the distribution or expose the negative codes.Source: §4.3 · p. 5
  • Activation Affine Correction shifts and scales activations before quantization, then folds the inverse transformation into the following linear or convolutional operation.The shift makes previously unused negative-side codes accessible.Source: §4.3 · p. 5

5. Method

VisionMX combines activation affine correction with bounded weight rounding to address two mismatches between vision-model values and MX quantization grids. Its reconstruction pipeline offers layerwise and blockwise variants.

5.1. Activation Affine Correction

  • Activation Affine Correction shifts and scales activations before MX quantization, then folds the inverse transform into the linear operation.Correction parameters are calibrated per channel and block, accounting for the shared block scale.Source: §5.1 · p. 5

5.2. RangeRound for MX Quantization

  • RangeRound searches a bounded range of legal levels, including the nearest strict lower and upper codes, rather than restricting weights to adjacent levels.The candidate window is defined in the normalized element domain.Source: §5.2 · p. 5
  • Under the stated assumptions, Proposition 5.1 shows that an unscaled candidate window preserves the minimum of the task-loss quadratic approximation.For MX quantization, the proposition requires corresponding assumptions on normalized weights and the transformed Hessian.Source: §5.2 · p. 6
  • Its multi-well penalty is zero at legal representable codes and largest at cell midpoints, steering relaxed weights toward discrete values.The penalty is summed over quantized weight tensors in the reconstruction unit.Source: §5.2 · p. 6, Figure 2 · p. 6

5.3. Optimization

  • VisionMX uses asymmetric reconstruction and provides layerwise VisionMX-LW and blockwise VisionMX-BW variants; block scales and AAC parameters remain fixed during rounding optimization.After optimization, relaxed weights are hardened to legal codes, and each reconstructed unit is persisted for subsequent calibration.Source: §5.3 · pp. 6–7

6. Experimental Results

Across evaluated vision tasks, VisionMX improves on direct conversion, with gains reported for ImageNet classification, object detection, and semantic segmentation. The experiments also compare PTQ baselines and assess the contributions of the method’s components.

6.1. Experimental Setup

  • The evaluation covers ImageNet-1K, COCO detection and segmentation, and low-light enhancement, comparing RTN, AdaRound, and QDrop with VisionMX variants across 4-bit block-scaled formats.Results are averaged over five random seeds; calibration sets contain 2,048 ImageNet or 256 COCO and low-light samples.Source: §6.1 · pp. 7–8

6.2. ImageNet Classification Results

  • VisionMX improves ImageNet top-1 accuracy across evaluated CNN and hybrid models; VisionMX-BW gains 5.64–9.59 AP and 3.04–48.51 mIoU over RTN on detection and segmentation.For ImageNet, UE5M3 block scaling consistently improves performance across models, with gains reaching around 3% in some cases.Source: Table 2 · p. 8, Table 3 · p. 8, §6.2 · p. 8

6.4. Ablation Studies

  • With RR and AAC, learned bias updates add 0.29 and 0.06 top-1 points; with RR and bias, AAC adds 8.11 and 0.26 points on MobileNetV2 and CaiT-XXS24, respectively.The blockwise ablation uses MXFP4 elements with UE8M0 block scales and QDrop.Source: §6.4 · p. 8
  • With bias and AAC enabled, RR adds 0.69 points on MobileNetV2 but 0.01 on CaiT-XXS24; all three components produce each model’s highest evaluated blockwise accuracy.Source: §6.4 · pp. 8–9

7. Conclusion

The conclusion identifies architecture-dependent sensitivity and two quantization mismatches, then presents VisionMX as a response and reports improvements over established PTQ baselines across vision tasks.

7. Conclusion

  • Compact convolutional and hybrid architectures tend to be more sensitive to MX quantization than transformers; nonuniform grids and unused negative codes for nonnegative activations are identified as error sources.Source: §7 · p. 9
  • VisionMX extends adaptive rounding across a bounded range of nonuniform-grid levels and adapts affine activation correction to shift and scale activations for better use of signed grids.Source: §7 · p. 9
  • The method outperforms established PTQ baselines on MXFP4-UE8M0, MXFP4-UE5M3, and NVFP4 across a broad range of vision tasks.Source: §7 · p. 9

A.1. Experimental Details

The appendix specifies quantization and reconstruction settings for the evaluated tasks, then details blockwise reconstruction and activation affine correction. It also gives assumptions and limitations for the second-order weight-quantization analysis.

A.1. Experimental Details

  • ImageNet-1K evaluation uses deterministic class-round-robin calibration and reports top-1 accuracy; COCO and low-light evaluation use task-specific calibration and metrics.ImageNet validation uses all 50,000 images; COCO evaluation uses 5,000 val2017 images, and low-light evaluation uses LOLv2-Real test images at native resolution.Source: §A.1 · p. 12

A.2. Quantization Format Implementation Details

  • MXFP4 uses signed E2M1 elements with blocks of 32 and no tensor-level scale, while NVFP4 uses E2M1 elements, E4M3 scales over groups of 16, and an FP32 tensor-level scale.NVFP4 local scales are selected with the tensor-level factor held fixed; weights use MSE-based scale selection and ties choose the smaller scale.Source: §A.2 · p. 12, §A.2.1 · pp. 12–13

A.2.1. Element rounding and scale selection

  • Element rounding uses nearest representable E2M1 values with ties to even and saturation at ±6; weight block scales minimize reconstruction MSE over the positive scale codebook.For MXFP4, the searched tensor is the original input; for NVFP4, it is normalized by the fixed tensor-level factor.Source: §A.2.1 · pp. 12–13

A.3. RangeRound Implementation Details

  • RangeRound initializes block scales from the active quantizer and keeps scales and zero-points fixed while optimizing rounding logits.Its implementation uses a rectified sigmoid with γ = −0.10 and ζ = 1.10, and draws one seeded cached calibration batch at each step.Source: §A.3 · p. 13

A.4. Blockwise Reconstruction Setup

  • Blockwise reconstruction treats each unit as a complete executable module and matches its student output to the floating-point reference output.The module passes include intervening operations, residual paths, nonlinearities, normalization, and internal biases; optimization includes rounding penalties and enabled bias updates.Source: §A.4 · p. 13

A.5. Activation Affine Correction Details

  • Activation affine correction applies per-channel scale and shift before dynamic block activation quantization, then folds the inverse transform into the following floating-point linear operation.For formats without tensor-level scales, calibration searches shared block parameters and can use them to initialize per-channel search; replacing activation quantization with identity preserves the original computation algebraically, up to floating-point rounding.Source: §A.5 · p. 13, Initialization of Activation Affine… · p. 14

B.1. Second-Order Analysis of Weight Quantization

  • The second-order weight-quantization analysis assumes a twice-continuously differentiable expected loss near the float weights and interchangeability of differentiation and expectation.The approximation requires a stationary float solution or a negligible linear contribution; large quantization errors and nonsmooth activation-boundary changes can invalidate it.Source: §B.1 · p. 14

B.2. Block Scale Format Analysis

The diagnostic varies unsigned block-scale formats for MXFP4 while holding the remaining setup fixed within each model and quantization mode. Results differ by model and mode, and the appendix separately states the diagonal-dominance condition used in its rounding proof.

B.2. Block Scale Format Analysis

  • The diagnostic evaluates ImageNet top-1 accuracy with group size 32 and round-to-nearest quantization, changing only the unsigned exponent scale format within each model and mode.The sweep covers UE8M0 through UE1M7; Figure 3 and Figure 4 show the ResNet-18 and ConvNeXt-Tiny diagnostics.Source: §B.2 · p. 14, Figure 3 · p. 15, Figure 4 · p. 15
  • UE8M0 uses exponent-only powers of two with floor rounding, while UE7M1 through UE1M7 round mantissas to the nearest code and clamp exponent and mantissa endpoints.The diagnostic selects a max-based, no-clipping scale before applying each scale format.Source: §B.2 · pp. 14–15
  • UE5M3 gives the highest weight-only accuracy among the tested formats for both ResNet-18 and ConvNeXt-Tiny, while activation-only preferences differ by model.ResNet-18’s highest activation-only result is UE4M4; ConvNeXt-Tiny improves most at UE4.Source: §B.2 · pp. 14, 16, Table 5 · p. 15
  • The plotted diagnostic’s scale-format preference is mode-dependent: the best activation-only format is UE4M4 for ResNet-18 and UE4 for ConvNeXt-Tiny.The corresponding weight-only best is UE5M3 for both models.Source: Table 5 · p. 15, §B.2 · p. 16

C.1. Proof of Theorem 4.1

  • The proof states a factor-two strengthening of the usual row-diagonal-dominance condition for its rounding results.The appendix identifies diagonal dominance of the task-loss Hessian as the assumption used in the proof.Source: §C.1 · p. 16
  • In the proof, a candidate coordinate update is chosen within the representable interval and yields a nonpositive loss change under the stated condition.The argument then invokes uniqueness to obtain a contradiction.Source: §C.1 · p. 16

C.2. Proof of Proposition 5.1

The appendix analyzes bounded candidate windows for weight rounding, illustrates their behavior for INT4 and FP4 under changing input correlation, and reports additional quantization diagnostics across vision tasks and models.

C.2. Proof of Proposition 5.1

  • For unscaled E2M1 FP4, a candidate-window width δ ≥ 4 suffices, without requiring uniqueness.The proposition’s bounded-window result is stated for normalized element coordinates; fixed-scale MX quantization is covered by a corollary.Source: §C.2 · pp. 16–17
  • The candidate-window argument requires the Hessian condition after transformation, since unequal block scales need not preserve factor-two row diagonal dominance.The window width δ is measured in normalized element coordinates.Source: §C.2 · p. 17

C.3. INT4–FP4 Reconstruction Example

  • In the fixed example, FP4’s full-grid and window optima coincide at (2, 4), reducing reconstruction loss from 0.365 to 0.315 despite greater squared weight error.The FP4 full-grid optimum lies outside the rounding up–down set; INT4’s full-grid optimum remains within it, and INT4 has lower absolute loss in this example.Source: §C.3 · p. 17, Table 7 · p. 17, Figure 6 · p. 18

C.4. Effect of Input Correlation

  • For the example’s FP4 weights, the outside candidate improves on every rounding up–down assignment when input correlation ρ exceeds 7/18.Enumeration finds that the full-grid and window minima coincide throughout the sweep; for INT4, the nearest and alternative pairs tie at ρ = 8/17, with the alternative optimal above that threshold.Source: §C.4 · pp. 17–18

D. MX Block Drop

  • MX block drop samples one activation-quantization bypass decision per block scale, and inference still uses fully quantized activations.The adaptation is applied at reconstruction-unit inputs and internal activation quantizers for QDrop and VisionMX-BW, but disabled for VisionMX-LW.Source: D. MX Block Drop · p. 18

E.1. Additional Results on Vision Transformers

  • Blockwise VisionMX achieves the highest reported accuracy on DeiT-T and Swin-T across all three evaluated 4-bit block-scaled formats.The additional results concern vision transformers.Source: §E.1 · p. 19, Table 8 · p. 19

E.2. Low-Light Image Enhancement Results

  • The low-light evaluation covers RetinexFormer and HVI-CIDNet on all 100 LOLv2-Real test pairs using three MXFP4/NVFP4 formats and RGB PSNR, SSIM, and LPIPS-Alex.The caption specifies that SSIM and LPIPS-Alex are multiplied by 100 and identifies the better direction for each metric.Source: §E.2 · p. 19, Table 9 · p. 20

E.3. Weight and Activation Analysis

  • Weight-only quantization degrades MobileNetV2 and DeepLabV3-MobileNetV3 more than activation-only quantization, while the evaluated low-light models show the opposite pattern.The dominant degradation source therefore differs across the evaluated models and tasks.Source: §E.3 · pp. 19–20

E.4. MXFP8 Results

The MXFP8 results show near-floating-point performance for many evaluated models, but EfficientViT-L2 remains sensitive to scale format. VisionMX-BW obtains similar reported top-1 accuracy with the two listed scales.

E.4. MXFP8 Results

  • 84.93% and 84.94% top-1 accuracy are reported for EfficientViT-L2 with VisionMX-BW using MXFP8-UE8M0 and MXFP8-UE5M3, respectively.These results use δ = 0.7; MXFP8 retains near-floating-point performance for many evaluated models, while EfficientViT-L2 remains highly sensitive to the scale format.Source: §E.4 · p. 21, Table 11 · p. 21

E.5. Layerwise Ablation without QDrop

The no-QDrop layerwise ablation combines RangeRound, bias updates, and AAC, but differing optimization settings prevent a matched comparison with the main-text blockwise ablation.

E.5. Layerwise Ablation without QDrop

  • The layerwise ablation and main-text blockwise ablation use different optimization settings, so their tables do not provide a matched comparison of reconstruction strategies.Source: §E.5 · p. 21
  • The ablation uses 20,000 optimization steps and 2,048 calibration samples, with Top-1 accuracy reported as a percentage.MobileNetV2 uses RangeRound δ = 3.00; CaiT-XXS24 uses δ = 2.00.Source: Table 12 · p. 22
  • 66.21% on CaiT-XXS24 is the highest reported accuracy, while MobileNetV2 reaches 65.11% with all three components.MobileNetV2's highest accuracy is 65.46% with bias updates and AAC but without RangeRound.Source: §E.5 · p. 22

Related papers

Loading 2610.03218v1…