PaperlayerView exact revision on arXiv

Source-linked AI summary

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, Xindian Ma

arXiv:2601.07475v2cs.LGcs.AI

TL;DR

Fine-grained NVFP4 creates accuracy and hardware-compatibility challenges for existing PTQ methods. ARCQuant addresses them by augmenting activations with quantized residual channels within a unified NVFP4 GEMM path. The method achieves W4A8-level performance, comparable worst-case error bounds to MXFP8, and practical speed improvements on supported Blackwell GPUs.

  • Problem

    Existing PTQ strategies struggle to preserve accuracy and unified hardware execution when adapting aggressive W4A4 quantization to fine-grained NVFP4.

  • Method

    ARCQuant augments the activation matrix with quantized residual channels and integrates dual-stage error compensation into the matrix reduction dimension.

  • Results

    ARCQuant consistently delivers W4A8-level performance across LLaMA and Qwen models, with a worst-case error bound comparable to MXFP8.

  • Takeaways & Limitations

    Unified NVFP4 residual compensation can combine competitive accuracy with standard optimized GEMM execution under W4A4 hardware constraints.

  • Takeaways & Limitations

    Practical throughput benefits depend on NVIDIA Blackwell hardware with native NVFP4 support, while offline calibration assumes relatively stable outlier-channel distributions during inference.

Abstract

from arXiv · show

The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference. However, it is difficult to adapt existing Post-Training Quantization (PTQ) strategies to these formats: rotation-based methods compromise fine-grained block isolation; smoothing techniques struggle with significant 4-bit quantization errors; and mixed-precision approaches often conflict with hardware constraints on unified-precision computation. To address these challenges, we propose ARCQuant, a framework that boosts NVFP4 performance via Augmented Residual Channels. Distinct from methods that compromise block isolation or hardware uniformity, ARCQuant maintains a strictly unified NVFP4 format by augmenting the activation matrix with quantized residual channels. This design integrates the error compensation process directly into the matrix reduction dimension, enabling the use of standard, highly optimized GEMM kernels with minimal overhead. Theoretical analysis confirms that the worst-case error bound of our dual-stage NVFP4 quantization is comparable to that of standard 8-bit formats such as MXFP8. Extensive experiments on LLaMA and Qwen models demonstrate that ARCQuant achieves state-of-the-art accuracy, comparable to full-precision baselines in perplexity and downstream tasks. Furthermore, deployment on RTX 5090 and RTX PRO 6000 GPUs confirms practical benefits, achieving up to 3x speedup over FP16. Our code is available at https://github.com/actypedef/ARCQuant.

1 Introduction

ARCQuant addresses NVFP4’s accuracy and hardware challenges by augmenting activations with quantized residual channels while preserving a unified NVFP4 data path. It combines dual-stage error compensation with optimized GEMM execution and reports W4A8-level performance and strong deployment gains.

  • Motivation: W4A4 quantization is needed for high inference throughput, but maintaining model fidelity under aggressive NVFP4 compression remains difficult.NVFP4 introduces fine-grained microscaling, while existing strategies face format-specific limitations.
  • Challenges: Global transformations can redistribute outliers across NVFP4 blocks, while mixed-precision methods conflict with unified Tensor Core execution.NVFP4’s block size and heterogeneous higher-precision granularities create hardware incompatibilities.
  • ARCQuant: ARCQuant augments the input tensor with quantized residual channels, using dual-stage quantization to capture high-magnitude structure and recover fine-grained residual information.The compensation process is mapped into the extended reduction dimension of a single matrix multiplication.
  • Results: ARCQuant consistently outperforms existing NVFP4 adaptation strategies across LLaMA and Qwen models, delivering W4A8-level performance under W4A4 constraints.The contribution is reported across multiple model families rather than a single evaluation setting.
  • Theory: The dual-stage mechanism has a worst-case error bound comparable to standard single-stage MXFP8, theoretically bridging the W4A4–W4A8 precision gap.The analysis provides a formal error-bound comparison rather than only empirical accuracy results.
  • Deployment: Custom kernels map residual accumulation into standard unified-precision GEMM calls, improving prefill speed and memory efficiency on RTX 5090 and RTX PRO 6000 versus FP16.The kernel design targets portability and efficient execution on Blackwell platforms.

2 Related Work

Prior NVFP4 quantization work includes mixed-precision, transformation-based, and compensation-based strategies. These approaches respectively preserve selected information, reshape distributions, or optimize reconstruction, but do not fully address ARCQuant’s target of unified runtime activation-error compensation.

  • Mixed-precision Methods: Mixed-precision methods preserve sensitive weights or activation channels in INT8 or FP16 while keeping the remaining computation in INT4.Decomposition-based variants isolate outliers in a separate higher-precision low-rank branch.
  • Transformation-based Methods: Transformation-based methods reshape activation distributions through scaling, rotations, or learnable affine transformations to suppress outliers.The cited approaches include SmoothQuant, QUIP, QuaRot, and FlatQuant.
  • Transformation-based Methods: Figure 2 contrasts ARCQuant’s outlier isolation and residual-error suppression with Hadamard’s spreading of outlier magnitudes across dimensions.The comparison concerns activation magnitudes and quantization errors on Llama 3.1-8B o_proj.
  • Compensation-based Methods: Compensation-based methods such as GPTQ and APTQ minimize reconstruction error with Hessian-based weight optimization.These methods target static parameter compression rather than dynamic runtime compensation of activation quantization errors.

3 Methodology

ARCQuant targets NVFP4 quantization errors and hardware constraints by adding unified-precision residual channels, integrating compensation into an extended GEMM dimension. Its dual-stage design achieves MXFP8-comparable worst-case error bounds while preserving NVFP4 execution.

  • Problem and design: ARCQuant targets W4A4 quantization while preserving unified NVFP4 data paths and hardware compatibility.It addresses the isolation problems of global rotations and the format incompatibility of mixed precision.
  • Augmented residual channels: ARCQuant augments activations with quantized outlier residual channels, using primary quantization for dominant structure and a second stage for residual information.Calibration identifies compensated channels; residuals are quantized and concatenated along the reduction dimension.
  • Unified GEMM execution: The augmented activation and weight tensors allow primary and residual computations to execute through one unified NVFP4 GEMM.The original reduction dimension becomes Kin+S, while the compensation term is accumulated alongside the main computation.
  • Kernel design: ARCQuant uses a fused kernel for channel reordering, RMSNorm, primary quantization, and residual quantization before standard CUTLASS GEMM execution.The resulting tensor remains strictly NVFP4, avoiding GEMM kernel modification.
  • Error bound analysis: ARCQuant’s dual-stage mechanism matches MXFP8’s worst-case error bound for compensated outlier channels.The analysis compares the dual-stage NVFP4 alignment factor with MXFP8 and reports sup α1α2 = 1.266 versus sup αmx = 2.

4 Experiments

ARCQuant is evaluated across model families, tasks, quantization baselines, calibration settings, architectures, and hardware. It combines strong accuracy with low overhead and substantial inference-throughput gains.

  • Experimental Setup: ARCQuant is evaluated on Llama 3.1 and Qwen2.5 models using perplexity, reasoning, and domain-task metrics.The evaluation includes WikiText2 perplexity, MMLU and zero-shot reasoning benchmarks, plus HumanEval, MBPP, GSM8K, and CMATH.
  • Main Results: ARCQuant consistently achieves the best results among W4A4 methods and can surpass the W4A8 RTN baseline on Llama 3.1-8B and Qwen2.5-7B.On Qwen2.5-7B, it reduces perplexity by 1.68 points compared with Atom; on Qwen2.5-32B, it nearly matches FP16.
  • Quantization Strategies: ARCQuant outperforms conventional NVFP4 strategies across reported metrics, while QuaRot can regress against RTN and SmoothQuant provides only marginal gains.The results support the stated concern that rotations disrupt fine-grained block isolation and that 4-bit weights limit smoothing benefits.
  • Efficiency Analysis: For S ≤512, ARCQuant adds only marginal kernel overhead over NVFP4 while retaining a speed advantage over W4A8 and MXFP8.End-to-end Qwen2.5-7B latency increases by 4.9%, and the fused quantization kernel contributes minimal cost.
  • End-to-End Inference: ARCQuant achieves 2.0×–2.5× speedup for Qwen2.5-7B on PRO 6000 and 3.5× for Llama 3.1-8B on RTX 5090, with 1.5×–2.8× lower memory use.Compared with uncompensated NVFP4, latency increases by only 3%–9%.
  • Generation Throughput: In vLLM decoding, ARCQuant reaches 1.96× speedup at sequence length 1024 and 2.08× at sequence length 2048 over FP16.It also substantially outperforms the FP8 reference in both settings.

5 Conclusion

ARCQuant addresses NVFP4’s outlier sensitivity with augmented residual channels while preserving unified-precision hardware compatibility. Its theory and experiments support accurate, efficient low-precision inference, with extensions planned for sub-4-bit formats and larger architectures.

  • Conclusion: ARCQuant uses augmented residual channels to address outlier-heavy activations without compromising NVFP4 block isolation or hardware uniformity.Its dual-stage mechanism combines primary high-magnitude capture with residual recovery through unified-precision GEMM operations.
  • Conclusion: The dual-stage mechanism has a worst-case error bound comparable to MXFP8 and enables practical speedups through custom fused kernels.The conclusion characterizes this as trading minimal compute dimensions for higher fidelity.
  • Conclusion: The paper identifies sub-4-bit formats and larger-scale architectures as directions for future work.

Limitations

ARCQuant’s limitations concern weight-quantization scope, hardware availability, and reliance on offline calibration data.

  • Integration with Advanced Weight Quantization: The current framework focuses on activation outliers and uses standard RTN for weights, leaving advanced weight quantization for future integration.GPTQ or AWQ could further improve fidelity, particularly below 4-bit weight compression.
  • Hardware Dependency: Practical throughput benefits depend on NVIDIA Blackwell support for native NVFP4 and unified-precision Tensor Core instructions.On legacy architectures without block-scaled acceleration, ARCQuant mainly simulates future performance rather than enabling immediate deployment.
  • Dependence on Offline Calibration: ARCQuant determines channel reordering indices and outlier count S offline from calibration data, assuming relatively stable outlier distributions during inference.This static choice may limit adaptability to extreme out-of-distribution inputs compared with slower dynamic strategies.

B.1 Experiment Setup

The experiments report quantization overhead using WikiText2 calibration and include a mathematical-reasoning figure for Qwen2.5-Math-7B-Instruct.

  • Quantization Overhead: Table 5 reports calibration latency, quantization time, and model memory on an RTX PRO 6000 GPU.The table summarizes quantization overhead and efficiency.
  • Calibration: Calibration uses 128 WikiText2 samples, each with sequence length 2048, followed by weight quantization using derived reordering indices and layer-specific outlier counts S.Table 5 reports the resulting quantized model size and total quantization time.
  • Math Evaluation: Figure 9 evaluates Qwen2.5-Math-7B-Instruct on the GSM8K and CMATH benchmarks.The figure covers two mathematical-reasoning datasets.

B.2 Results Breakdown

ARCQuant is evaluated across calibration settings, quantization formats, mathematical reasoning, inference efficiency, and larger dense and mixture-of-experts models.

  • Calibration Robustness: ARCQuant’s calibration robustness is assessed with C4, WikiText2, and HumanEval datasets matched for sample size, using zero-shot accuracy and WikiText2 perplexity.The study reports an ablation across three calibration datasets.
  • INT4 and MXFP4 Evaluation: ARCQuant consistently outperforms RTN across five standard benchmarks in both INT4 and MXFP4 formats.The benchmarks are ARC-Challenge, HellaSwag, Lambada, PIQA, and Winogrande.
  • Math Evaluation: ARCQuant retains over 99% of FP16 baseline accuracy on GSM8K and CMATH with Qwen2.5-Math-7B-Instruct.This evaluates domain-specific mathematical reasoning.
  • End-to-End Inference: End-to-end inference reports latency and peak memory across batch sizes and sequence lengths of 512, 1024, and 2048 on RTX PRO 6000 and RTX 5090 GPUs.The evaluation provides a detailed prefill-performance breakdown.
  • Larger-Scale and MoE Model Evaluation: ARCQuant consistently outperforms the NVFP4+RTN baseline on Llama 3.1-70B and Mixtral 8x7B-Instruct.The evaluation includes zero-shot accuracy, perplexity, and MMLU results for a larger dense model and a sparse mixture-of-experts model.

C Reproducibility Statement

The project supports reproduction with public code, calibration artifacts, datasets, checkpoints, fixed calibration settings, and documented hardware and baseline procedures.

  • Code and Data: Source code compatible with PyTorch 2.9.0 and CUDA 12.8 is available through the ARCQuant GitHub repository.Calibration artifacts are also provided in the supplementary material.
  • Code and Data: The experiments use publicly available datasets and official Hugging Face model checkpoints without modification.This supports reproduction from standard public resources.
  • Implementation Details: Calibration randomly selects 128 WikiText2 segments of length 2048, sets τ = 2−3M, and fixes the random seed to 0.These settings define the reported deterministic calibration procedure.
  • Baselines and Hardware: Baselines are reproduced from official repositories, and efficiency benchmarks run on NVIDIA RTX 5090 and RTX PRO 6000 GPUs.The baseline comparison includes methods such as Atom and FlatQuant.

D Kernel Implementation Details

ARCQuant implements residual compensation through a hardware-friendly interleaved channel layout while preserving standard GEMM execution. Its quantization logic remains decoupled from computation, supporting adaptation to other block-scaled formats.

  • Interleaved Channel Layout: ARCQuant interleaves each 16-channel primary block with its corresponding 16-channel residual block in physical memory.The layout matches NVFP4's block size and avoids the latency of strided global memory access.
  • Interleaved Channel Layout: The interleaved layout enables the fused quantization kernel to compute primary and residual quantizations locally.
  • Compatibility and Future Work: ARCQuant maps error compensation into the input data space, allowing compatibility with standard high-performance GEMM kernels such as cuBLAS and CUTLASS.
  • Compatibility and Future Work: Adapting ARCQuant to another emerging format requires updating the fused quantization kernel for that format's encoding while preserving downstream GEMM efficiency.The paper states that broader high-throughput serving support and deployment validation remain future work.
Loading 2601.07475v2…