PaperlayerView exact revision on arXiv

Source-linked AI summary

SoloQ: Calibration-Free Quantization for Diffusion Language Models

Donghyun Lee, Arkapravo Ghosh, Varun Manjunath, Bumjoon Kyle Rhee, Hyunho Kook, Shiting Xiao, Youngeun Kim, Priyadarshini Panda

arXiv:2610.07121v1cs.LG

TL;DR

dLLM inference repeatedly incurs compute and memory costs, and its activation distributions shift across masking states and denoising steps, complicating calibration-based quantization. SOLOQ uses a predictable rotated representation for calibration-free low-bit quantization, and across full-sequence and block-diffusion dLLMs it retains robust 4-bit accuracy while reducing memory and inference time with NVFP4.

  • Problem

    dLLMs repeatedly denoise tokens, increasing compute and memory demands, while shifting activation distributions make calibration-based quantization sensitive to masking states, denoising steps, and calibration data.

    Source: §1 · pp. 1–2

  • Method

    SOLOQ combines K-RPBH rotation and scale-only bias correction to quantize predictable rotated representations without calibration, with support for codebooks, NVFP4, and persistent KV states.

    Source: §1 · p. 2

  • Results

    Across full-sequence and block-diffusion dLLMs, SOLOQ achieves robust W4A4 accuracy and competitive or better benchmark performance; NVFP4 reduces peak memory up to 2.61× and accelerates end-to-end inference up to 2.24×.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    Predictable representation shaping provides a general foundation for efficient low-bit inference across full-sequence and block-diffusion language models.

    Source: §7 · p. 10

    TruthfulQA results for DLLMQuant and STaR-Quant are not reproducible because they are not open-sourced; their FP is 49.76 versus 43.96 in this setup.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Diffusion large language models dLLMs) have emerged as a promising alternative to autoregressive language models through bidirectional diffusion-based token generation. However, their growing model sizes and high inference costs make efficient deployment challenging: full-sequence denoising repeatedly invokes compute-intensive forward passes, while block-diffusion models additionally introduce a memory-intensive KV-cache. Low-bit weight-activation quantization is therefore attractive, yet existing dLLM post-training quantization methods rely on calibration data despite activation distributions shifting across masking states and denoising steps. We present SoloQ, a calibration-free quantization framework that maps weights and activations into a normalized rotated basis with a predictable marginal distribution, enabling data-independent quantization. SoloQ combines a structured K-RPBH rotation with a lightweight rescaling correction for calibration-free quantization. Its predictable post-rotation distribution supports both distribution-matched codebooks and hardware-native NVFP4. For block-diffusion models, SoloQ further applies commit-time KV-cache quantization to compress persistent states without perturbing the actively denoised block. Across full-sequence dLLMs (LLaDA and Dream) and block-diffusion dLLMs(Fast-dLLM v2 and Nemotron-Labs-Diffusion), SoloQ retains accuracy under 4-bit quantization and outperforms calibration-based baselines on knowledge- and reasoning-intensive benchmarks. With NVFP4, SoloQ reduces peak memory by up to 2.61X and accelerates end-to-end inference by up to 2.24X.

1 INTRODUCTION

dLLM inference repeatedly incurs compute and memory costs, while activation distributions shift across masking states and denoising steps, challenging calibration-based quantization. SOLOQ uses a predictable rotated representation for calibration-free quantization and extends it to persistent KV states in block-diffusion models.

1 INTRODUCTION

  • Repeated denoising and longer contexts raise compute and memory costs, as activation and KV-cache contributions grow and attention latency increases with sequence length.These costs make weight-only compression insufficient; low-bit quantization of weights, activations, and KV-cache can reduce computation and memory.Source: §1 · pp. 1–2
  • Activation distributions and quantization sensitivity vary across masking states and denoising steps, while calibration choices may not generalize across tasks.Changing calibration data from WinoGrande to MMLU alters accuracy across benchmarks.Source: §1 · p. 2
  • SOLOQ transforms weights and activations into a normalized rotated space with predictable marginals, using K-RPBH rotation and lightweight scale-only bias correction.The transformed representation supports data-independent quantization shared across diffusion states without calibration.Source: §1 · p. 2
  • SOLOQ supports distribution-matched codebooks and NVFP4, and extends quantization to persistent KV states in block-diffusion models.Its commit-time KV-cache quantization complements weight and activation quantization.Source: §1 · pp. 2–3
  • Across full-sequence and block-diffusion dLLMs, SOLOQ retains strong 4-bit accuracy while reducing peak memory up to 2.61× and accelerating end-to-end inference up to 2.24×.The framework supports both distribution-matched codebooks and hardware-native NVFP4.Source: §1 · pp. 2–3

2 RELATED WORK

Diffusion language models use iterative denoising, with block-based approaches reusing committed context. Related quantization work spans data-independent rotations and calibration-based PTQ for dLLMs; SOLOQ extends calibration-free quantization to persistent KV states in block-diffusion models.

2.1 DIFFUSION LARGE LANGUAGE MODELS

  • Full-sequence models such as LLaDA and Dream use iterative denoising, while block and hybrid approaches generate multiple tokens per block and reuse prior context.Recent systems also explore combinations of autoregressive, diffusion, and self-speculative decoding.Source: §2.1 · p. 3

2.2 CALIBRATION-FREE AND DATA-INDEPENDENT QUANTIZATION

  • Data-independent quantization methods use known distributions or rotations; SOLOQ extends this approach to arbitrary transformer dimensions and persistent KV states in block-diffusion models.Related approaches include TurboQuant, PolarQuant, OrbitQuant, ConvRot, and DVD-Quant.Source: §2.2 · p. 3

2.3 POST-TRAINING QUANTIZATION FOR DLLMS

  • Existing dLLM PTQ methods remain calibration-based and focus on full-sequence models; SOLOQ supports calibration-free KV-cache quantization for block-diffusion models as well as full-sequence dLLMs.Prior approaches address timestep, masking, activation-disparity, or token-commitment-state variation.Source: §2.3 · p. 3

3 PRELIMINARY

Diffusion language models progressively unmask tokens using bidirectional context, either across a full sequence or one block at a time. Under random rotation, normalized representations have predictable coordinate marginals that can support analytic quantizer design without calibration.

3.1 DIFFUSION LANGUAGE MODELS

  • At each diffusion timestep, tokens are independently preserved with probability α_t or replaced by a mask token, and the model predicts masked tokens from bidirectional context.Full-sequence dLLMs process the whole sequence repeatedly; block-diffusion models denoise one block at a time and reuse committed KV states.Source: §3.1 · pp. 3–4
  • The SOLOQ overview combines K-RPBH mixing with distribution-matched codebooks or NVFP4, and uses commit-time KV quantization while keeping the active denoising block in full precision.K-RPBH combines permutation, block-Hadamard, and cross-block mixing.Source: Figure 2 · p. 4

3.2 PREDICTABLE MARGINALS UNDER RANDOM ROTATION

  • Under a Haar-random rotation, a normalized vector is uniform on the unit sphere, giving each coordinate a fixed spherical marginal and a Gaussian-like approximation that improves as dimension grows.This predictable distribution enables analytic quantizer design without calibration.Source: §3.2 · p. 4
  • RPBH provides efficient O(d log h) mixing but can leave inter-block energy imbalance as the number of blocks grows because cross-block mixing is limited.The passage describes this as a limitation of the blockwise structure.Source: §3.2 · p. 4

4 METHOD

SOLOQ uses K-RPBH rotation to create predictable representations for calibration-free weight and activation quantization, with a lightweight scale-only correction. For block-diffusion models, it quantizes persistent KV states when blocks are committed, leaving the active denoising block in full precision.

4.1 OVERVIEW

  • SOLOQ combines K-RPBH rotation, two calibration-free quantization paths, and scale-only correction; block-diffusion models also quantize persistent KV states at commit time.The overview presents these as components of a shared design for full-sequence and block-diffusion models.Source: §4.1 · p. 4

4.2 K-RPBH ROTATION

  • K-RPBH adds a Haar-random cross-block mixer to RPBH, redistributing residual block statistics and bringing rotated activation marginals closer to Gaussian with lower quantization error.The rotation uses randomized Hadamard transforms within blocks; the cross-block stage mixes corresponding coordinates across blocks.Source: §4.2 · p. 5, Figure 3 · p. 6

4.3 CALIBRATION-FREE WEIGHT–ACTIVATION QUANTIZATION

  • SOLOQ rotates weights and activations into a shared K-RPBH basis and supports distribution-matched codebooks or hardware-native NVFP4 without calibration data.Weights are rotated offline and activations online; the codebook path uses E8P for weights and Lloyd–Max for activations.Source: §4.3 · pp. 5–6
  • NVFP4 quantizes the rotated representation using the fixed E2M1 format, with scales derived directly from the tensor and 16-coordinate groups.The passage states that this path needs no calibration data, pre-collected activation statistics, or learned parameters.Source: §4.3 · p. 6

4.4 SCALE-ONLY BIAS CORRECTION

  • Scale-only bias correction restores the quantized vector’s projection along the original vector without auxiliary payload or extra quantized bits.The correction is absorbed into the existing per-token activation or per-row weight scale, leaving the low-bit linear operation unchanged.Source: §4.4 · p. 6

4.5 BLOCK-DIFFUSION KV-CACHE QUANTIZATION

  • For block-diffusion models, SOLOQ quantizes KV states only when a block is committed, keeping the actively denoised block in full precision.Keys use KIVI-style per-channel asymmetric quantization; values use the SOLOQ activation path with K-RPBH and Lloyd–Max or NVFP4.Source: §4.5 · p. 7

5 EXPERIMENTS

SOLOQ retains strong 4-bit accuracy across full-sequence and block-diffusion dLLMs, while reducing inference memory and latency. Evaluations span five models and compare against quantized baselines and corresponding BF16 baselines.

5.1 EXPERIMENTAL SETUP

  • Evaluations cover three full-sequence and two block-diffusion models, using each model’s official evaluation stack and GPU-based accuracy testing.Full-sequence baselines include RTN, AWQ, QuaRot, GPTQ, DLLMQuant, and STaR-Quant; block-diffusion comparisons include RTN, AWQ, and QuaRot.Source: §5.1 · pp. 7–8

5.2 FULL-SEQUENCE DIFFUSION LANGUAGE MODELS

  • SOLOQ-C and SOLOQ-N outperform quantized baselines on full-sequence dLLMs, with reported average-accuracy gains of 3.45 points on LLaDA-Base and 4.17 points on LLaDA-1.5.On Dream-7B, SOLOQ-N achieves 62.53 average accuracy versus 61.90 for DLLMQuant++.Source: §5.2 · p. 8

5.3 BLOCK-DIFFUSION LANGUAGE MODELS

  • SOLOQ maintains strong accuracy under W4A4 and W4A4KV4 across Fast-dLLM v2 and Nemotron-Labs-Diffusion, where RTN and AWQ collapse under W4A4.QuaRot is not applicable to Fast-dLLM v2 because its rotation cannot support the model dimensions.Source: §5.3 · p. 8

5.4 MEMORY AND LATENCY EFFICIENCY

  • SOLOQ-N reduces peak memory by 2.31×–2.61× and achieves 1.94×–2.24× end-to-end speedups over BF16 across evaluated workloads.For Nemotron, the speed comparison uses the same BF16-recorded sequence of 75 forward passes because its EOS trajectory changes after quantization.Source: Figure 4 · p. 8, §5.4 · p. 9

6 ANALYSIS

The analysis finds that K-RPBH improves the accuracy–efficiency trade-off over alternative rotations, with its permutation and cross-block mixer providing complementary distribution-matching benefits. Scale-only bias correction improves quantization fidelity, and SOLOQ variants preserve accuracy under 2-bit KV-cache quantization.

6.1 ROTATION STRUCTURE COMPARISON

  • K-RPBH reduces rotation latency from 98.135 ms to 5.779 ms on Fast-dLLM v2 and from 0.559 ms to 0.048 ms on LLaDA-1.5, while retaining comparable accuracy to dense Haar.Compared with RPBH, K-RPBH consistently improves downstream accuracy.Source: §6.1 · p. 9

6.2 K-RPBH COMPONENT ABLATION

  • Combining permutation and cross-block mixing lowers KL divergence to 0.0041 and block-energy imbalance to 22.12%, outperforming either component alone on both measures.Permutation alone yields 0.0468 KL and 45.64% imbalance; cross-block mixing alone yields 0.0068 KL and 36.24% imbalance.Source: §6.2 · p. 9

6.3 EFFECT OF SCALE-ONLY BIAS CORRECTION

  • Scale-only bias correction reduces KL divergence across LLaDA-1.5 masking ratios and improves HellaSwag and GPQA accuracy for both SOLOQ variants.HellaSwag improves from 72.49/73.71 to 73.01/73.86 on LLaDA-1.5; GPQA improves from 30.03/31.92 to 30.58/33.71 on Fast-dLLM v2.Source: §6.3 · p. 9

6.4 KV-CACHE QUANTIZATION COMPARISON

  • At approximately 3 bits-per-entry, SOLOQ-C and SOLOQ-N achieve average accuracies of 53.31 and 53.53, outperforming KIVI, OScar, and TurboQuant.SOLOQ-C performs best on GSM8K, while SOLOQ-N achieves the highest IFEval accuracy; weights and activations remain FP16.Source: §6.4 · pp. 9–10

7 CONCLUSION

SOLOQ uses predictable rotated representations to quantize diffusion language models without calibration. Its results support robust low-bit quantization across full-sequence and block-diffusion models.

7 CONCLUSION

  • SOLOQ transforms weights and activations into predictable rotated representations for data-independent low-bit quantization without calibration samples or pre-collected activation statistics.The framework combines distribution-matched codebooks or hardware-native NVFP4 with commit-time KV-cache quantization for block-diffusion models.Source: §7 · p. 10
  • Across full-sequence and block-diffusion dLLMs, SOLOQ achieves robust W4A4 quantization and remains competitive with or outperforms calibration-based baselines across language benchmarks.Source: §7 · p. 10
  • The paper presents predictable representation shaping as a general foundation for efficient low-bit dLLM inference.Source: §7 · p. 10

A PROOF OF PROPOSITION 1

The proof passage describes K-RPBH’s cross-block mixing and the averaging argument involving its randomized orthogonal mixer. The algorithm passage specifies offline weight rotation and commit-time KV-cache quantization while keeping the active block in full precision.

A PROOF OF PROPOSITION 1

  • K-RPBH applies Qk ⊗ Ih to mix the same within-block coordinate across the k blocks.The proof states that, conditioned on P and Qk, variables are independent because the sign matrices are sampled independently.Source: A PROOF OF PROPOSITION 1 · p. 16

B ALGORITHM

  • The algorithm constructs K-RPBH rotations offline and applies them to target-layer weights before returning the quantized transformer.The algorithm passage defines SOLOQ weight and activation quantization operators, including scaling, quantization, and scale-only bias correction.Source: B ALGORITHM · p. 17
  • For block-diffusion models, the active KV block remains full precision until commit, when quantized keys and values are appended to the persistent cache.Source: B ALGORITHM · p. 17

C EVALUATION DETAILS

The evaluation retains each model’s official stack rather than imposing a unified harness, while matching full-precision and quantized settings within each model. It covers full-sequence and block-diffusion models with model-specific benchmarks, prompting, generation, and scoring protocols.

C EVALUATION DETAILS

  • The study preserves official model evaluation stacks because identically named benchmarks can differ in prompting, inference, and grading, while matching FP and quantized settings within each model.Prompts, few-shot settings, batch sizes, generation parameters, and graders are identical between FP and quantized runs for each model.Source: C EVALUATION DETAILS · p. 17

C.1 LLADA-8B-BASE

  • LLaDA-8B-Base uses its official harness, with 128 masking samples for most likelihood tasks, task-specific CFG and shot settings, and strict-match GSM8K and pass@1 HumanEval scoring.MMLU and C-EVAL use one masking sample; GSM8K and HumanEval use generation length, diffusion steps, and block length of 1,024.Source: §C.1 · p. 17

C.2 LLADA-1.5

  • LLaDA-1.5 uses OpenCompass for its official generation suite, while four additional likelihood-protocol benchmarks are supplementary rather than part of that suite.Generation tasks use greedy decoding and low-confidence remasking, with task-specific settings and scoring.Source: §C.2 · p. 18

C.3 DREAM-V0-BASE-7B

  • Dream-v0-Base-7B uses its native evaluator, including a Monte Carlo ELBO estimator, autoregressive logit shift, and BOS token, with 128 samples for likelihood tasks.TruthfulQA and C-EVAL are used only for controlled FP-to-quantized comparisons.Source: §C.3 · p. 18

C.4 FAST-DLLM V2

  • Fast-dLLM v2 follows the official NVLabs adapter, using block size 32 and task-specific likelihood or generation scoring across its benchmark suite.HumanEval and MBPP use EvalPlus, reporting base and Plus pass@1 after code sanitization.Source: §C.4 · p. 18

C.5 NEMOTRON-LABS-DIFFUSION-8B

  • Nemotron-Labs-Diffusion uses its official inference implementation and NeMo-Skills at a controlled operating point, not an exact reconstruction of the paper’s main table.The paper’s configuration does not specify block length or EOS policy for that table; cross-model evaluation includes GSM8K, MATH, IFEval, MMLU, GPQA, HumanEval, and MBPP.Source: §C.5 · p. 18

D ADDITIONAL ANALYSIS

Additional analyses compare SOLOQ with calibration-based and uniform-INT4 approaches, then examine the efficiency–accuracy trade-off of folding rotations. Results favor SOLOQ in the reported comparisons, while folding’s accuracy impact varies by model.

D.1 COMPARISONS WITH FAIR-CALIB

  • Under W4A4, SOLOQ achieves higher average accuracy than FAIR-Calib on LLaDA-Base and Dream-7B across five shared benchmarks.The comparison uses TruthfulQA, WinoGrande, PIQA, MMLU, and GSM8K; Table 6 reports task accuracies and averages.Source: §D.1 · p. 18, Table 6 · p. 19

D.2 UNIFORM INT4 QUANTIZATION

  • SOLOQ-I averages 56.98 on LLaDA-Base, 67.44 on LLaDA-1.5, and 55.32 on Fast-dLLM v2, remaining competitive with specialized quantizers.SOLOQ-I replaces codebook and NVFP4 quantizers with uniform INT4 while retaining K-RPBH rotation and the rest of the pipeline.Source: §D.2 · p. 19, Table 7 · p. 19, Table 8 · p. 20
  • SOLOQ’s predictable K-RPBH representation also supports conventional uniform INT4 quantization without calibration, with modest degradation relative to SOLOQ-C/N.The reported uniform-INT4 comparison covers three architectures.Source: §D.2 · p. 19

D.3 FOLDED ROTATION FOR EFFICIENT DEPLOYMENT

  • Static RMSNorm gain and residual-stream rotation can be folded into adjacent weights offline, but quantized folded and unfolded operations are not generally equivalent.Condition-dependent AdaLN scales and shifts vary at runtime, so residual-stream rotation cannot generally be absorbed into fixed adjacent weights; folding yields a relatively modest accuracy drop on LLaDA and a substantially larger one on Fast-dLLM v2.Source: §D.3 · p. 20, Table 9 · p. 20, Table 10 · p. 21

D.4 ATTENTION SINK IN DLLM

The analyses examine attention sinks, accelerator performance, and qualitative generations under quantization. They report that sink protection matters more at ultra-low KV precision, while selected outputs illustrate both preserved behavior and changed token-level decisions.

D.4 ATTENTION SINK IN DLLM

  • Fast-dLLM v2 has an attention sink at position 2 and Nemotron-Labs-Diffusion at position 0; preserving sinks helps under W4A4KV2 but has little effect under W4A4KV4.The accelerator analysis reports a 1.94× end-to-end speedup and 2.70× higher energy efficiency for NVFP4 versus BF16 on LLaDA-Base-8B.Source: §D.4 · pp. 21, 23, Figure 6 · p. 21, Table 12 · p. 22

F QUALITATIVE GENERATION EXAMPLES

  • The selected examples cover mathematical reasoning, code generation, and general instruction following, but quantitative claims rely on complete benchmark results.Examples use identical prompts and model-specific deterministic decoding configurations.Source: F QUALITATIVE GENERATION EXAMPLES · p. 23

Code generation — Correct Outputs

Mathematical reasoning — Incorrect Outputs

  • In the house-flipping example, FP16 and SOLOQ-C are marked incorrect, while SOLOQ-N computes a $70,000 profit.SOLOQ-C reports $80,000; SOLOQ-N calculates the $200,000 new value less $130,000 in total costs.Source: Mathematical reasoning — Incorrect… · p. 24

Mathematical reasoning — Divergent outputs

SOLOQ-C

  • SOLOQ-C’s generated explanation describes compression, efficient architectures, and hardware-aware training, with trade-offs involving speed, cost, and answer quality.Its response concludes that speed and memory efficiency must be balanced against answer quality.Source: SOLOQ-C · p. 25

SOLOQ-N

  • SOLOQ-N’s explanation presents compression, distillation, and hardware optimization, using a recipe analogy and noting possible quality, resource, and accuracy trade-offs.The sample is a qualitative generation, not a benchmark result.Source: SOLOQ-N · p. 25

General explanation — length-constrained instruction following

  • The general-explanation outputs can retain similar overall structure while varying in wording, examples, and response length across precisions.The paper notes that low-bit perturbations can sometimes correct an FP16 failure or introduce a new error.Source: SOLOQ-N · pp. 25–26

G LIMITATIONS

SOLOQ has deployment trade-offs involving online rotation, folding, quantizer kernels, and hardware support. These constraints may affect accuracy or the realized latency benefit.

G LIMITATIONS

  • Online activation rotations add inference overhead, while folding them into weights can reduce cost but may degrade low-bit accuracy, particularly for block-diffusion models.The limitation passage describes folding as an efficiency–accuracy trade-off.Source: G LIMITATIONS · p. 26
  • SOLOQ-C’s non-uniform codebook centroids use lossless INT8 representation, so its low-bit path relies on 8-bit kernels rather than native 4-bit Tensor Core execution.The passage associates this kernel path with lower speedups than hardware-native NVFP4.Source: G LIMITATIONS · p. 26
  • SOLOQ-N’s full latency benefit depends on native NVFP4 hardware support, and practical acceleration may be reduced on unsupported devices..Source: G LIMITATIONS · p. 26
Loading 2610.07121v1…