PaperlayerView exact revision on arXiv

Source-linked AI summary

ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization

He-Yen Hsieh, H. T. Kung

arXiv:2608.11045v1cs.LGcs.CL

TL;DR

Standard RTN leaves midpoint-ambiguous weight assignments unresolved, despite their matrix-level importance. ReRound uses diffusion-reconstructed weights and spectral selection to guide rounding, consistently improving 3- and 4-bit weight-only PTQ without calibration data.

  • Problem

    Standard RTN leaves midpoint-ambiguous rounding assignments unresolved, while prior work does not identify which alternative assignments should be chosen.

  • Method

    ReRound trains conditional diffusion models for weight reconstruction, guides midpoint rounding with a tolerance metric, and selects candidates by matching leading singular values.

  • Results

    ReRound consistently improves 3- and 4-bit weight-only PTQ across diverse LLMs without activation or text calibration data.

  • Takeaways & Limitations

    Structural patterns in pretrained weights can serve as an effective prior for resolving ambiguous rounding decisions while preserving the low-bit inference procedure.

  • Takeaways & Limitations

    ReRound requires a separate diffusion model for each pretrained LLM, and its weight-space spectral selection may not identify the best candidate for every downstream task.

Abstract

from arXiv · show

ReRound (Reconstructive Rounding) is a post-training quantization method that addresses the midpoint ambiguity inherent in standard round-to-nearest (RTN) schemes when quantizing weights near the centers of quantization intervals. Starting from a pretrained LLM, ReRound trains a conditional diffusion model to produce continuous reconstructions of low-bit weights for the LLM. These reconstructed weights act as a guidance signal to disambiguate the rounding direction of weights located close to interval midpoints. To integrate this reconstruction-guided rounding with conventional RTN, ReRound introduces a tolerance metric measuring how far the quantized weight (not the final quantized integer) is away from the midpoint: quantized weights within a tolerance region around midpoints are quantized using diffusion-based reconstructions, whereas weights closer to quantization boundaries are quantized with RTN. By sweeping the tolerance parameter, ReRound generates multiple candidate quantized integer weight matrices and selects the de-quantized weight matrix candidate whose leading singular values most closely match those of the original full-precision weights. This selected candidate determines the tolerance parameter ReRound uses. ReRound is particularly effective for smaller LLMs. Across a range of such models, it consistently outperforms standard RTN for 3-bit and 4-bit weight quantization. ReRound achieves superior accuracy compared to an extensive set of calibration-free methods, remains competitive with calibration-dependent approaches, and operates entirely offline, introducing no additional overhead during low-bit inference. The ReRound strategy represents a new approach for low-bit quantization. The method applies to AI models beyond LLMs. This paper focuses on its applications to small LLMs.

1 Introduction

ReRound addresses midpoint ambiguity in RTN by using diffusion-reconstructed weights to guide selected floor-or-ceiling assignments. A position-dependent tolerance and weight-only spectral criterion determine which changes to retain without calibration data or inference-time overhead.

  • Motivation: RTN becomes ambiguous near quantization midpoints because rounding up and down incur nearly identical scalar error, although matrix-level assignment changes can alter a layer’s rounding pattern.This creates an opportunity to improve rounding by selecting coordinated matrix-level alternatives rather than relying solely on scalar distance.
  • Method: ReRound trains a conditional diffusion model on pretrained weight regions and uses reconstructed continuous weights to guide midpoint-ambiguous floor-or-ceiling decisions.The reconstruction reflects the LLM’s weight distribution and can favor an assignment different from RTN when the reconstructed weight is sufficiently close to the original quantized weight.
  • Method: A position-dependent tolerance metric accepts reconstruction-guided changes near midpoints while preserving RTN more often when quantized weights lie closer to adjacent quantized integers.The tolerance depends on each quantized weight’s distance from the midpoint and reflects the confidence margin of the original RTN decision.
  • Candidate selection: ReRound selects among complete-matrix rounding candidates using the candidate whose de-quantized leading singular values most closely match those of the original full-precision matrix.This weight-only spectral criterion favors candidates that preserve the dominant structure of the original layer.
  • Practical properties: ReRound requires no activation or text calibration data, preserves the low-bit representation and inference procedure, and performs diffusion reconstruction and spectral selection offline.The final model differs from RTN only in selected floor-or-ceiling assignments; scales, zero-points, group size, and quantized layers remain unchanged.

2 Related Work

Prior work improves low-bit LLM quantization through parameter optimization, weight or activation transformations, and learned rounding assignments. Diffusion methods provide priors for reconstruction, while ReRound learns its diffusion prior from the pretrained LLM’s own weights and uses reconstructed weights.

  • LLM post-training quantization: LLM PTQ improves low-bit accuracy by optimizing quantization parameters or transforming weights and activations before quantization.GPTQ uses approximate second-order information, AWQ protects activation-salient channels, and OmniQuant learns clipping ranges and equivalent transformations from calibration data.
  • Learned rounding: With fixed quantization parameters, lower-versus-upper integer assignments can affect model accuracy, motivating learned rounding methods.AdaRound learns assignments through local output reconstruction, FlexRound uses element-wise divisive parameters, and SignRound jointly optimizes rounding and clipping with signed gradients.
  • Diffusion priors: Diffusion models serve as priors for inverse problems, including noisy measurements, quantized signal reconstruction, and image dequantization.DPS addresses noisy and nonlinear measurements; QCS-SGM and SIM-DMIS adapt diffusion priors to quantized reconstruction; Vavilala et al. reconstruct continuous colors.
  • Weight-space diffusion: Weight-space diffusion research generates, samples, edits, and inverts neural-network parameters, whereas ReRound learns its diffusion prior from the pretrained LLM’s own weights.Neural Network Diffusion and D2NWG generate parameters, while weights2weights supports sampling, editing, and inversion of customized diffusion models.

3 Method: ReRound

ReRound trains a conditional diffusion model to reconstruct local weight patches, then uses those reconstructions to selectively alter RTN assignments near quantization midpoints. It generates tolerance-dependent candidates and selects the one whose leading singular values best preserve the full-precision weight matrix.

  • Conditional reconstruction: ReRound trains one conditional diffusion model per pretrained LLM on local patches, learning a prior from the model’s own full-precision weights.The model jointly reconstructs each patch from its low-bit observation.
  • Conditional reconstruction: During training, stochastic low-bit conditions are paired with full-precision patches, while conditional diffusion models pθ(P | Plow).Only the condition is quantized; the reconstruction target remains the full-precision patch.
  • Reconstruction-guided rounding: Each tolerance parameter τ produces a candidate quantized integer matrix Qτ while keeping all other quantization settings fixed.The reconstructed matrix guides rounding only and is not deployed as the final weight matrix.
  • Reconstruction-guided rounding: Reconstructed weights can switch RTN only to the opposite adjacent integer, with proposals accepted when reconstruction deviation falls within a position-dependent tolerance.The tolerance is largest near midpoints and decreases toward adjacent quantized integers.
  • Spectral candidate selection: ReRound selects Qτ⋆ whose dequantized matrix has leading singular values closest to those of the full-precision matrix, without requiring activation data.The comparison uses k = min{m, 128, max(32, ⌊m/16⌋)} leading singular values.

4 Experiments

Experiments show that ReRound consistently improves matched group-wise RTN at 3- and 4-bit quantization across small LLMs, including gains over calibration-based methods. The improvements arise from changing selected rounding assignments while preserving RTN’s quantization parameters and inference procedure, with all added computation performed offline.

  • Calibration-free comparisons: ReRound improves matched group-wise RTN on every evaluated model, gaining 0.1–0.9 points at 3 bits and 0.2–1.3 points at 4 bits.At 3 bits, ReRound achieves the highest four-task average on every model; at 4 bits, it achieves or matches the best average on four of five models.
  • Calibration-based comparisons: 0.4 and 0.2 points are ReRound’s margins over the strongest calibration-based result on Gemma 2 2B and Gemma 3 1B, respectively.ReRound achieves four-task averages of 67.4 and 59.3, compared with SignRound’s 67.0 and 59.1, without activation or text calibration data.
  • Component ablation: 1.3 points is ReRound’s improvement over group-wise RTN in the OLMo 2 1B ablation, where full ReRound has the highest four-task average at both bit widths.The ablation replaces reconstruction guidance, position-dependent tolerance, or spectral selection one component at a time.
  • Transfer to SINQ parameters: 0.5 points is ReRound’s 4-bit accuracy-average improvement over reproduced RTN with SINQ parameters, while average perplexity falls from 18.53 to 18.46.At 3 bits, the accuracy average rises from 57.1 to 57.3 and average perplexity falls from 24.26 to 24.06.
  • Additional models: ReRound improves group-wise RTN on Llama 3.2 1B, Pythia 1.4B, and Phi-2 2.7B at both bit widths, with gains of 0.2–1.3 points at 4 bits and 0.1–1.6 points at 3 bits.These models extend the positive trend beyond the five-model main comparison.
  • Offline cost: 42 to 124 seconds is the average ReRound PTQ cost across W3 and W4 after shared diffusion reconstruction, and all additional computation occurs before deployment.The resulting quantized model uses the same low-bit representation and inference procedure as RTN.

5 Discussion and Limitations

ReRound can refine quantization assignments while retaining fixed scales and zero-points, but its candidate space and applicability remain limited. The method also incurs model-specific diffusion costs, and its selection and reconstruction strategies may miss task-relevant or long-range dependencies.

  • Quantization-parameter limitations: ReRound revises RTN assignments while keeping scales and zero-points fixed, allowing it to refine parameters from other PTQ methods.Its candidate set remains constrained by these fixed parameters, motivating joint optimization of quantization parameters and reconstruction-guided rounding.
  • Scope: Although ReRound may extend beyond LLMs, the evaluation covers only small LLMs.
  • Computational cost: ReRound requires a separate diffusion model for each pretrained LLM, creating model-specific offline cost that remains expensive when scaling to many models.Reusing reconstructed weights across bit widths can reduce this cost.
  • Selection limitations: Spectral selection operates only in weight space and may not identify the best candidate for every downstream task.
  • Reconstruction limitations: Patch-based reconstruction may miss long-range and cross-layer dependencies.

6 Conclusion

ReRound is a calibration-free framework that uses diffusion priors from pretrained weights to resolve midpoint-ambiguous RTN decisions and selects candidates through spectral preservation. It improves 3- and 4-bit weight-only PTQ without calibration data while preserving the quantization parameters, representation, and inference procedure.

  • Method: ReRound revisits midpoint-ambiguous RTN decisions after quantization parameters are fixed, using a diffusion prior learned from the model’s own weights.The prior guides floor-or-ceiling changes to selected quantized weights.
  • Results: ReRound selects candidate quantized matrices through spectral preservation and consistently improves 3- and 4-bit weight-only PTQ across diverse LLMs.These improvements require neither activation nor text calibration data.
  • Implications: Because ReRound changes only selected quantized integer assignments, its quantization parameters, low-bit representation, and inference procedure remain unchanged.The results indicate that structural patterns in pretrained weights can serve as an effective prior for resolving ambiguous rounding decisions.

A Quantization Notation

The notation defines a full-precision weight’s quantized floating-point value, its two neighboring quantized integers, and the corresponding dequantized weight. Fixed scale and zero-point parameters determine this mapping.

  • A Quantization Notation: A full-precision weight w is quantized within a group using fixed scale ∆ and zero-point z.These parameters define the quantization mapping for the weight.
  • A Quantization Notation: The quantized floating-point value ˜w is decomposed as ∆+z = ℓ+r, with ℓ=⌊˜w⌋, u=ℓ+1, and r∈[0,1).The neighboring integer candidates are the lower value ℓ and upper value u.
  • A Quantization Notation: Rounding ˜w selects a discrete quantized integer q∈{ℓ,u}, which maps back to dequantized weight ∆(q−z).˜w denotes the continuous value before rounding.

B Software Environment

The experiments use a unified software environment for diffusion-based weight recovery, post-training quantization, and language-model evaluation, with PyTorch compiled using CUDA 12.4.

  • B Software Environment: The shared environment covers diffusion-based weight recovery, post-training quantization, and language-model evaluation, using PyTorch compiled with CUDA 12.4.Weight recovery follows the DeepFloyd IF Stage-II implementation and initializes from the IF-II-M-v1.0 pretrained checkpoint.

C Diffusion Training and Inference Details

ReRound trains a ControlNet-conditioned diffusion model to recover normalized 64 × 64 weight patches from low-bit conditions, using frozen pretrained diffusion components and model-specific training. During inference, deterministic RTN conditions and 27-step DDPM sampling reconstruct all patches before mapping them back to full-precision weight space.

  • Diffusion architecture and conditioning: ReRound adds a ControlNet to the pretrained DeepFloyd IF Stage II U-Net, conditions it on low-bit weight patches, and updates only the ControlNet.The U-Net, ControlNet text-encoder pooling, and projection layers remain frozen.
  • Weight-patch construction: Target weight matrices, including the language-model head but excluding embeddings and normalization parameters, are converted to FP32 and divided into 64 × 64 patches.Quantization groups use [0, 1] normalization and asymmetric group-wise quantization with group size 128 for low-bit conditions.
  • Weight-patch construction: Training randomly samples matrices and 64 × 64 crops, pads smaller matrices with validity masks, flips patches with probability 0.5, and pairs targets with 2-bit stochastic-rounding conditions.The condition is generated by stochastic rounding, while the target remains the normalized full-precision patch.
  • Optimization and hardware: Each target LLM receives a separate model trained for five epochs on 1,000,000 patches per epoch using two RTX 4090 GPUs and an effective batch size of 64.Optimization uses AdamW with a constant learning rate of 2 × 10−5 and no scheduler.
  • Diffusion inference and matrix assembly: Inference uses deterministic 2-bit RTN conditions, batch size 128, and DDPM sampling with the super27 schedule’s 27 reverse-diffusion steps on one RTX 4090 GPU.All 64 × 64 patches are processed in fixed raster order, with edge-aligned final patches covering nondivisible matrix dimensions.

D ReRound Settings

ReRound maps full-precision and reconstructed weights into a shared quantized space, sweeps tolerance values, and selects candidates by matching leading singular values. Settings distinguish transformer linear layers from the output layer and specify β = 8, while SINQ experiments leave lm_head unquantized.

  • ReRound PTQ: ReRound constructs candidate quantized integer weight matrices over τ and selects each weight-matrix candidate by matching dequantized leading singular values to W.The procedure uses one GPU to map W and Wrec into the same quantized weight space.
  • ReRound PTQ: ReRound uses separate candidate sets for transformer linear layers and the output logit layer, lm_head.
  • ReRound with SINQ parameters: For Qwen3 1.7B experiments using SINQ quantization parameters, ReRound follows SINQ’s layer coverage and leaves lm_head unquantized.Table 9 reports the corresponding tolerance candidate sets.
  • ReRound PTQ: β = 8 is used at both bit widths, and τ = 0 retains the RTN assignments.

E Additional PTQ Results … F Baseline Implementation Details

Additional PTQ results show that ReRound improves four-task averages over group-wise RTN across the evaluated models and bit widths, and over AdaRound under W4A16. The paper also specifies standardized quantization settings and baseline implementations for reproducibility.

  • E.1 RTN Improvements on Additional Models: ReRound achieves a higher four-task average than group-wise RTN for every evaluated model at both W4A16 and W3A16.The comparison uses identical scales, zero-points, group size G = 128, and layer coverage.
  • E.2 Comparison with AdaRound: ReRound achieves a higher four-task average than AdaRound under W4A16 across six evaluated models.AdaRound uses 128 calibration samples from C4, whereas ReRound uses only pretrained model weights; both make layer-wise rounding decisions.
  • F Baseline Implementation Details: The main comparisons quantize transformer linear layers and the output logit layer, with group-wise methods using group size G = 128.Reimplemented baselines use the model checkpoints and evaluation protocol described in Section G unless otherwise stated.
  • F Baseline Implementation Details: RTN uses uniform asymmetric quantization with scale and zero-point computed from minimum and maximum weight values.Channel-wise RTN assigns one parameter pair per output channel, while group-wise RTN assigns one pair to each group of 128 consecutive input-dimension weights.
  • F Baseline Implementation Details: HQQ uses version 0.2.8.post1 with the PyTorch backend, quantization axis 1, and group size G = 128.BNB FP4 uses version 0.49.2 with FP4 weight quantization and BF16 computation and is evaluated only under W4A16.
  • F Baseline Implementation Details: Hadamard-transformed RTN applies a normalized Hadamard transform to groups of 128 weights, performs asymmetric RTN, and then applies the inverse transform.The groups are formed along the input-feature dimension.
  • F Baseline Implementation Details: GPTQ uses asymmetric group-wise quantization with G = 128, 128 C4 calibration sequences of length 2048, activation ordering, and true-sequential quantization.The output logit layer uses asymmetric group-wise RTN with G = 128.
  • F Baseline Implementation Details: AdaRound uses 128 C4 calibration sequences, optimizes rounding variables for 200 iterations, and processes the output layer in 1024-vocabulary blocks for 50 iterations.Its learning rate is 10^-2, regularization weight is 0.01, and each weight matrix uses up to 1024 calibration tokens.

G Models and Evaluation Protocol … K Weight Reconstruction and ReRound Procedures

ReRound combines conditional diffusion reconstruction, midpoint-aware revision rules, and spectral candidate selection to reconsider ambiguous RTN assignments while preserving the standard evaluation protocol and offline quantization workflow. The procedures reconstruct weights once, reuse them across W3 and W4, and select candidates using matrix-specific spectral comparisons.

  • G Models and Evaluation Protocol: Accuracy evaluations are zero-shot under LM Evaluation Harness v0.4.11, with 16-bit activations, fixed seeds, and task-specific accuracy metrics.The reported metrics are acc for WinoGrande, BoolQ, and SIQA, and acc_norm for PIQA; random seed 0 and NumPy/PyTorch seeds 1234 are used.
  • H Visualization of the Position-Dependent Tolerance Metric: The position-dependent tolerance equals τ within the midpoint region r ∈[0.4, 0.6] and decreases toward adjacent integers outside it, with β controlling sharpness.The region is defined by δ = 0.1; larger β values produce sharper boundary decreases.
  • I Matrix-Wise Revision of RTN Assignments: Revision rates depend on both the weight matrix and target bit width because W3 and W4 evaluate rounding candidates and perform spectral selection separately.The same reconstructed weights can therefore yield different revision rates for the same matrix at W3 and W4.
  • J Continuous Reconstruction for Low-Bit Rounding: A DDPM-based conditional diffusion model reconstructs continuous weights from low-bit observations, providing an additional signal for reconsidering midpoint-ambiguous RTN assignments.The reconstructed weights are not deployed as model parameters, and jointly reconstructed patch entries can depend on surrounding entries.
  • J.1 Low-Bit Conditioning: Deterministic low-bit quantization is many-to-one, so the RTN condition does not uniquely identify the original full-precision patch.Distinct continuous patches can produce the same RTN condition, motivating a conditional distribution over full-precision patches.
  • J.2 Conditional Denoising Relation: The ideal conditional denoiser’s clean estimate equals the conditional mean of the full-precision patch under Gaussian corruption, while inference uses a deterministic RTN condition.The final reverse-diffusion output is therefore based on a learned approximation under an inference condition differing from the stochastic training condition.
  • J.3 Midpoint Distance of Accepted Revisions: ReRound accepts a revision only when the quantized weight lies within the effective position-dependent tolerance of the midpoint.The rule allows revisions only for sufficiently midpoint-close assignments and does not require the denoiser to be ideal.
  • J.4 Scalar Error of an Accepted Revision; J.5 Spectral Comparison with RTN; K Weight Reconstruction and ReRound Procedures: The tolerance rule bounds each accepted revision’s additional scalar squared quantization error relative to RTN, while spectral selection chooses among valid candidates including RTN.For each candidate tolerance, ReRound forms a quantized matrix, dequantizes it, compares leading singular values, and selects the candidate with minimum spectral discrepancy; τ = 0 retains RTN.
Loading 2608.11045v1…