PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-training Quantization for Hybrid Iterative Generative Models

Jing Gao, Junyi Wu, Wei Wang, Yan Yan, Yao Zhao

arXiv:2608.13932v1cs.LG

TL;DR

Hybrid iterative generative models remain difficult to quantize because outliers and amplified anomalies can cause quality degradation or collapse. HyGenQ addresses these issues with outlier-channel decoupling and anomaly scaling, achieving high-fidelity W8A8 quantization that outperforms baselines and prevents collapse.

  • Problem

    Vanilla PTQ struggles with hybrid IGMs because excessive outliers and amplified anomalies disrupt activation quantization and can cause generation-quality degradation or model collapse.

  • Method

    HyGenQ combines Hierarchical Cluster Decoupling to independently quantize stable outlier and normal channels with Scaling Recalibration to constrain amplified anomalies.

  • Results

    8-bit W8A8 quantization consistently outperforms existing baselines, while preventing the complete model collapse observed when all linear layers are quantized.

  • Takeaways & Limitations

    HyGenQ preserves generation quality under aggressive quantization across representative hybrid IGMs and different model families.

  • Takeaways & Limitations

    Quantization still degrades performance, particularly for larger, deeper, and more iterative models such as MAR-H, leaving anomaly handling unresolved.

Abstract

from arXiv · show

Iterative Generative Models (IGMs) span autoregressive and diffusion paradigms, and hybrid variants that couple them can achieve remarkable image-generation fidelity. However, their iterative inference incurs substantial computational overhead, making Post-training Quantization (PTQ) appealing for acceleration, while directly applying vanilla PTQ to hybrid IGMs can trigger model collapse. By analyzing these failures, we identify two critical challenges: Excessive Outliers (EOs) in the activations create an irreconcilable trade-off between preserving normal precision and covering EOs, resulting in severe degradation in generation quality; Amplified Anomalies (AAs) arising unpredictably from minor quantization errors, create a mismatch between calibration and inference, thus iteratively triggering model collapse. To address these challenges, we introduce HyGenQ, a PTQ framework for hybrid IGMs. HyGenQ comprises Hierarchical Cluster Decoupling (HCD) and Scaling Recalibration (SR). HCD identifies and decouples outlier channels via a multi-stage clustering process, effectively isolating EOs while maintaining normal value precision, thereby alleviating performance degradation. SR scales AAs beyond Gaussian Bound, thereby avoiding model collapse caused by aggressive truncation. Extensive experiments demonstrate that HyGenQ successfully quantizes representative hybrid IGMs to 8-bit precision (W8A8), significantly outperforming existing baselines and validating its robustness across different model families.

I. INTRODUCTION · II. BACKGROUNDS AND RELATED WORKS · A. Iterative Generative Models

Hybrid iterative generative models combine autoregressive global-structure modeling with diffusion-style refinement, but their repeated inference steps make deployment costly. HyGenQ targets PTQ failures caused by activation outliers and accumulating quantization perturbations, enabling robust W8A8 quantization of representative IGMs.

  • I. INTRODUCTION: Hybrid IGMs combine token-by-token autoregression for global structure and long-range dependencies with diffusion-style refinement for local details.This coupling leverages complementary strengths of both paradigms for high-fidelity image generation.
  • I. INTRODUCTION: Sequential autoregressive prediction and step-by-step diffusion sampling create substantial inference costs, with matrix multiplications in linear layers often dominating runtime.These costs hinder the feasibility of real-time hybrid IGM applications.
  • II. BACKGROUNDS AND RELATED WORKS: PTQ reduces computational overhead without retraining and requires only a small calibration dataset, making it especially suitable for repeatedly executed hybrid IGM inference.Retraining is often costly for models whose generation involves many repeated steps.
  • I. INTRODUCTION: Vanilla PTQ is difficult for hybrid IGMs because activations are non-stationary and context-dependent, while small quantization perturbations can accumulate across iterations.This propagation may destabilize the entire generation process.
  • I. INTRODUCTION: Excessive Outliers create a trade-off between preserving normal-value precision and covering outlier ranges, so sacrificing either can cause information loss and model collapse.The outliers can remain critical to generation quality.
  • I. INTRODUCTION: Scaling Recalibration defines a fixed Gaussian Bound from full-precision activation distributions and dynamically scales channels exceeding it to mitigate anomaly-driven model collapse.The method monitors activation fluctuations across channels while preserving overall generation fidelity.
  • I. INTRODUCTION: 8-bit precision (W8A8) quantization demonstrates HyGenQ’s robustness and generality, while existing PTQ methods substantially degrade hybrid-IGM generation quality.The passage further reports that mainstream PTQ methods often result in complete model collapse when all linear layers are quantized.

B. Post-training Quantization

Post-training quantization converts full-precision models to low-bit representations by calibrating quantization parameters on representative samples for inference. Although effective for ViTs and diffusion models, existing PTQ methods remain inadequate for hybrid IGMs because of their token-by-token modeling and iterative inference.

  • PTQ Fundamentals: PTQ transforms full-precision parameters into low-bit integers to reduce storage and computational complexity while accelerating inference.It calibrates quantization parameters using a small set of representative samples, then applies them during low-bit inference.
  • PTQ Fundamentals: Affine uniform quantization maps a full-precision activation tensor to the integer range [0, 2b−1], where b is the quantization bit-width.The scale λ, zero-point β, and rounding operation determine the mapping.
  • PTQ Fundamentals: PTQ is advantageous for swift deployment and enhanced inference speed because it eliminates the need for retraining.The passage identifies PTQ as an effective model compression strategy.
  • PTQ for Hybrid IGMs: Existing PTQ methods succeed on Vision Transformers and diffusion models but fail to produce satisfactory results for hybrid IGMs.Hybrid IGMs’ token-by-token modeling and multiple iterations make direct application difficult, leaving their PTQ underexplored.

III. QUANTIZATION CHALLENGES IN HYBRID IGMS · A. Excessive Outliers · B. Amplified Anomalies

Hybrid IGMs are unusually vulnerable to PTQ because coupled autoregressive and diffusion iterations amplify activation irregularities and quantization errors. The resulting Excessive Outliers and Amplified Anomalies create precision trade-offs, calibration–inference mismatch, and eventually model collapse.

  • III. QUANTIZATION CHALLENGES IN HYBRID IGMS: Hybrid IGMs couple autoregressive context updates with diffusion sampling, making activations more sensitive to quantization perturbations than in conventional models.Repeated conditional interactions progressively accumulate and amplify abnormal activations, driving quantized inference away from the full-precision path.
  • III. QUANTIZATION CHALLENGES IN HYBRID IGMS: Quantization exposes two abnormalities: Excessive Outliers are unusually large activations, while Amplified Anomalies arise when minor perturbations grow during iterative denoising.These abnormalities are progressively accumulated and amplified during the coupled generation process.
  • A. Excessive Outliers: Hybrid IGMs generate tokens stepwise under partial context, while per-token diffusion gradients supervise the autoregressive backbone across multiple sampled timesteps.Composing autoregressive and diffusion refinement extends the error-propagation pathway and makes stable PTQ inference more challenging.
  • A. Excessive Outliers: Hybrid IGMs exhibit sparse but much larger high-magnitude channels than conventional ViTs, producing Excessive Outliers that make activation-range quantization inherently conflicting.Preserving outliers compresses normal values, whereas preserving normal precision requires truncating outliers, and both choices impair generation capability.
  • B. Amplified Anomalies: Quantization errors in diffusion boundary layers are injected into the reverse denoising step, where γt scales predicted-noise errors and perturbs the next denoising state.The resulting state enters the next boundary layer, enabling abnormal amplification at the input and output ends of the denoising update.
  • B. Amplified Anomalies: Amplified denoising perturbations propagate through autoregressive context updates, while stochasticity and shrinking tolerance make the effective activation range difficult to predict.A local perturbation can become a biased later condition, and values within calibration ranges may exceed boundaries after amplification.
  • B. Amplified Anomalies: Amplified Anomalies are quantized activations beyond the Gaussian Bound estimated from full-precision distributions, especially in boundary layers and early denoising timesteps.Vanilla PTQ with static calibration-derived clipping can cause model failure because early-timestep amplification enlarges boundary-layer inputs.

IV. METHODOLOGY · A. Hierarchical Cluster Decoupling

HyGenQ addresses hybrid IGM quantization with Hierarchical Cluster Decoupling (HCD) and Scaling Recalibration (SR). HCD identifies stable outlier channels through hierarchical clustering and decouples their quantization parameters from normal channels to reduce quantization error and mutual interference.

  • IV. METHODOLOGY: HyGenQ combines HCD, which decouples normal and outlier channels, with SR, which recalibrates amplified boundary-layer activations during diffusion.The modules target Excessive Outliers and Amplified Anomalies, respectively.
  • A. Hierarchical Cluster Decoupling: Outliers consistently concentrate in a few fixed channels across iterations, allowing HCD to distinguish stable outlier channels from normal channels.This separation addresses errors caused by sharing one quantization scale between dominant normal values and outliers.
  • A. Hierarchical Cluster Decoupling: Stage 1 reshapes FC2 activations into X ∈ R^n×C and represents each channel with its maximum absolute value, c_i = max(|X_i|).The resulting channel-wise feature vector is clustered according to activation-range similarity.
  • A. Hierarchical Cluster Decoupling: HCD uses k-means for channel-feature clustering and a Genetic Algorithm to improve robustness against suboptimal local minima caused by imbalanced channel-feature distributions.Candidate center proposals evolve through selection, crossover, and mutation over T generations.
  • A. Hierarchical Cluster Decoupling: For each candidate k, HCD pseudo-quantizes normal and outlier channels separately and selects k∗ by minimizing a weighted SSE-based Quantization Error Metric.The weighting emphasizes outlier-channel errors rather than averaging them away when outlier partitions are small.
  • A. Hierarchical Cluster Decoupling: The final partitions use the same bit-width but separate quantization parameters, enabling decoupled quantization without mutual interference.Normal and outlier activation ranges provide the respective quantization parameters.
  • A. Hierarchical Cluster Decoupling: Stage 3 fixes the identified outlier channels per layer and reuses them during inference without recomputation.Figure 7 describes the three-stage process as progressively separating outlier channels from normal ones.

B. Scaling Recalibration

Scaling Recalibration (SR) stabilizes quantized inference in diffusion boundary layers by using a Gaussian Bound derived from full-precision activations. It detects and rescales anomalous channels, while applying fixed-range quantization to mitigate calibration–inference range shifts and harmful truncation.

  • B. Scaling Recalibration: SR targets diffusion boundary layers, where timestep-dependent input-range shifts make calibration-time quantization ranges unstable during inference.These layers directly receive intermediate denoising states, so small quantization errors can alter their activation ranges.
  • B. Scaling Recalibration: The Gaussian Bound Φ−1(η) provides a high-confidence reference range derived from statistically stable standard-Gaussian boundary-layer activations in the full-precision model.Φ−1(·) denotes the inverse CDF of the standard normal distribution.
  • B. Scaling Recalibration: At each timestep, SR identifies channel c as anomalous when its maximum activation exceeds Φ−1(η).The anomalous-channel set is denoted At.
  • B. Scaling Recalibration: SR rescales anomalous channels by a shared factor sAt, mapping out-of-bound activations back into the Gaussian Bound before inverse scaling restores their magnitudes.This prevents harmful truncation during quantization.
  • B. Scaling Recalibration: SR uses fixed-range quantization based on Φ−1(η) to remain robust to moderate activation-range expansion without relying on dynamic scaling.Most timesteps remain within the bound, but calibration–inference distributional shifts can still cause suboptimal clipping.

V. EXPERIMENTS · A. Experimental Settings

The experiments evaluate HyGenQ under standardized ImageNet generation, calibration, and quantization protocols, comparing it with representative PTQ baselines. HyGenQ is configured through HCD and SR, with iteration- and timestep-specific quantization parameters to assess excessive outliers and amplified anomalies.

  • A. Experimental Settings: Each method generates 50,000 256×256 ImageNet images using official model configurations and identical iterative-generation settings on one NVIDIA RTX 4090.Experiments use PyTorch, and generation quality is measured with FID and IS using the official ADM implementation.
  • A. Experimental Settings: HyGenQ is compared with representative PTQ methods for vision transformers and diffusion models, including RepQ-ViT, SmoothQuant, PTQ4DM, TFMQ-DM, TaQ-DiT, and OCS.The baselines address distribution mismatch, activation burden, denoising-step variance, temporal information, sensitive Post-GELU activations, or outlier-induced error.
  • A. Experimental Settings: The calibration set randomly selects 32 samples from 256×256 ImageNet evaluation and collects statistics across autoregressive iterations and diffusion timesteps.All methods use the same calibration set and sampling protocol for fair comparison.
  • A. Experimental Settings: Deterministic experiments use a uniform quantizer for weights and activations, with separate quantization parameters for each autoregressive iteration and diffusion timestep.This setup controls activation shifts and focuses evaluation on excessive outliers and amplified anomalies.
  • A. Experimental Settings: HyGenQ combines Hierarchical Cluster Decoupling and Scaling Recalibration, using candidate cluster numbers k ∈[2, 20] and weighting factor ω = 0.001 for HCD.Its genetic algorithm uses population size P = 50, generations T = 20, crossover probability pc = 0.5, and mutation probability pm = 0.2.
  • A. Experimental Settings: For MAR-H, Scaling Recalibration is additionally enabled on bou.The supplied passage truncates the remainder of this configuration.
  • A. Experimental Settings: The performance comparison uses 64 iterations and 100 timesteps, distinguishing full-precision diffusion boundary layers from quantized linear layers.The table caption marks these conditions with ◦ for full-precision boundary layers and • when all linear layers are quantized.

B. Quantization Performance of Hybrid IGMs

The section evaluates PTQ methods on MAR as a representative hybrid IGM under demanding iterative inference settings. HyGenQ avoids the severe degradation and collapse affecting existing baselines, remaining stable across MAR scales even when all linear layers are quantized.

  • Evaluation Settings: MAR serves as the representative hybrid IGM because its tightly coupled multi-step inference produces the most severe error accumulation.Evaluation includes 64 autoregressive iterations with a 100-step DDPM solver and a more stringent 32-iteration setting.
  • Baseline Analysis: PTQ4DM, TFMQ-DM, TaQ-DiT, and OCS each address specific activation or temporal-information issues, but leave important outlier-related limitations unresolved.PTQ4DM retains more high-magnitude information than direct clipping; TFMQ-DM preserves timestep-dependent information; TaQ-DiT mitigates Post-GELU outliers; OCS splits extreme-value channels.
  • Full Linear-Layer Quantization: FID values around 250 and IS close to 2 occur for SmoothQuant, PTQ4DM, TFMQ-DM, and OCS on MAR-L, while TaQ-DiT reaches FID 139.90 and IS 2.68.This stringent setting quantizes all linear layers, exposing error accumulation that boundary-layer full precision can otherwise suppress.
  • HyGenQ Results: FID 3.14 and IS 255.35 on MAR-L demonstrate HyGenQ’s stability, while MAR-H retains FID 6.89 and IS 210.95.HyGenQ remains stable across MAR scales by suppressing anomaly propagation and limiting accumulated degradation.

C. Robustness and Generalization Across IGM Families

HyGenQ is evaluated for robustness across autoregressive, diffusion, and hybrid IGMs under a controlled ImageNet 256×256 protocol. Across these settings, it consistently improves over PTQ4DM.

  • Evaluation Scope: Experiments cover representative autoregressive, diffusion, and hybrid IGMs, generating 8,000 ImageNet 256×256 samples per model.The evaluation targets robustness and generalization across diverse IGM paradigms.
  • Controlled Protocol: Calibration data, bit-width configuration, and sampling settings remain consistent across methods, with baselines generally built upon PTQ4DM.Different sample counts should not be directly compared across tables.
  • Model Families: LlamaGen and LDM-4 represent autoregressive and diffusion settings with distinct quantization risks: heavytailed activations and timestep-accumulated perturbations, respectively.LlamaGen is sensitive to range selection, while LDM-4 accumulates quantization perturbations through iterative denoising.
  • Generalization Results: HyGenQ consistently improves over PTQ4DM across the evaluated autoregressive, diffusion, and hybrid model settings.This result is reported in Table II and Figure 9.

D. Extending Outlier Suppression to Hybrid IGMs · E. Ablation Study · F. Inference Efficiency

HyGenQ remains more robust than transferred outlier-suppression baselines on hybrid IGMs, especially when all linear layers are quantized or bit widths are reduced. Ablations attribute this robustness to HCD and SR, while the implementation achieves substantial end-to-end acceleration through optimized INT8 inference and sampling components.

  • D. Extending Outlier Suppression to Hybrid IGMs: QuaRot and HadaNorm provide competitive but unstable results across model scales, indicating that outlier redistribution only partially benefits hybrid IGM quantization.Both methods mainly alleviate outlier effects rather than fully resolving the underlying issues.
  • D. Extending Outlier Suppression to Hybrid IGMs: HyGenQ outperforms all baselines in MAR visualization results with 64 iterations and 100 timesteps under both boundary-layer and fully quantized settings.The ◦ setting keeps diffusion boundary layers in full precision; the • setting quantizes all linear layers.
  • D. Extending Outlier Suppression to Hybrid IGMs: W4A8 and W4A4 cause severe generation collapse for most methods under lower-bit quantization.This visualization further highlights the difficulty of aggressive quantization for hybrid IGMs.
  • D. Extending Outlier Suppression to Hybrid IGMs: When all linear layers are quantized, transferred outlier-suppression methods suffer severe collapse, whereas HyGenQ remains substantially more stable.The instability reflects amplified anomalies and accumulated error propagation that outlier suppression alone cannot address.
  • E. Ablation Study: The ablation study evaluates HCD and SR on MAR-B with 64 autoregressive iterations and 100 diffusion timesteps under both ◦ and • settings.SR is unnecessary under ◦ because that setting does not suffer degradation from amplified anomalies.
  • E. Ablation Study: HCD handles excessive outliers, SR suppresses amplified anomalies, and their integration into HyGenQ preserves generation quality under aggressive quantization.The complete framework combines both modules after the ablation variants are evaluated.
  • F. Inference Efficiency: End-to-end efficiency evaluation uses cuBLASLt INT8 GEMM through a lightweight C++ and CUDA wrapper across MAR model scales.All methods are tested under the same sampling configuration.
  • F. Inference Efficiency: HyGenQ achieves substantial acceleration over full-precision inference by baking per-(step,i) scale/zero-point tables and replaying the diffusion loss network with CUDA Graphs.The tables track drifting activation distributions, while CUDA Graph replay reduces kernel launch overhead during sampling.

G. Lower-Bit Quantization … J. Hyperparameter Sensitivity

Across lower-bit settings, calibration protocols, HCD grouping strategies, and hyperparameter studies, HyGenQ shows strong comparative performance while revealing sensitivity to activation precision and the benefits of per-step calibration and robust parameter choices.

  • G. Lower-Bit Quantization: Under W8A6, HyGenQ consistently outperforms SmoothQuant and PTQ4DM across all MAR scales, but performance drops sharply versus W8A8.More aggressive W4A8 and W4A4 quantization further visualize degradation from accumulated errors.
  • H. Analysis of Calibration Protocol: Global calibration strategies improve over Global Max but remain inferior to per-step calibration, which better handles activation shifts across autoregressive iterations and diffusion timesteps.Separate quantization parameters are assigned to each autoregressive iteration and diffusion timestep.
  • I. Comparison of HCD Clustering Strategies: Genetic search achieves the best IS across all model scales, while k-means++ performs worse and threshold grouping can suppress or misassign informative outlier channels.The threshold-based strategy achieves competitive FID on MAR-B and MAR-H but lower IS than genetic search.
  • J. Hyperparameter Sensitivity: HyGenQ remains stable across different HCD cluster-number ranges, so nearby ranges produce similar performance without requiring fine-grained tuning of k.The default upper bound is k=20.
  • J. Hyperparameter Sensitivity: Performance remains stable across a wide range of small ω values, with ω=0.001 used as the default weighting factor in HCD.ω emphasizes abnormal channels while downweighting normal channels during clustering.
  • J. Hyperparameter Sensitivity: The default genetic-algorithm configuration achieves the best trade-off between performance and stability, while alternatives still clearly outperform PTQ4DM.The implementation follows the DEAP framework.
  • J. Hyperparameter Sensitivity: Boundary-layer activations support the Gaussian approximation used for SR, with a mean KS value of 0.0211 and maximum value of 0.0617.The Gaussian Bound Φ−1(η) is derived from the FP model’s activation distribution.

VI. OPEN ISSUES AND POTENTIAL INSIGHTS · VII. CONCLUSION

HyGenQ addresses Excessive Outliers and Amplified Anomalies in hybrid iterative generative models through Hierarchical Cluster Decoupling and Scaling Recalibration, achieving high-fidelity W8A8 quantization. Remaining issues include inference and computational overhead, performance degradation in larger iterative models, and the broader limits of PTQ for complex generative models.

  • VI. OPEN ISSUES AND POTENTIAL INSIGHTS: Extra Scaling Recalibration operations noticeably weaken acceleration gains on MAR-H, making inference-latency reduction an important improvement direction.The added latency arises from isolating normal values and outliers as much as possible.
  • VI. OPEN ISSUES AND POTENTIAL INSIGHTS: Stability of HCD-detected outlier channels on MAR-B is evaluated across different seeds and input-class splits using detected sets and the core anomalous-channel set.The table defines Sa and Sb as seed-specific sets, Si as the i-th input-class split, and Score as the core anomalous-channel set.
  • VI. OPEN ISSUES AND POTENTIAL INSIGHTS: Sensitivity of the Gaussian Bound G used in Scaling Recalibration is explicitly evaluated.The supplied passage identifies this as a dedicated sensitivity analysis.
  • VI. OPEN ISSUES AND POTENTIAL INSIGHTS: Amplified-Anomaly fluctuations between calibration and inference can exceed PTQ’s effective applicability range, although HyGenQ’s added computational overhead is described as negligible.Further consideration is needed for this overhead and the calibration mismatch.
  • VI. OPEN ISSUES AND POTENTIAL INSIGHTS: HyGenQ prevents baseline model collapse under low-bit versions but still degrades versus full-precision versions, especially in larger, deeper, and more iterative models such as MAR-H.This reflects severe accumulated quantization errors and a trade-off between generative performance and quantization.
  • VII. CONCLUSION: HyGenQ isolates stable outlier channels from normal channels with Hierarchical Cluster Decoupling and constrains amplified anomalies through Gaussian-Bound-based Scaling Recalibration.The framework targets Excessive Outliers and Amplified Anomalies in multi-step generation.
  • VII. CONCLUSION: High-fidelity W8A8 quantization consistently outperforms strong baselines under identical iterative generation settings.Scaling Recalibration rescales anomalous channels during denoising to prevent severe generation collapse.
  • VII. CONCLUSION: The paper emphasizes severe quantization challenges in hybrid iterative generative models and encourages further exploration of quantization for complex generative models.Excessive Outliers and Amplified Anomalies become more pronounced in iterative paradigms.
Loading 2608.13932v1…