Channel-wise Vector Quantization

Wei Song; Tianhang Wang; Yitong Chen; Tong Zhang; Zuxuan Wu; Min Li; Jiaqi Wang; Kaicheng Yu

Source-linked AI summary

Channel-wise Vector Quantization

Wei Song, Tianhang Wang, Yitong Chen, Tong Zhang, Zuxuan Wu, Min Li, Jiaqi Wang, Kaicheng Yu

arXiv:2605.26089v2May 25, 2026cs.CVcs.AI

TL;DR

Conventional patch-wise VQ faces codebook-utilization and sequence-ordering limitations. The paper introduces CVQ, which quantizes channels, and CAR, which predicts them sequentially in a coarse-to-fine order; CVQ reports high utilization and reconstruction fidelity, while CAR supports autoregressive image generation.

Problem
Conventional patch-wise VQ suffers codebook collapse, while raster-scan patch sequences are structurally misaligned with next-token prediction.
Method
CVQ quantizes feature channels into a 1D sequence, and CAR performs next-channel prediction with nested dropout establishing coarse-to-fine ordering.
Results
CVQ maintains nearly 100% codebook utilization at large codebook sizes and reduces rFID from 3.64 to 2.32, while CAR achieves high text-to-image evaluation scores.
Takeaways & Limitations
Channel-wise tokens provide a promising alternative visual representation for autoregressive image generation, combining progressive visual detail with high utilization and reconstruction fidelity.
Takeaways & Limitations
The authors identify combining CVQ with newer VQ advances and extending channel-wise quantization to video as future work.

Abstract

from arXiv · show

We present Channel-wise Vector Quantization (CVQ), a novel image tokenization paradigm that replaces patch-wise tokens with channel-wise tokens. Unlike conventional vector quantization, which assigns a discrete token to each patch feature vector, CVQ quantizes each channel of the feature map. This formulation represents an image as discrete levels of visual details, rather than as a grid of spatial patches. Based on CVQ, we introduce a new visual autoregressive framework with "next-channel prediction". Instead of rendering images patch by patch in raster order, our Channel-wise Autoregressive (CAR) model predicts image channels sequentially, producing progressively enriched visual details. Specifically, it first sketches global structure and then refines fine-grained attributes, akin to a human artist's workflow. Empirically, we show that: (1) CVQ achieves 100% codebook utilization with a 16K+ codebook size without any bells and whistles, and substantially improves reconstruction quality over conventional VQ; and (2) CAR attains a DPG score of 86.7 and a GenEval score of 0.79, demonstrating strong effectiveness for text-to-image generation.

1 Introduction

The paper identifies codebook collapse and spatial patch ordering as limitations of conventional VQ, then proposes channel-wise tokenization and next-channel autoregressive modeling. CVQ represents images as progressively enriched 1D channel sequences, while CAR uses that ordering for generation.

Conventional VQ assigns discrete indices to local 1×1×c spatial feature vectors, following a patch-wise paradigm.
CVQ discretizes feature channels rather than spatial vectors, representing images as 1D sequences with progressively enriched visual content.
Patch redundancy concentrates updates on a small subset of codebook entries, causing dead codes and codebook collapse.
Channel-wise partitioning produces more separable embeddings across images, yielding broader codebook coverage and improved utilization.
CVQ changes only the quantization axis to mitigate codebook collapse without additional modules or constraints.
CAR reformulates autoregressive generation as progressive next-channel prediction, using nested dropout to create a coarse-to-fine channel ordering.

2 Related Works

Related work establishes VQ as a foundation for discrete visual representations and describes persistent challenges in codebook utilization and autoregressive token ordering. Existing remedies improve utilization or modeling but often add complexity, reduce capacity, or retain structurally misaligned raster sequences.

VQ provides the learnable discrete latent representations underlying major visual tokenization and generation methods.
Prior VQ methods address quantization error and codebook collapse through multi-stage quantization, modulation, factorization, fixed values, or modified training pipelines.
Token factorization and fixed-value quantization can prevent collapse but substantially limit representational capacity.
Existing autoregressive image methods commonly flatten 2D patch grids into raster-scan sequences, creating structurally misaligned ordering for next-token prediction.

3 Method

CVQ replaces conventional patch-wise quantization with channel-wise quantization, using global channel features to improve codebook utilization and reconstruction. CAR then predicts the resulting channel tokens sequentially, with nested dropout establishing a coarse-to-fine ordering for visual generation.

Channel-wise Vector Quantization: Conventional VQ suffers codebook collapse because redundant, overlapping image patches cluster around the same codebook vectors during training.Only a small portion of the codebook receives gradient updates, leaving other cluster centers inactive.
Channel-wise Vector Quantization: Channel-wise partitioning produces more distinguishable embeddings and broader codebook coverage, activating a significantly larger portion of the codebook.The t-SNE analyses connect separable channel embeddings with reduced overlap and higher utilization throughout training.
Channel-wise Autoregressive Generation: CAR reformulates autoregressive generation as next-channel prediction, conditioning each channel token on textual context rather than predicting spatial patches.A decoder-only transformer models the 1D sequence of discrete channel tokens; a two-layer MLP aligns token dimensions with the backbone.
Channel-wise Autoregressive Generation: Nested channel dropout orders channels from coarse to fine by retaining a random prefix, so early channels encode global structure and later channels encode finer details.This training strategy establishes an ordered sequence because channels otherwise have no inherent order.
Channel-wise Autoregressive Generation: CVQ improves reconstruction fidelity over existing VQ methods across 256- and 1024-token budgets on ImageNet-1K.The comparison includes established baselines such as MoVQ and VQ-LC at comparable vocabulary scales.

4 Experiments

Experiments evaluate CVQ for reconstruction and CAR for text-to-image generation, including codebook scaling, nested dropout, tokenizer comparisons, and generation fidelity. CVQ maintains high codebook utilization and improves reconstruction, while CAR produces progressively detailed images and competitive benchmark results.

Visual Reconstruction: 100% codebook utilization is achieved by CVQ with a 16,384-codebook setting, whereas conventional VQ collapses to 4.5%.The comparison uses ImageNet-1K validation reconstruction metrics and reports no additional modifications for CVQ.
Visual Reconstruction: With 256 tokens, CVQ reaches 2.60 rFID versus 4.99 for vanilla VQGAN and 2.63 for SimVQ.
Visual Reconstruction: At 1024 tokens, CVQ attains 0.88 rFID and 25.02 dB PSNR, surpassing MoVQGAN at 1.05 rFID and VQGAN-LC at 1.29 rFID.
Visual Generation: CAR generates progressively detailed content as channels are added and reaches 0.79 GenEval and 86.72 DPG overall at 8B scale.The 8B model is reported as competitive with strong VAR methods and the 4B model is competitive or superior to listed unidirectional baselines.
Visual Generation: CAR achieves 6.42 FID on MJHQ-30K, outperforming both 1D masked-token baselines and the standard 2D-token baseline.
Discussions: At codebook size 65K, CVQ maintains nearly 100% utilization and achieves a 52% reconstruction-fidelity improvement over the VQ baseline.Across scales, CVQ reduces rFID from 3.64 to 2.32, while patch-wise VQ utilization falls from 20.3% to 1.1%.
Discussions: Nested channel dropout improves CVQ generation by 0.12 GenEval and 9.38 DPG while maintaining reconstruction quality.The ablation attributes the improvement to inducing an autoregressive-friendly channel ordering.

5 Conclusion and Future Works

The paper concludes that channel-wise quantization provides high codebook utilization and reconstruction fidelity without architectural modifications or auxiliary losses. It identifies combining CVQ with newer VQ methods, unified vision models, and temporal quantization as future directions.

Conclusion: CVQ discretizes images along the channel dimension and achieves high codebook utilization and reconstruction fidelity without architectural modifications or auxiliary loss terms.
Conclusion: CAR shifts autoregressive image generation from spatial patch prediction to next-channel prediction.
Conclusion: Channel-wise tokens are presented as a promising direction for autoregressive image generation and for rethinking the fundamental unit of visual tokenization.
Future Works: Future work includes combining CVQ with SimVQ and IBQ, applying CAR to unified vision models, and extending channel-wise quantization to video.

B.1 Implementation Details of Nested Channel Dropout

Nested channel dropout trains truncated latent representations to preserve critical information in early channels while balancing hierarchical learning with full-resolution reconstruction. An adaptive GAN weight stabilizes training when sparse channel configurations challenge the discriminator.

Nested channel dropout: Nested dropout retains the first c_keep channels and masks the remaining channels, forcing lower-indexed channels to encode the most critical visual information.The mechanism is applied to latent representations Z ∈ R^{h×w×c}.
Optimization objective: The nested objective combines reconstruction, quantization, perceptual, and adversarial losses for each retained-channel configuration.The quantization loss is computed only over active channels to maintain valid reduced-dimensional codebook mappings.
Adaptive adversarial weighting: An adaptive sigmoid-weighted GAN term gradually increases adversarial influence as channel capacity grows, supporting stable convergence for sparse configurations.The formulation uses η = 0.05 and λ0 = 1.
Stochastic training: Training samples c_keep uniformly from U(1, c) with probability α and uses the full channel configuration with probability 1 − α.The hybrid objective combines nested and full-configuration losses.
Ablation: At α = 25% and α = 50%, channel dropout improves generation performance while reconstruction performance remains stable.The ablation evaluates dropout effects on reconstruction fidelity and text-to-image generation.

B.2 Analysis of Nested Channel Dropout

Nested channel dropout establishes ordered representations that support autoregressive generation while preserving reconstruction quality. Its theoretical guarantees are narrower than its empirical applicability because they require semi-linear architectures and L2 reconstruction loss.

Theoretical basis: Nested dropout enforces an ordered representation in which feature importance decreases with dimension index.Prior theory preserves global performance under stated semi-linear autoencoder conditions.
Limitations: Theoretical guarantees require a semi-linear architecture optimized with L2 reconstruction loss.These conditions constrain the direct scope of the cited guarantees.
Empirical evidence: Empirical evidence indicates that nested token dropping can benefit autoregressive generation in deeper architectures, complex-loss settings, and diffusion frameworks without significant expressivity loss.The cited experiments report preserved reconstruction quality and improved generation at 25% and 50% dropout.
Ablation results: GenEval rises from 0.61 to 0.72 and DPG rises from 72.76 to 82.13 when dropout increases from 0% to 25%.Increasing dropout to 50% produces comparable generation performance to the 25% setting.

C Progressive Channel Analysis

Progressively revealing channels produces a coarse-to-fine reconstruction hierarchy: early channels recover global semantics and later channels refine local structure and texture.

Progressive channel analysis: From 32 to 128 channels, rFID, SSIM, and PSNR improve rapidly as early channels recover global appearance, object identity, and coarse structure.The rapid improvement identifies the main semantic content as concentrated in early channels.
Progressive channel analysis: After 128 channels, rFID improvement slows while SSIM and PSNR continue increasing as later channels refine local structure and high-frequency texture.This provides quantitative evidence for a global-to-fine organization of visual information.

D Variable-Resolution Extension

CVQ extends to variable resolutions through lightweight resampling while decoupling channel-token count from image resolution. Under matched token budgets, it maintains stronger reconstruction quality than patch-wise VQ across resolutions.

Fixed-resolution setting: The main controlled comparison fixes resolution at 256 × 256, producing codewords with spatial size 16 × 16.This setting enables direct comparison with VQ.
Variable-resolution extension: Learnable query resampling maps arbitrary h × w features to a fixed h0 × w0 size before quantization and projects quantized channels back afterward.The method uses fixed queries before lookup and dynamically generated target queries after lookup.
Experimental protocol: For inputs at 256, 512, and 1024 resolution, CVQ fixes the embedding dimension at 256 and uses a 1024-token budget for fair comparison with VQ.Patch-wise VQ produces 256, 1024, and 4096 tokens at those resolutions.
Results: CVQ consistently outperforms the corresponding VQ baseline across resolutions, including when token budgets are matched.With fixed token budgets, higher-resolution use may trade some fidelity for efficiency; matched-budget results remain stronger than VQ.
Autoregressive efficiency: CAR keeps channel-sequence length nearly constant as resolution increases, whereas patch-wise token counts grow quadratically with the latent grid.CVQ encodes spatial resolution within each channel token rather than increasing token count.

E Hyperparameters and Data Source

CAR training uses a two-stage optimization schedule, and the model is trained on a diverse mixture of filtered image-text datasets.

Hyperparameters: Stage I trains only the MLP projector and LLM head while keeping the LLM backbone frozen.The learning rate is 1 × 10−4.
Hyperparameters: Stage II trains all model components with a learning rate of 2 × 10−5.Both stages use AdamW with β1 = 0.9, β2 = 0.96, and weight decay of 1 × 10−3.
Data Source: The training data combines filtered subsets of ImageNet-21K, LAION-Aesthetics-12M, CC12M, Megalith-10M, BLIP-3o-short, BLIP-3o-long, and a 6M in-house aesthetics dataset.Megalith-10M is re-captioned by Qwen3-VL.