PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Generative Modeling with Residual Vector Quantization-Based Tokens

Jaehyeon Kim, Taehong Moon, Keon Lee, Jaewoong Cho

arXiv:2412.10208v3cs.LG

TL;DR

RVQ can improve fidelity through deeper quantization, but deeper hierarchies complicate efficient generation. ResGen predicts cumulative embeddings of collective tokens and frames masking with discrete diffusion and variational inference. Across conditional image generation and zero-shot speech synthesis, it outperforms autoregressive counterparts and shows improved fidelity or faster sampling as RVQ depth scales.

  • Problem

    RVQ improves reconstruction fidelity with shorter, deeper token sequences, but generative modeling can make sampling complexity depend on both sequence length and depth.

    Source: §1 · p. 1

  • Method

    ResGen predicts cumulative vector embeddings of collective RVQ tokens and formulates token masking and multi-token prediction using discrete diffusion and variational inference.

    Source: §1 · pp. 1–2

  • Results

    ResGen outperforms autoregressive counterparts on conditional image generation and zero-shot text-to-speech, with increased RVQ depth associated with improved fidelity or faster sampling than similarly sized baselines.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Across the evaluated image and speech tasks, ResGen combines strong fidelity with fast sampling while mitigating the trade-off between RVQ depth and inference speed.

    Source: §6 · p. 9

    The latent variational formulation assumes that its conditional distributions correspond to RVQ quantization and dequantization of masked tokens.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

We introduce ResGen, an efficient Residual Vector Quantization (RVQ)-based generative model for high-fidelity generation with fast sampling. RVQ improves data fidelity by increasing the number of quantization steps, referred to as depth, but deeper quantization typically increases inference steps in generative models. To address this, ResGen directly predicts the vector embedding of collective tokens rather than individual ones, ensuring that inference steps remain independent of RVQ depth. Additionally, we formulate token masking and multi-token prediction within a probabilistic framework using discrete diffusion and variational inference. We validate the efficacy and generalizability of the proposed method on two challenging tasks across different modalities: conditional image generation on ImageNet 256x256 and zero-shot text-to-speech synthesis. Experimental results demonstrate that ResGen outperforms autoregressive counterparts in both tasks, delivering superior performance without compromising sampling speed. Furthermore, as we scale the depth of RVQ, our generative models exhibit enhanced generation fidelity or faster sampling speeds compared to similarly sized baseline models.

1. Introduction

ResGen predicts cumulative embeddings of collective RVQ tokens to decouple generation iterations from sequence length and depth. On conditional ImageNet image generation and zero-shot text-to-speech, it outperforms autoregressive counterparts.

1. Introduction

  • High-fidelity generation remains challenging when computational efficiency must be balanced against detailed, high-resolution outputs across image, video, and audio domains.The paper situates this challenge alongside the growing demand for long-sequence and complex-structure modeling.Source: §1 · p. 1
  • RVQ improves reconstruction fidelity with shorter, deeper token sequences, but modeling them can make autoregressive sampling scale with sequence length and depth.Existing non-autoregressive approaches do not eliminate complexity from both dimensions simultaneously.Source: §1 · p. 1
  • ResGen predicts cumulative embeddings of collective tokens rather than individual tokens, decoupling inference steps from sequence length and depth.The approach models token masking and multi-token prediction using discrete diffusion and variational inference.Source: §1 · pp. 1–2
  • ResGen outperforms autoregressive counterparts on conditional ImageNet 256×256 image generation and zero-shot text-to-speech synthesis.As RVQ depth increases, it also shows improved fidelity or faster sampling than similarly sized baseline models.Source: §1 · p. 2

2. Background

The background introduces masked token modeling and residual vector quantization. RVQ quantizes residuals iteratively, forming a deeper hierarchy of tokens whose embeddings are summed to reconstruct the vector.

2. Background

  • Masked token modeling randomly masks input tokens and trains a model to predict the masked tokens using cross-entropy loss.The tokens are derived from quantized encoder outputs of a VQ-VAE.Source: §2 · p. 2
  • The forward masking process selects a scheduled number of tokens, while decoding starts from a fully masked sequence and iteratively unmasks predictions.Figure 1 depicts forward masking right-to-left and reverse unmasking left-to-right; its detailed reverse process predicts cumulative RVQ embeddings, quantizes them, and partially masks again.Source: §2 · p. 2, Figure 1 · p. 3
  • RVQ iteratively quantizes residuals from previous quantization steps, producing a token sequence with length L and depth D.Each depth selects an embedding from its codebook, and the reconstructed vector sums embeddings across depths.Source: §2 · p. 2
  • Increasing RVQ depth can more closely approximate the encoder output, with lower layers capturing significant features and higher layers progressively capturing finer details.This hierarchy makes deeper quantization a means of improving reconstruction fidelity.Source: §2 · p. 2

3. Method

ResGen fills RVQ tokens in a coarse-to-fine manner, predicting collective embeddings to separate generation iterations from token length and depth. It frames masking and prediction probabilistically through discrete diffusion and variational inference.

3.1. Masking and Prediction Task Design for RVQ Tokens

  • The RVQ masking schedule masks a total of n = ⌈γ(r) · L · D⌉ tokens, with r sampled uniformly during training and γ decreasing from one to zero.Masked-token counts are distributed across sequence positions, with tokens masked from highest depth toward lower depths.Source: §3.1 · p. 3
  • ResGen predicts the sum of residual vector embeddings for collective masked tokens, rather than predicting each token individually.It uses a mixture of Gaussians to model the embedding distribution.Source: §3.1 · pp. 3–4
  • Predicting vector embeddings avoids assuming conditional independence across token depths, which could harm performance.This choice reflects that the VQ-VAE decoder operates on vector embeddings.Source: §3.1 · p. 4
  • At sampling time, ResGen predicts cumulative masked-token embeddings, quantizes the vectors into RVQ tokens, and iteratively unmasks a schedule-determined subset.The process starts from a fully masked sequence and fills it in coarse-to-fine order.Source: §3.1 · p. 4

3.2. Formulation within a Probabilistic Framework

  • The paper casts token masking as forward discrete diffusion and token reconstruction as the reverse process, deriving a variational lower bound on data log-likelihood.Incremental masking without replacement permits closed-form intermediate marginal and conditional distributions.Source: §3.2 · pp. 4–5
  • The variational objective combines a prior loss, diffusion losses at each step, and a reconstruction loss; the prior term is zero when the final sequence is fully masked.A simplified loss gives equal emphasis to predicting the original tokens at each step.Source: §3.2 · p. 5
  • Variational inference models dependencies across token depths by predicting cumulative vector embeddings instead of individual tokens.This aligns with RVQ dequantization and decouples generation time complexity from token depth.Source: §3.2 · p. 5
  • The latent formulation assumes that p(x(0)|z, x(t)) corresponds to RVQ quantization and q(z | x(0), x(t)) to RVQ dequantization of masked tokens.Under this assumption, the method focuses on the remaining variational term, which matches the prediction loss in Equation 1.Source: §3.2 · p. 5

4. Related Work

Related work covers masked discrete diffusion and alternative token-generation paradigms, then reviews methods for hierarchical RVQ tokens. ResGen predicts cumulative vectors to separate sampling complexity from sequence length and depth.

4. Related Work

  • Discrete diffusion models learn to reverse corruption processes on discrete data, including masked-token approaches that iteratively refine or unmask sequences.VQ-Diffusion and MaskGIT exemplify masked diffusion for flat token sequences, while GIVT uses mixture-of-Gaussians vector prediction.Source: §4 · p. 5
  • VAR, MAR, and HART offer alternative paradigms to token-based autoregression, including coarse-to-fine prediction and continuous or hybrid latent modeling.These approaches are described separately from the discrete diffusion methods.Source: §4 · p. 5
  • Prior RVQ methods include autoregressive modeling over sequence length and depth in RQ-Transformer, and depth-linear sampling in SoundStorm [Borsos et al., 2023].Vall-E [Wang et al., 2023] predicts the first depth autoregressively and remaining depths sequentially; CLaM-TTS predicts vectors but remains autoregressive over sequence length.Source: §4 · pp. 5–6
  • ResGen predicts cumulative vector embeddings for masked tokens, decoupling sampling complexity from both sequence length and token depth.This strategy is presented as a way to handle hierarchical RVQ tokens while targeting sampling efficiency and high-fidelity generation.Source: §4 · p. 6

5. Experiments

The experiments evaluate ResGen on conditional ImageNet image generation and text-to-speech, comparing quality, speed, and efficiency with prior models. Across the tasks, results show competitive or better performance, with deeper RVQ improving fidelity or efficiency.

5. Experiments

  • The study evaluates ResGen on conditional image generation and text-to-speech to assess its effectiveness and generalizability across modalities.The TTS experiments include continuation and cross-sentence synthesis.Source: §5 · p. 6, §5.1 · p. 6

5.1. Experimental Setting

  • The evaluation covers ImageNet at 256 × 256 and two TTS tasks, using FID for images and CER, WER, and speaker-similarity metrics for audio.The comparisons include autoregressive and non-autoregressive model families.Source: §5.1 · p. 6

5.2.1. VISION TASK: ABLATION STUDIES

  • The parallel discrete-token variant is competitive with baselines but is outperformed by final ResGen, while increasing ResGen from 576 M to 1 B parameters further improves FID.The comparison supports the cumulative-embedding prediction design and reports improved FID for the larger model.Source: §5.2.1 · pp. 6–7
  • With identical 16-depth RVQ tokens, ResGen generates higher-quality images than MaskGIT and RQ-Transformer while requiring fewer inference steps than RQ-Transformer.The comparison uses models with comparable parameter counts.Source: §5.2.1 · p. 6, Table 1 · p. 7
  • At 8 refinement iterations, AR-ResGen reaches FID 5.22 and reduces decoding time from 5.35 to 3.05 s (×1.8 faster) than RQ-Transformer.At two iterations it reaches FID 5.33 in 1.56 s, a ×3.4 speed-up at comparable quality.Source: §5.2.1 · p. 7

5.2.2. VISION TASK: BENCHMARK COMPARISONS

  • FID 1.93 with CFG: ResGen-rvq16 achieves competitive ImageNet quality, ranks second only to VAR in speed, and supports a maximum latent batch size of 1,915 versus MAR-B’s 1,738.The quality comparison notes slightly higher FID than MAR-L despite comparable quality and faster sampling.Source: §5.2.2 · p. 7, Table 2 · p. 8
  • ResGen-rvq16 achieves better sample quality than ResGen-rvq8 at the same sampling steps with minimal additional inference time.The experiments attribute improved reconstruction and generation quality to greater RVQ depth.Source: §5.2.2 · p. 8

5.2.3. AUDIO TASK

  • ResGen lowers WER and CER and raises SIM-o and SIM-r versus the autoregressive baseline, using 25 inference iterations for 32-depth RVQ.With RVQ depth increased to 72, it achieves the lowest WER and CER in cross-sentence synthesis and ranks second to DiTTo-en-XL on continuation WER and CER.Source: Table 3 · p. 9, §5.2.3 · p. 9

6. Conclusion

ResGen is an RVQ-based discrete diffusion model designed to maintain high fidelity and fast sampling. Experiments on image generation and text-to-speech show strong performance, with deeper RVQ improving fidelity or efficiency.

6. Conclusion

  • ResGen generates high-fidelity samples while maintaining fast sampling speeds.The conclusion presents it as an efficient RVQ-based discrete diffusion model.Source: §6 · p. 9
  • ResGen predicts vector embeddings of collective tokens to mitigate the trade-off between RVQ depth and inference speed.The conclusion also describes token masking and multi-token prediction in a probabilistic framework using discrete diffusion and variational inference.Source: §6 · p. 9
  • On conditional image generation and zero-shot text-to-speech, ResGen performs comparably to or better than baselines in fidelity and sampling speed, with depth scaling improving fidelity or efficiency.Source: §6 · p. 9

Impact Statement

The impact statement frames ResGen as a memory-efficient approach for high-fidelity generation as real-world data resolution and size increase. It also notes potential benefits and misuse risks associated with synthetic media.

Impact Statement

  • ResGen is presented as a scalable approach intended to reduce memory and computational burdens in high-resolution generative tasks.The statement points to increasing data resolution and size in real-world applications.Source: Impact Statement · p. 10
  • The authors identify accessibility technologies, creative industries, and scientific simulations as areas where resource-efficient, high-quality generation could be useful.They suggest this could make state-of-the-art synthetic media generation more accessible to small-scale researchers and organizations.Source: Impact Statement · p. 10
  • The statement flags deceptive synthetic media and unauthorized replication of copyrighted content as potential misuse risks.It encourages responsible-use policies, including detection, authentication, and transparency measures.Source: Impact Statement · p. 10

A.1. Configurations for Training

The section details ResGen’s training and sampling implementation and reports improved reconstruction and generation quality at greater RVQ depth.

A.1. Configurations for Training

  • Vision and audio models use DiT XLarge configurations, while the audio setup adds a duration predictor and a 72-depth waveform autoencoder.Vision training uses 2.8M iterations; the TTS model uses 4 GPUs for 310M iterations, and its waveform autoencoder matches MelVAE’s 10.7 Hz temporal resolution.Source: §A.1 · p. 13

A.2. Implementation Techniques for Training

  • The training objective combines classification and regression losses, with an auxiliary distribution that gives closer mixture means higher probability while retaining nonzero probability for every component.The stated purpose is to encourage use of every mixture component and promote diversity in predictions.Source: §A.2 · p. 14
  • The mixture-of-Gaussians implementation uses low-rank mean projections to limit the computational cost of modeling many components.The appendix says this reduces overhead for higher-dimensional audio embeddings and makes the effective cost similar to a roughly 64K-vocabulary model.Source: §A.2 · pp. 13–14

A.3. Pseudo-code for Training

  • The training pseudocode samples token positions without replacement and then takes a gradient-descent step on the training objective.Source: §A.3 · p. 15

A.4. Scalability of our generative modeling with RVQ depth

  • At RVQ depth 16 versus 8, the autoencoder has lower rFID and ResGen has lower FID despite the increased token count.The passage reports the direction of change, not specific metric values.Source: Figure 3 · p. 15

B.1. Sampling with Confidence Scores

  • Confidence-based sampling derives token log probabilities from embedding-to-latent distances, sums them across depths, and uses temperature-scaled Gumbel noise to prioritize unmasking.Table 4 identifies CFG scheduling, top-p, and temperature among the sampling hyperparameters.Source: §B.1 · pp. 15–16, Table 4 · p. 16

B.2. Ablation Studies on Sampling

Sampling quality varies with steps, top-p, and temperature; ResGen’s RVQ hierarchy also differs from VAR in how it represents and generates tokens.

B.2. Ablation Studies on Sampling

  • More sampling steps improve generation quality both with and without classifier-free guidance (CFG).The authors attribute the gain to additional steps refining outputs.Source: §B.2 · p. 16
  • Higher top-p improves quality with CFG, whereas lower top-p helps without CFG by favoring confident predictions.The reported top-p trends differ depending on whether CFG is used.Source: §B.2 · p. 16
  • Moderate temperature introduces controlled stochasticity into confidence-guided unmasking and balances diversity with fidelity.The authors describe this as mitigating the monotonicity of the RVQ-token unmasking order.Source: §B.2 · p. 16

B.3. Pseudo-code for Sampling

  • The sampling pseudocode initializes an all-zero mask, applies RVQ to masked tokens, and updates the mask through random or confidence-based selection.Source: §B.3 · p. 17

C.1. Elaborating Differences with VAR

  • Unlike VAR’s fixed depth-to-resolution schedule and sequential generation across depths, ResGen predicts cumulative embeddings on one latent grid and supports length-agnostic tokenization.Increasing RVQ depth can represent lower effective spatial resolutions without extending sequence length.Source: §C.1 · pp. 17–18

C.2. Further comparison with faster MAR

Controlled comparisons report that ResGen-rvq16 outperforms tested faster MAR-B configurations in image quality while sampling at lower latency.

C.2. Further comparison with faster MAR

  • ResGen reports single-image wall-clock latency on one A100 GPU, whereas MAR reports seconds per image averaged over a large batch, complicating direct speed comparisons.The authors say this difference in measurement protocols largely explains the apparent latency discrepancy.Source: §C.2 · p. 18
  • FID 4.11 for MAR-B at 16 AR steps took roughly 5.6 seconds per image, versus ResGen-rvq16’s FID 1.93; at 25 diffusion steps, MAR-B’s FID 3.38 took about 22.5 seconds per image.The comparisons vary AR steps with diffusion steps fixed, then diffusion steps with AR steps fixed.Source: §C.2 · p. 18, Table 5 · p. 18, Table 6 · p. 18

C.3. Sensitivity to Masking Strategies

The section examines masking-schedule sensitivity and compares ResGen-rvq16 with recent masked generators, while describing flexibility in RVQ depth and resolution.

C.3. Sensitivity to Masking Strategies

  • ResGen-rvq16 achieved its best reported masking-schedule result at FID 9.53 with 64 sampling steps and CFG when trained with circle masking and sampled with exponential masking.The cross-evaluation tested each model with its original and a different sampling schedule.Source: §C.3 · p. 18, Table 7 · p. 19
  • Exponential sampling unmasks fewer tokens early and then increases the unmasking rate; it performed strongly even when the model was not trained with that schedule.The authors describe this as a coarse-to-fine process that appears to benefit global structure before finer details.Source: §C.3 · p. 19

C.4. Comparison with Recent Masked Generative Models

  • MAGVIT-v2 and MaskBit report FID scores of 1.78 and 1.62, respectively, compared with ResGen-rvq16’s 1.93 with CFG at a comparable number of sampling steps.The cited table compares image-generation performance; the passage notes that the other models’ FID scores are lower.Source: §C.4 · p. 19, Table 8 · p. 19
  • ResGen predicts cumulative embeddings to model cross-depth dependencies and supports 16-depth RVQ with an 8×8 token map, balancing representational capacity and memory footprint.The passage contrasts this with typical VQ methods using 16×16 spatial tokens and notes that independent bit-group performance can degrade as groups increase beyond two.Source: §C.4 · p. 19

D. Limitations and Future Directions

ResGen has favorable memory efficiency, sampling speed, and generation quality, but the authors identify KV caching, FSQ support, and a theoretical account of low-step sampling as future directions. The method is designed around RVQ tokens, and extending it to FSQ is not straightforward.

D. Limitations and Future Directions

  • KV caching is not explored; progressively filled token positions could reuse precomputed values to reduce redundant computation and potentially improve sampling speed.The authors describe KV caching as a future direction, alongside favorable memory efficiency, sampling speed, and generation quality for the proposed method.Source: D. Limitations and Future Directions · p. 20
  • ResGen is designed around RVQ tokens, and supporting FSQ is not straightforward because its tokenization and embedding processes differ.The authors suggest exploring FSQ as a possible route to novel quantization strategies and improved generative performance.Source: D. Limitations and Future Directions · p. 20
  • The authors identify the lack of a theoretical explanation for why high-quality generation needs relatively few iterations as a future research direction.They hypothesize that unmasking may be easier than denoising, but do not provide a formal justification.Source: D. Limitations and Future Directions · p. 20

Related papers

Loading 2412.10208v3…