PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversity Has Always Been There in Your Visual Autoregressive Models

Tong Wang, Guanyu Yang, Nian Liu, Kai Wang, Yaxing Wang, Abdelrahman M Shaker, Salman Khan, Fahad Shahbaz Khan, Senmao Li

arXiv:2511.17074v1cs.CV

TL;DR

VAR models generate images efficiently through next-scale prediction, yet can produce highly similar samples for a text prompt. DiverseVAR analyzes early-scale pivotal components and suppresses them at input while amplifying them at output, without additional training. The paper reports improved diversity while preserving image quality and text–image alignment.

  • Problem

    Despite their efficient next-scale generation, VAR models can suffer diversity collapse, producing highly similar samples for a given prompt.

    Source: §1 · p. 2

  • Method

    DiverseVAR analyzes early-scale diversity formation and applies training-free input suppression and output amplification of the pivotal component during inference.

    Source: §1 · p. 2

  • Results

    DiverseVAR improves generative diversity while preserving image quality and text–image alignment.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The pivotal component plays a critical role in shaping diversity formation at early scales in text-to-image VAR models.

    Source: §5 · p. 8

    Soft-suppression can weaken text-semantic alignment, especially for numerical descriptions.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Visual Autoregressive (VAR) models have recently garnered significant attention for their innovative next-scale prediction paradigm, offering notable advantages in both inference efficiency and image quality compared to traditional multi-step autoregressive (AR) and diffusion models. However, despite their efficiency, VAR models often suffer from the diversity collapse i.e., a reduction in output variability, analogous to that observed in few-step distilled diffusion models. In this paper, we introduce DiverseVAR, a simple yet effective approach that restores the generative diversity of VAR models without requiring any additional training. Our analysis reveals the pivotal component of the feature map as a key factor governing diversity formation at early scales. By suppressing the pivotal component in the model input and amplifying it in the model output, DiverseVAR effectively unlocks the inherent generative potential of VAR models while preserving high-fidelity synthesis. Empirical results demonstrate that our approach substantially enhances generative diversity with only neglectable performance influences. Our code will be publicly released at https://github.com/wangtong627/DiverseVAR.

1. Introduction

DiverseVAR targets diversity collapse in efficient next-scale VAR models by intervening on early-scale pivotal components. The paper reports greater diversity with preserved image fidelity and slight performance influences across evaluated benchmarks.

1. Introduction

  • Structural formation occurs predominantly at early scales, where the pivotal token shapes structure while auxiliary tokens carry semantic information and image fidelity.Later scales mainly refine and stabilize existing structures.Source: §1 · p. 2
  • DiverseVAR uses training-free input soft-suppression and output soft-amplification of pivotal components to promote diversity while preserving fidelity and semantic alignment.The method intervenes during inference without additional training.Source: §1 · p. 2
  • On COCO, GenEval, and DPG, DiverseVAR improves generative diversity while maintaining image fidelity, with slight influences on generation performance.The paper describes extensive evaluation across numerous metrics.Source: §1 · p. 2

2. Related Wrok

VAR replaces AR next-token decoding with progressive next-scale generation, aiming for high-quality images in around ten steps. The related-work discussion contrasts this efficiency with diversity collapse and notes that existing diversity methods do not directly transfer to VAR.

2.1. Visual Autoregressive Generation

  • VAR generates high-quality images through next-scale prediction in around ten steps, but samples for a given prompt tend to be highly similar.Traditional AR methods use many iterative next-token steps.Source: §2.1 · p. 2

2.2. Alternative Visual Generation Paradigms

  • Few-step diffusion distillation can also produce diversity collapse; proposed remedies include trajectory alignment, a parallel decoder, teacher seeding, or feature amplification.These approaches have trade-offs including training needs, higher memory use, or reliance on creativity-oriented prompts.Source: §2.2 · pp. 2–3
  • Prior diversity work for diffusion and AR models does not directly transfer to next-scale VAR, where diversity-focused study remains scarce.The paper proposes analyzing which VAR scales govern diversity and using those properties to enhance inference while preserving semantic consistency and visual fidelity.Source: §2.2 · p. 3

3. Method

DiverseVAR builds on VAR’s coarse-to-fine next-scale prediction, focusing on early scales where structures form and diversity is governed by pivotal feature components. It uses input suppression and output amplification during inference to encourage diverse generation while retaining the original process at later scales.

3.1. Preliminaries

  • VAR predicts a sequence of resolution-specific token maps, each conditioned on earlier residuals, progressively approximating the continuous image feature map.The first scale uses text embeddings; later scales use a downsampled feature map from the previous scale as input.Source: §3.1 · p. 3

3.2. Motivation

  • Structures form predominantly at early scales and stabilize later, while pivotal tokens primarily contribute structure and auxiliary tokens carry semantics and image fidelity.Zeroing pivotal tokens at early scales changes structure while largely retaining meaning; zeroing auxiliary tokens degrades semantics and fidelity, whereas changes at later scales have little effect.Source: §3.2 · pp. 4–5
  • Naive Component Partition identifies pivotal tokens using token-feature distance from the mean and an elbow point, but zeroing them can cause unexpected variations.The elbow point is computed with the Maximum Distance to Chord method.Source: §3.2 · pp. 4–5

3.3. Diverse Visual Generation for VAR

  • DiverseVAR applies soft-suppression to the pivotal input component and soft-amplification to the output, intervening during inference without additional training.The framework explores diversity formation at early scales and retains the original VAR inference process at later scales.Source: §3.3 · p. 5, Figure 8 · p. 6
  • Soft-Suppression Regularization scales dominant singular values of the early-scale input feature map to suppress the pivotal component and promote diversity.The feature map is decomposed with SVD and reconstructed using the modified singular values; resetting the top two values to zero increases diversity but reduces fidelity.Source: §3.3 · pp. 5–6
  • Soft-Suppression disperses output-logit distributions across samplings, but isolated high-probability peaks can impair numerical text-image alignment; Soft-Amplification further guides diversity.Soft-Amplification augments singular values of the output feature to encourage more dispersed logits while avoiding isolated peaks.Source: §3.3 · p. 6

4. Experiment

DiverseVAR improves reported diversity while largely preserving generation performance, fidelity, and text–image alignment. Experiments compare it with vanilla Infinity and other model families, test its regularization choices, and show it retains variable-aspect-ratio generation.

4.1. Experimental Setup

  • Experiments use Infinity-2B and Infinity-8B at 1024×1024, evaluate diversity and text–image alignment across COCO, GenEval, and DPG, and apply the method at scales 4 and 6.The remaining listed scales use vanilla inference; settings are α = 1.0, β = 0.01, ˆα = 1.0, and ˆβ = 0.001.Source: §4.1 · pp. 6–7

4.2. Main Results

  • On COCO2014 and COCO2017, DiverseVAR reports higher Recall, Coverage, and FID than vanilla Infinity with comparable CLIP scores; on AFHQ and CelebA-HQ, this pattern holds except for Infinity-8B Recall.Qualitative results also show more diverse outputs and closer resemblance to real AFHQ images.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • DiverseVAR surpasses most competing approaches on GenEval and DPG, except DALL-E 3 on DPG, while performing on par with vanilla Infinity.Both DiverseVAR and Infinity score over 0.7 on GenEval and around 83.0 on DPG.Source: §4.2 · p. 7, Table 3 · p. 8

4.3. Additional Analysis

  • The ablations compare removing SAR, applying SAR and SSR to different features, and reversing their order on the input feature.SSR is applied to the input feature and SAR to the output feature in the described full method.Source: §4.3 · pp. 7–8
  • The full design outperforms ablated variants in perceptual quality and diversity, except for Relation on DPG and Coverage on COCO2017.The ablations remove a regularizer or change which feature receives it and the order of operations.Source: §4.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8
  • DiverseVAR retains Infinity’s variable-aspect-ratio generation capability and continues to support efficient image generation.The paper presents qualitative comparisons for generations with diverse aspect ratios.Source: §4.3 · p. 8, Figure 10 · p. 8

5. Conclusion

The conclusion identifies the pivotal component as important to early-scale diversity formation in text-to-image VAR models. DiverseVAR suppresses that component at the input and augments it at the output, improving diversity while preserving image quality and text–image alignment.

5. Conclusion

  • The pivotal component plays a critical role in early-scale diversity formation, and input-level suppression plus output-level augmentation improves diversity while preserving image fidelity and text–image alignment.The authors report extensive experiments supporting these outcomes.Source: §5 · p. 8

Supplementary Material

The supplement provides implementation details, algorithm descriptions, and extended experiments supporting the main paper.

Supplementary Material

  • The supplement includes dataset, metric, and baseline configurations; DiverseVAR pseudocode; and ablations, scale and block analyses, and additional qualitative comparisons.These materials are presented as support for reproducibility and for the main paper’s analyses of image quality and diversity.Source: A. Overview · p. 12

B. Implementation Details

The implementation details describe evaluation datasets and baselines, a rule for partitioning pivotal and auxiliary tokens, and the text prompts used for generation.

B.1. Configure

  • Diversity evaluation uses AFHQ and CelebA-HQ, while Infinity-8B runs on an A100 40GB are evaluated at scale 48 because larger runs cause out-of-memory errors.Infinity-2B uses the official implementation with default hyperparameters.Source: §B.1 · p. 12

B.2. The Pivotal and Auxiliary Token

  • The pivotal score measures each token’s L2 deviation from the mean feature map; the MDC elbow in the descending scores separates pivotal tokens from auxiliary tokens.The elbow is the intermediate point with maximum perpendicular distance from the chord joining the sorted sequence’s endpoints.Source: §B.2 · p. 12

B.3. Text Prompts

  • The supplement lists text-generation prompts, including prompts for scenes such as a library, a train crossing a canyon bridge, and a temple among flowers.It also provides prompts used for the qualitative comparison in Fig. S3.Source: §B.3 · p. 12, Figure S3 · p. 13

D.1. Ablation Study of Scales

Applying DiverseVAR at a single early scale improves diversity with only marginal quality degradation. Increasing the number of applied scales raises diversity but can sharply reduce quality; scales 4 and 6 are selected as the trade-off.

D.1. Ablation Study of Scales

  • Using a single early scale consistently improves Recall, Coverage, and FID on COCO 2014 and COCO 2017, with only marginal degradation on GenEval and DPG.The tested single-scale configuration uses scales 2 and 4 (li ∈{4, 6}).Source: §D.1 · p. 13
  • Applying DiverseVAR at four early scales yields the highest diversity metrics but the lowest generation quality, with GenEval decreasing by more than 0.1.The authors choose scales 4 and 6 (li ∈{4, 6}) as the default trade-off.Source: §D.1 · p. 13
  • Increasing the number of applied scales raises diversity but rapidly degrades generation quality.The authors therefore use scales 4 and 6 as the default setting.Source: §D.1 · p. 13

D.2. Ablation Study of Blocks

Applying DiverseVAR across all eight Infinity blocks is the default trade-off. Partial-block subsets either substantially reduce image quality or leave diversity suboptimal, while model-level application preserves quality but yields almost no diversity improvement.

D.2. Ablation Study of Blocks

  • Applying the method to partial block subsets significantly degrades either image quality or diversity compared with applying it to all eight blocks.Blocks {0, 1, 2, 3, 4, 5, 6} give superior diversity but severely degrade image quality; blocks {4, 5, 6, 7} preserve quality but have suboptimal diversity.Source: §D.2 · p. 13
  • The default setting applies DiverseVAR across all eight Infinity backbone blocks to balance image quality and diversity.The blocks are indexed 0 through 7.Source: §D.2 · p. 13
  • Applying DiverseVAR at the model level maintains quality, with GenEval at 0.73, but yields almost no diversity improvement on COCO 2014 and COCO 2017.The model-level configuration is reported in the last row of Table S2.Source: §D.2 · p. 13

D.3. Ablation Study on logits

Applying DiverseVAR to logits preserves generation quality but produces only slight diversity gains. The authors report that varying the logits distributions is intended to address the similarity across vanilla VAR samplings.

D.3. Ablation Study on logits

  • Vanilla VAR samplings have similar logits distributions; DiverseVAR introduces variation into those distributions to encourage diverse generations while maintaining alignment and visual quality.The authors compare logits-level application with their block-level approach.Source: §D.3 · p. 13
  • The logits ablation evaluates image generation quality with GenEval and DPG and diversity with COCO 2014 and COCO 2017.Source: Table S3 · p. 14
  • Applying DiverseVAR to logits preserves generation quality but yields only a slight diversity improvement on COCO 2014 and COCO 2017.The reported GenEval scores are 0.72 and 0.73.Source: §D.3 · p. 14

E. Additional Results

Additional qualitative comparisons show that DiverseVAR generates more varied samples than vanilla VAR while maintaining text–image alignment. The supplementary figures report this pattern across multiple prompts.

E.1. Diversity Comparison

  • Across text prompts, DiverseVAR generates more varied images than vanilla VAR while maintaining good text–image alignment.The vanilla model tends to produce high-quality but less diverse samples.Source: §E.1 · p. 14
  • The comparisons cover samples generated from multiple text prompts.Source: §E.1 · p. 14
  • Figures S1–S3 provide additional qualitative diversity comparisons, which the paper describes as showing superior diversity.Source: §E.1 · p. 14, Figure S1 · p. 15, Figure S2 · p. 16

Related papers

Loading 2511.17074v1…