PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Autoregressive Models Beat Diffusion Models on Inference Time Scaling

Erik Riise, Mehmet Onurcan Kaya, Dim P. Papadopoulos

arXiv:2510.16751v2cs.CV

TL;DR

Search strategies have delivered limited benefits for diffusion-based image generation, leaving open whether visual generation can benefit from inference-time computation. This paper applies search to discrete visual autoregressive models and finds that beam search lets a 2B model outperform a 12B diffusion model across benchmarks with fewer function evaluations.

  • Problem

    Search has improved language-model inference, but translating those gains to image generation has been difficult, and search on continuous diffusion models has shown limited benefits.

    Source: §1 · p. 1

  • Method

    The paper applies tree search to discrete autoregressive image generation, using verifiers to guide exploration and exploiting shared prefixes for computational reuse.

    Source: §1 · p. 1, §3 · p. 3, §3.1 · p. 3

  • Results

    Across benchmarks, a 2B autoregressive model with beam search achieves twice the gains of search on a 12B diffusion model while using 46% fewer function evaluations.

    Source: §1 · p. 2, §4.5 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    The results support architectural compatibility with search as a factor in inference-time scaling for visual generation, beyond model size alone.

    Source: §5 · p. 9

    NFEs across autoregressive and diffusion architectures are not directly comparable in FLOPs, so they support order-of-magnitude efficiency comparisons.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

While inference-time scaling through search has revolutionized Large Language Models, translating these gains to image generation has proven difficult. Recent attempts to apply search strategies to continuous diffusion models show limited benefits, with simple random sampling often performing best. We demonstrate that the discrete, sequential nature of visual autoregressive models enables effective search for image generation. We show that beam search substantially improves text-to-image generation, enabling a 2B parameter autoregressive model to outperform a 12B parameter diffusion model across benchmarks. Systematic ablations show that this advantage comes from the discrete token space, which allows early pruning and computational reuse, and our verifier analysis highlights trade-offs between speed and reasoning capability. These findings suggest that model architecture, not just scale, is critical for inference-time optimization in visual generation.

1 INTRODUCTION

Search for image generation has shown limited benefits in diffusion models, prompting a test of whether visual autoregressive models can benefit from inference-time search. The study reports that beam search lets a 2B model outperform a 12B diffusion model with fewer function evaluations.

1 INTRODUCTION

  • Earlier search strategies for diffusion models showed limited improvement, with simple random sampling often outperforming search, unlike the consistent benefits reported for language-model search.This contrast raises whether continuous latent spaces are poorly matched to discrete search algorithms.Source: §1 · p. 1
  • Visual autoregressive models generate images token by token across scales, giving beam search opportunities to explore alternatives, prune weak paths early, and reuse computation across branches.The passage contrasts this discrete structure with diffusion models’ continuous noise spaces.Source: §1 · p. 1
  • The study combines discrete visual autoregressive models with verifiers ranging from lightweight preference models to vision-language models for guided exploration.The goal is to generate high-quality, compositionally accurate images.Source: §1 · p. 1
  • A 2B autoregressive model with beam search delivers twice the gains of search on a 12B diffusion model while using 46% fewer function evaluations and surpassing it on all compositional metrics.Figure 1 also shows examples of corrections to object counts, spatial relationships, and colors.Source: Figure 1 · p. 2, §1 · p. 2

2 RELATED WORK

Related work describes inference-time search as a way to improve model performance and traces why image-generation search may depend on architecture. It contrasts recent autoregressive models’ discrete structure with limited gains from search in continuous diffusion models.

2 RELATED WORK

  • Hierarchical visual autoregressive models such as VAR and Infinity generate discrete tokens across scales, creating a structure naturally compatible with search algorithms.The passage describes VAR’s coarse-to-fine next-scale prediction and Infinity’s bitwise tokenization.Source: §2 · p. 2
  • Language models improve through inference-time computation, including tree search and process reward models, and compute-optimal inference can let small models match models up to 14× larger.The passage presents strategic computation as a possible substitute for scale.Source: §2 · p. 2
  • Random search selects the best of independent images, greedy token optimization commits to one continuation at each step, and beam search retains top-scoring parallel sequences.Beam search preserves more diverse paths than the greedy strategy.Source: Figure 2 · p. 3
  • Diffusion-model search has often been outperformed by random sampling, while gains from other approaches remain limited and the potential of search for autoregressive image generation remains unexplored.The passage frames architectural compatibility as a possible missing ingredient for effective inference-time scaling in visual generation.Source: §2 · pp. 2–3

3 METHOD

The method applies tree search to discrete autoregressive image generation, pairing a scale-wise model with complementary verifiers and search strategies. It measures compute using evaluated images and transformer forward passes, while noting that NFEs across architectures are not directly comparable in FLOPs.

3.1 AUTOREGRESSIVE IMAGE GENERATION

  • Infinity generates 1024 × 1024 images across 13 scales using next-scale prediction, with tokens at each scale generated simultaneously and cached key-value representations reused across branches sharing a prefix.This scale-wise setup provides 13 decision points for search.Source: §3.1 · p. 3

3.2 VERIFICATION FRAMEWORK

  • ImageReward provides lightweight preference scoring, complemented by CLIPScore, Aesthetic Score, a rank-aggregation ensemble, and LLaVA-OneVision prompt-alignment assessment.The ensemble averages each candidate’s rank across the three metrics.Source: §3.2 · pp. 3–4

3.3 SEARCH STRATEGIES

  • Random search selects among independent images, while Greedy Token Optimization commits to a single optimized path and risks local optima; the study tests c values of 4, 15, and 30.The strategies represent different exploration-exploitation trade-offs.Source: §3.3 · p. 4
  • Beam search expands w parallel hypotheses with c candidates at each scale, retaining the top w prefixes to balance exploration breadth and computational tractability.The tested configurations are (w, c) = (2, 2), (3, 5), and (3, 10).Source: §3.3 · p. 4
  • Prefix reuse reduces beam-search complexity from O(n · K) for independent generation to approximately O(n · K/w).Here, n is the number of candidate images and K the number of scales.Source: §3.3 · p. 4

3.4 COMPUTATIONAL BUDGET METRICS

  • The study reports total images evaluated and NFEs; one NFE is a transformer forward pass generating tokens for one of Infinity’s 13 scales.For 195 images, beam search requires 1,365 NFEs versus 2,535 for equivalent random search, a 46% efficiency gain.Source: §3.4 · p. 4
  • NFEs across autoregressive and diffusion architectures are not directly comparable in FLOPs, though they provide meaningful order-of-magnitude efficiency comparisons.Source: §3.4 · p. 4

4 EXPERIMENTAL RESULTS

The experiments evaluate search for autoregressive image generation across budget scaling, DrawBench, verifier trade-offs, compositional benchmarks, and comparison with diffusion search. The results favor beam search, especially against diffusion search at much larger model scale, while showing that verifier costs and task capabilities shape practical choices.

4.1 BUDGET SCALING ANALYSIS

  • Expected maximum verification scores scale logarithmically with budget across all three verifiers, so each additional quality gain requires exponentially more samples.The analysis uses random samples from Infinity-2B outputs on 200 DrawBench prompts, scored by ImageReward, Aesthetic Score, and CLIPScore.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 PERFORMANCE ON DRAWBENCH

  • On DrawBench, beam search is the most effective tested strategy and surpasses Infinity-8B, while verifier choice trades off visual quality, prompt adherence, and aesthetic appeal.Beam search with 195 images (1,365 NFEs) surpasses random search with 390 images (5,070 NFEs); aesthetic optimization improves visual appeal but degrades CLIPScore prompt adherence.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 VERIFIER ANALYSIS

  • CLIPScore needs 14 ms and about 1.6 GB of GPU memory per image, while LLaVA-OneVision needs 500 ms and 15.3 GB; verifier performance also depends on task.ImageReward leads on color, shape, and texture by 0.02 each, whereas LLaVA-OneVision leads spatial reasoning by 0.07; narrow verifier objectives can produce verifier hacking.Source: §4.3 · pp. 6–7, Table 2 · p. 6, Table 3 · p. 6

4.4 COMPOSITIONAL VALIDATION

  • On GenEval, beam search improves all categories, with an overall average improvement of 16%.Reported gains include +25% on counting tasks and +26% on position.Source: §4.4 · p. 7, Table 5 · p. 7
  • On T2I-CompBench++, beam search improves compositional performance using 54% of random search’s NFE budget, while verifier strengths vary with task complexity.ImageReward is competitive for attribute binding; LLaVA-OneVision performs better on reasoning-heavy spatial and numeracy tasks, and qualitative examples show it correcting spatial errors ImageReward misses.Source: §4.4 · pp. 7–8, Table 4 · p. 7, Figure 4 · p. 8
  • Increasing ImageReward-guided beam-search temperature from 1.0 to 2.0 boosts most tasks, including numeracy by 2.0%, but slightly reduces spatial performance.The authors attribute the trade-off to greater candidate diversity and ImageReward’s weakness in evaluating spatial layouts.Source: §4.4 · p. 8, Table 6 · p. 8

4.5 COMPARISON WITH CONTINUOUS DIFFUSION MODELS

  • On T2I-CompBench++, the 2B autoregressive model with beam search scores higher than the 12B diffusion model in every category, with average gains of 11.3% versus 5.7%.On DrawBench, medium-budget beam search also exceeds the diffusion model’s best ImageReward score with less than half its compute; discrete tokens support early pruning and prefix caching.Source: §4.5 · pp. 8–9, Table 8 · p. 9, Table 7 · p. 9
  • Discrete token space makes guided search more tractable than continuous diffusion search by enabling early pruning and computational reuse through prefix caching.The reported performance scaling suggests that search compatibility can matter more than raw parameter count for compositional image generation.Source: §4.5 · p. 9

5 CONCLUSION

The conclusion argues that autoregressive image models’ discrete token space gives them an architectural advantage for inference-time search, with gains across benchmarks despite a smaller model size.

  • Autoregressive image models’ discrete token space enables efficient pruning and computational reuse for inference-time search.Source: §5 · p. 9
  • With fewer function evaluations, a 2B autoregressive model using beam search surpasses a 12B diffusion model, with gains consistent across benchmarks.Source: §5 · p. 9
  • These results challenge quality-by-model-size assumptions and highlight co-designing models and inference algorithms for more efficient, capable text-to-image generation.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The authors commit to making their implementation publicly available upon publication and state that their study relies on publicly accessible models and datasets.

AUTHOR STATEMENT ON THE USE OF LARGE LANGUAGE MODELS

The authors state that large language models were used only for editorial refinement and grammatical corrections, while the scientific work originated with the authors.

A QUALITATIVE RESULTS: BASELINE VS. BEAM SEARCH

The appendix presents qualitative comparisons on T2I-CompBench++ between baseline generation and LLaVA-OneVision-guided search, with each row pairing a prompt and two generations.

A QUALITATIVE RESULTS: BASELINE VS. BEAM SEARCH

B RESULTS FROM DIFFERENT SEARCH STRATEGIES

Figure 5 compares four text-to-image search strategies using Infinity-2B under the same generation parameters and ImageReward guidance.

B RESULTS FROM DIFFERENT SEARCH STRATEGIES

  • The comparison includes baseline generation, random search, greedy token optimization, and beam search.Each column represents one strategy.Source: Figure 5 · p. 19
  • All images use Infinity-2B with identical generation parameters and ImageReward guidance.Source: Figure 5 · p. 19
  • The visual comparison uses 390 verified images and prompts from Drawbench.Source: Figure 5 · p. 19

C SCALING BEHAVIOR FOR DIFFERENT VERIFIERS

The scaling plots compare search performance against computational budget, measured in NFEs and images, and indicate stronger compute efficiency for guided methods than random search.

C SCALING BEHAVIOR FOR DIFFERENT VERIFIERS

  • Beam search’s scaling lead over GTO widens as the computational budget increases.Source: Figure 6 · p. 20
  • At low budgets, both GTO and beam search outperform random search and are more compute-efficient.Source: Figure 6 · p. 20
  • Table 14 reports no clear benefit for variance-based dynamic allocation: it is less efficient at equal candidate-image counts and underperforms at similar NFE counts.Source: Table 14 · p. 21

D ANALYSIS OF A HEURISTIC-BASED DYNAMIC BUDGET ALLOCATION

Variance-based dynamic allocation assigns more candidate slots to steps with higher verifier-score variance, concentrating search early. It performs worse than fixed-budget GTO in the reported comparisons, while a verifier example shows a trade-off between prompt adherence and aesthetic appeal.

D ANALYSIS OF A HEURISTIC-BASED DYNAMIC BUDGET ALLOCATION

E VERIFIER IMAGE SELECTION

  • For “a metallic ring and a fluffy hat,” the aesthetic verifier selects an appealing image without the ring, while other verifiers favor adherence but yield poorly generated hands.The example uses random search with 390 verified images.Source: Figure 8 · p. 22
Loading 2510.16751v2…