PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale Local Speculative Decoding for Image Generation

Elia Peruzzo, Guillaume Sautière, Amirhossein Habibian

arXiv:2601.05149v2cs.CV

TL;DR

Autoregressive image generation faces high inference latency as resolution increases and token sequences grow. MULO-SD combines low-resolution drafting, upsampling, local verification, and parallel decoding. It reports up to 5× speedup over speculative- and parallel-decoding baselines while maintaining comparable semantic alignment and perceptual quality.

  • Problem

    Autoregressive image generation has high inference latency, particularly at high resolutions where token counts grow rapidly.

    Source: §1 · p. 1

  • Method

    MULO-SD drafts at low resolution, upsamples candidates for high-resolution verification, and uses local resampling with parallel decoding.

    Source: §1 · p. 2, §3.2 · p. 3, §3.4 · p. 5

  • Results

    Up to 5× speedup over EAGLE-2, LANTERN, and ZipAR, with comparable semantic alignment and perceptual quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    When paired with parallel decoding, MULO-SD delivers the best speed–quality trade-off among existing methods.

    Source: §5 · p. 8

    The method requires two model checkpoints and KV-caches, which can strain memory-constrained devices; lower drafter–verifier alignment can reduce acceptance rates and speedup gains.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Autoregressive (AR) models have achieved remarkable success in image synthesis, yet their sequential nature imposes significant latency constraints. Speculative Decoding offers a promising avenue for acceleration, but existing approaches are limited by token-level ambiguity and lack of spatial awareness. In this work, we introduce Multi-Scale Local Speculative Decoding (MuLo-SD), a novel framework that combines multi-resolution drafting with spatially informed verification to accelerate AR image generation. Our method leverages a low-resolution drafter paired with an up-sampling step to propose candidate image tokens, which are then verified in parallel by a high-resolution target model. Crucially, we incorporate a local rejection and resampling mechanism, enabling efficient correction of draft errors by focusing on spatial neighborhoods rather than raster-scan resampling after the first rejection. When integrated with parallel decoding resampling, MuLo-SD achieves substantial speedups -- up to $\mathbf{5\times}$ -- outperforming both speculative decoding and parallel decoding baselines in terms of acceleration, while maintaining comparable semantic alignment and perceptual quality. These results are validated using GenEval, DPG-Bench, and FID/HPSv2 on the MS-COCO 5k validation split. Extensive ablations highlight the impact of up-sampling design, probability pooling, and local rejection and resampling with neighborhood expansion. Our approach sets a new state-of-the-art in speculative decoding for image synthesis, bridging the gap between efficiency and fidelity. Project page is available at https://qualcomm-ai-research.github.io/mulo-sd-webpage/ .

1. Introduction

Autoregressive image generation faces high latency as image resolution increases, while adapting next-scale generation to next-token models remains difficult. MULO-SD combines multi-scale drafting, local verification, and parallel decoding; it reports up to 5× speedup over speculative- and parallel-decoding baselines while maintaining comparable alignment and perceptual quality.

1. Introduction

  • Unified MLLMs combine language and vision generation and understanding, and tend to outperform diffusion models on text-to-image alignment and semantic understanding.The supplied introduction also notes their performance on complex prompts and knowledge-driven generation tasks.Source: §1 · p. 1
  • AR decoding has high inference latency, especially for large models and high-resolution images, whose token counts grow quadratically with resolution.Even modest 1024p outputs can involve thousands of tokens.Source: §1 · p. 1
  • Next-scale prediction can accelerate coarse-to-fine generation, but its objective differs from next-token prediction and complicates adaptation to unified MLLMs.The paper focuses on acceleration under the next-token prediction objective.Source: §1 · pp. 1–2
  • MULO-SD drafts low-resolution image tokens, upsamples them for high-resolution AR verification, and resamples rejected tokens within local neighborhoods.It also incorporates parallel decoding at drafting and resampling stages to reduce sequential steps without sacrificing generated-token quality.Source: §1 · p. 2
  • Up to 5× speedup over EAGLE-2, LANTERN, and ZipAR, while maintaining comparable semantic alignment and perceptual quality.Results are validated on GenEval, DPG-Bench, and FID/HPSv2 using the MS-COCO 2017 5k validation split.Source: §1 · p. 2

2. Related art

Prior work accelerates autoregressive image generation through speculative decoding, multi-scale sampling, or spatial locality, but these approaches leave gaps in image-specific verification or compatibility with next-token frameworks. MULO-SD combines multi-scale drafting with local resampling and can pair that approach with parallel decoding.

2. Related art

  • Text speculative-decoding methods do not generalize to images; LANTERN adapts draft-and-verify to image synthesis using relaxed acceptance for latent-token ambiguity.LANTERN reports 1.75–1.82× speedups over greedy decoding on LlamaGen.Source: §2 · p. 2
  • MULO-SD extends image speculative decoding by leveraging a multi-scale prior, unlike LANTERN’s focus on token ambiguity..Source: §2 · p. 2
  • Multi-scale AR models use coarse-to-fine generation, but their bespoke sampling schedules hinder integration with next-token prediction frameworks and unified MLLMs.MULO-SD instead injects coarse-to-fine structure into drafting for next-token prediction models.Source: §2 · pp. 2–3
  • ZipAR uses spatial adjacency for inter-row parallel decoding, while LPD separates context and query tokens for parallel, arbitrary-order sampling but requires full model retraining.Both approaches exploit spatial coherence to improve image-generation efficiency.Source: §2 · p. 3
  • MULO-SD resamples rejected tokens in local neighborhoods rather than raster-scan order, and parallel decoding can be combined with its drafting and resampling stages.The combined approach pairs locality-driven sampling with target-model verification.Source: §2 · p. 3

3. Method

MuLo-SD combines low-resolution drafting and high-resolution parallel verification with spatially local correction. It further parallelizes drafting and resampling by processing rejection islands concurrently.

3.1. Preliminaries

  • Speculative decoding uses lightweight sequential drafts for parallel target verification; LANTERN pools nearest-codebook probability to improve acceptance under a TVD bound.Visual-token ambiguity lowers acceptance, and the TVD bound controls distributional drift.Source: §3.1 · p. 3

3.2. Multi-Scale Drafting

  • The low-resolution drafter emits full rows, which are upsampled by r^2 in sequence length for parallel high-resolution verification; verified tokens are downsampled as the next draft prefix.The drafter has the target model’s capacity, so speedup relies on fewer function evaluations and the sequence-size gap; sequential drafting remains a bottleneck.Source: §3.2 · pp. 3–4

3.3. Local Verification

  • Because raster-scan rejection after the first failure produced low acceptance and negligible speedup, MuLo-SD uses pooled neighborhood probability and locally expands rejected positions before target resampling.Higher τ more closely approximates the target but slows inference; expansion excludes positions before the first rejection, and omitting it degrades performance.Source: §3.3 · pp. 4–5

3.4. Integration with Parallel Decoding

  • Parallel decoding accelerates low-resolution drafting, while 8-connected rejection islands are resampled concurrently using accepted tokens as context and ZipAR within each island.The pipeline combines parallel drafting, batched verification, and islandwise parallel resampling for end-to-end latency reduction.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate MULO-SD across image resolutions, model settings, and decoding baselines. At 1024p it achieves 4–5× faster generation with a slight metric drop, while ablations examine its upsampling, verification, and resampling choices.

4.1. Implementation Details; 4.2. Setting

  • Experiments use 256p drafts with target-resolution verifiers, compare learned latent-space and training-free pixel-space upsamplers, and evaluate speed, alignment, and perceptual quality.Evaluations include GenEval, DPG-Bench, FID, and HPSv2; latency is measured on an A100 at batch size one.Source: §4.1 · p. 5, §4.2 · p. 5, Table 1 · p. 6

4.3. Main Results

  • At 1024p, MULO-SD achieves 4–5× faster end-to-end generation than standard sampling with a slight metric drop, while visual comparisons report quality comparable to LANTERN and higher speedups.The reported table compares speedup with GenEval, DPG-Bench, FID, and HPSv2 on the MS-COCO 5k validation split.Source: Table 1 · p. 6, §4.3 · p. 7, Figure 4 · p. 7
  • EAGLE-2 often has negative speedups on image data, whereas LANTERN improves latency with small metric degradation; MULO-SD ranks best overall when matched to LANTERN’s GenEval scores.The paper attributes lower LANTERN speedups on Tar than LlamaGen to Tar’s stronger model distribution, which is harder to approximate.Source: §4.3 · pp. 6–7

4.4. Ablation Studies

  • Pixel-space reconstruction losses improve fidelity over token classification, while the adopted off-the-shelf upsampler is training-free; codebook probability pooling brings only modest gains over no pooling.Pooling improves acceptance and stabilizes performance particularly beyond 1.2× speedup, but behaves similarly to the acceptance threshold.Source: §4.4 · pp. 7–8
  • Local verification with neighborhood expansion corrects rejected tokens and surrounding context, while parallel decoding adds consistent latency gains at matched thresholds with negligible metric changes.The comparison includes raster-scan rejection and naive local resampling without context modification.Source: §4.4 · p. 8

4.5. Discussion and Limitations

  • The framework requires separate low-resolution drafter and high-resolution verifier checkpoints, whose weights and KV-caches can be prohibitive on memory-constrained devices.Independently trained or architecturally different models can also reduce acceptance rates and multi-scale gains.Source: §4.5 · p. 8

5. Conclusion

MULO-SD combines multi-scale drafting and local verification to accelerate autoregressive image generation. Experiments report strong semantic alignment and perceptual quality, with parallel decoding producing the best speed–quality trade-off among existing methods.

5. Conclusion

  • Combining low-resolution drafting, upsampling, and local verification accelerates autoregressive image generation while maintaining strong semantic alignment and perceptual quality.With parallel decoding, MULO-SD delivers the best speed–quality trade-off among existing methods.Source: §5 · p. 8
  • MuLo-SD consistently outperforms speculative decoding baselines EAGLE-2 and LANTERN and parallel decoding method ZipAR across Tar-1.5/7B at 512p and 1024p.Ablations support the multi-scale design, probability pooling, and local verification and expansion mechanisms.Source: §5 · p. 8
  • MuLo-SD integrates with next-token prediction objectives and unified MLLMs; extending the framework to video generation and other multimodal tasks is future work.The authors describe it as a practical and scalable solution for high-resolution image synthesis.Source: §5 · p. 8

6. Primer on Tar

Tar uses an MLLM to condition an autoregressive generative detokenizer, which produces VQ-VAE tokens for image decoding. The detokenizer sequence grows from 256 tokens at 256p to 4096 at 1024p, making the highest-resolution setting the most suitable for accelerating visual-token generation.

6. Primer on Tar

  • Tar comprises an MLLM that generates a conditioning sequence and a generative detokenizer that maps it to VQ-VAE tokens for pixel-space decoding.The MLLM is fine-tuned from QwenVL and predicts visual tokens at conditioning lengths of 81, 169, or 729.Source: §6 · p. 11
  • VQ-VAE downsamples each spatial dimension 16×, yielding AR-DTok sequence lengths of 256, 1024, and 4096 tokens at 256p, 512p, and 1024p, respectively.AR-DTok is based on LlamaGen and uses the MLLM output as conditioning.Source: §6 · p. 11
  • The MLLM conditioning sequence is sampled with topk = 1200, topp = 0.95, and length 729; AR-DTok uses the full logit distribution with classifier-free guidance scale 4.0.Both stages use logits temperature 1.0; the negative prompt is an empty sequence.Source: §6 · p. 11

7. MULO-SD

MULO-SD drafts at lower resolution, upsamples candidate tokens, and verifies them with a high-resolution target model. Rejections trigger local neighborhood resampling, while parallel decoding can reduce the latency of drafting and resampling.

7. MULO-SD

  • The up- and down-samplers are lightweight residual convolutional networks using pixel shuffle, trained progressively for the 2× and 4× settings.For 2×, each module has about 20M parameters and is trained on LAION-COCO-Aesthetic for 150k steps with MSE, LPIPS, commitment, and discriminator losses.Source: §7 · p. 11
  • MULO-SD downsamples the target prefix, samples a draft sequence at lower resolution, upsamples it, and parallel-verifies candidate tokens with the target model.The drafter is AR-DTok at 256p, fixed across experiments.Source: §7 · pp. 11–12
  • Rejected tokens expand to a local neighborhood of radius l, and the target model resamples the rejected positions instead of restarting raster-scan resampling after the first rejection.The algorithm sorts the expanded rejection indices before sequential target-model resampling.Source: §7 · p. 12
  • The drafter cost is fixed and can bottleneck 512p generation, but becomes negligible at higher resolutions; the added overhead is about 5% at 512p and 3% at 1024p.The empirical latency analysis generates 500 MS-COCO prompts on one NVIDIA A100 GPU at batch size 1.Source: §7 · p. 13
  • Parallel decoding at the drafter and target-sampling stages markedly reduces their latency impact and yields a larger overall speedup.Sampling from the drafter or target model still dominates the latency budget.Source: §7 · p. 13

8. Experiments

The experiments compare MuLo-SD with ZipAR, EAGLE-2, and LANTERN across semantic-alignment and perceptual-quality metrics. MuLo-SD leads the speculative methods on semantic alignment, while its perceptual-quality trade-offs and the local-radius ablation depend on the setting.

Qualitative Evaluation

  • Across 512p and 1024p, acceleration slightly reduces perceptual quality while leaving semantic alignment mostly unaltered; τ = 1e−4 appears to offer the best speed–quality trade-off.The τ = 1e−5 setting gives the greatest speedup but largest quality degradation, whereas τ = 1e−3 is closest to the original and slower for complex prompts.Source: §8 · p. 13

Additional Ablation

  • At 256p →512p, local expansion radius l = 3 gives the best GenEval boost across the 1–1.5× speedup range; l = 1 follows, while l = 5 lags.The authors expect optimal radius to depend on resolution and use l = 3 at 1024p because they lacked resources and time for a higher-resolution ablation.Source: §8 · p. 14, Figure 9 · p. 15

Discussion on LANTERN

  • The LANTERN-style drafter uses a transformer layer trained with next-token cross-entropy and L1 regression on target hidden states.Training uses 100k LAION-COCO-Aesthetic samples and reserves 1k for evaluation; the paper notes LANTERN does not specify its drafter-training dataset.Source: §8 · p. 14

Quantitative Evaluation

  • ZipAR dominates the evaluated methods overall; MuLo-SD dominates EAGLE-2 and LANTERN on semantic-alignment metrics, while FID tends to suffer and HPSv2 is slightly better.Figure 8 plots GenEval and DPG-Bench alignment metrics and FID and HPSv2 quality metrics on MS-COCO 2017 Val 5k while sweeping acceptance threshold τ.Source: §8 · p. 13, Figure 8 · p. 15

Drafter Evaluation

  • The Tar drafter reaches Top-1 accuracy 0.12 at 512p and 0.22 at 1024p; comparison with LlamaGen is not strictly valid because test sets differ.The authors attribute the lower 512p score than LlamaGen's reported 0.38 partly to Tar being a stronger model, while noting the test-set difference.Source: §8 · p. 15
Loading 2601.05149v2…