PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Training with Foresight: Multi-Token Auxiliary Supervision for Autoregressive Image Generation

Guo Niu, Xiongfei Yao, Teng Wang, Nannan Zhu

arXiv:2608.25386v1cs.CV

TL;DR

Traditional next-token prediction for autoregressive image generation provides sparse supervision, weakly constrained token representations, and costly dense training computation. MTAR addresses these issues with multi-token prediction, token-level contrastive regularization, and semantic dropping, achieving better generation quality and training efficiency than LlamaGen on ImageNet. Its training-only components leave autoregressive inference unchanged and remain effective under smaller training budgets.

  • Problem

    Traditional next-token prediction provides sparse supervision, lacks explicit constraints on hidden token representations, and performs dense computation despite uneven token importance.

  • Method

    MTAR combines multi-token prediction for denser supervision, token-level contrastive regularization for more discriminative representations, and semantic dropping for more efficient training computation.

  • Results

    MTAR improves FID from 3.80 to 2.85 and makes training 39% faster than LlamaGen under the same number of ImageNet training iterations.

  • Takeaways & Limitations

    MTAR achieves a better trade-off between generation quality and training efficiency, while remaining effective with smaller training budgets and leaving autoregressive inference unchanged.

  • Takeaways & Limitations

    TCR can in principle include a small number of false negatives among its in-batch negatives.

Abstract

from arXiv · show

Autoregressive (AR) image generation has shown strong potential for scalable high-fidelity synthesis by modeling images as discrete token sequences. However, traditional next token prediction (NTP) continues to suffer from sparse and myopic supervision, insufficiently discriminative representations, and high training cost caused by dense computation over the full token sequence. To address these issues, we propose multi-token autoregressive (MTAR), a unified training framework that improves autoregressive image generation from three aspects: prediction objectives, representation regularization, and training efficiency. Specifically, MTAR introduces multi-token prediction (MTP) to alleviate the sparsity and myopia of traditional NTP by imposing joint supervision on multiple future tokens; employs token-level contrastive regularization (TCR) to explicitly enhance the separability of sampled token representations and thereby improve representation discriminability; and incorporates semantic dropping (SD) as a semantics-aware training acceleration strategy to reduce redundant computation on low-information tokens while preserving informative learning signals. All three components are applied only during training and introduce no additional overhead during autoregressive inference. On ImageNet, MTAR achieves a better balance between generation quality and training efficiency. Compared with LlamaGen, MTAR achieves up to 0.95 lower FID and 39\% faster training. Moreover, even with only 1/3 of the training iterations, it still attains performance comparable to or better than the baseline, substantially reducing training time.

1 Introduction

MTAR addresses sparse supervision, weak token representations, and expensive full-sequence training in autoregressive image generation through MTP, TCR, and SD. On ImageNet, it improves FID and training efficiency relative to LlamaGen.

  • Motivation: Autoregressive image generation models images as token sequences, but conventional NTP predicts only the next token from the generated prefix.This creates short-range supervision over a flattened sequence.
  • Limitations: Sparse single-step supervision, unconstrained hidden representations, and dense computation over unevenly informative tokens limit training quality and efficiency.These limitations can increase risks such as repetitive textures and representation collapse.
  • Method: MTP jointly supervises multiple future tokens to provide denser, more forward-looking signals and improve contextual modeling.Its supervision is designed to better exploit image geometry.
  • Method: SD uses offline patch-importance scores from an external vision encoder to retain salient patches and reduce redundant training computation.The strategy aims to preserve informative learning signals while lowering training cost.
  • Results: 39% faster training accompanies an FID improvement from 3.80 to 2.85 under the same training iterations as LlamaGen.With one-third of the iterations, MTAR remains comparable to or better than the baseline.
  • Contributions: MTAR combines prediction supervision, representation regularization, and training efficiency improvements in a unified framework.The framework’s stated contributions include MTP, TCR, and SD.

2 Related Work

Prior autoregressive visual methods model images through token prediction, while multi-token prediction has mainly been developed for one-dimensional language sequences. MTAR adapts auxiliary targets to the two-dimensional spatial structure of images.

  • Autoregressive Visual Generation: Autoregressive visual generation commonly predicts image tokens in raster order, while other methods use masked, grouped, or bidirectional prediction schemes.LlamaGen is presented as a representative raster-order autoregressive model.
  • MTAR: MTAR introduces an additional MTP head alongside the original NTP head to supervise a spatially lower token.The framework overview also depicts TCR’s matched-pair attraction and different-token separation.
  • Multi-Token Prediction: Language-modeling MTP jointly predicts future tokens and can support parallel decoding or inference acceleration.Its benefits have been reported to become more pronounced at larger model scales.
  • Multi-Token Prediction: Visual MTP cannot directly copy language-modeling designs because image tokens originate from a two-dimensional spatial grid.Auxiliary targets should account for spatial locality and neighborhood structure rather than only raster-order offsets.

3 MTAR Framework

MTAR combines multi-token supervision, token-level contrastive regularization, and semantic dropping to improve autoregressive image-generation training across prediction, representations, and efficiency.

  • 3.1 Multi-Token Prediction: MTP adds auxiliary heads that jointly supervise future tokens at different raster-order offsets, providing denser signals and better alignment with image geometry.The auxiliary objective aggregates supervision from all auxiliary MTP heads, while the NTP head handles the immediate next-token prediction.
  • 3.1 Multi-Token Prediction: During inference, auxiliary MTP heads are discarded, so traditional autoregressive decoding uses only the NTP head.This keeps the additional multi-token supervision confined to training.
  • 3.2 Token-Level Contrastive Regularization: TCR samples image-token features, creates two independently dropout-perturbed views, and projects them through a shared MLP with ℓ2 normalization.The sampled features exclude the condition token and are drawn from the B × N image-token features in a batch.
  • 3.2 Token-Level Contrastive Regularization: TCR uses matched tokens across views as positives and remaining sampled tokens as in-batch negatives in a one-sided InfoNCE objective.The cross-view similarity uses a temperature parameter, with diagonal terms matching the same token and off-diagonal terms comparing other sampled tokens.
  • 3.3 Semantic Dropping: SD prioritizes semantically salient patches by offline scoring, retaining M = ⌊N(1 − r)⌋ tokens and processing them as a spatially ordered compressed sequence.Retained tokens receive 2D RoPE encodings from their original coordinates, and NTP, MTP, and TCR operate on the compressed sequence during training.

4 Experiments

Experiments evaluate MTAR on ImageNet 256 × 256, comparing generation quality, training efficiency, component choices, and qualitative outputs against LlamaGen. Results show improved FID and speed, with semantic dropping and selected MTP, TCR, and SD configurations supporting the quality–efficiency trade-off.

  • Main Results: MTAR-L achieves the best result among autoregressive models of similar size.
  • Main Results: MTAR improves FID by 0.96 for MTAR-B and 0.95 for MTAR-L while achieving training speedups of 1.27× and 1.39× over LlamaGen.
  • Training Efficiency: MTAR remains superior under smaller budgets, with MTAR-B surpassing LlamaGen-B at about one-third of the training iterations.
  • Semantic Dropping: DINOv3-guided semantic dropping preserves semantic content and foreground structure better than SigLIP2 when retaining the top-50% patches.
  • Ablation Studies: Sampling 2,048 tokens for TCR yields the best overall performance, while larger sampling sizes increase FID to 19.83 and 20.70.
  • Semantic Dropping: The 80%:20% SD schedule achieves FID 18.65 with a 1.60× speedup, balancing quality and efficiency better than shorter recovery schedules.

5 Conclusion

MTAR improves autoregressive image generation through denser supervision, more discriminative token representations, and more efficient computation while leaving inference unchanged. ImageNet experiments show a better generation-quality and training-efficiency trade-off than LlamaGen, including under smaller training budgets.

  • MTAR combines MTP, TCR, and SD to improve supervision, token representations, and training efficiency, respectively.
  • All three components are applied only during training and leave the autoregressive inference pipeline unchanged.
  • ImageNet 256 × 256 experiments show better FID and substantially faster training than LlamaGen, including under smaller training budgets.

Overview

The supplementary material organizes additional implementation details, ablations, analyses, and generated samples across Sections A–E.

  • Section A provides implementation and hyper-parameter details.
  • Section B presents additional ablations on SD schedules, transfer to RAR, MTP-head optimization, and CFG sensitivity.
  • Section C analyzes token discriminability, potential false negatives, and Semantic Dropping behavior.
  • Sections D and E provide additional MTAR samples and comparisons with LlamaGen samples.

A Implementation Details

The supplementary material summarizes the training and sampling hyper-parameters used for MTAR in Table 10.

  • Table 10 summarizes the training and sampling hyper-parameters used for MTAR.

B Additional Ablation Studies

The ablations examine SD schedules, MTP transfer beyond LlamaGen, MTP-head configurations, and CFG sensitivity. They identify a practical SD schedule and show that auxiliary supervision generalizes while spatially aligned heads are most effective.

  • B.2 Three-Stage SD Schedules: 80%:20% SD offers a better overall quality–efficiency trade-off than the three-stage alternatives, despite 60%:5%:35% achieving the best three-stage FID.The three stages use patch-drop rates of 50%, 25%, and 0%; the final model adopts the simpler two-stage schedule.
  • B.3 Generalization to RAR: 0.88 lower FID: MTP(B) improves RAR-B from 18.70 to 17.82, showing transfer beyond the primary LlamaGen baseline.The comparison uses the official RAR implementation with the LlamaGen VQGAN tokenizer for consistency.
  • B.4 MTP-Head Configurations: MTP (R1, B) achieves the lowest primary autoregressive loss among evaluated prediction sets.The results indicate that supervision from the spatially lower token is particularly effective.
  • B.4 MTP-Head Configurations: Adding more auxiliary heads does not consistently improve optimization, making spatial alignment more important than simply increasing prediction targets.The comparison motivates limiting auxiliary heads to avoid excessive interference with the main autoregressive objective.
  • B.5 CFG Sensitivity: 2.00 guidance scale: the CFG sweep reports FID values of 4.94, 4.50, and 4.85 at scales 1.75, 2.00, and 2.25, respectively.The MTAR-B model trained for 300 epochs uses guidance scale 2.00.

C.1 Token Discriminability

The token-discriminability analysis evaluates TCR through linear probing, local neighborhood purity, and similarity statistics. MTAR-L produces more separable token representations, while sampling and Semantic Dropping reduce potential contrastive false negatives.

  • Linear Probing: 3.10%–8.76% gains: MTAR-L with TCR consistently outperforms LlamaGen-L in token-level linear probing from Top-1000 to Top-50.The larger gains at smaller N indicate particularly stronger representations for frequent visual tokens.
  • kNN Purity: 17.54% kNN purity: MTAR-L exceeds LlamaGen-L’s 15.78% by 1.76% absolute with k=5.The analysis measures the proportion of five nearest neighbors sharing the same VQ token ID.
  • Potential False Negatives: 2.01% of token pairs exceed cosine similarity 0.9 and 8.73% exceed 0.8, indicating highly similar within-batch pairs are uncommon.The statistics are computed from DINOv3 features extracted from 10,000 ImageNet training images.
  • Potential False Negatives: Randomly sampling K tokens and discarding 50% of patch tokens through Semantic Dropping further reduces the probability of false negatives in contrastive learning.The procedure combines token sampling with semantics-aware patch dropping.
  • Semantic Dropping: 50% token retention: Semantic Dropping removes substantial background redundancy while preserving primary foreground structures in representative texture-rich samples.Figure 9 shows examples from the picket fence and spider web classes before and after SD.

D Additional Generated Samples

The supplementary figures present MTAR-L samples for four ImageNet classes and compare MTAR with LlamaGen on uncurated and class-conditional ImageNet images at 256 × 256 resolution.

  • Class-Conditional Samples: MTAR-L generated samples are shown for ImageNet class ID 269, wolf.Figure 10 provides class-specific examples.
  • Class-Conditional Samples: MTAR-L generated samples are shown for ImageNet class ID 323, butterfly.Figure 11 provides class-specific examples.
  • Class-Conditional Samples: MTAR-L generated samples are shown for ImageNet class ID 90, lorikeet.Figure 12 provides class-specific examples.
  • Class-Conditional Samples: MTAR-L generated samples are shown for ImageNet class ID 979, valley.Figure 13 provides class-specific examples.
  • Baseline Comparisons: Uncurated random samples from LlamaGen and MTAR are compared.Figure 14 provides the uncurated comparison.
  • Baseline Comparisons: Class-conditional samples from LlamaGen-L and MTAR-L are compared on ImageNet 256 × 256.Figures 15 and 16 provide the class-conditional comparisons.
Loading 2608.25386v1…