PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Training Dynamics in Scale-wise Autoregressive Generation

Gengze Zhou, Chongjian Ge, Hao Tan, Feng Liu, Yicong Hong

arXiv:2512.06421v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Scale-wise autoregressive models face train–test mismatch and uneven learning difficulty across scales, which can undermine generation quality. SAR uses lightweight self-generated rollouts and contrastive student-forcing supervision to refine pretrained models. It consistently improves FID across model sizes with limited additional compute.

  • Problem

    Scale-wise AR models are trained on ground-truth contexts but must use their own imperfect predictions at inference, while coarse and fine scales face unequal learning demands.

    Source: §1 · p. 2

  • Method

    SAR combines Stagger-Scale Rollout, which exposes training to self-generated latents, with Contrastive Student-Forcing Loss for stable supervision.

    Source: §1 · p. 2, §4.2 · p. 8

  • Results

    SAR yields 5.2% / 2.5% / 3.1% lower FID on 300M, 600M, and 1B parameter models, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SAR provides a practical, scalable post-training approach that improves perceptual quality and robustness with less than 5.5% of pretraining compute.

    Source: §6 · p. 16

    Naïve student-forcing inputs can drift, amplify errors, and destabilize supervision.

    Source: §5.4.2 · p. 15

Abstract

from arXiv · show

Recent advances in autoregressive (AR) generative models have produced increasingly powerful systems for media synthesis. Among them, next-scale prediction has emerged as a popular paradigm, where models generate images in a coarse-to-fine manner. However, scale-wise AR models suffer from exposure bias, which undermines generation quality. We identify two primary causes of this issue: (1) train-test mismatch, where the model must rely on its own imperfect predictions during inference, and (2) imbalance in scale-wise learning difficulty, where certain scales exhibit disproportionately higher optimization complexity. Through a comprehensive analysis of training dynamics, we propose Self-Autoregressive Refinement (SAR) to address these limitations. SAR introduces a Stagger-Scale Rollout (SSR) mechanism that performs lightweight autoregressive rollouts to expose the model to its own intermediate predictions, thereby aligning train-test patterns, and a complementary Contrastive Student-Forcing Loss (CSFL) that provides adequate supervision for self-generated contexts to ensure stable training. Experimental results show that applying SAR to pretrained AR models consistently improves generation quality with minimal computational overhead. For instance, SAR yields a 5.2% FID reduction on FlexVAR-d16 trained on ImageNet 256 within 10 epochs (5 hours on 32xA100 GPUs). Given its efficiency, scalability, and effectiveness, we expect SAR to serve as a reliable post-training method for visual autoregressive generation.

1 Introduction

Scale-wise autoregressive models face both train–test mismatch from teacher forcing and uneven learning difficulty across scales. SAR addresses these issues through lightweight self-generated rollouts and targeted supervision, improving FID across model sizes with limited compute.

Challenges

  • Teacher forcing creates a train–test mismatch because inference conditions on the model’s imperfect predictions, allowing scale-wise errors to accumulate and degrade output quality.Each scale predicts many tokens, so errors propagate through the hierarchy.Source: §1 · p. 2
  • Uniform supervision overlooks that coarse scales must create global structure while fine scales mainly reconstruct details, leaving later scales unable to correct early structural errors.This imbalance makes training uneven and fine-scale refinement suboptimal.Source: §1 · p. 2

SAR

  • SAR uses Stagger-Scale Rollout to expose scales to self-generated latents and Contrastive Student-Forcing Loss to stabilize supervision, requiring only one additional forward pass.It is presented as a lightweight post-training method for pretrained VAR models.Source: §1 · p. 2

Experimental results

  • SAR yields 5.2% / 2.5% / 3.1% lower FID on 300M, 600M, and 1B parameter models, respectively, using 160 A100 GPU hours.The evaluation used ImageNet-256 for 10 epochs, amounting to approximately 5.5% of pretraining cost.Source: §1 · p. 2

2 Related Work

Related work spans discrete-latent autoregression, masked prediction, and scale-wise generation. Scale-wise methods improve parallelism, but their teacher-forced training leaves accumulated errors in model-generated trajectories unaddressed.

Vector-Quantized Latent Modeling

  • VQ-VAE established a two-stage approach that quantizes image latents before modeling their discrete indices with an autoregressive prior such as PixelCNN (Van Den Oord et al., 2017; Van Den Oord et al., 2016).This connected continuous visual data with discrete sequence modeling.Source: §2 · p. 3

Raster-scan Autoregression

Random-scan Generation

Scale-wise Generation

  • VAR replaces next-token with next-scale prediction, reducing inference steps while preserving global consistency, but teacher-forced training provides no exposure to model-generated trajectories or a mechanism for correcting accumulated multi-scale errors (Tian et al., 2024).Subsequent work extends scale-wise generation to text conditioning, faster decoding, alternative tokenizers, and flow-, diffusion-, or bitwise-based prediction.Source: §2 · p. 3

3 Preliminaries

Scale-wise AR predicts progressively finer latent maps conditioned on earlier scales, but conventional supervision and teacher forcing can undermine the intended coarse-to-fine hierarchy. Analyses find that smoothing supervision and adding early-scale computation do not resolve the imbalance.

3.1 Scale-wise Autoregressive Visual Generation

  • Scale-wise AR represents an image with progressively higher-resolution latent maps and predicts each scale conditioned on all preceding scales.Depending on the formulation, latents encode ground-truth maps or residual differences; the model generates a full latent map per step.Source: §3.1 · p. 3
  • Under latent-space supervision, coarse scales receive targets with little semantic structure while the finest scale must reconstruct most details; image-space smoothing instead leaves later scales with mostly sharpening work.Both paths weaken effective coarse-to-fine learning in different ways.Source: Figure 2 · p. 4

3.2 Teacher Forcing in Scale-wise AR

  • Teacher forcing is parallelized by precomputing ground-truth latents and shifting scales to predict the next latent, but inference uses the model’s own predictions, creating a train–test gap.The figure contrasts ideal ground-truth inputs during training with self-generated coarse latents at inference.Source: §3.2 · p. 4, Figure 3 · p. 5

3.3 Diagonosing Scale-wise Autoregression

  • Image-level supervision produces smoother input–target pairs but higher FID scores of 7.5–7.9 versus 3.9 for latent-level supervision.The comparison used FlowAR-B on ImageNet 256×256 and included standard and equivariant VAEs.Source: §3.3 · pp. 5–6, Table 1 · p. 5
  • Teacher-forced predictions refine cleanly from global structure to local detail, but inference can collapse when a poor earliest coarse latent propagates because later scales were trained mainly for super-resolution.Later scales magnify erroneous early predictions rather than repairing their semantics.Source: §3.3 · p. 6
  • Hybrid masked-autoregressive modeling improves coarsest-scale 4×4 FID from 27 to 22 for d=16 and from 26 to 20 for d=20, while full-resolution FID worsens from 3.05 to 5.02 and from 2.41 to 4.74.The later scales consequently function as near-super-resolution refinement rather than exploiting scale-wise AR dependencies.Source: §3.3 · p. 6, Table 2 · p. 6
  • Neither smoothing the supervision path nor allocating more computation to early scales resolves the imbalance; both can shift modeling burden toward the earliest scale and leave later scales overly simplified.The analyses motivate rethinking training objectives and dependency structures beyond supervision design or residual formulations.Source: §3.3 · pp. 5–6

4 Self-Autoregressive Refinement

This section motivates student forcing as a response to train–test discrepancy and scale-wise learning difficulty, then develops SAR as an efficient, stable refinement strategy.

4.1 Naïve Integration of Student Forcing

  • Naïve student-forcing schedules increase training cost and destabilize supervision, and none approaches teacher-forcing stability or quality.The comparison covers student-forcing schedules, including mixtures with teacher forcing.Source: §4.1 · pp. 7–8, Table 3 · p. 8

4.2 SAR: A Stable Student-Forcing Strategy

  • SAR combines two-step Stagger-Scale Rollout (SSR) with Contrastive Student-Forcing Loss (CSFL) for efficient, stable optimization across scales.The proposed strategy uses only two rollout steps and consistent supervision.Source: §4.2 · p. 8

4.2.1 Stagger-Scale Rollout (SSR)

  • SSR uses a controlled, parallelized student-forcing rollout from teacher-forced predictions, exposing the model to self-generated inputs at the cost of one extra NFE.Limiting the rollout to two student-forcing steps keeps intermediate inputs structured; deeper rollouts accumulate noise without additional benefit.Source: §4.2.1 · pp. 8–9

4.2.2 Contrastive Student-Forcing Loss (CSFL)

  • CSFL aligns student-forced predictions with teacher-forced counterparts rather than ground truth, while the teacher-forced branch stabilizes training.The student-forced branch learns to follow the teacher-forced trajectory when conditioned on self-generated latents.Source: §4.2.2 · pp. 9–10

5.1 Implementation Details

The implementation details describe the SAR post-training setup, optimization settings, and comparison metrics.

5.1 Implementation Details

  • SAR continues pretrained FlexVAR checkpoints with 10 additional epochs on ImageNet-1K at 256×256, keeping total training epochs fixed across baselines.The three checkpoints correspond to 170, 240, and 340 prior epochs.Source: §5.1 · p. 10
  • The generative-model comparison covers class-conditional ImageNet 256×256 and reports FID, IS, precision, recall, generation steps, and relative inference time.The table includes models of size ≤1B; inference time is relative to VAR-d30.Source: Table 4 · p. 11
  • Training uses AdamW with β1 = 0.9, β2 = 0.95, weight decay 0.05, and learning rate 1e−4 without warm-up.Experiments use NVIDIA A100 GPUs with 80GB memory.Source: §5.1 · p. 11

5.2 State-of-the-Art Image Generation

SAR improves FlexVAR’s ImageNet generation results across model sizes, converges quickly when initialized from pretrained weights, and offers a favorable throughput–FID trade-off among autoregressive models.

5.2.1 Overall comparison on ImageNet 256×256.

  • FID improves from 3.05 to 2.89 for the 310M model, from 2.41 to 2.35 for the 600M model, and from 2.21 to 2.14 for the 1B model.Across all three scales, SAR improves the FlexVAR baselines and achieves state-of-the-art results among scaling-scan autoregressive models, with parameter sizes ≤1B.Source: §5.2.1 · p. 11

5.2.2 Training Dynamics

  • Within a few epochs, pretrained SAR surpasses the best fully trained FlexVAR performance and converges to the lowest FID among the variants.SAR trained from scratch also descends sharply early and maintains a lower FID across training; by epoch 180, it has a lower FID than FlexVAR.Source: §5.2.2 · p. 12
  • The training analysis concludes that SAR improves sample quality while making student-forcing training in scale-wise autoregression effective and stable.The smoother convergence curve is reported as suggesting improved stability in token prediction during rollouts.Source: §5.2.2 · p. 12

5.2.3 Model Comparison

  • SAR attains the highest throughput among autoregressive models and the lowest FID across autoregressive baselines.The comparison covers diffusion, next-token autoregressive, and next-scale autoregressive model families.Source: §5.2.3 · p. 12

5.3 Quanlitive Results

The qualitative comparisons visualize SAR outputs and compare SAR with FlexVAR from the same initial latent and scale schedule.

5.3.1 Visualization of the Generation Process

  • Both models start from the same 4 × 4 latent and follow identical scale schedules in the generation-process comparison.The visualization shows intermediate outputs at each scale.Source: §5.3.1 · p. 13, Figure 7 · p. 14
  • SAR shows smoother transitions between scales, with fewer abrupt structural or textural changes than FlexVAR.The comparison reports smoother refinement across the examples.Source: §5.3.1 · p. 14, Figure 7 · p. 14
  • When early predictions are distorted or misaligned, SAR corrects errors at later scales, whereas FlexVAR often amplifies them and degrades high-resolution details.The visualizations associate SAR’s student-forcing-aligned training with more stable hierarchical latents and more coherent, robust refinement.Source: §5.3.1 · p. 14

5.4 Ablation Study

The ablations report that stochastic student-prediction sampling improves over argmax and that classifier-free guidance performs best. The rollout visualizations show how errors can grow under naïve student-forcing and how SSR and CSFL keep student-forced trajectories structured and aligned.

5.4.1 Effect of sampling strategy in student-forcing rollout.

  • FID is 2.89 with classifier-free-guidance sampling, which the study reports as the best-performing variant.Stochastic sampling improves FID and recall over deterministic argmax, while classifier-free guidance further enhances image fidelity and overall generation quality.Source: §5.4.1 · p. 14, Table 5 · p. 14

5.4.2 Visualization of Student-Forcing Inputs During Training

  • Deviations introduced at coarse scales typically propagate and enlarge at finer scales, illustrating why naïve student-forcing destabilizes training.The section describes naïve student-forcing as exposing the model to inference-like conditions while suffering from drift, error amplification, and destabilized supervision.Source: §5.4.2 · p. 15
  • SSR starts from teacher-forced predictions, keeping student-forced inputs structurally meaningful so the model can correct local distortions rather than diverge.The visualization reports smaller difference maps for the resulting student-forced latents than for naïve student-forcing.Source: §5.4.2 · pp. 15–16
  • CSFL aligns student-forced latents with the stable teacher-forced trajectory, resolving the supervision inconsistency and train–test gap described for scale-wise autoregression.Together, SSR and CSFL keep self-generated inputs structured while guiding student-forced latents along the predictive trajectory.Source: §5.4.2 · p. 16

6 Conclusion

The conclusion presents SAR as a lightweight post-training framework for reducing train–test mismatch and scale-wise learning imbalance. It reports consistent quality and robustness improvements across model sizes with less than 5.5% of pretraining compute.

6 Conclusion

  • SAR combines Stagger-Scale Rollout for efficient exposure to self-generated latents with Contrastive Student-Forcing Loss for stable refinement.The conclusion describes the framework as aligning training dynamics with inference behavior at minimal additional computational cost.Source: §6 · p. 16
  • Applied to state-of-the-art autoregressive models, SAR consistently improves perceptual quality and robustness across model sizes.The conclusion reports up to 5.2% FID reduction on ImageNet-256 using less than 5.5% of pretraining compute.Source: §6 · p. 16
  • The paper concludes that fine-grained autoregressive refinement is practical and effective, offering a scalable path toward stronger and more reliable visual autoregressive generation.Source: §6 · p. 16
Loading 2512.06421v1…