Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation

1 Introduction

• Prior semantic-enhancement methods described here rely on pretrained representations, while next-token prediction faces challenges when adapted from language to vision.The paper frames these challenges against next-token prediction’s use as a contextual pre-training approach in NLP.Source: §1 · p. 1
• LlamaGen analysis identifies local and conditional dependence, declining semantic probe accuracy across steps, and token changes under slight view shifts as barriers to visual understanding.Attention favors the conditioning token and spatially adjacent steps; probe accuracy declines after early steps, while slight image perturbations can produce different visual tokens.Source: §1 · p. 2
• ST-AR combines random attention-map token masking with inter-step and inter-view contrastive losses alongside token prediction, while retaining autoregressive sampling.The framework uses the added objectives during training only.Source: §1 · p. 2
• 21.00% to 55.23% linear probing top-1 accuracy for LlamaGen-B and approximately 49% FID improvement over baseline for LlamaGen-XL are reported with ST-AR.ST-AR also boosts LlamaGen-B by 7.82 FID score on class-conditional ImageNet.Source: §1 · p. 2

Autoregressive Image Generation

• Causal autoregressive image methods use raster-order next-token prediction, whereas masked and parallelized methods use bidirectional attention or token-grouping and next-scale strategies.The passage names VQ-GAN and LlamaGen as causal examples, MaskGiT and MAR as masked examples, and VAR, PAR, and NPP as parallelized examples.Source: §2 · p. 3
• Masked and parallelized methods adjust sampling strategies; ST-AR instead targets visual-modality modeling without altering autoregressive sampling.The passage describes this as preserving compatibility with language models.Source: §2 · p. 3

Self-Supervised Learning

• Contrastive learning aligns representations of positive samples, while masked image modeling reconstructs masked image content to learn visual context.ST-AR uses random masking on attention maps and MoCo-like losses to align representations across time steps and views.Source: §2 · p. 3

3.1 Preliminaries

• Images are quantized into discrete token sequences, and the autoregressive model predicts each token from the condition and preceding tokens.The condition can be a class label or text vector.Source: §3.1 · p. 3

3.2 Observations

• Autoregressive models show local and conditional dependence, semantic inconsistency across steps, and visual-token variability across transformed views.Attention favors conditioning and spatially adjacent tokens; probing declines after step 192, while transformations can yield different tokens for the same object.Source: §3.2 · pp. 4–5

3.3 Self-Guided Training for Autoregressive Models

• ST-AR uses a non-trainable teacher with the student’s architecture, updated by exponential moving average, to provide objectives without pretrained representation models.The framework combines reconstruction with inter-step and inter-view contrastive losses.Source: §3.3 · p. 5

3.3.1 Masked Learning for Longer Contexts

• Masked learning applies random masks to student attention maps, rather than input tokens, and aligns student final-layer features with teacher features.The feature distance defaults to cosine distance.Source: §3.3.1 · pp. 5–6

3.3.2 Contrastive Learning for Consistency

• Inter-step contrastive learning uses same-view, different-position teacher features as positives and features from other images as negatives.Training uses augmented image views, a projector on student features, and sampled token positions; the implementation sets the number of views to two.Source: §3.3.2 · p. 6
• Inter-view contrastive learning pairs a student feature with a teacher feature from the same token position in another view of the same image, using other images as negatives.Source: §3.3.2 · p. 6

3.3.3 Training Losses.

• ST-AR combines masked image modeling and two contrastive losses with the conventional next-token prediction objective.The reconstruction and contrastive losses are weighted by α and β.Source: §3.3.3 · p. 6

4.1 Implementation Details

• Experiments use ImageNet-256×256 tokens from LlamaGen's VQGAN and assess linear-probe top-1 accuracy and ADM-suite FID.Training follows LlamaGen's settings but uses 256×256 images rather than its 384×384 training images; inference uses the same sampling strategy as LlamaGen.Source: §4.1 · pp. 7–8

4.2 Main Results

• At 50 ST-AR epochs, LlamaGen-XL's FID falls from 19.42 to 9.81; at 300 epochs it reaches 6.20, outperforming LlamaGen-3B with 4× parameters.Table 2 also reports that ST-AR beats baseline LlamaGen across model sizes and training costs, with performance comparable to other state-of-the-art methods.Source: Table 2 · p. 8, §4.2 · pp. 8–9

4.3 Ablation Studies

• 18.68% to 45.27%: adding all three losses raises LlamaGen-B linear-probing accuracy; each improves probing and generation, with L_view aiding probing more than L_step.The reported loss comparison is shown in Table 3.Source: Table 3 · p. 8, §4.3 · p. 9

4.2 Main Results

• ST-AR raises linear-probing performance across sampling steps, avoids accuracy decline after step 192, and attends more broadly to semantically relevant regions.The attention comparison is against the baseline model.Source: §4.2 · p. 8

4.3 Ablation Studies

• Larger K improves generation, but gains are marginal above K=4, so K=4 is the default.The number-of-steps ablation is reported in Table 6.Source: Table 6 · p. 9, §4.3 · p. 9
• Applying L_step and L_view at the 6th layer, half the network depth, yields the best performance in the contrastive-loss-depth ablation.The comparison is reported in Table 5.Source: Table 5 · p. 9, §4.3 · p. 9
• A mask ratio of 0.35 gives the lowest FID, while higher ratios degrade sFID, indicating harm to low-level spatial-structure learning.The mask-ratio ablation is reported in Table 4.Source: Table 4 · p. 9, §4.3 · p. 9

• ST-AR addresses three challenges to high-level visual semantics with masked image modeling to broaden attention and contrastive learning to maintain consistency across steps and views.Source: §5 · p. 9
• Experiments validate stronger visual understanding and, consequently, improved image-generation quality.Source: §5 · p. 9
• Increased training costs are the work's main limitation, slated for future research.Source: §5 · p. 9