PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlocking Any-Order Generation in Pretrained Autoregressive Image Models

Rishav Pramanik, Marco Pedersoli, Zhaozheng Yin

arXiv:2610.39293v1cs.CVcs.LG

TL;DR

Fixed raster-scan ordering limits autoregressive image models to one factorization, restricting conditioning on non-contiguous context. The paper adapts pretrained models with future-aware positional embeddings and an ordering curriculum. The adapted models outperform MaskGIT on ImageNet inpainting and outpainting, with a measured quality-flexibility trade-off.

  • Problem

    Fixed raster-scan ordering hardcodes one factorization and leaves autoregressive image models unable to condition on non-contiguous context at inference.

    Source: §1 · p. 1

  • Method

    The recipe adapts pretrained raster-scan models using future-aware positional embeddings to identify target positions and an ordering curriculum from raster-scan to random permutations.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    FID 6.21 vs. 10.44 on ImageNet inpainting and FID 5.84 vs. 12.17 on outpainting; adapted models are competitive with MAR.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Inpainting, outpainting, and editing are accessible from existing autoregressive models while retaining the standard AR inference stack.

    Source: §1 · p. 3

    Experiments cover class-conditional ImageNet generation at 256 × 256; extension to text-to-image AR models and higher resolutions remains untested.

    Source: §5 · p. 10

Abstract

from arXiv · show

Autoregressive (AR) image generators are powerful in principle but inflexible in practice: training with a fixed raster-scan order hardcodes a single factorization of the image distribution into the model, leaving it unable to inpaint, outpaint, or edit at inference time. Existing solutions, including the modern discrete-diffusion family, train from scratch on a mixture of token orderings, paying roughly 3x the compute of standard raster-scan training; discrete diffusion further sacrifices compatibility with the AR inference ecosystem (KV caching, vLLM) by construction. We argue that this cost is unnecessary. A pretrained raster-scan AR model has already learned the joint distribution over image tokens; what it lacks is not knowledge but the ability to express that knowledge through non-sequential conditional pathways. We propose a recipe for adapting pretrained raster-scan models into any-order generators using two components: future-aware positional embeddings that resolve a target-position identifiability problem at negligible overhead, and an ordering curriculum that traces a stable optimization path from the pretrained solution to a good any-order basin. Applied to four base models across two architectures and two scales (LlamaGen-L/XL, RAR-L/XL), the recipe substantially outperforms MaskGIT on ImageNet inpainting (FID 6.21 vs. 10.44) and outpainting, is competitive with MAR-L, trades a 1.2-1.9 increase in class-conditional FID for this flexibility, and costs 13-33% additional compute beyond pretraining, while leaving the standard AR inference stack untouched. Inpainting, outpainting, and editing, our results suggest, need not be added to autoregressive image models. They can be unlocked from the ones that exist. Code is available at https://github.com/Ordered-AR/unlockedadaptation.

1 Introduction

Fixed raster-scan order prevents standard autoregressive image models from conditioning on non-contiguous context. The paper adapts pretrained models for any-order generation, enabling inpainting, outpainting, and editing while retaining the autoregressive inference stack.

1 Introduction

  • A fixed raster-scan order hardcodes one factorization of the image distribution, preventing conditioning on non-contiguous context at inference.Existing from-scratch any-order methods require roughly 3× the compute of raster-scan training, while discrete diffusion falls outside the standard AR inference ecosystem.Source: §1 · pp. 1–2
  • Any-order AR predicts any subset of image tokens from any other subset, supporting inpainting, outpainting, and region-conditioned editing with the same weights.The adapted model varies which tokens are observed at inference, without task-specific training, sampler modification, or auxiliary networks.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The approach starts from pretrained raster-scan models because they already encode image statistics and structure; adaptation adds non-sequential conditional pathways rather than relearning the distribution from scratch.The paper hypothesizes that raster-trained representations are sufficiently order-agnostic for light adaptation.Source: §1 · p. 2
  • The recipe combines future-aware positional embeddings, which identify the next target, with an ordering curriculum that transitions from raster-scan to random permutations.Both components are necessary: removing either causes divergence or a noticeable drop in generation quality.Source: §1 · pp. 2–3
  • On ImageNet, the adapted model substantially outperforms MaskGIT on inpainting (FID 6.21 vs. 10.44) and outpainting (5.84 vs. 12.17), and is competitive with MAR, despite no training on either task.Source: §1 · p. 3

2 Related Work

Prior work explored autoregressive and non-autoregressive routes to any-order image generation, often with added training or architectural costs. This work adapts pretrained autoregressive models to a new factorization of the same data distribution.

Autoregressive image generation

  • Standard autoregressive image models tokenize images into discrete codes and predict them sequentially, preserving language-model inference features but committing to raster-scan order.The work relaxes that ordering constraint while retaining those inference-time properties.Source: §2 · p. 3

Any-order generation

  • Permutation language modeling and image-domain follow-ups use causal models for arbitrary orderings, while MAR uses a diffusion head and discrete diffusion uses bidirectional mask-token denoising.Prior any-order approaches can require from-scratch training, architectural overhead, or several times the standard raster-scan training budget [29].Source: §2 · p. 3

Adaptation of pretrained models

  • Unlike task- or domain-focused adaptation, this work targets a new factorization of the same data distribution; its results suggest pretrained representations transfer with light adaptation.ControlNet adds a branch for one fixed control signal, whereas this approach supports arbitrary observed-token subsets.Source: §2 · p. 3

3 Method

The method adapts a pretrained raster-scan AR model to varied token orders by resolving target-position identifiability and using an ordering curriculum to reach an any-order solution. A learned target-position embedding and task-specified generation orders support conditional generation while preserving the standard single-stream inference stack.

3 Method

  • The method continues raster-scan pretraining under a permutation-distributed likelihood, addressing target identifiability and the difficulty of reaching an any-order solution from pretrained initialization.The two mechanisms address orthogonal problems: identifiability makes the loss well-posed, while the curriculum makes it reachable from the pretrained initialization.Source: §3 · p. 4

3.1 Future-Aware Positional Embeddings

  • At each step, a learned embedding for the next target position is added at the input layer, disambiguating which spatial token the model must predict.The embedding table is shared across sequence positions and permutations; a learned null embedding is used for the final token.Source: §3.1 · p. 4
  • Using spatial indices σ(t) for positional operations preserves pretrained spatial-relational priors, while the additive table leaves the single-stream architecture and KV caching intact.The embedding adds (n + 1) · d parameters and an elementwise sum.Source: §3.1 · pp. 4–5

3.2 Ordering Curriculum

  • The ordering curriculum mixes raster-scan and uniformly random permutations, linearly increasing the random-order share until training uses fully random permutations.At α = 0, adaptation is continued raster-scan training; at α = 1, examples use uniformly random permutations.Source: §3.2 · p. 5

3.3 Training Objective

  • The adaptation objective combines the future-aware embedding with curriculum sampling, while otherwise retaining the continued raster-scan training pipeline and pretrained initialization.The embedding table is trained at 10× the base learning rate.Source: §3.3 · p. 5

3.4 Inference

  • At inference, users place observed tokens first in a chosen order and sample the remainder sequentially, enabling conditional generation with unchanged KV caching.The procedure supports inpainting, outpainting, and region-conditioned editing without task-specific training, sampler modification, or auxiliary networks.Source: §3.4 · p. 5

4 Results

The results evaluate class-conditional quality, conditional image completion, recipe ablations, and training and inference efficiency. The adapted models gain any-order capabilities while trading some raster-scan quality for adaptation cost and changes to generation speed.

4.1 Experimental Setup

  • Experiments use ImageNet-1k pretrained LlamaGen and RAR checkpoints, evaluate class-conditional generation on 50,000 samples, and share masks across completion methods.Region completion is evaluated on 5,000 held-out images; the reported evaluation uses no other datasets.Source: §4.1 · pp. 5–6

4.2 Class-Conditional Image Generation

  • Across four base models, FID increases by 1.17–1.93 relative to the pretrained checkpoint, while IS, Precision, and Recall remain comparable.The models cover LlamaGen and RAR at two scales; the reported FID increase is the trade-off for any-order capability.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Any-Order Capabilities: Inpainting, Outpainting, and Editing

  • Adapted models outperform both MaskGIT schedules on inpainting and outpainting, though Halton narrows the gap; against MAR-L, results are competitive, with lower FID but higher outpainting Precision and Recall for MAR-L.The qualitative comparison reports sharper structure and more coherent lighting for inpainting, and plausible outpainting that preserves identity and palette.Source: Figure 3 · p. 7, §4.3 · p. 8, Table 3 · p. 8
  • On held-out region completion, the model is best on LPIPS, PSNR, and SSIM; these reference-based metrics do not measure instruction following or identity preservation.LPIPS uses random regions covering 30–50% of 5,000 images; PSNR and SSIM are measured over generated regions in center-mask examples.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 Ablations

  • Removing future-aware embeddings makes training loss rise, removing the curriculum weakens fidelity, and stable convergence requires a 10× embedding learning-rate multiplier.Teacher regularization also raises LlamaGen-L FID from 4.24 to 9.62.Source: §4.4 · p. 8

4.5 Training and Inference Efficiency

  • Adaptation costs 1.13–1.33× the raster-scan pretraining budget across base models, versus about 3× for from-scratch any-order training, while retaining standard AR inference compatibility.On LlamaGen-XL, parallel decoding uses 9× fewer forward passes at a cost of 1.12 FID.Source: §4.5 · p. 9, Table 6 · p. 9, Table 7 · p. 9

5 Conclusion

The paper adapts pretrained raster-scan autoregressive image generators into any-order generators, trading a modest increase in class-conditional FID for flexible generation. The experiments show gains over MaskGIT on inpainting and outpainting, while remaining within a limited evaluation scope.

5 Conclusion

  • FID increases by 1.17–1.93 relative to pretrained checkpoints as adapted models outperform MaskGIT on inpainting and outpainting and are competitive with MAR-L.The reported MaskGIT comparisons are inpainting FID 6.21 vs. 10.44 and outpainting FID 5.84 vs. 12.17.Source: §5 · p. 10
  • The recipe combines future-aware positional embeddings to resolve target identifiability with an ordering curriculum that traces a stable path from the pretrained solution to an any-order basin.The reported experiments span four base models across two architectures and two scales: LlamaGen-L/XL and RAR-L/XL.Source: §5 · p. 10
  • The authors argue that pretrained raster-scan representations can support other token orderings, and frame their contribution as reusing causal checkpoints rather than replacing diffusion or flow matching.The chain rule guarantees equivalent factorizations of the true joint distribution, while the authors describe the empirical applicability of this principle as suggested by their results.Source: §5 · p. 10
  • Experiments cover class-conditional ImageNet generation at 256 × 256; extension to text-to-image AR models and higher resolutions remains untested.Editing evaluation covers region completion only; the random-initialization control matches compute but not schedule.Source: §5 · p. 10

A Extended Motivation

The appendix frames the ordering curriculum as a continuation method that addresses imbalanced gradient magnitudes and high variance when adapting a raster-scan checkpoint. Gradually increasing the random-order contribution is intended to preserve pretrained competence while order-flexible routing develops.

A Extended Motivation

  • At pretrained initialization, loss is small for raster-like permutations but large for random permutations, so uniformly sampling permutations produces imbalanced gradients.The appendix explains that high-loss random permutations dominate the expected gradient and can rapidly modify parameters encoding the raster-scan distribution.Source: A Extended Motivation · p. 13
  • Different random permutations induce different dependency structures and gradient directions, producing high-variance updates that can oscillate and destabilize the pretrained basin.The appendix describes the combined effect of gradient magnitude imbalance and variance as catastrophic forgetting, with raster-scan competence eroded before order-flexible routing develops.Source: A Extended Motivation · p. 14
  • The curriculum gradually increases random-order training so each objective is a small perturbation of the previous one, preserving prior ordering competence as adaptation proceeds.At small α, raster-scan terms dominate; as α increases, the model has partially adapted and random-permutation loss and variance decrease.Source: A Extended Motivation · p. 14
  • The appendix notes that adding future-aware embeddings only at the input layer was sufficient, consistent with their role in disambiguating the prediction target.The authors considered injecting the embedding at every transformer layer but found a single input-layer addition sufficient.Source: A Extended Motivation · p. 14

B Hyperparameters

Adaptation uses each base model’s released setup and resolution, with a linearly ramped curriculum coefficient and model-specific adaptation budgets. The implementation leaves the base architecture, tokenizer, and conditioning unchanged.

B Hyperparameters

  • The curriculum coefficient α ramps linearly from 0 to 1 over the first Sramp epochs, then remains at 1 for the rest of adaptation.The ramp duration is reported per model, and the adaptation budget is set per base model to amortize against its pretraining cost.Source: B Hyperparameters · p. 14
  • Architecture, tokenizer, and conditioning remain unchanged from pretraining, while LlamaGen adds (n + 1) · d future-aware embedding parameters.The added parameters are 0.26M for LlamaGen-L and 0.74M for LlamaGen-XL; RAR loads its existing target-aware positional embedding from its checkpoint.Source: B Hyperparameters · p. 14
  • Global batch size varies with GPU allocation, while gradient accumulation keeps the total effective batch within each base model’s pretraining range.Per-run batch size was chosen to maximize GPU utilization on the allocated node.Source: B Hyperparameters · p. 14

C Training and Compute Details

The appendix details the adaptation hardware, permutation sampling, and evaluation protocol. Runs use data-parallel training and evaluate generation on ImageNet at 256 × 256, with masks matched across inpainting and outpainting comparisons.

C Training and Compute Details

  • LlamaGen-L and RAR-L use RTX PRO 6000 nodes, while LlamaGen-XL and RAR-XL use H200 nodes.LlamaGen-XL adaptation took approximately 64 H200-GPU-hours; per-run GPU-hours were not logged for the other models.Source: C Training and Compute Details · p. 14
  • Table 8 reports adaptation hyperparameters by base model, with notation matching Section 3.The table provides the per-model configuration referenced in the appendix.Source: Table 8 · p. 15
  • RAR-L and RAR-XL follow the original RAR paper’s per-step CFG schedule for a like-for-like comparison.The passage identifies the schedule as following the original RAR paper.Source: C Training and Compute Details · p. 15
  • Adaptation uses standard data-parallel training without tensor or pipeline parallelism, with activation checkpointing for XL variants and bfloat16 mixed precision.Float32 master weights are used for optimizer state.Source: C Training and Compute Details · p. 15
  • At each training step, each example independently samples raster-scan order with probability 1 − α(s) or a fresh uniform random permutation with probability α(s).Permutations are neither pre-sampled nor shared across examples in a batch.Source: C Training and Compute Details · p. 15
  • Evaluation uses 50,000 class-conditional ImageNet samples at 256 × 256 and the corresponding pretrained model’s number of decoding steps.For inpainting and outpainting, methods receive identical masks; FID, IS, and Precision/Recall use full images, while PSNR and SSIM use generated regions.Source: C Training and Compute Details · p. 15

D Adaptation Algorithm

The adaptation loop adds a curriculum over token orders, spatial positional inputs, and future-aware embeddings to standard causal AR fine-tuning. Inference can use arbitrary orders while retaining the standard AR caching path.

D Adaptation Algorithm

  • Each step uses spatial image positions and adds the next target’s future-aware embedding; the final step instead uses a null embedding.The spatial positions are passed to the model’s positional encoding, and the future-aware embedding is added before the transformer pass.Source: D Adaptation Algorithm · pp. 15–16
  • The loop otherwise follows standard causal AR fine-tuning, updating θ at the base learning rate and E and E∅ at 10× that rate.Source: D Adaptation Algorithm · pp. 15–16
  • The curriculum ramps the probability of a uniformly random token permutation from zero to one, using raster-scan order otherwise.The schedule is α(s)=min(s/Sramp,1).Source: D Adaptation Algorithm · p. 16
  • At inference, arbitrary orders—including task-induced orders with observed tokens first—reuse the standard causal transformer’s key-value cache.The future-aware embedding affects only the current step’s input representation.Source: D Adaptation Algorithm · p. 16

E Broader Impacts

The recipe extends existing class-conditional image generators, inheriting their applications and misuse risks without introducing new categories of risk.

E Broader Impacts

  • Because the base models are publicly released and restricted to ImageNet classes, the recipe adds no risk categories beyond those models’ existing capabilities.Source: E Broader Impacts · p. 16

F Licenses for Existing Assets

The paper uses existing model, tokenizer, dataset, and software assets under their stated terms, and released adapted checkpoints inherit the more restrictive base-model license.

F Licenses for Existing Assets

  • LlamaGen checkpoints use the MIT License and RAR checkpoints Apache License 2.0; both permit research use, redistribution, and publication of derived models.Source: F Licenses for Existing Assets · p. 16
  • The generation procedure accepts an arbitrary token order and can place observed tokens first before sampling the remaining tokens.It reuses the KV cache and decodes the resulting tokens with the base-model VQ tokenizer.Source: F Licenses for Existing Assets · p. 17
  • The VQ tokenizers are released alongside their corresponding base models under the same licenses and are used unmodified.Source: F Licenses for Existing Assets · p. 17
  • Adaptation and evaluation use ImageNet under its standard research-use terms; the paper reports metric-level results rather than redistributing imagery.Source: F Licenses for Existing Assets · p. 17
  • Inference benchmarks use vLLM under Apache 2.0, while other software dependencies are used under their standard open-source licenses.Source: F Licenses for Existing Assets · p. 17
  • Released adapted checkpoints inherit the more restrictive license of their base models and will include a model card covering provenance, intended use, and known limitations.Source: F Licenses for Existing Assets · p. 17

Related papers

Loading 2610.39293v1…