Source-linked AI summary
Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modeling
Meimingwei Li, Stefan Andreas Baumann, Felix Krause, Björn Ommer
TL;DR
VAR’s parallel within-scale decoding is a mean-field-style approximation that discards spatial dependencies, producing locally incoherent samples despite correct per-token predictions. The Logit Refiner restores those dependencies through sequential intra-scale sampling over frozen backbone features, adding little training cost and improving generation across scales and conditioning settings. Its gains are attributed to joint intra-scale sampling rather than extra capacity or training, while sequential computation remains a practical limitation.
Problem
VAR’s parallel within-scale decoding independently samples tokens, discarding spatial dependencies and causing locally incoherent samples even when per-token predictions are accurate.
Method
The Logit Refiner is a lightweight autoregressive module that sequentially samples within each scale conditioned on frozen VAR backbone features.
Results
Across 310M–2B-parameter backbones and class- and text-conditional generation, the refiner consistently improves generation quality, with a 1.1B-parameter model surpassing a twice-as-large 2B-parameter VAR.
Takeaways & Limitations
Controlled ablations identify joint intra-scale sampling, rather than additional capacity or training, as the critical ingredient for alleviating the mean-field bottleneck.
Takeaways & Limitations
Autoregressive within-scale sampling introduces sequential computation, although selective application at early scales retains most quality gains while reducing overhead.
Abstract
from arXiv · showhide
Visual Autoregressive Models (VAR) generate images through next-scale prediction, producing all tokens within each scale in parallel. We show that this parallel decoding constitutes a mean-field-style approximation that discards spatial dependencies among same-scale tokens, causing locally incoherent samples regardless of backbone capacity -- a limitation of the decoding rule. Addressing this limitation, we introduce the Logit Refiner, a lightweight autoregressive module that restores intra-scale dependencies by sequentially sampling tokens conditioned on frozen backbone features. Adding only ~10% parameters and less than 5% of the base model's training compute, it plugs into any pretrained VAR checkpoint without retraining. Controlled ablations isolate joint intra-scale sampling -- rather than additional capacity or training -- as the critical ingredient. Across backbones from 310M to 2B parameters on class-conditional ImageNet 256x256, the refiner consistently improves generation quality, enabling a 1.1B-parameter model to surpass one twice its size. The approach further generalizes to text-to-image generation, confirming that the mean-field bottleneck persists across VAR variants and is effectively alleviated by our method. Project page: https://compvis.github.io/logit-refiner/
1 Introduction
VAR improves image-generation efficiency through coarse-to-fine next-scale prediction, but parallel within-scale decoding discards spatial dependencies and produces locally incoherent samples. The Logit Refiner restores these dependencies with a lightweight plug-in, improving generation across model scales.
- Motivation: Parallel within-scale decoding makes VAR efficient but can produce locally incoherent samples despite plausible individual predictions.Observed artifacts include mismatched textures, structural discontinuities, and implausible combinations.
- Motivation: The failure comes from a mean-field-style factorization that samples same-scale tokens independently, yielding individually likely but jointly inconsistent combinations.A checkerboard example shows that correct per-token probabilities can still produce invalid global patterns.
- Method: The Logit Refiner sequentially samples tokens within each scale from frozen backbone features, restoring intra-scale dependencies without changing the pretrained VAR backbone.It is designed as a small causal module that captures residual dependencies ignored by independent decoding.
- Method: The plug-in adds approximately 10% parameters and less than 5% extra training compute, while requiring only modest inference overhead.The backbone remains frozen, and the module can be applied to an already pretrained VAR model.
- Results: Across 310M–2B-parameter ImageNet backbones, the refiner consistently improves performance, enabling a 1.1B-parameter model to surpass a 2B-parameter VAR.A matched bidirectional-attention refiner with independent sampling fails to match the autoregressive variant, isolating joint intra-scale sampling as critical.
2 Related Work
Prior autoregressive image-generation work explores token ordering, grouping, backbones, and refinement strategies to balance image quality and efficiency. This work is orthogonal to those changes, correcting the independence assumption within parallel decoding groups.
- Autoregressive Image Generation: Autoregressive image generators have progressed from pixel sequences to discrete-token models with alternative orderings, token grouping, and shared backbones.These directions seek efficient or flexible ways to represent and generate images.
- Refinement and Post-hoc Correction: Unlike prior refinement and decoding modifications, Logit Refiner corrects independence within parallel groups without changing token order or the backbone.Its intervention targets a specific shortcoming of VAR-style models.
- Scale-wise Autoregressive Image Generation: VAR-style methods generate progressively finer token maps while sampling each scale in parallel, and later variants extend the paradigm across tasks and model components.Extensions include text-to-image generation, more efficient backbones, altered prediction targets, and applications such as video.
- Scale-wise Autoregressive Image Generation: Across scale-wise extensions, tokens within each scale remain at least partially independent, suggesting a structural mean-field-style approximation across the paradigm.The passage connects this shared decoding pattern to a limitation broader than the original VAR model.
3 Mean-Field-style Approximation in Scale Autoregression
VAR factorizes image generation across progressively finer scales and predicts all tokens within each scale in parallel. This fully factorized decoding ignores same-scale spatial dependencies, so correct marginals can still yield structurally incoherent joint samples.
- VAR Factorization: VAR represents an image as K discrete token scales, with each scale containing L_k tokens at a corresponding spatial resolution.The model factorizes the image distribution autoregressively across scales.
- Within-Scale Decoding: At each scale, the transformer computes hidden states for all positions in parallel and parameterizes per-token categorical distributions.The resulting conditional distribution is then approximated by independent token distributions.
- Mean-Field-style Approximation: This fully factorized mean-field-style approximation ignores spatial dependencies among same-scale tokens, causing structurally incoherent samples even at large model scales.Natural images exhibit strong spatial dependencies that the approximation does not faithfully model.
- Toy Example: Correct per-token marginals do not guarantee valid joint samples: independent sampling of checkerboard tokens produces many invalid outcomes.The toy example makes the decoding-rule failure concrete by contrasting valid checkerboards with the larger set of independently sampled configurations.
4 Logit Refiner
The Logit Refiner replaces VAR’s independent within-scale sampling with lightweight autoregressive decoding over frozen backbone features. It preserves parallel backbone computation while restoring intra-scale dependencies during generation.
- 4.1 Restoring Joint Within-Scale Sampling: Autoregressive within-scale sampling restores dependencies discarded by VAR’s parallel mean-field-style decoder without modifying the backbone.The refiner models each token conditioned on previously sampled same-scale tokens and frozen backbone features.
- 4.2 Architecture: The backbone computes current-scale hidden states once in parallel, while only the lightweight refiner samples tokens sequentially within that scale.This design preserves efficient across-scale generation and avoids rerunning the expensive backbone for every token.
- 4.2 Architecture: The refiner consumes frozen hidden states and previous tokens, fuses them through a causal transformer, and predicts logits for sequential categorical sampling.Its inputs combine backbone representations with a start-of-sequence vector or the preceding token embedding.
- 4.3 Training and Inference: Teacher forcing enables fully parallelized refiner training across positions and scales while the VAR backbone remains frozen.The refiner is optimized with cross-entropy over autoregressive token predictions.
- 4.3 Training and Inference: Identity initialization makes the refiner initially reproduce the base VAR predictions, focusing optimization on residual corrections for joint sampling.Copied output-head and embedding weights, disabled context integration, and zero-initialized projections preserve functional equivalence at initialization.
5 Experiments
Experiments show that joint intra-scale autoregressive sampling, rather than added capacity or training, drives the Logit Refiner’s gains across image and text-to-image generation.
- 5.1 Ablation Studies: Controlled ablations show that only causal attention with autoregressive sampling yields substantial gains, isolating joint intra-scale dependency modeling rather than capacity or extra training.The parallel and autoregressive refiners use the same architecture, differing in attention mask and sampling.
- 5.2 Main Results: VAR + Refiner improves FID by 0.16 to 0.49 across backbone scales, with ∼10% additional parameters, and VAR-d24 + Refiner surpasses the twice-sized VAR-d30.The 1.1B-parameter VAR-d24 + Refiner exceeds the 2B-parameter VAR-d30 on class-conditional ImageNet-256^2.
- 5.2 Main Results: The refiner shifts the FID scaling curve downward across backbone sizes without changing the overall scaling trend, indicating persistent spatial-consistency problems in vanilla VAR.The reported qualitative and quantitative scaling behavior attributes the issue to the sampling rule rather than insufficient model capacity.
- 5.2 Main Results: Applying the refiner only at early scales reduces overhead by 84% or 71% while retaining 88% or 99% of the full FID improvement.For VAR-d16, early scales contain fewer tokens and concentrate the quality gains.
- 5.2 Main Results: On text-to-image synthesis, the refiner improves HPSv3 from 9.79 to 9.91 while improving structural coherence and reducing texture inconsistencies.The refiner is trained with a frozen backbone and applied selectively to early stages.
6 Conclusion
The Logit Refiner addresses the mean-field limitation of parallel within-scale decoding and consistently improves generation quality across model sizes and conditioning settings. Its main cost is sequential computation within each scale, although early-scale use retains most quality gains at lower cost.
- Controlled ablations identify joint intra-scale sampling, rather than extra capacity or training, as the critical ingredient.
- The lightweight autoregressive refiner restores dependencies discarded by parallel decoding, adding about 10% parameters and less than 5% of the base model’s training compute.
- Sequential computation at each scale remains an overhead, but applying the refiner selectively at early scales retains most quality gains at a fraction of the cost.
- Combining the refiner with other VAR improvements and exploring non-autoregressive within-scale sampling are identified as future directions.
A Implementation Details
The implementation evaluates inference sensitivity, training variability, and token-ordering choices across model configurations. Results indicate that inference hyperparameters vary by base model, while ordering changes preserve the refiner’s gains.
- Inference performance measured by FID varies with top-k and classifier-free guidance scales, whose optimal values depend on base model size.The study sweeps both hyperparameters individually with respect to FID.
- Across four training runs, FIDs of 2.77, 2.80, 2.81, and 2.83 produce a sample standard deviation of 0.025.The reported gains over all Tab. 3 baselines are at least eight standard deviations.
- All tested intra-scale token orderings retain the refiner’s gain, indicating that the improvement is not an artifact of raster-scan ordering.
B Additional Quantitative Evaluation Details
Additional evaluations show that the refiner’s gains persist across metrics, model scales, and inference settings. The improvement is larger for large models, while some secondary metrics shift alongside the preferred guidance scale.
- The extended comparison reports a major FID improvement alongside minor decreases in IS and precision and an increase in recall.
- The reduction in IS is primarily attributed to the refiner requiring lower CFG scales for optimal FID.Higher guidance scales are generally associated with higher IS values.
- Refiner gains are more pronounced for large models than for small models.
- Refiner gains persist without classifier-free guidance and are significant across scales.The supplementary table compares the no-guidance setting with the typical CFG setting.
C Additional Qualitative Samples
Supplementary qualitative evaluation includes additional comparisons between the base Infinity-2B model and its refiner version. These comparisons are presented in two supplementary figures.
- The supplementary evaluation compares the base Infinity-2B model with its refiner version.
- The additional qualitative comparisons are shown in Supplementary Figures C.2 and C.3.
- The passage identifies the refiner version as the comparison counterpart to the base Infinity-2B model.
C.2 ImageNet
This section presents supplementary ImageNet visual results for VAR with and without the refiner across base-model scales, alongside uncurated 256×256 samples from VARd30 with the refiner.
- C.2 ImageNet: Supplementary Figure C.4 compares VAR samples with and without the refiner across different base-model scales.The supplied passage identifies this figure as additional comparative results.
- C.2 ImageNet: Supplementary Figures C.5–C.10 show uncurated 256×256 ImageNet samples from VARd30 with the refiner.These figures cover various classes.
- C.2 ImageNet: Figures C.2 and C.3 present selected Infinity-2B refiner samples generated from HPSv3 prompts.These are supplementary visual examples for text-conditioned generation.
- C.2 ImageNet: The VARd30 refiner samples use classifier-free guidance scale 2.5 and topk = 500 for sampling.The same settings are stated for the uncurated sample figures.