PaperlayerView exact revision on arXiv

Source-linked AI summary

GEAR: Guided End-to-End AutoRegression for Image Synthesis

Bin Lin, Zheyuan Liu, Chenguo Lin, Sixiang Chen, Yunyang Ge, Yunlong Lin, Jianwei Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Li Yuan

arXiv:2606.32039v1cs.CV

TL;DR

GEAR addresses the mismatch between reconstruction-focused tokenizers and generators by jointly training them with differentiable representation guidance. It reduces ImageNet gFID from 6.00 to 4.95 at 111M parameters versus LlamaGen-REPA and accelerates convergence by up to 10×.

  • Problem

    Two-stage training freezes a tokenizer optimized for reconstruction without accounting for whether its induced latent space is easy for the downstream generator to model.

  • Method

    GEAR jointly trains a VQ tokenizer and autoregressive generator using soft assignment for differentiable alignment guidance and hard one-hot tokens for prediction.

  • Results

    At matched 300-epoch ImageNet settings, CFG gFID drops from 6.00 to 4.95 for the 111M model versus LlamaGen-REPA, with consistently higher IS.

  • Takeaways & Limitations

    GEAR shifts representation alignment toward the AR model, producing less DINOv2-like tokenizer features and more DINOv2-like, spatially coherent AR features.

  • Takeaways & Limitations

    GEAR remains limited by its discrete tokenizer: rFID 1.64 bounds generation at gFID 2.52, trailing REPA-E’s rFID 0.28 and gFID 1.12.

Abstract

from arXiv · show

Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.

1 Introduction

GEAR removes the fixed tokenizer–generator boundary by jointly training both models, while routing only a differentiable alignment signal into the tokenizer. This preserves reconstruction, improves predictable structure, and accelerates ImageNet training.

  • Motivation: GEAR jointly trains a VQ tokenizer and autoregressive generator through representation alignment.The method addresses the mismatch created when reconstruction-only tokenizers are frozen before generator training.
  • Challenge: The VQ index is non-differentiable, and naive straight-through training is unstable and collapses the codebook.GEAR avoids sending the next-token-prediction loss into the tokenizer because that encourages low-entropy code usage.
  • Method: A hard one-hot branch trains the AR model, while a soft differentiable branch carries alignment gradients to the tokenizer.This separates inference-time discrete-token training from tokenizer guidance.
  • Results: Up to 10× faster ImageNet gFID convergence is achieved relative to LlamaGen-REPA.The comparison is reported in figure 1.
  • Representation alignment: GEAR shifts DINOv2-like structure from tokenizer features into AR features while preserving reconstruction.The tokenizer becomes less DINOv2-like, whereas AR features become more patch-level and spatially coherent.
  • Generality: The mechanism generalizes across VQVAE, LFQ, and IBQ quantizers and accelerates text-to-image generation.The end-to-end-tuned tokenizer transfers as a drop-in across settings.

2 Related Work

GEAR extends the broader move toward end-to-end visual generation to the hardest discrete VQ–AR setting. Its dual hard/soft assignment avoids the codebook collapse caused by routing autoregressive gradients through an STE.

  • Two-stage pipelines: Most visual generation pipelines train a reconstruction tokenizer first and freeze it before fitting a generator.This applies to discrete AR or masked generation and continuous diffusion pipelines.
  • Representation alignment: Representation-alignment methods either align generator features or make the tokenizer latent itself semantic.Examples include REPA, VA-VAE, MAETok, and related encoder-alignment approaches.
  • End-to-end generation: End-to-end training dissolves the two-stage boundary, but discrete VQ–AR generation is difficult because its index is a non-differentiable arg max.Naively sending autoregressive gradients into the tokenizer destabilizes training and collapses the codebook.
  • GEAR: GEAR separates hard inference-time token training from soft tokenizer guidance through dual assignment read-outs.The hard branch carries NTP and hard REPA, while the soft branch carries differentiable alignment.

3 Method

GEAR couples VQ tokenization and AR generation in one loop without allowing prediction gradients to reshape the codebook. A hard branch matches inference, while a truncated soft branch guides the tokenizer through alignment.

  • VQ tokenization: A VQ tokenizer encodes an image into a downsampled grid of latent vectors and quantizes each position against a learnable codebook.The decoder reconstructs the image from the quantized latent grid.
  • Tokenizer objective: GEAR retains the standard VQ objective, including reconstruction, perceptual, adversarial, entropy, and commitment terms.The displayed objective weights these terms as LVQ = Lrec + 0.1 LLPIPS + 0.1 LGAN + 0.05 Lent + 0.25 Lcommit.
  • AR generation: The AR generator predicts the same discrete indices causally using its own learnable embedding table.Each token index selects an AR embedding rather than directly reusing the VQ codebook vector.
  • Dual read-out: GEAR forms hard and soft read-outs from each tokenizer assignment and maps both into the AR embedding table.The soft read-out is a temperature-controlled mixture, with τ > 0 controlling guidance softness.
  • Dual-branch forward: The hard branch reaches the final AR layer for next-token logits, while the soft branch stops at alignment depth ℓ to reduce computation.The hard path matches the discrete tokens used at inference; the soft path remains differentiable.
  • Decoupled optimization: The soft alignment gradient updates only tokenizer parameters, while NTP and hard alignment update only the AR model.This decoupled optimization prevents prediction loss from inducing codebook collapse.

4 Experiment

GEAR improves autoregressive image generation while relocating semantic alignment from the tokenizer to the generator. Across ImageNet, text-to-image, quantizer, resolution, and ablation studies, its end-to-end guidance improves generation and preserves reconstruction.

  • Main Results: At matched 300-epoch ImageNet settings, GEAR lowers CFG gFID from 6.00 to 4.95, 3.15 to 2.95, and 2.68 to 2.52 across 111M, 343M, and 775M models.The corresponding AR variants are B, L, and XL, with consistently higher IS.
  • Main Results: GEAR consistently lowers GPIC FDD with CFG to 256.9/177.4/138.0/115.3 across 50k/100k/200k/390k steps, versus 279.6/198.6/153.5/127.9 for LlamaGen-REPA.All models use the same Qwen3-1.7B text encoder and 100M-image corpus.
  • Training Dynamics: GEAR reaches the baseline’s final REPA-alignment loss 11.1× faster and its NTP loss 2.5× faster when both ARs use frozen tokenizers.The difference therefore reflects tokenizer quality rather than ongoing tokenizer updates.
  • Representation Analysis: GEAR’s tokenizer becomes less DINOv2-like, while the AR becomes more patch-level aligned and carries richer locally coherent spatial structure.Tokenizer CKA falls from 0.173 to 0.107 at the patch level, while AR patch-level similarity exceeds LlamaGen-REPA across mid-to-deep layers.
  • Representation Analysis: The codebook becomes more predictable: usage concentration peaks near 30k steps and converges after ∼120k to lower entropy than the warm-up tokenizer.The codebook remains broadly used and reconstruction is preserved, avoiding few-code collapse.
  • Ablation Studies: Replacing the soft-assignment bridge with a straight-through estimator collapses generation to gFID 104.9 and reconstruction to rFID 59.7.Removing adversarial loss also degrades rFID from 1.640 to 5.857 and gFID from 10.630 to 16.353.

5 Discussion

GEAR narrows the gap to diffusion while exposing a structural ceiling: autoregressive token count, compression, and compute remain tightly coupled.

  • Reconstruction ceiling: GEAR still trails REPA-E because its discrete tokenizer reconstructs less faithfully.GEAR reports rFID 1.64 and gFID 2.52 with CFG, versus REPA-E’s rFID 0.28 and gFID 1.12.
  • Reconstruction ceiling: Closing the reconstruction gap is the largest remaining lever for improving VQ-AR generation.
  • Compression and compute: Autoregressive models tie compression to compute because each latent token becomes one decoding step.A 16× tokenizer maps a 256 × 256 image to 256 tokens, while diffusion can retain 1024 latent positions and regroup them for transformer processing.
  • Compression and compute: Milder tokenizers paired with AR-side grouping could raise reconstruction fidelity without increasing sequence length.
  • Unified generation: Discrete next-token generation remains attractive for unified, long-context multimodal systems.Discrete tokens bound per-step error and support uniform, scalable pipelines across modalities.

6 Conclusion

GEAR jointly trains the tokenizer and autoregressive generator by separating prediction from guidance. The result is faster, stronger generation with improved tokenizer reconstruction and broad quantizer portability.

  • Method: GEAR routes only representation-alignment gradients through a differentiable soft assignment while hard tokens train next-token prediction.This avoids the codebook collapse caused by sending the prediction loss through a non-differentiable index.
  • Results: Up to 10× faster ImageNet gFID convergence distinguishes GEAR from the LlamaGen-REPA baseline.The method also improves final quality and reconstruction of the co-trained tokenizer.
  • Results: GEAR improves patch-level, spatially coherent structure that makes next-token prediction easier.
  • Generality: The guidance mechanism transfers across VQVAE, LFQ, and IBQ quantizers.

A Ablation Training Configurations

The ablation studies fine-tune GEAR-L while changing one experimental axis at a time, with shared optimization and sampling settings documented separately.

  • Study design: Each ablation varies a single axis while holding the remaining configuration fixed.Most studies fine-tune from a warm-up tokenizer; the initialization study also trains the tokenizer from scratch.
  • Study design: Table 15 assigns one column to each ablation and bolds the swept axis while underlining the default.
  • Shared settings: Shared optimization, loss-weight, and sampling settings are listed separately in Table 16.
  • Alignment setup: Alignment is applied at one-third of backbone depth across model sizes.The corresponding layers are 4, 8, and 12 for the B, L, and XL backbones.
  • Text-to-image setup: The text-to-image configuration uses a LlamaGen-1B model trained from scratch on the approximately 100M-image GPIC corpus.The recipe specifies one epoch, about 390k steps, and batch size 256.

B Text-to-Image Training Configurations

The text-to-image system is a strict causal autoregressor over concatenated text and image tokens, trained from scratch under a controlled GPIC recipe.

  • Architecture: The model autoregresses over the concatenation of text and image tokens with purely causal attention.
  • Architecture: The first third of transformer blocks uses dual streams, while the remaining two thirds share projections.
  • Conditioning: Qwen3-1.7B supplies 300-token text conditioning with 0.1 caption dropout for classifier-free guidance.
  • Controlled training: GEAR and LlamaGen-REPA differ only in whether the frozen tokenizer is end-to-end fine-tuned.Both use the same LlamaGen-1B parameter count and GPIC training recipe.

C Tokenizer Reconstruction and Evaluation Interpolation

GEAR is instantiated with three 16384-entry-codebook tokenizers, while reconstruction evaluation depends materially on the image-resizing interpolation.

  • Tokenizer Setup: GEAR uses VQ-16, LFQ-16, and IBQ-16 tokenizers, each with a 16384-entry codebook.VQ-16 comes from LlamaGen, LFQ-16 from Open-MAGVIT2, and IBQ-16 is the third tokenizer.
  • Tokenizer Setup: Reconstruction quality is reported on the ImageNet validation set using the tokenizers' sizes and reconstruction metrics.The cited setup reports reconstruction quality for the three tokenizer variants rather than generation quality.
  • Evaluation Interpolation: Bicubic interpolation is used for evaluation because it preserves more high-frequency content than bilinear interpolation.PSNR, SSIM, and rFID are computed against the resized reference image, so interpolation changes the reference itself.

D Classifier-Free Guidance Sweep for Text-to-Image

The controlled GPIC sweep compares GEAR with LlamaGen-REPA across CFG scales from 1 to 20. GEAR is consistently stronger, although continued gains at high guidance reveal underfitting in the small-budget text-to-image model.

  • Experimental Design: The sweep evaluates both tokenizers across CFG scales 1–20 on DPG-Bench and GenEval using short and long prompts.The comparison is conducted under the controlled GPIC setting for LlamaGen-REPA and GEAR.
  • DPG-Bench: +1.65 best-vs-best points: GEAR reaches 72.881 at CFG=16, versus 71.229 at CFG=18 for LlamaGen-REPA on DPG-Bench.At CFG ≤2 the methods are within noise, while GEAR leads stably by +1.3 to +2.5 points from CFG ≥4.
  • GenEval: ≈+0.05 short-prompt and ≈+0.06 long-prompt gaps favor GEAR at every CFG scale on GenEval.Long-prompt scores peak near CFG=16, whereas short-prompt scores are still climbing at CFG=20.
  • Interpretation: Continued quality gains at CFG 16–20 indicate that the single-epoch GPIC model is underfitting and does not strongly exploit text conditioning.The study isolates tokenizer effects under a matched recipe rather than targeting state-of-the-art text-to-image performance.

E Per-Layer Representation Analysis

GEAR's representation alignment appears at different depths for global semantics and local spatial structure. Image-level similarity peaks near the alignment depth, while patch-level similarity emerges deeper in the causal AR stack.

  • Global Features: Image-level similarities and Moran’s I peak near alignment depth ℓ=8.Global semantic similarity reflects the representation alignment injected directly at layer 8.
  • Patch Features: Patch-level similarity curves peak only in deeper layers after causal processing accumulates sufficient spatial context.The deeper peak reflects intra-image spatial structure rather than an anomaly in the measurements.
  • Interpretation: The analysis distinguishes global semantic alignment from the spatially coherent features needed for generation.Image-level and patch-level probes therefore identify different useful depths in the AR generator.
Loading 2606.32039v1…