VA-$\pi$: Variational Policy Alignment for Pixel-Aware Autoregressive Generation

• Token-only optimization can yield high-likelihood sequences that decode into visually poor images, while direct pixel-level autoregression is intractable in practice.The paper calls such sequences off-manifold because their decoded images deviate from the image manifold.Source: §1 · pp. 1–2
• Earlier noisy-context approaches did not directly address pixel-level misalignment, and excessive noise can reduce decoder sharpness and visual fidelity.Tokenizer-centered approaches increase tolerance to off-manifold sequences rather than preventing the AR generator from producing them.Source: §1 · p. 2
• VA-π treats sampled token sequences as latent variables of the image and derives an ELBO that combines tokenizer reconstruction with the AR model’s token-likelihood prior.The reconstruction term supplies pixel-level supervision, while the prior term retains token-level likelihood modeling.Source: §1 · p. 2
• Because the variables are discrete, the method uses reinforcement-based policy optimization to distribute gradients across sampled sequences rather than only the ground-truth path.The tokenizer’s teacher-forcing reconstruction loss supplies the intrinsic reward, and variational regularization keeps the updated policy close to the base AR model.Source: §1 · p. 2
• FID falls from 14.36 to 7.65 and IS rises from 86.55 to 116.70 for LlamaGen-XXL; VA-π uses 1% of pretraining data and 25 minutes of post-training.The contribution summary reports these settings without external reward models.Source: §1 · pp. 2–3

2.1. Auto-Regressive Visual Generation

• Visual AR generation uses discrete tokenizers to make pixel generation tractable, but generator likelihood training and tokenizer reconstruction objectives remain separated.Prior component-level advances address tokenizer artifacts or tokenization structure without resolving this objective-level bottleneck.Source: §2.1 · p. 3

2.2. Tokenizer-Generator Alignment

• Generator- and tokenizer-centered methods reduce mismatch through noisy contexts, randomized token order, decoder robustness, or token-structure changes, but often require costly retraining and intuitive regularization.The authors contrast those approaches with their own principled and efficient objective.Source: §2.2 · p. 3

2.3. Reinforcement Learning in Visual Generation

• The authors state that their work is the first to apply reinforcement learning to optimize AR models directly for pixel-space reconstruction quality.Prior visual-generation RL has targeted reasoning, human preferences, controllability, or prompt alignment.Source: §2.3 · p. 3

3.1. Visual Autoregressive Models

• A visual tokenizer encodes images into discrete codes and decodes them for reconstruction, using pixel, perceptual, and vector-quantization losses.The AR model learns the code distribution by maximizing sequence likelihood with ground-truth preceding tokens as teacher-forced context; conditioning inputs are omitted from the notation.Source: §3.1 · pp. 3–4
• At inference, the AR model samples tokens sequentially in free-running mode and the decoder maps the completed sequence back to an image.The class or text input can serve as an additional condition.Source: §3.1 · pp. 3–4

3.2. Reinforcement Learning

• GRPO samples groups of outputs, normalizes their rewards into relative advantages, and combines a clipped policy-ratio objective with a KL penalty that constrains drift from the reference policy.The paper adopts GRPO for stability through group reward normalization and KL regularization.Source: §3.2 · p. 4

4.1. Evidence Lower Bound for Alignment

• The ELBO treats discrete tokens as latent variables, approximates their posterior with teacher-forced predictions, and combines pixel reconstruction with prior regularization while keeping the tokenizer frozen.The pixel-space likelihood is intractable to evaluate directly; VA-π updates the AR generator rather than jointly tuning it and the tokenizer.Source: §4.1 · pp. 4–5

4.2. Regularization with Next Token Prediction

• The prior regularization term is treated as reducing exposure bias and implemented with contextual noise and next-token prediction.The contextual noise level controls corruption of the ground-truth token sequence.Source: §4.2 · p. 5

4.3. Learning with Reconstruction Reward

• Because quantization and discrete sampling block direct gradients, VA-π treats negative reconstruction loss on teacher-forced token samples as a policy reward.The method reuses noisy token sequences from next-token prediction regularization to avoid multiple AR-model forwards.Source: §4.3 · p. 5

4.4. VA-π Policy Optimization

• GRPO combines reconstruction reward with next-token prediction regularization, using teacher-forced trajectories without extra reference-model storage or free-running rollouts.The authors report that VA-π outperforms GRPO models trained to maximize external rewards.Source: §4.4 · pp. 5–6
• VA-π achieves competitive ImageNet-1K FID and IS at substantially lower training cost, and the highest overall GenEval score for LlamaGen-XL with fine-grained attribute gains for Janus-Pro.The supplied captions report these comparisons but provide no numerical values.Source: Table 3 · p. 6, Table 2 · p. 6

5.1. Experimental Setup

• Experiments cover class-conditioned generation with LlamaGen-XL and XXL, text-conditioned generation with LlamaGen-XL and Janus-Pro 1B, and GenEval’s six compositional dimensions.C2I uses ImageNet-1k; the reported training setups use 100 steps and 12.8K samples for C2I, 200 steps for LlamaGen T2I, and 100 steps for Janus-Pro 1B.Source: §5.1 · pp. 6–7

5.2. Main Results on Class-to-Image Generation

• LlamaGen-XXL post-training reduces FID from 14.35 to 7.65 and increases IS by 30.16; VA-π also reports the best CFG-scale-1.75 FID of 2.28 and IS of 273.53.For LlamaGen-XL, VA-π improves FID from 15.55 to 9.23 and IS from 79.16 to 111.59 without CFG; its CFG variant reaches IS 299.63 and trains 7.5× faster than AR-GRPO, while the XXL model trains 15× faster than STE.Source: §5.2 · p. 7

5.3. Main Results on Text-to-Image Generation

• On GenEval, VA-π raises LlamaGen-XL’s overall score from 0.324 to 0.339 and Janus-Pro 1B’s from 0.725 to 0.744.The LlamaGen gains include color understanding (+0.013), counting (+0.010), and two-object composition (+0.065); for Janus-Pro 1B, attribute binding improves by +0.045 and two-object relations by +0.034.Source: §5.3 · pp. 7–8

5.4. Ablation Study

• For regularization without CFG, CE outperforms KL on FID and IS, with weight 0.1 best; for T2I contextual noise, ξ=0.5 gives the best GenEval overall score of 0.339.No regularization yields FID 38.63, while β=1.0 oversmooths gradients and suppresses diversity; no noise or ξ>0.75 is suboptimal.Source: Figure 4 · p. 8, §5.4 · p. 8, Table 5 · p. 8
• Ablations report that reconstruction reward alone fails to align meaningfully without prior regularization, while combining reconstruction and prior regularization gives the best balance.Table 4 examines pixel reconstruction (LMSE), perceptual similarity (Lp), and token-level cross-entropy regularization (Lprior).Source: §5.4 · p. 8, Table 4 · p. 8

6. Conclusion

• VA-π substantially improves visual fidelity and compositional alignment on class-conditional and text-conditional generation tasks.The conclusion frames the method as aligning visual AR generators to pixel space through a variational objective grounded in probabilistic modeling.Source: §6 · p. 8
• VA-π reduces training cost by 86.6% compared with conventional RL fine-tuning.Source: §6 · p. 8
• The conclusion reports that VA-π generalizes to the unified multimodal model Janus-Pro 1B.It characterizes the framework as a lightweight, theoretically grounded path for bridging token-level modeling and pixel-level generation in scalable multimodal generation.Source: §6 · p. 8

A. Theoretical Details of VA-π

• The section restates the ELBO proof using a discrete token sequence as the latent variable.Source: A. Theoretical Details of VA-π · p. 9
• It examines the formulation’s relationship to VAE and autoregressive models based on VQVAE.Source: A. Theoretical Details of VA-π · p. 9
• It further justifies the formulation of the prior regularization term.Source: A. Theoretical Details of VA-π · p. 9

A.1. Proof of Alignment ELBO

• The proof sets up image likelihood using a prior over token sequences and a pixel-space likelihood, noting that marginalizing over tokens is generally intractable.It introduces a variational posterior qϕ(x | I), applies Jensen’s inequality to define an ELBO, and notes that matching the true posterior makes the KL term zero and ELBO maximization equivalent to MLE.Source: §A.1 · p. 9

A.2. Comparison with VAE and AR on VQVAE

• Unlike VQVAE’s encoder-and-quantizer posterior and uniform prior, VA-π teacher-forces the encoded discrete code through the AR model to define its posterior.This keeps the ELBO valid and gives the AR model pixel-space reconstruction supervision; the formulation uses the pretrained AR model as prior and supports post-training alignment with the tokenizer.Source: §A.2 · pp. 9–10, Table 6 · p. 10

A.3. Details of Prior Regularization

• VA-π relates prior KL minimization to next-token prediction under perturbed prefixes, subject to assumptions that are not guaranteed in experiments.The appendix reports empirical effectiveness with a suitable corruption kernel and describes uniform token noise applied at rate ξ.Source: §A.3 · pp. 10–11

B. Tokenizer Training Objectives

• The tokenizer’s total objective combines reconstruction and quantization terms to support high-fidelity image reconstruction and stable codebook learning.The resulting discrete tokens balance compression and visual quality for subsequent autoregressive modeling.Source: B. Tokenizer Training Objectives · p. 11
• Pixel-wise reconstruction uses mean squared error between the original image and its reconstruction, while perceptual loss compares their pretrained VGG feature activations.Perceptual loss follows.Source: B. Tokenizer Training Objectives · p. 11
• Vector quantization encourages encoder outputs to commit to the closest codebook embedding; its terms update codebook entries and penalize deviations from assigned embeddings.The commitment strength is controlled by β.Source: B. Tokenizer Training Objectives · p. 11

C. Implementation Details

• The implementation covers STE fine-tuning and tokenizer-decoder fine-tuning baselines, then VA-π experiments with LlamaGen for class-to-image and text-to-image generation and Janus-Pro for text-to-image.The STE baseline uses more data and training time than VA-π, which updates the model under the ground-truth path.Source: C. Implementation Details · p. 11, §C.1 · p. 12

C.1. Straight-Throught Estimator Variant

• STE decodes discrete hard tokens in the forward pass but reuses soft-token gradients in backpropagation, copying decoder reconstruction gradients to generator logits through the frozen tokenizer.The method applies this mechanism to reconstruction and regularization objectives.Source: §C.1 · p. 12

C.2. Tokenizer Post-training

• Tokenizer post-training freezes the autoregressive model, quantizer, and codebook while updating only the decoder to improve reconstruction of autoregressive-generated tokens.The resulting tokenizer is described as a drop-in replacement that improves decoding fidelity without inducing distributional mismatch.Source: §C.2 · p. 12

C.3. Implementaton Details of VA-π

• LlamaGen fine-tuning uses AdamW with a 1 × 10^-6 learning rate, while Janus-Pro 1B uses 1 × 10^-7; training runs range from 100 to 200 steps across the three settings.LlamaGen models use 384 × 384 resolution and Janus-Pro 1B uses 256 × 256 resolution.Source: §C.3 · p. 12, Table 7 · p. 13

D.1. Visualization of Learning Curves

• Reward steadily increases during VA-π fine-tuning across C2I LlamaGen-XXL and T2I LlamaGen-XL and Janus-Pro 1B scenarios.Figure 5 presents the learning curves for these three scenarios.Source: §D.1 · p. 13, Figure 5 · p. 13

D.2. Class-to-Image Generation

• Decoder-only fine-tuning cannot correct off-manifold token sequences and instead amplifies train–inference mismatch while smoothing away meaningful details.The reported degradation can leave coarse structure intact while reducing sharpness and perceptual richness.Source: §D.2 · p. 13
• Tokenizer post-training preserves coarse structure but smooths textures, while STE partly improves details yet retains off-manifold transitions; VA-π yields sharper textures and more coherent semantics.The comparisons use identical decoding configurations with CFG scale 1.0.Source: §D.2 · pp. 13–14
• VA-π generations are more consistent with class semantics and show improved structural fidelity and perceptual realism than pretrained LlamaGen-XXL.The qualitative comparison also reports better semantic alignment and image quality against LlamaGen-XXL, tokenizer post-training, and STE-based post-training.Source: §D.2 · p. 14, Figure 6 · p. 14

D.3. Text-to-Image Generation

• GenEval text-to-image comparisons show Jans-Pro 1B and VA-π generations for attribute binding, counting, position, and two-object combination, using CFG scale 5.0, temperature 1.0, top-k 0, and top-p 1.0.Source: §D.3 · p. 14