Source-linked AI summary
Representation Forcing for Bottleneck-Free Unified Multimodal Models
Yuqing Wang, Zhijie Lin, Ceyuan Yang, Yang Zhao, Fei Xiao, Hao He, Qi Zhao, Zihan Ding, Fuyun Wang, Shuai Wang, Youliang Zhang, Haoqi Fan, Xihui Liu
TL;DR
Existing unified multimodal models rely on a frozen pretrained VAE, creating a bottleneck for end-to-end pixel generation. Representation Forcing makes visual representation prediction native to the decoder, and the resulting pixel-space model matches VAE-based generation while improving understanding.
Problem
Existing unified multimodal models depend on a separately pretrained, frozen VAE, creating a structural bottleneck that limits end-to-end pixel-space generation.
Method
Representation Forcing trains the decoder to autoregressively predict discretized visual representation tokens before pixels, using them as in-context structural guidance for pixel-space diffusion.
Results
RF closes the quality gap with VAE-based generation while improving multimodal understanding, with pixel-space RF matching VAE-based generation and outperforming its counterpart on understanding.
Takeaways & Limitations
RF offers an effective step toward pixel-space, bottleneck-free unified multimodal models by integrating perception and generation through shared representations.
Takeaways & Limitations
The study focuses on still-image generation and does not extend Representation Forcing to video or other temporal modalities.
Abstract
from arXiv · showhide
Unified multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before pixels; these tokens then stay in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.
1 Introduction
Unified multimodal models still depend on separately pretrained VAEs for image generation, creating a structural bottleneck and a quality gap when removed. Representation Forcing addresses this by predicting visual representations before pixels, improving both generation and understanding in pixel-space and VAE-based models.
- Motivation: Prevailing unified multimodal models use a frozen, separately pretrained VAE for image generation, creating a structural bottleneck that further UMM training cannot overcome.Removing this bottleneck is presented as an important step toward end-to-end unified multimodal models.
- Motivation: Direct pixel-space generation in UMMs fails to match VAE-based quality because the model must learn semantic structure and fine-grained detail from raw pixels.The challenge is attributed to UMMs’ broader image distribution and richer text conditioning.
- Representation Forcing: Representation Forcing trains the decoder to autoregressively predict encoder-derived visual representations as intermediate tokens, which guide pixel diffusion within the same backbone.The predicted representations are learned under the same next-token prediction objective used for language.
- Results: With controlled architecture, data, and training budgets, pixel-space RF matches the VAE-based baseline on generation and outperforms its VAE-based variant on understanding.The paper reports preserved rich textural details in generation and argues that pixel-space generation is more compatible with unified multimodal modeling.
- Results: Ablations show that RF is critical for pixel-space generation and also improves VAE-based settings, benefiting both directions of unified multimodal modeling.The work positions RF as a step toward bottleneck-free UMMs with a shared, end-to-end-learned representation space.
2 Related Work
Related unified multimodal models either generate within one backbone or connect an LLM to an external diffusion model, while typically relying on separately pretrained visual tokenizers or VAEs. Parallel work explores end-to-end pixel-space generation and richer pretrained visual representations to address these limitations.
- Unified Multimodal Models: Unified multimodal models generate within one backbone using discrete image tokens or diffusion in a VAE latent space.Examples include Chameleon and Emu3 for discrete tokens, and Transfusion, Show-o, and JanusFlow for VAE-latent diffusion.
- Unified Multimodal Models: These unified models address understanding-generation interference with decoupled visual encoders or modality-specific experts, but depend on separately pretrained VQVAEs or continuous VAEs.The Janus series uses decoupled visual encoders, while BAGEL uses modality-specific experts.
- Unified Multimodal Models: A second family stitches an LLM with an external diffusion model, using predicted visual representations to condition a separately trained decoder.Examples include Emu2, SEED-X, BLIP3-o, and MetaQueries; Omni instead rolls out intermediate multimodal context within its backbone before decoding.
- Pixel-Space Generation: Pretrained VAE latent spaces reduce compute and enable high-resolution synthesis but prevent end-to-end training, motivating direct pixel-space generation from raw data.JiT demonstrates raw-pixel generation with plain Vision Transformers and x-prediction.
- Representation Learning for Generation: Representation-learning approaches improve generation by aligning diffusion features with frozen representations or replacing VAEs with frozen pretrained encoders that provide richer semantic latent spaces.REPA targets faster convergence, while RAE and related works use encoders such as DINOv2 and SigLIP.
3 Representation Forcing
Representation Forcing uses jointly trained visual features as discrete intermediate targets that the decoder predicts before rendering images in pixel space. These predicted representations remain in context to guide semantic layout and low-level pixel generation within one backbone, without an external latent space.
- Representation formulation: The method uses the understanding encoder’s jointly trained features as an internal intermediate representation so pixel diffusion can focus on low-level rendering.The encoder’s features capture high-level visual structure and are intended to replace an external latent space.
- Representation formulation: Online vector quantization discretizes EMA encoder patch features without a separate pretrained tokenizer, using slow-moving targets for stable assignments.Features come from the last layer of an exponential moving average encoder before final normalization.
- Representation formulation: Sinkhorn–Knopp normalization balances feature-to-prototype assignments, preventing popular prototypes from absorbing most features and collapsing the codebook.The normalization softly constrains every prototype to receive an equal share of the batch.
- Representation prediction: During training, the decoder predicts EMA encoder representation tokens autoregressively with cross-entropy; during inference, it predicts them from text without the encoder.The predicted representations force the decoder to learn high-level visual structure and guide pixel generation toward the intended semantic layout.
- Pixel-space generation: Predicted representation tokens remain in the sequence as in-context conditioning while the shared backbone generates visible pixel patches through flow matching.The representation tokens are not part of the visible output.
- Pixel-space generation: Generation first autoregressively produces the full representation sequence from text, then iteratively denoises Gaussian noise into pixels conditioned on text and representations.Classifier-free guidance is applied to both conditioning sources.
4 Experiments
Experiments show that Representation Forcing closes the pixel-space generation gap while improving visual understanding, with benefits depending on the representation mechanism and encoder choice.
- Text-to-image generation: 0.84 GenEval without an LLM rewriter slightly exceeds BAGEL’s 0.82 and matches BLIP3-o, while 0.88 with a rewriter matches unified-model state of the art.RF-Pixel also scores 84.15 on DPG-Bench without a rewriter and uses no pretrained VAE.
- Image understanding: RF improves understanding in both pathways: Pixel+RF improves 6/8 benchmarks, while VAE+RF improves 5/8.Pixel-space gains include MMMU +4.3, MME +3.6, BLINK +3.6, AI2D +4.5, and RealWorldQA +2.7; VAE gains include HalluBench +5.6 and MME +8.0.
- Image understanding: Pixel+RF outperforms VAE+RF on 6 of 8 understanding benchmarks, consistent with tighter sharing of a single representation space after removing the external VAE latent space.The comparison uses identical architecture and training data, with no post-training applied.
- Representation ablations: 0.25 GenEval without RF in pixel space rises to 0.76 with discrete representation tokens, whereas VAE-based generation without RF reaches 0.52.The result indicates that representation prediction is critical for structurally capable pixel-space generation.
- Representation ablations: 0.26 from continuous feature regression provides no improvement over the no-RF baseline, while discrete tokens achieve 0.76 under the same setting.The paper attributes the difference partly to error accumulation when causally predicting high-dimensional continuous vectors.
- Encoder and codebook ablations: DINOv3 outperforms SigLIP2 on 4 of 5 understanding benchmarks, while codebook sizes K=16,384 and K=32,768 perform comparably at 0.76 and 0.77.The experiments use K=16,384 elsewhere and attribute DINOv3’s advantage to richer spatial features.
5 Discussion
Representation Forcing enables pixel-space image generation in unified multimodal models by autoregressively predicting understanding representations before rendering pixels. The method closes the quality gap with VAE-based generation while improving multimodal understanding, though the study is limited to pretrained initialization and still images.
- Limitations: The model is initialized from a pretrained large language model rather than trained from scratch on multimodal data because of computational constraints.Fully from-scratch multimodal pretraining may yield richer joint representations and remains future work.
- Limitations: The study focuses on still-image generation and does not extend Representation Forcing to video or other temporal modalities.Extending RF beyond still images is identified as an important direction for future work.
- Conclusion: Representation Forcing lets the decoder predict its own understanding representations before rendering pixels, guiding pixel-space diffusion within the same sequence.The same representation supports both visual interpretation and pixel generation.
- Conclusion: The single mechanism closes the quality gap with VAE-based generation while improving multimodal understanding.This demonstrates benefits in both generation and understanding within unified multimodal models.
Appendix · A Implementation Details
The appendix specifies training hyperparameters, staged learning-rate schedules, sequence packing, EMA-based inference, and a two-stage representation-conditioned generation procedure. Generation autoregressively predicts representations before pixel denoising.
- A Implementation Details: AdamW training uses β1=0.9, β2=0.95, ϵ=10−8, weight decay 0.1, and gradient clipping 1.0.
- A Implementation Details: 5×10−5 is the base learning rate in Stages 1–2, while Stage 3 uses 2.5×10−5 for high-resolution stability.
- A Implementation Details: 4× multiplier applies to newly initialized generation-related parameters, while the LLM backbone retains the base learning rate.
- A Implementation Details: 32,768 tokens per GPU are processed using NaViT-style variable-resolution batchin.
- A Implementation Details: 0.9999 EMA decay is used, and inference runs with the EMA model parameters.
- A Implementation Details: Generation first autoregressively predicts the full representation-token sequence from the text prompt with top-k sampling, then denoises Gaussian noise into pixel patches.The denoising stage uses 25 flow-matching steps with dynamic timestep shifting and conditions on text plus predicted representation tokens.
B Online Vector Quantization Algorithm
Online Vector Quantization extracts normalized visual features with an EMA understanding encoder, assigns them to visual prototypes using temperature-scaled Sinkhorn-normalized similarities, and updates prototypes by momentum. The algorithm uses default momentum 0.9999 and temperature 0.5.
- Hyperparameters: The implementation uses temperature t with default 0.5 alongside the EMA encoder and prototype matrix.The pseudocode defines the encoder, samples, prototypes, momentum, and temperature as core algorithm inputs.
- Feature extraction: The algorithm extracts continuous features with an EMA encoder, normalizes them, and represents samples as B×L feature vectors of dimension D.The encoder output is reshaped from the batch and spatial dimensions before normalization.
- Prototype assignment: It computes temperature-scaled pairwise cosine similarities between normalized features and K visual prototypes, then applies one Sinkhorn-Knopp normalization iteration before argmax assignment.The score is row-normalized and then column-normalized, producing one discrete prototype assignment per feature.
- Prototype update: Prototype updates aggregate assigned features and counts, normalize each prototype mean, then apply momentum m and renormalize the prototype matrix.The default momentum is 0.9999.
C Broader Impact
Representation Forcing systems, like other text-to-image models, could generate misleading or harmful visual content, but standard unified-model safeguards apply.
- C Broader Impact: RF-based systems could be misused to generate disinformation, non-consensual imagery, or deepfakes.Applicable safeguards include safety filters, output watermarking, and controlled access.