PaperlayerView exact revision on arXiv

Source-linked AI summary

RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space

Xichen Pan, Aashu Singh, Satya Narayan Shukla, Xiangjun Fan, Shlok Kumar Mishra, Saining Xie

arXiv:2606.14700v1cs.CV

TL;DR

Existing T2I systems largely reserve LLMs for text encoding, leaving denoising to newly trained backbones, and it is unclear how pretrained language priors could process noisy visual representations. RepFusion feeds noisy RAE latents through a pretrained MLLM to condition a diffusion transformer, outperforming comparable-capacity baselines under similar inference budgets.

  • Problem

    It remains unclear whether pretrained language models can consume noisy visual representations beneficially during denoising, especially because conventional VAE latents lack semantic structure.

  • Method

    RepFusion reuses a pretrained MLLM to encode noisy RAE latents alongside text and conditions a diffusion transformer on its evolving outputs.

  • Results

    Under similar inference FLOPs, RepFusion outperforms baselines that allocate comparable capacity to newly initialized denoisers, including static text-embedding systems.

  • Takeaways & Limitations

    Frozen MLLMs can provide useful denoising priors for noisy RAE latents, with evolving conditioning offering a way to spend test-time compute across denoising steps.

Abstract

from arXiv · show

Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. These results demonstrate that MLLMs provide strong priors for denoising visual representations and that, by conditioning on evolving noisy representations, test-time compute can be productively spent on repeated MLLM conditioning in modern T2I systems.

1 Introduction

RepFusion repurposes a pretrained MLLM as a noisy representation encoder in the semantically structured latent spaces enabled by RAEs. By conditioning a DiT on evolving noisy representations, it provides an efficient denoising prior that can outperform comparable capacity allocated to newly initialized denoisers.

  • Motivation: RAEs replace reconstruction-oriented VAE latents with semantically structured visual representations, bridging text-to-image generation and MLLM feature spaces.Examples include CLIP and DINO features.
  • Method: RepFusion feeds noisy RAE latents into an off-the-shelf MLLM through its MLP projector, freezes the LLM backbone, and uses its output to condition a DiT denoiser.Only the projector and DiT are fine-tuned.
  • Capacity allocation: Under similar inference FLOPs, RepFusion uses a 7B frozen MLLM and 1.3B DiT, whereas TextEmbed uses an 8B DiT and Transfusion an 8B joint denoising transformer.All compared systems allocate roughly 8B parameters to modules processing noisy visual latents or denoising them.
  • Test-time scaling: RepFusion makes conditioning evolve across denoising steps by repeatedly processing the changing noisy RAE latents, unlike pipelines that reuse static text embeddings.This introduces a distinct axis for scaling test-time compute.
  • Empirical findings: Switching from VAEs to RAEs helps unified baselines such as Transfusion, but explicitly repurposing a frozen MLLM as a noisy encoder produces larger gains.The comparison is reported in Figure 1 across TextEmbed, Transfusion, and RepFusion.
  • Contributions: The paper reports that frozen pretrained MLLMs provide useful denoising priors beyond static text conditioning, outperform further joint optimization, and can beat comparable newly initialized denoisers.These findings define RepFusion’s central contributions.

2 Related Work

Prior T2I systems evolved from small text encoders to frozen pretrained encoders, while latent diffusion shifted denoising from pixels to autoencoder latents. Recent work improves semantic latent representations and integrates language models more tightly with denoisers through unified architectures or compact interfaces.

  • Text encoders in T2I: Diffusion models standardized T2I text conditioning with frozen pretrained token encoders, with Stable Diffusion 1.5 popularizing CLIP.Earlier conditional GANs used LSTMs that produced either global sentence embeddings or token-level embeddings.
  • From VAEs to RAEs: Latent diffusion makes high-resolution generation tractable by denoising in autoencoder latent space rather than pixel space.VAEs remain common, but their heavy compression and reconstruction-focused optimization limit semantic expressiveness; RAEs address this bottleneck with a decoder paired with a richer representation.
  • Integration of Language Models and Denoisers: Recent approaches integrate conditional encoders and denoisers using unified transformer stacks or compact MLLM-to-diffusion interfaces.RepFusion reuses a pretrained MLLM for text prompts and noisy RAE latents, projecting the latents with an MLP and conditioning every DiT block through AdaLN modulation.

3 RepFusion

RepFusion conditions a diffusion transformer on MLLM encodings of evolving noisy visual representations, extending conditioning beyond text. Ablations show that noisy-state conditioning, representation-compatible latents, and multimodal perception pretraining drive its gains over comparable baselines.

  • 3.2 RepFusion conditioning: RepFusion appends projected noisy visual tokens to text tokens, recomputing MLLM conditioning as the representation evolves at each denoising step.Timestep information is added before projection, and the DiT receives the selected MLLM states through adaptive layer normalization.
  • 3.3 Noisy representation inputs: 0.70 GenEval: RepFusion outperforms the learnable-query baseline’s 0.55 with a frozen 7B MLLM and 1.3B DiT.The comparison keeps training FLOPs similar; RepFusion’s advantage comes from conditions that depend on the current denoising state.
  • 3.4 Multimodal perception pretraining: Perception-pretrained MLLMs improve both Transfusion-RAE and RepFusion, while fine-tuning helps language-only backbones but degrades RepFusion-RAE with pretrained multimodal backbones.The ablation suggests preserving multimodal perception pretraining rather than re-optimizing it for generation.
  • 3.5 Ingredient analysis: 0.47 to 0.54 GenEval: feeding noisy VAE latents into the LLM improves over text-only conditioning, while RAE latents raise the score further to 0.64.Joint LM-loss training with bidirectional attention provides a minor additional gain to 0.65, and multimodal perception pretraining further improves RAE-space denoising.
  • 3.5 Ingredient analysis: 0.64 to 0.68 GenEval: reallocating 1.3B trainable parameters from replicated LLM layers to a separate DiT improves Transfusion-RAE, while freezing the LLM reaches 0.70.The comparison uses a stronger 8.0B Transfusion-RAE baseline and keeps the trainable-parameter budget matched before freezing.

4 Experiments

Experiments show that RepFusion achieves strong text-to-image alignment and competitive benchmark performance with limited training data, while SFT reaches state-of-the-art levels. Ablations further identify AdaLN-Single as the preferred conditioning interface and show that larger DiTs generally provide better iso-FLOPs scaling than larger MLLMs.

  • T2I Generation: The 7B MLLM and 3.2B DiT configuration performs competitively on GenEval, GenEval++, GenEval2, and DPG-Bench, while RepFusion-SFT reaches state-of-the-art levels.The evaluation reports Soft-TIFAGM for GenEval2.
  • T2I Generation: RepFusion compares favorably with BAGEL with Self-CoT despite BAGEL being pretrained on over 1 billion web-scale examples.
  • Reasoning-Based Generation: RepFusion leverages a frozen LLM to follow complex prompts involving world knowledge and reasoning, matching state-of-the-art performance on WISE.
  • Conditioning Interface: AdaLN-Single achieves a slightly higher GenEval score with fewer parameters than cross attention and is therefore used as the default conditioning interface.Its token-wise adaptive normalization directly exploits the alignment between MLLM outputs and DiT tokens.
  • Scaling and Inference Compute: At around 280T FLOPs and around 540T FLOPs, configurations with larger DiTs outperform those allocating more compute to the MLLM on the reported comparisons.Increasing either component can improve performance, with clearer scaling trends on GenEval and GenEval++.

5 Conclusion

The conclusion identifies the conditional encoder as an underused degree of freedom in modern T2I systems. RepFusion instead uses a frozen MLLM to read noisy visual representations during denoising, motivating repeated conditioning when input-dependent information warrants its test-time cost.

  • The conditional encoder is an underused degree of freedom in modern T2I systems.
  • A frozen MLLM can read noisy visual representations, making the encoder an active part of the denoising loop rather than a static text encoder.
  • Future models should expose pretrained MLLM priors to evolving noisy representations and repeat conditioning only when it carries input-dependent information.The conclusion also recommends preserving those priors while spending test-time compute on repeated conditioning.

Appendix · A Details on Training TextEmbed and Transfusion Baselines

The appendix describes controlled training setups for the TextEmbed and Transfusion baselines, matching RepFusion’s setup and model-scale comparisons. TextEmbed follows Sana’s static text-encoder design with RMSNorm-normalized embeddings conditioning a newly initialized DiT denoiser.

  • A Details on Training TextEmbed and Transfusion Baselines: TextEmbed and Transfusion are trained with the same training setup as RepFusion.
  • A Details on Training TextEmbed and Transfusion Baselines: The appendix identifies Transfusion as another baseline trained under the shared comparison setup.
  • A Details on Training TextEmbed and Transfusion Baselines: The controlled comparisons use the same text-encoder family across methods.
  • A Details on Training TextEmbed and Transfusion Baselines: The comparisons use newly initialized denoising components at the corresponding model scale.
  • A Details on Training TextEmbed and Transfusion Baselines: TextEmbed follows recent T2I practice used in Sana by treating the LLM as a static text encoder.
  • A Details on Training TextEmbed and Transfusion Baselines: TextEmbed conditions a newly initialized DiT denoiser using the LLM’s last-layer text-token embeddings.
  • A Details on Training TextEmbed and Transfusion Baselines: Following Sana, TextEmbed applies RMSNorm after the decoder-only text encoder.
  • A Details on Training TextEmbed and Transfusion Baselines: RMSNorm normalizes the variance of TextEmbed’s text embeddings to 1.0.

B Impact of Decoders

RepFusion shows a performance gap between the RAE and diffusion decoders, but decoding the same CLIP feature produces similar images and preserves prompt-following ability. The remaining gap on GenEval and DPG-Bench appears partly attributable to blurrier RAE textures that hinder reliable evaluation.

  • Decoder impact: RepFusion exhibits a performance gap between the RAE decoder and the diffusion decoder.The paper reports results with both decoders in Table 1.
  • Decoder impact: The same CLIP feature decoded by either decoder yields similar layouts and colors, with differences mainly in fine-grained textures.This indicates that object layout and colors are determined upstream during CLIP-representation denoising, while decoder choice mainly affects texture.
  • Decoder impact: Decoder choice does not affect RepFusion’s prompt-following ability.The decoded images appear very similar at the object and layout level despite slight texture differences.
  • Decoder impact: Blurrier textures from the RAE decoder may partly explain its lower GenEval and DPG-Bench performance by making evaluation less reliable.Detectors and vision-language models may find these images harder to evaluate reliably.

C Additional Latent Space Comparison

This section isolates latent-space token count by evaluating DC-AE with 576 tokens, matching RAE’s token count. Matching the count does not improve the TextEmbed baseline.

  • C Additional Latent Space Comparison: DC-AE uses N=256 tokens versus N=576 for RAE in the main experiments, with comparable output resolutions.The comparison controls output resolution across latent spaces while retaining different token counts.
  • C Additional Latent Space Comparison: Increasing DC-AE to N=576 tokens does not improve the TextEmbed baseline.This setting isolates token count by matching DC-AE’s token count to RAE’s.
Loading 2606.14700v1…