PaperlayerView exact revision on arXiv

Source-linked AI summary

UniFusion: Vision-Language Model as Unified Encoder in Image Generation

Kevin Li, Manuel Brack, Sudeep Katakol, Hareesh Ravi, Ajinkya Kale

arXiv:2510.12789v1cs.CVcs.AIcs.LG

TL;DR

Separate text and image encoders complicate cross-modal image-generation workflows, while some VLM-based approaches have struggled to preserve fine visual detail. UniFusion conditions a diffusion model on a frozen VLM through multi-layer feature pooling and rewritten prompts. It reports competitive generation and editing, along with zero-shot generalization and positive transfer from editing training.

  • Problem

    Separate text and image representations differ in abstraction, and prior VLM-conditioned approaches have struggled to preserve fine-grained visual details needed for high-fidelity editing.

    Source: §1 · p. 1, §2.1 · p. 7

  • Method

    UniFusion uses a frozen VLM as a unified text-and-image encoder, combining multi-layer LAP features with Verifi rewritten prompts for DiT conditioning.

    Source: §1 · p. 5

  • Results

    UniFusion achieves competitive text-to-image generation and editing, generalizes to multi-reference inputs after single-reference editing training, and editing training improves text-to-image prompt adherence and aesthetic quality.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The findings support a robust, flexible strategy for using VLMs as unified encoders, with reported zero-shot capabilities and efficient adaptation of existing models.

    Source: §6 · p. 21

    Text rendering remains problematic, with InternVL particularly poor at spelling and often producing misspellings.

    Source: §6 · p. 20

Abstract

from arXiv · show

Although recent advances in visual generation have been remarkable, most existing architectures still depend on distinct encoders for images and text. This separation constrains diffusion models' ability to perform cross-modal reasoning and knowledge transfer. Prior attempts to bridge this gap often use the last layer information from VLM, employ multiple visual encoders, or train large unified models jointly for text and image generation, which demands substantial computational resources and large-scale data, limiting its accessibility.We present UniFusion, a diffusion-based generative model conditioned on a frozen large vision-language model (VLM) that serves as a unified multimodal encoder. At the core of UniFusion is the Layerwise Attention Pooling (LAP) mechanism that extracts both high level semantics and low level details from text and visual tokens of a frozen VLM to condition a diffusion generative model. We demonstrate that LAP outperforms other shallow fusion architectures on text-image alignment for generation and faithful transfer of visual information from VLM to the diffusion model which is key for editing. We propose VLM-Enabled Rewriting Injection with Flexibile Inference (VERIFI), which conditions a diffusion transformer (DiT) only on the text tokens generated by the VLM during in-model prompt rewriting. VERIFI combines the alignment of the conditioning distribution with the VLM's reasoning capabilities for increased capabilities and flexibility at inference. In addition, finetuning on editing task not only improves text-image alignment for generation, indicative of cross-modality knowledge transfer, but also exhibits tremendous generalization capabilities. Our model when trained on single image editing, zero-shot generalizes to multiple image references further motivating the unified encoder design of UniFusion.

1 Introduction

UniFusion uses a frozen VLM as a shared text-and-image encoder to address the limits of separate conditioning spaces and the costs of joint multimodal training. Its components support generation and editing, with reported zero-shot generalization and positive transfer across tasks.

1 Introduction

  • Separate text and image encoders operate at different abstraction levels, and prior VLM-conditioned models have struggled to preserve fine-grained visual details needed for high-fidelity editing.The paper argues that heterogeneous feature alignment uses DiT capacity that could otherwise support image synthesis.Source: §1 · p. 5
  • UniFusion uses a frozen VLM as a unified encoder, with LAP aggregating multiple layers and Verifi conditioning the DiT on a rewritten target prompt.The framework is presented as a way to capture both fine-grained visual details and high-level semantics while reducing distribution shift between prompt formats.Source: §1 · pp. 5–6
  • UniFusion achieves competitive text-to-image generation and editing, generalizes to multi-reference inputs after single-reference editing training, and improves prompt adherence and aesthetic quality when trained on editing.It also performs image-to-image variation after training exclusively on text-conditional generation.Source: §1 · pp. 5–6
  • The model showcases complex-instruction prompt following, reference-image use for content and style, text-driven editing, and visual reasoning.The introduction also identifies multi-image inputs and generalization to unseen tasks as capabilities of the unified-encoder setup.Source: §1 · pp. 5–6
  • A single model performs text-to-image generation and editing using VLM input features, without separate VAE-based image-reference conditioning.The paper reports competitive performance on both task types compared with strong contemporaries.Source: §1 · pp. 5–6

2 Architecture Selection

The section compares ways to condition a diffusion transformer on a unified VLM encoder, then evaluates feature extraction and injection choices. Layerwise Attention Pooling (LAP) is the strongest candidate, with image-detail preservation depending on providing enough VLM image tiles.

2.1 VLM conditioning candidates

  • LAP aggregates representations across VLM layers, avoiding the single-layer detail limitation while retaining flexibility in DiT architecture design.Key-Value Fusion and Hidden State Injection tie the encoder and generator more tightly through layer and dimension requirements.Source: §2.1 · p. 7

2.2 Experimental Evaluation

  • LAP gives the strongest prompt adherence among the tested conditioning candidates, while Key-Value Fusion performs worst and naive Llama replacement does not surpass T5.The comparisons use text-to-image VQA on GenAI-Bench after 200k training steps.Source: Figure 6 · p. 8, §2.2 · pp. 8–9
  • At 10 image tiles, reconstruction errors become largely imperceptible; small, fine-grained images require more tiles to preserve detail.The study concludes VLM features can encode image details without additional feature injection.Source: Figure 7 · p. 9, §2.2 · p. 10

Representation Injection

  • A single pooled LAP representation strongly outperforms injecting LAP features into individual DiT layers.The comparison controls for the total parameter count across the LAP modules.Source: Representation Injection · p. 10

3 UniFusion Design

The design analysis refines LAP by selecting VLM layers, adding a bidirectional refiner, and using VLM-generated rewrites for conditioning. It also finds that adapting a T5-trained checkpoint can preserve performance relative to training from scratch.

3.1 Layer Selection

  • Sampling every third VLM layer reduces redundancy and overhead; dropping middle-layer information changes image composition more than dropping the first or last layers.The revised LAP assigns more uniform weights, and the penultimate layer contributes least despite its common use in other methods.Source: §3.1 · pp. 10–11

3.2 Position Bias

  • A two-block bidirectional refiner addresses causal-attention position bias, and the authors report that both it and LAP are crucial for optimal performance.Causal masking can leave a late-mentioned subject insufficiently represented.Source: §3.2 · pp. 11–12

3.3 VLM-Enabled Rewriting Injection with Flexible Inference (Verifi)

  • Verifi conditions the DiT on rewritten text tokens while retaining original prompt context, and improves prompt following, including on long, detailed prompts.In multimodal settings, image tokens are also injected to preserve fine-grained details; the authors also report zero-shot reasoning over complex inputs.Source: §3.3 · pp. 12–13

3.4 Finetuning vs. Training from Scratch

  • After 100k T5-conditioned training steps, roughly 10k steps with InternVL-2.5-8B LAP suffice for coherent images, and the continued model performs similarly to training from scratch at matched sample counts.The authors report preserved benchmark performance and qualitative capabilities, including self-rewrite, multimodal use, and zero-shot editing.Source: §3.4 · pp. 13–14

4 Final UniFusion Model

The final UniFusion model combines a scaled-up DiT with LAP-based VLM features and rewritten prompts. The authors report competitive qualitative performance against larger models, while flagging substantial reliability problems in popular prompt-following benchmarks.

4.1 UniFusion Design & Training

  • The final model uses an 8-billion-parameter DiT, InternVL3-8B, and LAP features extracted from every third VLM layer.A two-block refiner follows LAP; image inputs use up to 10 tiles, and the model is trained on generation, reconstruction, and text-and-image data before instruction training for editing and reference workflows.Source: §4 · p. 14, §4.1 · pp. 14, 16

4.2 Evaluation

  • GenEval and DPG-Bench have high error rates, with benchmark noise exceeding model differences of a few percentage points.The authors also report issues including penalization of photorealistic outputs and inaccurate DPG-Bench score aggregation.Source: §4.2 · pp. 16–17
  • UniFusion remains competitive with significantly larger, heavily post-trained models and is less likely to produce over-saturated colors or smoothed textures.The direct comparisons use the same checkpoint for text-to-image and editing only for Bagel and UniFusion; Flux and Qwen-Image use task-specific versions.Source: Figure 14 · p. 15, §4.2 · pp. 16, 18, Figure 15 · p. 17

5 Emergent Abilities

UniFusion’s unified VLM encoder supports reasoning and transfer to tasks and modalities unseen during DiT training. Editing and reference training also improve text-to-image generation, while single-reference training generalizes to multi-reference composition.

5 Emergent Abilities

  • UniFusion exhibits zero-shot capabilities that the authors attribute to retained VLM capabilities and a unified contextualized text-image space with smaller distribution shifts between tasks.Figure 17 describes LAP-extracted VLM features enabling generalization to unseen tasks and modalities.Source: §5 · p. 18, Figure 17 · p. 18

5.1 Reasoning & Complex Prompts

  • Verifi uses VLM reasoning to interpret abstract prompts and decompose hypothetical scenarios into image-editing instructions that satisfy user intent.For example, InternVL resolves a zodiac prompt to Taurus and rewrites it as a bull scene.Source: §5.1 · pp. 18–19

5.2 Generalization to unseen modalities

  • Models trained only on text or image sequences can still perform image editing, while text-to-image-only training transfers image semantics imperfectly but meaningfully.For editing, conditioning on image tokens for the first 10–20% of steps and text tokens for the remaining 80–90% preserves much of the input content and semantics.Source: §5.2 · p. 19

5.3 Cross-Task Improvements

  • Over 2 percentage points in DPG-Bench Micro Avg accompanied improved text-to-image quality after editing and reference training; annotators preferred UniFusion-Edit across all aspects.The human study compared checkpoints using 180 annotators and 616 prompts.Source: §5.3 · p. 19, Figure 18 · p. 19, Table 2 · p. 20

5.4 Zero-shot multi-reference capabilities

  • Despite training only on single-reference editing data, UniFusion composes scenes from multiple references and handles different input and output aspect ratios and resolutions.It can expand a scene and shift perspective while preserving fine-grained details.Source: §5.4 · p. 20

6 Conclusion

The conclusion presents UniFusion as a unified VLM encoder for image generation, with LAP outperforming alternative architectures and supporting generalization. It also notes encoding-time costs and weaknesses in InternVL’s spelling when prompts are rewritten.

6 Conclusion

  • Rewriting all prompts with an 8B transformer increases encoding compute and runtime, though the authors say this limitation is not unique to UniFusion.InternVL’s spelling weaknesses can also produce incorrect or illegible text in generated scenes.Source: §6 · pp. 20–21
  • Gemma-based models work similarly well, leading the authors to conclude that UniFusion is not limited to one VLM family..Source: §6 · p. 21
  • UniFusion uses a single VLM as a unified encoder for generative image models, with LAP aggregating features across frozen-VLM layers.The framework is presented as a flexible strategy for using VLMs as unified encoders.Source: §6 · p. 21
  • LAP outperforms last-layer encoding and key-value fusion in prompt adherence and fine-grained detail preservation, and UniFusion eliminates the need for multiple image encoders.The authors also identify layer selection, bidirectional refinement, and Verifi as design considerations.Source: §6 · p. 21
  • UniFusion’s VLM reasoning and world knowledge support zero-shot generalization to unseen use cases, including complex prompts, visual reasoning, and image-reference tasks without explicit training.The conclusion describes the framework as computationally viable for adapting existing models.Source: §6 · p. 21

A Additional Results & Experimental Details

The appendix describes LAP’s feature aggregation and compares ways to inject its representations into the DiT. A single pooled representation outperforms dedicated layerwise injection in the reported comparison.

A.1 UniFusion layerwise representation aggregation

  • LAP extracts features from every third VLM layer, aggregates corresponding tokens across layers with two transformer blocks, then pools them into a dense representation and applies a bidirectional refiner.The refiner uses two bidirectional transformer blocks to mitigate position bias across the input sequence.Source: §A.1 · p. 27
  • The appendix also gives implementation details for the LAP transformer blocks, including 32 attention heads, RMS normalization, and a SiLU feed-forward block.The feed-forward block expands and contracts hidden dimension by a factor of 1.3.Source: §A.1 · p. 27

A.2 Representation Injection

  • A single LAP representation input to the DiT without later-layer injections strongly outperforms dedicated per-layer LAP representations injected through hidden states.The comparison suggests that injecting conditioning into later DiT layers may be counterproductive.Source: §A.2 · p. 27

A.3 Bidirectional Refiner

The bidirectional-refiner comparison finds that InternVL with a refiner closes the text-to-image capability gap to the T5 baseline, while LAP still beats last-layer extraction.

A.3 Bidirectional Refiner

  • The separate injection comparison reports no performance improvement from injecting aggregated representations at different DiT depths.It compares dedicated per-layer representations with a single pooled representation at 200k training steps.Source: Figure 20 · p. 28
  • Combining InternVL with a bidirectional refiner closes the text-to-image capability gap to the T5 baseline in the reported comparison.The comparison evaluates InternVL-2.5 8B models using a refiner on penultimate-layer features and using LAP with a refiner.Source: §A.3 · pp. 27–28
  • LAP with bidirectional refinement still outperforms representation extraction from last layers.The authors conclude that both multi-layer feature extraction and bidirectional refinement are crucial for decoder-only autoregressive encoders.Source: §A.3 · p. 28

A.4 Continued Training vs Finetuning

Continued training from a T5-conditioned model and training with InternVL2.5-8B from scratch reach the same performance, substantially outperforming the T5 baseline. LAP also outperforms last-layer representation extraction.

A.4 Continued Training vs Finetuning

  • Both continued-training and from-scratch InternVL2.5-8B models reach the same performance and substantially outperform the T5 baseline.The comparison uses 250k total steps; the continued checkpoint switches from T5 to InternVL2.5-8B at 100k steps.Source: §A.4 · p. 28, Figure 22 · p. 29
  • Continuing late training from an existing T5-conditioned model can save compute compared with training the unified conditioning from scratch.The passage states this conclusion but does not provide a compute-savings amount.Source: §A.4 · p. 28
  • LAP still outperforms last-layer representation extraction, even when a bidirectional refiner narrows the performance gap to the T5 baseline.The comparison is at 250k training steps.Source: Figure 21 · p. 28, §A.4 · p. 28

A.5 Gemma-Based UniFusion

The Gemma-based UniFusion variant shows strong VQA performance and qualitative generation and reconstruction examples, with slight reconstruction variations at one input tile. The appendix also discusses limitations in image-generation benchmark reliability.

A.5 Gemma-Based UniFusion

  • 84.4% VQA score on GenaiBench is reported for UniFusion-Gemma with VERIFI.The appendix also provides text-to-image generation and reconstruction examples.Source: §A.5 · p. 29, Figure 23 · p. 29
  • With one input tile, Gemma-based image reconstruction shows slight variations that the authors expect increased input resolution to resolve.The figure describes the one-tile input as a thumbnail.Source: §A.5 · p. 29

B On the reliability of Image Generation Benchmarks

B.1 Evaluating popular benchmarks

The section identifies automated evaluator errors, ill-formulated questions, and benchmark priorities that may not match human preferences as reliability problems. These issues can make benchmark scores difficult to interpret.

Automated evaluation error

  • Automated evaluators can fail on object detection, counting, and attribute binding, with some evaluation errors exceeding 70% and potentially surpassing differences between models.The figure illustrates cases where benchmarks incorrectly score simple examples, including clear subjects.Source: §B.1 · p. 30, Figure 24 · p. 31

Ill-formulated tasks

  • DPG-Bench includes questions that are subjective, ungroundable in one image, or impossible to assess without context, such as nationality or historical significance.Questions about subjective qualities such as whether a painting “radiates” are also described as unsuitable for objective evaluation.Source: §B.1 · pp. 30–31

Questionable capability prioritization

  • Current benchmarks favor literal prompt adherence, although incomplete objects or out-of-focus backgrounds may serve aesthetic or compositional goals and strict adherence does not necessarily correlate with human preference.The authors report that this disconnect also applies to human-preference metrics such as ImageReward.Source: §B.1 · p. 31

B.2 Refined DPG benchmark

The refined DPG-Bench evaluation upgrades its question-answering model and inference procedure, corrects score aggregation, and reports additional averages and best-of-n results. Its prompt directs answers to rely on visible image evidence.

B.2 Refined DPG benchmark

  • The revised evaluation uses Gemma-3-27b-it for question answering and extends inference with chain-of-thought reasoning before a yes/no answer.The authors selected the strongest model from the Gemma family for evaluation.Source: §B.2 · pp. 31–32, Table 4 · p. 33
  • The authors reimplemented DPG-Bench score aggregation because the official script used only the last image for subcategory scores.The overall score was calculated across all images per prompt.Source: §B.2 · p. 32
  • Reporting adds category-wise Macro Avg alongside Micro Avg and best-out-of-n performance alongside means over multiple seeds.The authors say best-out-of-n better reflects users who can choose among multiple generated seeds.Source: §B.2 · p. 32
  • The question-answering prompt requires decisions based on visible evidence, with a four-step process to analyze the image, parse the question, reason, and conclude.It permits some interpretation of out-of-focus, background, or partially obscured objects.Source: §B.2 · p. 33
Loading 2510.12789v1…