Source-linked AI summary
Steering Visual Generation in Unified Multimodal Models with Understanding Supervision
Zeyu Liu, Zanlin Ni, Yang Yue, Cheng Da, Huan Yang, Di Zhang, Kun Gai, Gao Huang
TL;DR
Unified multimodal models often separate understanding and generation, leaving their potential mutual enhancement uncertain. UNO uses understanding supervision to steer generative representations, improving image generation and editing across benchmarks without degrading understanding performance.
Problem
Unified multimodal models commonly decouple understanding and generation, leaving their potential for mutual enhancement empirically uncertain.
Method
UNO jointly supervises generative representations with captioning and visual understanding objectives conditioned on noised generative representations.
Results
UNO consistently improves image generation and editing over strong baselines, including BAGEL from 71.7 to 75.1 on GenEval2 and without degrading understanding performance.
Takeaways & Limitations
Understanding supervision can serve as an effective catalyst for improving generation in unified multimodal models.
Takeaways & Limitations
UNO does not leverage specialized data for vertical domains and remains largely orthogonal to tasks requiring capabilities such as knowledge retrieval.
Abstract
from arXiv · showhide
Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.
1 Introduction
Unified multimodal models combine understanding and generation, but decoupled representations weaken their potential capability synergy. UNO addresses this gap by supervising generative representations with understanding signals and improves image generation performance without degrading understanding.
- Motivation: Unified Multimodal Models integrate language comprehension, visual understanding, and visual generation to support versatile any-to-any interaction.The introduction connects this integration to multimodal reasoning, free-form image manipulation, and interleaved world modeling.
- Motivation: State-of-the-art architectures increasingly decouple representations to preserve task-specific performance, weakening the desired mutual enhancement between understanding and generation.The decoupled paradigm is intended to alleviate optimization conflicts between high-level semantic abstraction and generation.
- UNO: UNO explicitly supervises generative representations with understanding signals, rerouting information flow through frozen understanding experts conditioned on intermediate noised generative representations.This design strengthens direct gradient flow from understanding to generation.
- Results: UNO improves BAGEL on image generation benchmarks: GenEval2 71.7 →75.1, DPG-Bench 84.03 →86.12, and UniGenBench++ 61.53 →65.03.The reported evaluation spans diverse generation and editing tasks and indicates consistent improvements over strong baselines without degrading understanding performance.
2 Related Work
Related work has pursued unified multimodal models by combining understanding and generation, using autoregressive visual tokens, hybrid MLLM–diffusion designs, or parallel unified backbones. A central concern is representation conflict, motivating semantically informed priors and latent spaces for improved generation.
- Unified multimodal models combine multimodal understanding and generation within a single framework, following advances in large language models and separate multimodal systems.
- Early unified approaches modeled visual content autoregressively by transforming images into sequences of discrete tokens.
- Hybrid architectures pair an MLLM backbone sequentially with a diffusion decoder, predicting either special query tokens or intermediate latent representations.
- Unified representation design is challenging because different tasks impose conflicting requirements, and a single shared visual representation can degrade task-specific performance.
- Understanding-oriented priors improve generative modeling through feature alignment methods such as REPA and REPA-E, and through semantically rich latent spaces in RAE and SVG.
3 Approach
The approach restores information flow from understanding to generation by jointly supervising generative representations with language-based semantic objectives and visual feature regression. It combines complementary signals through unified data packing while masking information flow to prevent leakage.
- Motivation: Unified multimodal models use specialized understanding and generation experts, but generation receives only low-level flow-matching gradients, leaving understanding signals isolated.Generation is conditioned on understanding representations, yet supervision remains effectively unidirectional.
- Language Supervision: Language supervision reroutes noised generation representations through the understanding expert, whose outputs are trained with an image-captioning objective.This propagates abstract pretrained semantics into the generative pathway through backward gradients.
- Language Supervision: Semantic re-captioning supplies lexically different but semantically consistent targets, preventing prompt-token copying and forcing semantic extraction from generation representations.Conditional prompt tokens are masked during supervision to reduce information leakage.
- Visual Understanding Supervision: Visual understanding supervision uses MetaQuery metaqueries to regress dense target-image features from the pretrained visual encoder, adding fine-grained visual and spatial structure.The metaquery hidden states are trained against visual representations using cosine similarity.
- Joint Supervision: Joint supervision combines captioning, visual regression, and standard generation objectives, packing signals into one sequence while modifying attention masks to prevent cross-task leakage.The configuration makes the generative pathway optimize generation and understanding signals simultaneously.
4 Experiments
UNO consistently improves image generation and editing over BAGEL and strong baselines, while ablations show complementary benefits from language and visual understanding supervision. Analyses attribute these gains to better-organized generative features, non-conflicting optimization gradients, and improved semantic augmentation.
- Image Generation: UNO consistently improves image generation across GenEval2, DPG-Bench, and UniGenBench++ over BAGEL and generation-only and unified-model baselines.Gains on UniGenBench++ are pronounced for compound, attribute, action, and relationship dimensions, with only slight degradation in world-knowledge scores.
- Image Editing: UNO improves image editing across semantic consistency, perceptual quality, and overall metrics on GEdit-Bench-EN/CN, achieving the best overall GEdit-Bench-EN score.The improvements include notable perceptual-quality gains while preserving edit intent and transfer to Chinese evaluations despite English-only training.
- UNO as an effective post-training approach: Language and visual understanding supervision each improve post-training performance independently, while their combination achieves the best results for both generation and editing.The editing ablation states that each signal improves over the SFT baseline and that their combination performs best on GEdit-Bench.
- UNO improves generative features: Understanding supervision improves generative representations by reducing noise and preserving better semantic information and fine-grained details in highly noised latent features.With generative supervision alone, intermediate representations show substantial noise and weak semantic organization; UNO produces more structured features.
- Optimization gradient directions: Understanding and generative gradients are largely orthogonal across layers, with some positive alignment, suggesting that understanding supervision avoids optimization conflicts with denoising.The gradient-direction analysis supports compatibility between the two objectives during generative training.
- Ablations and extensions: Semantic augmentation with different captions reduces leakage and enhances performance, while UNO also consistently outperforms CoT alone and complements CoT when combined.Using the same prompt remains beneficial but is more prone to leakage; connector architectures and supervision targets otherwise yield similar generation performance, motivating identity projection with cosine similarity.
5 Conclusion
UNO explores directly supervising generation with understanding in unified multimodal models through a lightweight post-training framework. Its complementary objectives strengthen gradient flow from understanding to generation, and experiments across generative tasks validate its effectiveness as a catalyst for enhanced generation.
- UNO presents a preliminary exploration of directly supervising generation with understanding in unified multimodal models.
- The lightweight post-training framework combines complementary objectives to strengthen gradient flow from understanding to generation.
- Extensive experiments across different generative tasks validate UNO’s effectiveness in enhancing generation.
A Training Settings
The paper provides detailed post-training settings for BAGEL on image generation and editing tasks in Table 11.
- Training Settings: The reported settings apply to both image generation and image editing.These task-specific parameters are documented as part of the training setup.
- Training Settings: Table 11 presents the detailed training parameters for post-training BAGEL.The table covers image generation and editing tasks.
B Training Sample Statistics
UNO training samples incur only a marginal token overhead compared with standard text-to-image training. Table 12 reports average token counts for each packed-sample component.
- B Training Sample Statistics: UNO reports average token counts for each component in its packed training samples.These statistics are presented in Table 12.
- B Training Sample Statistics: The added text captions and meta-query tokens create only marginal overhead relative to standard text-to-image text conditions and image tokens.The comparison is made against standard text-to-image training components.
C Image Generation Training Data
The image-generation training data combines diverse open-source resources spanning natural, synthetic, and design imagery, then applies resolution-based filtering. Captions are generated by prompting Qwen2.5-VL-7B to describe each target image accurately and in detail.
- Data curation: The dataset draws from LAION, JourneyDB, OpenImages, and other open-source resources, covering natural, synthetic, and design imagery across many categories.Categories include objects, landscapes, human subjects, animals, plants, food, indoor scenes, artistic designs, and sports.
- Data curation: Images below 512 pixels are excluded through a systematic resolution-based filtering pipeline.The supplied passage truncates the remaining filtering criteria.
D Further Ablations · E More Visualization Comparisons
Further ablations show that visual supervision is sensitive to metaquery count and prompt masking, while unfreezing the understanding expert offers no consistent generation gains and may weaken supervision. Additional qualitative comparisons evaluate UNO against representative generative and unified baselines.
- D Further Ablations: Metaquery counts above 256 degrade performance, indicating that 224 × 224 visual understanding supervision captures sufficient semantic details for generation.The study varies the number of metaqueries used for visual supervision and reports the results in Table 13.
- D Further Ablations: Unfreezing and fine-tuning the understanding expert does not consistently improve generation performance.The effect of jointly training the understanding expert is summarized in Table 14.
- E More Visualization Comparisons: Figure 9 provides more complete qualitative image-generation comparisons between UNO and competitive generation and unified model baselines.The figure is described as containing more comprehensive comparisons.
- D Further Ablations: Optimizing the understanding expert solely with the proxy objective may degrade standard understanding benchmarks and weaken supervision for the generation expert.This links reduced understanding-benchmark performance to weaker supervision signals for generation.
- D Further Ablations: Without masking condition prompts, information leakage mostly undermines the effect of understanding supervision.The masking-condition experiment is reported in Table 15.
- D Further Ablations: Causal versus bidirectional prediction of metaquery tokens does not significantly affect performance.The prediction-order experiment is reported in Table 16.
- E More Visualization Comparisons: The qualitative comparison includes FLUX, Janus-Pro, BLIP3-o, Show-o2, and BAGEL as representative generative and unified baselines.These baselines are named in the accompanying description of Figure 9.
F Editing Evaluations
The paper evaluates image editing more comprehensively on ImgEdit and KRIS-Bench, reporting quantitative comparisons in separate tables.
- Editing Evaluations: The editing evaluation uses ImgEdit and KRIS-Bench as benchmarks.The paper identifies these as the datasets for its more comprehensive image-editing evaluation.
- Editing Evaluations: Table 17 reports quantitative comparisons on ImgEdit.
- Editing Evaluations: Table 18 reports quantitative comparisons on KRIS-Bench.
G Evaluation Robustness · H Detailed Prompt List · I Societal Impact and Safeguards
The paper reports evaluation means and standard deviations across four random seeds, provides detailed prompts for qualitative generations, and discusses deepfake misuse alongside Apache-2.0 safeguards. These materials cover evaluation robustness, prompt reproducibility, and societal-risk mitigation.
- G Evaluation Robustness: Evaluation robustness is assessed by reporting the main evaluation’s mean and standard deviation across four random seeds.The paper states that these statistics demonstrate the evaluation’s robustness.
- H Detailed Prompt List: Detailed prompts for generated images shown in Figure 5 are provided in Table 19.This documents the prompts associated with the qualitative image-generation results.
- G Evaluation Robustness: Table 20 presents UNO’s mean and standard deviation results.The supplied table caption identifies these statistics as belonging to UNO.
- H Detailed Prompt List: One documented prompt describes a hyper-realistic digital sketch of an egg containing a miniature city of London.The scene includes London landmarks such as Tower Bridge, the River Thames, Big Ben, and Buckingham Palace.
- I Societal Impact and Safeguards: The paper frames its contribution as foundational research rather than work tied to specific applications or deployments.This scope statement appears in the societal-impact discussion.
- I Societal Impact and Safeguards: Improved generative-model quality could enable deepfakes for disinformation, creating a potential abuse risk.The paper recommends using released models according to the Apache-2.0 license and accompanying terms of use to mitigate abuse.