PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Penghao Wu, Haiwen Diao, Weichen Fan, Lewei Lu, Dahua Lin, Ziwei Liu

arXiv:2609.01607v1cs.CV

TL;DR

The paper asks whether unifying visual understanding and generation produces genuine learning synergy rather than mere coexistence or competition. Using controlled native pixel-in, pixel-out UMMs, it finds architecture-dependent representation transfer, bidirectional gains for tasks sharing knowledge, and an end-to-end advantage when both capabilities must interact closely.

  • Problem

    UMMs expose understanding and generation through one model, but it remains unclear whether joint learning reinforces, competes for, or merely coexists within shared capacity.

  • Method

    The study evaluates representation, task, and system interactions in a controlled native pixel-in, pixel-out setting without pretrained vision representations.

  • Results

    Understanding and generation mutually provide useful signals, task-decoupled routing avoids asymmetric degradation, shared-knowledge tasks benefit bidirectionally, and end-to-end UMMs outperform matched pipelines on tightly coupled image editing.

  • Takeaways & Limitations

    UMMs provide value beyond a unified interface when conflicting visual computation is specialized and training and inference are unified for shared knowledge and decisions.

  • Takeaways & Limitations

    Conclusions are centered on autoregressive text with continuous diffusion-based image modeling, and the broader architecture design space remains insufficiently explored.

Abstract

from arXiv · show

While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.

1 INTRODUCTION

The paper studies whether visual understanding and generation truly benefit one another in native UMMs, examining their interaction at representation, task, and system levels. It finds that synergy depends on architecture, shared task knowledge, and tight end-to-end interaction.

  • UMMs unify visual understanding and generation, enabling interleaved visual-textual reasoning and iterative image interpretation and modification.
  • The study addresses limited and conflicting evidence about whether joint learning reinforces, competes for, or merely coexists within shared model capacity.
  • Representation level: Shared computation creates asymmetric trade-offs: dense sharing improves understanding while degrading generation, whereas modality-decoupled routing reverses this pattern.
  • Representation level: Generation enriches understanding representations and understanding improves vision–language alignment for generation, but task-decoupled routing is developed to avoid dominance through one computation path.
  • Task level: Joint training benefits both directions when understanding and generation tasks share knowledge or capabilities across geometry, vector graphics, and 3D spatial intelligence.
  • The controlled study examines understanding–generation interaction across representation, task, and system levels in a native multimodal setting.
  • System level: An end-to-end UMM outperforms a matched agentic pipeline on reasoning-intensive image editing that requires understanding, reasoning, and generation.

2 NATIVE UNIFIED MODELING

The native unified model uses a pixel-in, pixel-out interface while comparing controlled routing and parameter-sharing choices. Its architectures vary which branches process text, understanding visuals, and generation visuals.

  • The adopted modeling paradigm combines autoregressive text modeling with continuous diffusion or flow matching for image generation.
  • Native visual modeling means visual information enters and leaves in pixel space without importing representations from external pretrained vision models.
  • The model converts images into patch tokens and processes multimodal sequences with causal cross-sequence attention and bidirectional within-image attention.
  • Routing settings: Dense sharing routes text, clean understanding visual tokens, and noised generation visual tokens through the same LLM decoder.
  • Routing settings: Modality-decoupled MOT keeps text in the pretrained LLM branch while routing all visual tokens through a scratch-initialized branch.
  • Routing settings: Task-decoupled MOT keeps understanding visual tokens language-anchored and specializes a separate visual branch for generation tokens.

3 REPRESENTATION-LEVEL SYNERGY

Joint training can improve visual representations and transfer between understanding and generation, but shared routing creates asymmetric degradation. Task-decoupled visual computation preserves semantic interaction while avoiding these opposite trade-offs.

  • Visual representation changes: Joint training strengthens visual representations, with clearest gains in early and middle layers across semantic and geometric probing tasks.Probes cover classification, semantic segmentation, and monocular depth estimation; PCA maps also show more coherent object regions and clearer spatial structure.
  • Dense sharing: In dense sharing, joint training improves visual understanding but substantially degrades generation relative to the generation-only model.The pretrained language model can act as a semantic anchor, while competition within shared parameters degrades generation capability.
  • Modality-decoupled routing: In modality-decoupled routing, joint training improves generation but makes visual understanding worse than understanding-only training.Generation dominates the shared visual computation in this setting, while understanding primarily provides auxiliary supervision.
  • Vision-language alignment: Understanding supervision improves generation representations by producing consistently higher text-image CKA than generation-only training.The CKA analysis uses pooled text and generative image features from 5,000 COCO validation text-image pairs.
  • Task-decoupled routing: Task-decoupled MOT specializes conflicting visual pathways while retaining shared semantic interaction, avoiding the opposite trade-offs of dense and modality-decoupled models.Text and clean image tokens use the pretrained language branch, while noised generation tokens use a generation-specialized visual branch; the branches attend to shared textual features.

4 TASK-LEVEL SYNERGY

Task-level synergy emerges when understanding and generation share underlying domain knowledge or capabilities. Across geometry, SVG, and 3D spatial intelligence, joint supervision transfers positively between tasks, often improving both directions.

  • 4.1 CASE I: GEOMETRY PROBLEM SOLVING: Geometry generation and reasoning share domain knowledge because both must represent entities, relations, constraints, and valid visual structures.
  • 4.1 CASE I: GEOMETRY PROBLEM SOLVING: Joint training improves geometry understanding across four benchmarks and transfers gains to general mathematical reasoning.Using the same additional examples as text-description tasks yields smaller and less consistent benefits, indicating generation is more effective auxiliary supervision in this setting.
  • 4.2 CASE II: SVG UNDERSTANDING AND GENERATION: Joint training improves both SVG image understanding and SVG-conditioned image generation.The two directions share an executable relationship between vector operations and their visual consequences.
  • 4.2 CASE II: SVG UNDERSTANDING AND GENERATION: Joint SVG training strengthens visual simulation from code, improving blind code-conditioned VQA and enabling earlier recovery of contours and spatial layout.A single generation step also visualizes how readily the model forms global shape and layout from SVG code.
  • 4.3 CASE III: 3D SPATIAL INTELLIGENCE: In 3D spatial intelligence, joint training improves eight of nine understanding benchmarks, raises average performance from 57.15 to 59.01, and improves the evaluated generation metrics.Generation performance improves on Spatial-Edit and across all three VSC metrics.
  • 4 TASK-LEVEL SYNERGY: Across the three case studies, shared underlying knowledge or capabilities make understanding and generation mutually reinforcing.

5 SYSTEM-LEVEL SYNERGY

The paper compares an end-to-end UMM with a matched planner–executor pipeline on reasoning-intensive image editing. The end-to-end model outperforms the modular pipeline on both evaluated benchmarks when understanding, reasoning, and generation must interact closely.

  • 5 SYSTEM-LEVEL SYNERGY: Reasoning-intensive image editing requires understanding a source image, inferring the intended concrete edit, and generating the target image.The task supports both end-to-end and planner–executor implementations.
  • 5 SYSTEM-LEVEL SYNERGY: The end-to-end format performs reasoning, produces an explicit edit instruction, and then generates the target image.
  • 5 SYSTEM-LEVEL SYNERGY: The agentic pipeline separates planning from execution by composing a planner that outputs an edit instruction with an executor that generates the target image.
  • 5 SYSTEM-LEVEL SYNERGY: The end-to-end UMM outperforms the planner–executor pipeline on both RISEBench and KRIS-Bench.The comparison uses the same task-decoupled checkpoint, matched base model, and matched task data.
  • 5 SYSTEM-LEVEL SYNERGY: These results show that end-to-end unification can provide a performance advantage for complex tasks explicitly requiring visual understanding and generation.

6 RELATED WORK

Related work establishes that UMMs combine visual understanding and generation, while prior evidence on whether the objectives synergize remains mixed. The paper addresses this uncertainty in a controlled native setting spanning representation, task, and system levels.

  • Unified multimodal models: UMMs integrate visual understanding and generation in one model, enabling text and images to be consumed and produced as interleaved content.They support interactions such as generating mixed visual-textual content and iteratively interpreting and modifying images.
  • Native multimodal models: Native multimodal models may be defined by multimodal training from the beginning or by processing visual inputs and outputs directly in the multimodal backbone.
  • Native multimodal models: This study initializes from a pretrained LLM and focuses on how visual understanding and generation are learned and interact.
  • Understanding and generation: conflict or synergy?: Prior studies report both interference and transfer between understanding and generation, leaving no clear consensus about whether they intrinsically benefit each other.
  • Understanding and generation: conflict or synergy?: The paper isolates this relationship by studying representation, task, and system interactions in a controlled native setting.The motivation is to disentangle effects otherwise confounded by specialized modules, auxiliary losses, task formulations, or pretrained visual components.

7 LIMITATIONS

The study’s conclusions are scoped primarily to discrete autoregressive text modeling combined with continuous diffusion-based image modeling and to a largely task-decoupled architecture.

  • 7 LIMITATIONS: The generality of the conclusions across alternative multimodal formulations remains empirically unvalidated.The paper focuses on discrete autoregressive text and continuous diffusion-based image modeling.
  • 7 LIMITATIONS: The experiments do not extensively explore the broader architecture design space or identify the optimal balance between sharing and specialization.

8 CONCLUSION

The paper finds that understanding–generation synergy in native UMMs depends on architecture, shared task knowledge, and end-to-end interaction. These factors extend UMM value beyond a unified interface.

  • Mutual benefit between understanding and generation is architecture-dependent.
  • Task-aware specialization allows both capabilities to coexist without asymmetric degradation.
  • Joint training improves both directions when tasks share underlying knowledge.
  • An end-to-end UMM outperforms a matched modular pipeline when understanding, reasoning, and generation must interact closely.
  • UMMs provide value beyond a unified interface through knowledge transfer and end-to-end optimization across the two capabilities.

A.1 DATA

The data pipeline uses tailored sampling for understanding and generation, with a fixed mixture for joint training and a specified model configuration. Figure 7 summarizes the visual-understanding category distribution.

  • Understanding data are drawn from SenseNova-U1 using a tailored sampling mixture.
  • Generation data exclude infographics, downweight text-rendering samples, and retain only English prompts.
  • Joint training samples visual-understanding, text-to-image, and image-editing data in a relative ratio of 3 : 6 : 1.
  • Figure 7 reports the category distribution of the visual-understanding training data.
  • Models are initialized from Qwen3-1.7B with a 16-layer pre-buffer and use MLP-based timestep and noise-scale embeddings added directly to visual inputs.
  • Representation-level training configurations are provided in Table 7.

A.4 EVALUATION

Evaluation uses selected benchmark subsets and fixed decoding settings across visual understanding and generation. Representation-level training configurations and full understanding results are documented in Table 7 and Table 8.

  • Visual-understanding evaluation uses selected subsets of SEED-Bench, MME-RealWorld, and MindCube, plus MCQ questions from PerceptionBench.
  • All benchmarks are evaluated with temperature set to 0.
  • Table 7 presents representation-level training configurations, while Table 8 reports full visual-understanding benchmark results.
  • Generation evaluation produces 1024 × 1024 images with classifier-free guidance scale 4.0, 50 sampling steps, and timestep shift 3.0.

A.5 PROBING EXPERIMENTS

The probing experiments evaluate frozen visual representations through linear classification and dense prediction, alongside detailed generation benchmarks and an SVG-specific VQA diagnostic.

  • ImageNet probing freezes the multimodal backbone, globally pools visual features, and trains a linear classification head for 100 epochs with LARS.
  • Segmentation and depth probing freeze the backbone and use three convolution-and-upsampling blocks to recover spatial resolution from downsampling factor 32.
  • GenEval2 and DPG-Bench provide generation results by skill and level-1 category, respectively, with higher scores better.
  • The SVG Blind VQA diagnostic contains 1,370 filtered multiple-choice examples testing visual-appearance inference from SVG code.
  • The diagnostic examples are shown in Figure 8.

B.2 GENERATION TASKS FOR 3D SI

The paper constructs four generation tasks targeting different aspects of 3D spatial intelligence, spanning camera-conditioned view synthesis, multi-view inference, temporal completion, and layout-conditioned scene rendering.

  • Four generation tasks probe distinct aspects of 3D spatial intelligence: ego-motion transition, multi-view reconstruction, view sequence completion, and layout-to-image generation.Together, they cover camera transformations, unobserved-view inference, missing-frame recovery, and scene synthesis from structured or textual layouts.
  • Ego-motion transition predicts a target view from a source image and sequential camera transformations.
  • Multi-view reconstruction generates an unobserved object view from two provided views.
  • View sequence completion recovers a missing middle frame while maintaining temporal and geometric consistency.
  • Layout-to-image generation renders a scene from structured 3D object annotations or natural-language descriptions of scene layout.
Loading 2609.01607v1…