PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Generative Tuning for Unified Multimodal Models

Songsong Yu, Yuxin Chen, Ying Shan, Yanwei Li

arXiv:2605.18714v2cs.CVcs.AI

TL;DR

UMMs’ separate text-supervised understanding and pixel-optimized generation can leave their representations misaligned. The paper systematically evaluates hierarchical visual tasks as generative proxies and introduces SGT, using segmentation to align the capabilities. SGT consistently improves multimodal comprehension and generative fidelity, with reported gains including 6.02% over BAGEL on CV-Bench and 90.0% on GenEval.

  • Problem

    Divergent sparse-text and dense-pixel supervision leaves visual understanding and generation isolated within UMMs, limiting their mutual reinforcement.

  • Method

    SGT systematically evaluates hierarchical visual tasks and uses image segmentation as a semantic generative proxy for coupling understanding and generation.

  • Results

    SGT consistently improves visual understanding and generation across mainstream architectures and benchmarks, including a 6.02% CV-Bench gain over BAGEL and a 90.0% GenEval score.

  • Takeaways & Limitations

    Semantic generative tuning aligns multimodal capabilities through structural semantics while improving feature linear separability and visual-textual attention allocation.

  • Takeaways & Limitations

    Using segmentation data exclusively constrains performance on symbolically dense and knowledge-intensive tasks, so SGT is best treated as a foundational alignment strategy.

Abstract

from arXiv · show

Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture. However, prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives. Such a decoupled strategy yields misaligned representation spaces, isolating visual understanding from generation and hindering their mutual reinforcement. This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs. Our empirical investigation reveals that high-level semantic tasks, particularly image segmentation, serve as optimal proxies. Unlike low-level tasks that distract models with texture details, segmentation provides structural semantics that significantly enhance both vision-centric perception and generative layout fidelity. Building upon these insights, we introduce Semantic Generative Tuning (SGT), a novel paradigm that leverages segmentation as a generative proxy to align and synergize multimodal capabilities. Mechanistic analyses further demonstrate that SGT fundamentally improves feature linear separability and optimizes visual-textual attention allocation pattern. Extensive evaluations show that SGT consistently improves both multimodal comprehension and generative fidelity across mainstream benchmarks. Our code is available on the https://song2yu.github.io/SGT/.

1 Introduction

UMMs seek unified understanding and generation, but divergent supervision leaves their capabilities weakly coupled. The paper finds that semantic generative tuning with segmentation aligns these capabilities and improves both perception and generation.

  • Motivation: Traditional UMM training uses sparse text supervision for understanding and low-level visual objectives for generation, isolating the two capabilities.This shared architecture therefore retains disjointed optimization processes rather than capturing dependencies between understanding and generation.
  • Motivation: Pixel-level reconstruction improves generation but overemphasizes granular texture details that are less relevant to semantic visual comprehension.The paper questions whether low-level reconstruction is an optimal proxy for coupling understanding and generation.
  • Proxy-task investigation: The study systematically compares low-, mid-, and high-level visual objectives and identifies image segmentation as the optimal generative proxy.Segmentation aligns more closely with the semantic demands of visual comprehension than low-level texture-focused tasks.
  • Semantic Generative Tuning: Semantic Generative Tuning (SGT) uses image segmentation as a generative proxy to couple visual understanding and generation.Mechanistic analysis links SGT to improved feature linear separability and optimized visual-textual attention allocation.
  • Results: 6.02% performance increase over BAGEL on CV-Bench and 90.0% on GenEval show SGT’s reported gains in understanding and generation.The paper also reports consistent improvements across diverse benchmarks and mainstream UMM architectures.

2 Related Work

Prior UMM research unifies processing through token-based or hybrid architectures, while generative representation learning often relies on reconstruction objectives. This work distinguishes itself by systematically testing hierarchical visual proxies and using semantic tasks to bridge discriminative and generative capabilities.

  • Unified multimodal models: Recent UMMs pursue unified any-to-any processing through discrete visual tokenization or hybrid causal-language-modeling and denoising architectures.These trajectories aim to combine interleaved reasoning, mixed-modality processing, and synthesis quality within unified systems.
  • Generative representation learning: Generative models have also been used for visual representation learning through data augmentation and self-supervised reformulations of generative processes.This line of work extends generative modeling beyond direct image synthesis.
  • Reconstruction for understanding and alignment: Existing alignment frameworks commonly rely on exact pixel reconstruction, whereas this work abandons raw pixel recovery to reduce representational redundancy.The paper instead validates how hierarchical visual proxy tasks affect generative tuning of UMMs.
  • Reconstruction for understanding and alignment: The proposed unified optimization establishes a shared semantic space intended to capture structural abstraction across discriminative and generative capabilities.The paper positions this systematic proxy-task validation as a bridge between perception and synthesis.

3 Semantic Generative Tuning

This section formulates generative tuning for UMMs and systematically compares visual proxy tasks across semantic levels. The findings favor high-level semantic objectives, especially segmentation, as a basis for SGT because they improve understanding while generative objectives improve spatial fidelity.

  • 3.1 Formulation: UMMs combine semantic vision features, VAE image latents, and noise differently for understanding, generation, and editing tasks.Understanding uses z_vit for textual responses, generation uses text and Gaussian noise for images, and editing integrates z_vit, z_vae, and noise.
  • 3.2 Motivation and Hierarchical Visual Task Taxonomy: Generative tuning models visual task learning as conditional generation from a text instruction and optional visual features toward a visual target.The comparison controls input RGB images and training volume while evaluating high-, mid-, and low-level visual tasks.
  • 3.3 From Empirical Observations to the SGT Paradigm: High-level semantic tasks, particularly image segmentation, outperform mid- and low-level objectives for multimodal understanding.The authors attribute this advantage to stronger alignment between semantic supervision and the reasoning requirements of understanding models.
  • 3.3 From Empirical Observations to the SGT Paradigm: Generative tuning strengthens vision-centric perception, spatial reasoning, and hallucination resistance, while chart recognition and mathematical knowledge remain static or marginally decline.The results indicate that visual supervision improves representation quality without adding knowledge or logical reasoning skills.
  • 3.3 From Empirical Observations to the SGT Paradigm: Various generative objectives consistently improve spatial fidelity, producing comparable overall gains across the evaluated generation dimensions.Figure 3 reports averages across twelve random seeds for position, colors, color attributes, counting, single-object, two-object, and overall dimensions.
  • 3.3 From Empirical Observations to the SGT Paradigm: SGT uses high-level visual proxies, especially image segmentation, to refine UMM representations and harmonize visual understanding with generation.Semantic instance, panoptic, and class-agnostic segmentation reportedly yield comparable improvements.

4 Experiments

Experiments evaluate SGT across UMM architectures, benchmarks, proxy tasks, data recipes, scaling behavior, feature spaces, and attention patterns. Results show consistent gains from segmentation-based tuning, with improved convergence, separability, and visual grounding.

  • Main Results: SGT-BAGEL and SGT-Gen2 consistently outperform their original architectures and competitive UMMs across multiple understanding and generation benchmarks.The enhanced variants use SAM segmentation data alongside visual understanding instruction tuning.
  • Main Results: SGT improves adherence to complex textual prompts, including spatial and color instructions, over the baseline model.The qualitative comparison focuses on compositional text-to-image generation.
  • Ablation Study: Segmentation, reconstruction, and edge proxy tasks improve most perception-centric understanding benchmarks, with SGT using segmentation targets.The ablation isolates SFT data and compares proxy tasks across semantic levels.
  • Optimal Data Recipe: A 2:1 segmentation-to-VQA ratio is optimal, while expanding segmentation data from 2k to 100k samples improves performance by 3.3% for BAGEL and 2.0% for OmniGen2.The ratio is measured within each training batch, and the scaling results concern multimodal understanding.
  • Scaling Properties: Segmentation objectives accelerate convergence on challenging benchmarks, including CV-Bench and Hallusion, as segmentation data increases.Average normalized performance rises monotonically with segmentation-data volume.
  • Mechanistic Analysis: SGT produces more discriminative embeddings with clear separation between Grand Piano and Upright Piano categories.The baseline feature space is entangled, whereas BAGEL+Segmentation yields clearer class separation.
  • Mechanistic Analysis: SGT increases deeper-layer attention to visual tokens and focuses text attention on critical Object, Color, and Relation tokens.These attention changes counter linguistic over-reliance and anchor reasoning more firmly in visual evidence.

5 Limitations

SGT aligns understanding and generation effectively for natural scenes, but segmentation-only training constrains performance on symbolically dense and knowledge-intensive tasks.

  • Scope Boundary: Segmentation-only SGT is constrained on symbolically dense and knowledge-intensive tasks, so it functions best as a foundational alignment strategy rather than a standalone solution.Augmenting SGT with VQA data retains symbolic proficiency.

6 Conclusion

The paper proposes Semantic Generative Tuning to address optimization divergence between visual understanding and generation by aligning them through semantic rather than pixel-level objectives. Its analyses and evaluations indicate improved feature separability, attention allocation, visual understanding, and generative layout fidelity.

  • Conclusion: SGT shifts the alignment proxy from pixel-space reconstruction to semantic space to mitigate representational misalignment in UMMs.The paradigm uses semantic integration rather than raw pixel recovery.
  • Conclusion: Mechanistic analyses report improved feature linear separability and optimized attention allocation, alongside gains in visual understanding accuracy and generative layout fidelity.These findings are reported across mainstream UMM architectures.

7 Appendix

The appendix describes the vision-task data, benchmark suite, training configurations, and additional analyses used to evaluate generative tuning. Results identify segmentation as the primary target, report a 1% overall understanding gain, and note a limitation on complex instruction parsing.

  • Data Processing: The study evaluates high-level segmentation and detection alongside low-level restoration and enhancement tasks within a generative-tuning framework.MS COCO annotations provide supervision for segmentation and detection, while other tasks use task-specific degradations or extracted targets.
  • Benchmarks: The evaluation covers six visual-understanding competencies across benchmarks for perception, hallucination resistance, spatial reasoning, VQA, document and chart comprehension, and mathematical and knowledge reasoning.The benchmark suite includes CV-Bench, MMVP, HallusionBench, POPE, SIBench-mini, MMMU-val, MMStar, DocVQAval, ChartQA, MathVista-mini, and ScienceQA.
  • Detailed Results: Segmentation yields the most significant improvements for both BAGEL and OmniGen2, motivating its adoption as the primary generative-tuning target named Semantic Generative Tuning.The appendix compares multiple classic vision tasks and recommends segmentation based on the detailed results.
  • Detailed Results: 1% overall performance gain is achieved from segmentation generative tuning under a setting without additional supervised fine-tuning data.The gain is aggregated across 12 benchmarks and is reported as evidence of improved perceptual understanding.
  • Detailed Results: Mixed training with panoptic segmentation, image reconstruction, and edge detection produces smaller gains than using segmentation data exclusively.The mixed setup uses 20,000 samples distributed equally among the three task categories.
  • Mechanistic Analysis and Configurations: Semantic generative tuning treats semantic information as an intermediate representation connecting sparse text supervision with dense visual generation signals.The appendix also reports model architectures, optimizer settings, training schedules, and additional mixed SFT and SGT results.
  • Inference and Additional Results: SGT does not substantially improve complex instruction parsing on GenEval, suggesting that specialized high-complexity instruction-following data may still be needed.This limitation concerns editing proficiencies rather than the reported gains in visual understanding.
Loading 2605.18714v2…