Semantic Generative Tuning for Unified Multimodal Models

1 Introduction

• UMMs train comprehension with sparse text supervision and generation with low-level pixel or token objectives, isolating capabilities and hindering mutual reinforcement.The result is a shared architecture with disjointed optimization processes.Source: §1 · p. 2
• Because robust comprehension relies on semantics rather than texture memorization [1], pixel-space reconstruction [80] can improve generation yet distracts with granular details.The paper questions whether low-level reconstruction is an optimal proxy for coupling understanding and generation.Source: §1 · p. 2
• Across a hierarchy of visual objectives, high-level semantic tasks—especially image segmentation—are optimal proxies and outperform low-level reconstruction.The investigation distinguishes low-, mid-, and high-level tasks by spatial granularity and semantic information.Source: §1 · p. 3
• SGT uses segmentation as a generative proxy to couple understanding and generation, improving feature linear separability and visual-textual attention allocation.The paper reports enhanced vision-centric perception and generative layout fidelity across mainstream architectures and benchmarks.Source: §1 · p. 3
• SGT improves performance by 6.02% over BAGEL [9] on CV-Bench and attains 90.0% on GenEval.The paper reports consistent gains in both visual understanding and generation across diverse benchmarks.Source: §1 · p. 3

2.1 Unified Multimodal Models

• Recent UMMs pursue unified any-to-any processing through discrete-token autoregression [18] [67] or hybrid language-modeling and denoising architectures [9] [69] [73].Examples include Emu3 [67], Janus-Pro [6], VARGPT [93], Show-o [81] [82], and Transfusion [92]; TokenFlow and Chameleon [57] address semantic abstraction and structural integrity.Source: §2.1 · pp. 3–4

2.2 Representation Learning via Generative Objectives

• Prior work uses diffusion for data augmentation [43], self-supervised objectives [7] [17], and transferable semantic features, while this work studies classical visual tasks in UMMs.Examples include SODA's diffusion bottleneck and DDAE's reconstruction-based masked autoencoding; existing approaches primarily emphasize pixel-space reconstruction [26] [63] [80].Source: §2.2 · p. 4

2.3 Reconstruction for Understanding and Alignment

• Unlike exact pixel-reconstruction methods [80], [66], [63],, this work systematically validates hierarchical visual proxy tasks for UMM tuning and finds advanced visual tasks yield maximum performance improvements.It distinguishes its active alignment of discriminative and generative capabilities from isolated proxy-task study [55] and observed mutual influence [61], establishing a shared semantic space for structural abstraction.Source: §2.3 · p. 4

3.1 Formulation

• UMMs use semantic ViT tokens for understanding, noise for generation, and ViT tokens, VAE latents, and noise for high-fidelity editing.The dual-encoder formulation represents semantic and structural or textural information at different granularities.Source: §3.1 · p. 5

3.2 Motivation and Hierarchical Visual Task Taxonomy

• The study compares high-level segmentation and object detection, mid-level depth estimation and inpainting, and low-level edge detection using matched visual inputs and training-data volume.It isolates task-granularity effects by excluding VQA, text-to-image, and standard image-editing data during this investigation.Source: §3.2 · p. 6

3.3 From Empirical Observations to the SGT Paradigm

• High-level semantic proxies, especially image segmentation, yield optimal UMM enhancements, motivating SGT to harmonize visual understanding and generation.Semantic instance, panoptic, and class-agnostic segmentation also yield comparable improvements.Source: Figure 3 · p. 7, §3.3 · pp. 7–8
• Visual supervision strengthens vision-centric perception, spatial reasoning, and hallucination resistance, while chart recognition and mathematical knowledge remain static or may decline marginally.The authors attribute this divergence to visual supervision improving representations without imparting additional knowledge or logical reasoning skills.Source: §3.3 · pp. 7–8, Figure 3 · p. 7
• Across proxy tasks, generative tuning consistently improves generation quality, particularly on position-aware tasks, by imposing spatial constraints that support layout alignment.The position gains hold regardless of proxy-task semantic granularity.Source: Figure 3 · p. 7, §3.3 · p. 8

4.1 Experimental Setup

• The experiments use 190k SAM-sourced segmentation samples with LLaVA-OneVision SFT data, evaluating SGT on BAGEL and OmniGen2 across understanding, text-to-image, and image-editing benchmarks.The two UMMs differ in architecture and training strategy; SAM images are kept separate from evaluation data.Source: §4.1 · p. 9

4.2 Main Results

• SGT-BAGEL and SGT-Gen2 outperform their baselines and a broad range of UMMs across benchmarks, while improving adherence to complex spatial and color prompts.The prompt-adherence improvement is shown in compositional text-to-image generation.Source: §4.2 · pp. 9–10, Table 2 · p. 10, Figure 4 · p. 11
• Across both architectures, segmentation yields the largest understanding gains among the tested proxy tasks; all three proxies improve text-to-image synthesis, while image-editing gains are smaller.The authors suggest explicit image-editing data may be needed for larger editing gains.Source: §4.2 · p. 10, Table 3 · p. 11

4.3 More Explorations

• A 1:2 VQA-to-segmentation batch ratio optimizes aggregate understanding scores for both models; scaling segmentation data raises scores and accelerates convergence on CV-Bench and Hallusion.From 2k to 100k segmentation samples, the average normalized score improves by BAGEL: +3.3% and OmniGen2: +2.0%.Source: §4.3 · pp. 11–12, Figure 5 · p. 12, Figure 6 · p. 12

4.4 Mechanistic Insights: Why Semantic Proxies Unlock Synergy?

• Segmentation supervision improves visual-feature separability, shifts deeper-layer understanding attention toward visual tokens, and increases generation attention to position, color, and object identity.Its attention shift toward visual tokens is more pronounced than that induced by low-level objectives.Source: §4.4 · pp. 13–14, Figure 7 · p. 13, Figure 8 · p. 13

• Segmentation-only SGT constrains performance on symbolically dense and knowledge-intensive tasks, so it works best as a foundational alignment strategy rather than a standalone solution.The limitation is shown in 3a.Source: §5 · p. 14
• Augmenting SGT with VQA data retains its symbolic proficiency, as shown in Table 2.Source: §5 · p. 14
• Future work will integrate SGT alignment with understanding data, generative targets, and reinforcement learning to pursue optimal cross-modal performance.Source: §5 · p. 14

6 Conclusion

• SGT shifts the alignment proxy from pixel space to semantic space, addressing the granular visual noise introduced by pixel reconstruction.The paradigm is designed to mitigate optimization divergence between visual understanding and generation.Source: §6 · p. 14
• Mechanistic analyses show that semantic integration improves feature linear separability and optimizes attention allocation to mitigate representational misalignment.Source: §6 · p. 14
• Across mainstream architectures, SGT consistently improves visual understanding accuracy and generative layout fidelity.The paper presents semantic-level alignment as a foundation for cohesive and versatile UMMs.Source: §6 · p. 14

7.1 Data Preparation

• Each task used 20,000 sample pairs from MS COCO; annotations supervised segmentation and detection, while depth targets were cross-checked and filtered for consistency.Segmentation annotations became pseudo-color targets, and detection boxes and labels were rendered on images. Depth estimates from two models were aligned, with samples discarded above a 0.4 discrepancy threshold; task overlap exceeded 95%.Source: §7.1 · pp. 15–16
• Other proxy tasks simulated image corruption or information loss, including masked-image inpainting, downsampled-and-resized super-resolution, blur, low-light noise, denoising, and Canny edge detection.Super-resolution used downsampling factors of 2, 4, 6, and 8; deblurring used random angles and kernel sizes of 10, 20, and 30, while low-light inputs used brightness factors of 0.1–0.5 and noise intensities of 0.01–0.04.Source: §7.1 · pp. 16–17

7.2 Detailed Results in Section 3.3.

• Segmentation tuning improved aggregate performance by 1% across 12 benchmarks without additional supervised fine-tuning; mixed three-task training gained less at the same 20,000-sample volume.Segmentation produced the strongest improvements for both BAGEL and OmniGen2. Generative tuning alone slightly degraded table interpretation and knowledge reasoning; the authors attribute this to alignment without added reasoning skills or prior knowledge.Source: §7.2 · pp. 17–18, Table 6 · p. 18

7.3 Training Configurations

• OmniGen2 and BAGEL used AdamW with β1 = 0.9, β2 = 0.95, and 0.01 weight decay, but differed in model size, learning rate, warmup, and training duration.OmniGen2 (7B parameters) trained at 4 × 10^-4 for 2,500 steps; BAGEL (14B) trained at 1 × 10^-4 for 10,000 steps. Both used global batch size 60.Source: §7.3 · pp. 19–20

7.4 Inference

• Joint SFT and SGT maintained knowledge-intensive and OCR performance, while DPGBench showed no significant change; limited complex instruction parsing may require specialized editing data.Inference followed each model’s original mechanisms; BAGEL editing used visual, VAE, and noise tokens, whereas OmniGen2 used VAE and noise tokens.Source: §7.4 · pp. 20–21, Table 9 · p. 20

7.5 Mechanism Analysis

• Attention analysis found stronger focus on salient prompt tokens after segmentation tuning, with attention to “tie” rising from 4.70% to 7.45% and “right” from 9.59% to 12.64%.The analysis averaged attention over the first three denoising steps for 20 prompts with spatial and color attributes; features were visualized by reducing dimensions with PCA before t-SNE.Source: §7.5 · p. 22, Figure 13 · p. 24