PaperlayerView exact revision on arXiv

Source-linked AI summary

Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation

Zihan Su, Hongyang Wei, Kangrui Cen, Yong Wang, Guanhua Chen, Chun Yuan, Xiangxiang Chu

arXiv:2601.21406v3cs.CVcs.LG

TL;DR

UniMRG addresses the limited exploration of using generation to improve visual understanding in Unified Multimodal Models. It post-trains UMMs to generate pixel, depth, and segmentation representations alongside understanding objectives, improving perception, hallucination, spatial understanding, and generation across diverse architectures.

  • Problem

    Using generation to improve visual understanding in Unified Multimodal Models remains largely unexplored, despite efforts in the reverse direction.

  • Method

    UniMRG is an architecture-agnostic post-training method that adds pixel reconstruction, depth, and segmentation generation to standard visual understanding objectives.

  • Results

    Across diverse UMM architectures, UniMRG improves fine-grained perception, reduces hallucinations, strengthens spatial understanding, and enhances generation quality.

  • Takeaways & Limitations

    Generating complementary appearance, geometric, and structural representations can encourage UMMs to internalize regularities that support broader visual understanding.

  • Takeaways & Limitations

    UniMRG may yield only marginal understanding gains in models with limited generative representation capacity, such as Show-o's small VQ codebook.

Abstract

from arXiv · show

Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework. Their ultimate aspiration is to create a cycle where understanding and generation mutually reinforce each other. While recent post-training methods have successfully leveraged understanding to enhance generation, the reverse direction of utilizing generation to improve understanding remains largely unexplored. In this work, we propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method. UniMRG enhances the understanding capabilities of UMMs by incorporating auxiliary generation tasks. Specifically, we train UMMs to generate multiple intrinsic representations of input images, namely pixel (reconstruction), depth (geometry), and segmentation (structure), alongside standard visual understanding objectives. By synthesizing these diverse representations, UMMs capture complementary information regarding appearance, spatial relations, and structural layout. Consequently, UMMs develop a deeper and more comprehensive understanding of visual inputs. Extensive experiments across diverse UMM architectures demonstrate that our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.

1. Introduction

UniMRG addresses the underexplored use of generation to improve UMM understanding by training models to generate complementary intrinsic image representations alongside understanding tasks. Across architectures, it improves perception, hallucination, spatial understanding, and generation.

  • Motivation: Prior post-training methods mainly use UMM understanding signals to improve generation, leaving the reverse direction largely unexplored.This motivates asking whether generation capabilities can enhance visual understanding.
  • Method: Depth and segmentation provide dense supervision for spatial reasoning, hallucination reduction, and structural understanding beyond what pixel reconstruction alone constrains.Depth encodes geometry and relative distance, while segmentation provides object boundaries and region partitions.
  • Method: UniMRG trains UMMs to generate pixel reconstructions, depth maps, and segmentation maps alongside standard visual understanding objectives.These targets capture appearance, geometry, and structural layout.
  • Results: Across autoregressive, masked autoregressive, and diffusion-based UMMs, UniMRG consistently improves fine-grained perception, reduces hallucinations, strengthens spatial understanding, and enhances generation.The evaluation spans diverse UMM architectures.

2. Related Work

Related work establishes UMM architectural paradigms, prior post-training methods that use understanding to improve generation, and depth and segmentation as complementary structural representations. UniMRG reverses the usual direction by using these representations as auxiliary generation targets without changing the UMM architecture.

  • UMM Architectures: UMMs use autoregressive, masked autoregressive, or diffusion-based architectural paradigms to combine visual understanding and generation.Examples include discrete-token models, shared masked-autoregressive representations, and diffusion decoders conditioned on multimodal hidden states.
  • Post-training for UMMs: Prior UMM post-training methods exploit understanding representations or scores to improve image generation through reconstruction and self-rewarding training.RecA guides reconstruction with understanding representations, while SRUM uses understanding capacity to score synthesized images.
  • Intrinsic Representations: Depth estimates scene geometry and relative distance, whereas segmentation decomposes scenes into regions or object-wise partitions.Together, they provide complementary geometric and structural cues.
  • UniMRG: UniMRG uses depth and segmentation as auxiliary generation targets so richer geometric and structural knowledge transfers to downstream understanding.This differs from prior methods that primarily use understanding to improve generation.

3. Unified Multi-Representation Generation

UniMRG jointly trains UMMs on reconstruction, depth generation, segmentation generation, and standard understanding tasks. The design targets complementary intrinsic factors because pixel reconstruction alone weakly constrains geometry and structure, while inference requires no architectural modification or extra computational overhead.

  • Multi-Task Strategy: UniMRG jointly trains UMMs on image reconstruction, image-to-depth, image-to-segmentation, and image understanding.The four tasks combine generation objectives with standard vision-language supervision.
  • Understanding Objective: The visual understanding loss uses standard vision-language tasks, with cross-entropy for understanding objectives.The question prompt and answer define the supervised understanding pair.
  • Generation Objectives: Reconstruction models RGB appearance, while depth and segmentation generation target geometric cues, spatial relations, structural cues, and region partitions.Depth targets are produced with Depth Anything V2, and segmentation targets use automatic mask generation with Segment Anything.
  • Optimization: The total objective combines pixel, depth, segmentation, and understanding losses, with all four loss weights set to 1 in experiments.The loss weights are denoted λpixel, λdepth, λseg, and λund.
  • Implementation: During inference, UniMRG operates like a standard UMM without architectural modifications or additional computational overhead.The understanding encoder is updated only for UMMs with a shared generation-understanding encoder; otherwise it is frozen.
  • Design Rationale: Pixel-level supervision alone is often insufficient for fine-grained perception, spatial understanding, and hallucination reduction because textures and colors weakly constrain intrinsic factors.UniMRG therefore adds intrinsic representations as complementary auxiliary targets beyond RGB appearance.

4. Experiments

UniMRG is evaluated across three UMM generation paradigms and multiple understanding and generation benchmarks. It improves fine-grained perception, hallucination mitigation, spatial understanding, and generation, while representation-specific ablations clarify the roles of depth, segmentation, and pixel generation.

  • Experimental Setup: UniMRG is validated on Show-o, Harmon, and OpenUni, representing AR, AR+MAR, and AR+Diffusion generation paradigms.The experiments use three representative UMM architectures with different generation paradigms.
  • Experimental Setup: The evaluation covers general understanding, fine-grained perception, hallucination, spatial understanding, and image generation.Understanding is measured with MMBench, MMVP, HallusionBench, RealWorldQA, and VSR; generation uses GenEval and DPGBench.
  • Main Results: UniMRG improves understanding across architectures while preserving generation capability, unlike SFT, which sharply degrades generation, and RecA, which provides no understanding gains.On Harmon, SFT reduces GenEval from 71.37 to 0.30 and DPGBench from 80.52 to 2.85.
  • Main Results: On OpenUni-3.6B, MMVP increases from 71.67 to 74.67, HallusionBench from 60.88 to 64.56, and VSR from 66.69 to 73.90.These gains correspond to fine-grained perception, hallucination mitigation, and spatial understanding.
  • Ablation Study: Depth and segmentation generation improve understanding in ablations, while pixel generation maintains natural image quality.Adding depth raises VSR from 59.00 to 60.39, adding segmentation raises Hallusion from 48.26 to 49.32, and pixel generation prevents depth- or segmentation-biased outputs.
  • Discussion: UniMRG generalizes depth generation to synthetic out-of-distribution images, although Show-o gains only marginally because its 4,096-token VQ codebook limits representational capacity.Depth similarity rises from 0.623 to 0.822 for Harmon and from 0.617 to 0.834 for OpenUni, while Show-o rises from 0.637 to 0.664.

5. Conclusion

UniMRG improves UMM understanding by training auxiliary generation of pixel, depth, and segmentation representations alongside standard understanding objectives. Across diverse architectures, it improves perception, hallucination reduction, spatial understanding, and generation quality.

  • UniMRG trains UMMs to generate pixel, depth, and segmentation representations alongside standard visual understanding objectives.These representations encode appearance, geometry, and structure.
  • Across diverse UMM architectures, UniMRG improves fine-grained perception, reduces hallucinations, and strengthens spatial understanding while enhancing generation quality.
  • The authors plan to extend UniMRG to additional intrinsic representations, including pose and sketches, and to video settings.

Impact Statement

The paper describes potential benefits of more reliable visual grounding alongside risks from stronger image-generation capabilities. It also situates UMMs within several generation paradigms and their associated training objectives.

  • Improved fine-grained perception, reduced hallucinations, and stronger spatial understanding may support more reliable grounding in assistive tools, education, and human-computer interaction.
  • Stronger image-generation capabilities can increase misuse risks, including deceptive or misleading synthetic content.
  • Autoregressive Modeling: Autoregressive UMMs generate discrete visual tokens sequentially using causal context and log-likelihood objectives.
  • Masked Prediction: Masked prediction enables parallel token prediction from bidirectional context, followed by iterative confidence-based decoding.
  • Diffusion Modeling: Diffusion models generate data by reversing a continuous corruption process, including stochastic DDPM and deterministic Flow Matching formulations.

A.3. Masked Autoregressive Modeling

Masked Autoregressive modeling combines autoregressive sequence modeling with diffusion-based density estimation for continuous feature tokens. Its diffusion loss supports sequential high-fidelity generation without discrete codebooks.

  • Masked Autoregressive modeling combines autoregressive sequence modeling with diffusion-based density estimation and avoids discrete codebooks.
  • Continuous feature sequences are factorized autoregressively, conditioning each token on its preceding clean history.
  • A diffusion loss replaces a simple unimodal loss for each continuous token, reducing the risk of blurry predictions.
  • The model denoises noisy token representations conditioned on clean history to generate continuous tokens sequentially.
  • Using a diffusion model as the autoregressive head enables high-fidelity continuous-token generation without quantization artifacts.

B.1. Training Details

Training uses architecture-specific optimization settings across Harmon, OpenUni, and Show-o, with Depth-Anything-V2 producing normalized depth targets. The experiments use task-specific batches, training steps, and compute budgets.

  • Harmon: Harmon is trained with AdamW at 1e-5 for 4,000 steps using batch size 16 per task and approximately 5 hours of training.
  • OpenUni: OpenUni is trained with AdamW at 1e-5 for 2,000 steps using batch size 4 per task, followed by 3,000 pixel-generation steps with its understanding component frozen.
  • Show-o: Show-o is trained with AdamW at 1e-6 for 2,500 steps using batch size 2 per task and approximately 8 hours of training.

B.3. Segmentation Map Generation

This section lists prompt templates for generating depth and segmentation representations during UniMRG training, including varied phrasings for each task.

  • Segmentation Mask Construction: The segmentation procedure uses automatic mask generation with sampled point prompts and filters candidate masks using quality thresholds and duplicate suppression.The passage specifies a 32 × 32 grid of point prompts, predicted IoU and stability thresholds, and non-maximum suppression.
  • Prompt Diversity: UniMRG uses multiple prompts with similar meanings but different phrasings to reduce overfitting to specific prompts.
  • Depth Generation Prompts: Depth-generation prompts request depth maps, estimations, per-pixel values, or extracted depth information.
  • Segmentation Generation Prompts: Segmentation-generation prompts request segmentation maps, masks, object separation, or foreground-background separation.

C. Qualitative Results on Representation Generation

Qualitative results show that UniMRG improves depth and segmentation generation for Harmon and OpenUni, while Show-o remains constrained by a representational bottleneck.

  • Base-model Outputs: Before training, Harmon, OpenUni, and Show-o produce depth and segmentation outputs resembling image reconstruction rather than structured representations.
  • Post-training Improvements: After UniMRG training, Harmon and OpenUni generate semantically correct objects in depth and segmentation representations.The examples correctly represent two boys and grassland from the input image.
  • Post-training Improvements: The generated depth maps distinguish relative distances by rendering distant backgrounds black and closer foreground objects white.This behavior is described as demonstrating learned relative-distance distinctions and enhanced spatial understanding.
  • Show-o Limitation: Show-o produces purely black depth and segmentation outputs after UniMRG, likely because its 4,096-token codebook creates a representational bottleneck.
  • Figure Comparison: The figure compares original images, ground truth, and outputs from Harmon, OpenUni, and Show-o before and after UniMRG training.
Loading 2601.21406v3…