PaperlayerView exact revision on arXiv

Source-linked AI summary

TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models

Zhiheng Liu, Weiming Ren, Haozhe Liu, Zijian Zhou, Shoufa Chen, Haonan Qiu, Xiaoke Huang, Zhaochong An, Fanny Yang, Aditya Patel, Viktar Atliha, Tony Ng, Xiao Han, Chuyan Zhu, Chenyang Zhang, Ding Liu, Juan-Manuel Perez-Rua, Sen He, Jürgen Schmidhuber, Wenhu Chen, Ping Luo, Wei Liu, Tao Xiang, Jonas Schult, Yuren Cong

arXiv:2512.02014v1cs.CV

TL;DR

Unified multimodal models need visual representations that serve understanding and generation without the mismatches introduced by separate encoders. TUNA cascades a VAE encoder with a representation encoder into one continuous visual space, achieving state-of-the-art results across multimodal tasks and showing benefits from stronger encoders and joint training.

  • Problem

    Separate visual encoders in unified multimodal models introduce representation-format mismatches, while existing unified designs can favor understanding over generation.

  • Method

    TUNA cascades a VAE encoder with a representation encoder to build a unified visual representation used by one model for multimodal understanding and generation.

  • Results

    TUNA achieves state-of-the-art performance across image and video understanding, image and video generation, and image editing, outperforming prior unified and decoupled designs.

  • Takeaways & Limitations

    Stronger pretrained representation encoders and joint training on understanding and generation data improve performance across multimodal tasks.

Abstract

from arXiv · show

Unified multimodal models (UMMs) aim to jointly perform multimodal understanding and generation within a single framework. We present TUNA, a native UMM that builds a unified continuous visual representation by cascading a VAE encoder with a representation encoder. This unified representation space allows end-to-end processing of images and videos for both understanding and generation tasks. Compared to prior UMMs with decoupled representations, TUNA's unified visual space avoids representation format mismatches introduced by separate encoders, outperforming decoupled alternatives in both understanding and generation. Moreover, we observe that stronger pretrained representation encoders consistently yield better performance across all multimodal tasks, highlighting the importance of the representation encoder. Finally, in this unified setting, jointly training on both understanding and generation data allows the two tasks to benefit from each other rather than interfere. Our extensive experiments on multimodal understanding and generation benchmarks show that TUNA achieves state-of-the-art results in image and video understanding, image and video generation, and image editing, demonstrating the effectiveness and scalability of its unified representation design.

1 Introduction

TUNA addresses representation mismatches and task imbalance in native unified multimodal models with a unified visual representation. It achieves state-of-the-art results across multimodal understanding and generation tasks while outperforming decoupled alternatives.

  • 1 Introduction: Decoupled representations increase parameters and costs, while separate vision encoders produce incompatible spatial, temporal, and channel formats.The cited examples differ in spatial compression, temporal compression, and channel dimension, motivating a single representation space.
  • 1 Introduction: Existing unified models can favor understanding or generation, whereas TUNA directly connects a VAE encoder to a representation encoder to address this imbalance.The design then fuses unified visual features with text tokens for LLM decoding and multimodal generation.
  • 1 Introduction: 61.2% on MMStar and 0.90 on GenEval accompany TUNA’s reported state-of-the-art performance on multimodal understanding and generation benchmarks.These values are representative results cited alongside the model’s three-stage training and unified representation design.
  • 1 Introduction: TUNA uses a unified visual representation to support image/video understanding, image/video generation, and image editing within one native multimodal model.The model is jointly pretrained on understanding and generation objectives rather than assembled from separate task-specific models.
  • 1 Introduction: An ablation study reports that TUNA’s unified representation is superior to Show-o2 and other decoupled representation designs.This comparison evaluates the representation design rather than only the overall model architecture.

2 Our Method: Tuna

TUNA constructs continuous unified visual representations by combining VAE latents with higher-level representation features, then uses one decoder for understanding and generation. Its architecture and training pipeline combine autoregressive text generation with flow-matching visual generation.

  • 2 Our Method: Tuna: TUNA uses continuous visual representations because continuous VAE latents support generation while continuous semantic features support understanding.The method combines these complementary properties rather than relying on discrete visual representations alone.
  • 2 Our Method: Tuna: The LLM decoder performs autoregressive text generation for understanding and flow-matching-based visual generation after combining visual and language tokens.During visual generation, noise is added to visual tokens for diffusion-based generation.
  • 2 Our Method: Tuna: TUNA cascades a VAE encoder with a representation encoder to construct unified visual representations for both understanding and generation.The method uses continuous VAE latents and higher-level semantic features, motivated by their support for both task types.
  • 2 Our Method: Tuna: For video, reshaping latent frames into the batch dimension lets the representation encoder process each 4-frame window independently, improving efficiency.The operation avoids flattening all latent frames into one excessively long sequence.
  • 2 Our Method: Tuna: TUNA trains progressively in three stages, beginning with representation-encoder and flow-matching-head pretraining using image captioning and text-to-image generation.The first stage freezes the LLM decoder while adapting the visual representation and generation head.

3 Experiments

Across understanding, generation, editing, and video tasks, Tuna achieves strong benchmark performance with a unified visual representation. Ablations and representation analyses attribute this consistency to balanced fusion and joint training.

  • Main Results: Tuna achieves state-of-the-art or leading results across image understanding, image generation, image editing, video understanding, and video generation benchmarks.It reaches state-of-the-art results on nearly all image-understanding benchmarks, all three image-generation benchmarks, and video generation, while leading unified models on image editing.
  • Ablation: Visual Representation Design: Unified representations consistently outperform decoupled representations across understanding and generation benchmarks.The unified setting also surpasses understanding-only and generation-only comparisons on most corresponding benchmarks, indicating fewer representation conflicts.
  • Ablation: Representation Encoder: Stronger representation encoders improve Tuna consistently across benchmarks, with SigLIP 2 and DINOv3 outperforming SigLIP.The comparison holds regardless of which VAE encoder is used.
  • Ablation: Understanding-Generation Synergy: Joint training on understanding and generation data improves both task families relative to training on either data type alone.Model 12 surpasses Model 3 on understanding benchmarks and Model 6 on generation benchmarks.
  • Unified Representation Analysis: Tuna’s representation aligns strongly with semantic features while aligning more with generation features than Show-o2.Both models exceed 0.5 CKNNA with SigLIP 2, whereas Show-o2 correlates with its generation branch at only 0.07.
  • Qualitative Results: Qualitative results show accurate text rendering, compositional placement, nuanced image edits, and high-fidelity text-to-video generation.Examples include precise shelf contents, style transfer, environment changes, object replacement, and left-side lighting.

4 Related Work

Related work develops multimodal understanding, visual generation, and unified multimodal models through increasingly capable encoders and fusion strategies. Existing unified approaches improve efficiency but often remain imbalanced across understanding and generation, motivating Tuna’s balanced representation design.

  • Multimodal Understanding: Multimodal understanding research improves instruction following, vision encoders, interleaved image and video understanding, and reasoning through stronger models and data.The LLaVA paradigm typically encodes visual inputs and concatenates them with text tokens for joint decoding.
  • Visual Generation: Visual generation research commonly uses VAE-defined continuous latent spaces with diffusion models for high-fidelity and efficient image and video synthesis.Latent-space diffusion reduces computational cost and supports higher-resolution scaling compared with pixel-space approaches.
  • Unified Multimodal Models: Unified multimodal models combine multimodal inputs and outputs, while prior systems either connect separate task-specific representations or share one visual representation.Decoupled systems often use MoE routing, whereas unified systems prioritize a single representation space.
  • Unified Multimodal Models: Existing unified models are more efficient but often excel at one task while underperforming on the other.This imbalance reflects the difficulty of using one representation for both understanding and generation.
  • Unified Visual Representations: Unified visual tokenizers increasingly merge understanding and generation features through late fusion or single-encoder designs.TokenFlow and MUSE-VL use late fusion, while DualToken, UniTok, and TokLIP train a single encoder.

5 Conclusion

Tuna constructs a unified visual representation space by cascading a VAE encoder with a representation encoder. It achieves strong performance across multimodal understanding, generation, and image editing, while ablations support the benefits of its unified design.

  • Tuna constructs a unified visual representation space by cascading a VAE encoder with a representation encoder.
  • Tuna achieves strong performance across image and video understanding, image and video generation, and image editing.It also performs competitively with leading understanding-only and generation-only models.
  • Tuna’s unified representation outperforms Show-o2-style unified representations and decoupled representation designs.
  • Ablations show that stronger pretrained representation encoders improve performance and that understanding and generation mutually enhance each other.
Loading 2512.02014v1…