PaperlayerView exact revision on arXiv

Source-linked AI summary

Ovis-U1 Technical Report

Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, Yang Li, Qing-Guo Chen

arXiv:2506.23044v2cs.CVcs.AI

TL;DR

Unified multimodal models raise questions about adding image generation to understanding models and training them across both tasks. Ovis-U1 addresses these questions with a unified model spanning understanding, generation, and editing, and reports strong performance across evaluated tasks.

  • Problem

    As unified multimodal understanding and generation frameworks emerge, the report asks how to add image generation and effectively train models across both tasks.

    Source: §1 · p. 2

  • Method

    Ovis-U1 is a 3-billion-parameter unified model trained across multimodal understanding, text-to-image generation, and image editing using a diffusion-based visual decoder and bidirectional token refiner.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Ovis-U1 shows outstanding performance across all evaluated tasks and surpasses the compared 3B-range models on OpenCompass Multi-modal Academic Benchmarks.

    Source: §5.1 · p. 8

  • Takeaways & Limitations

    Unified training collaboratively enhances Ovis-U1’s understanding and generation performance.

    Source: §6 · p. 13

    The authors note that smaller image-generation models may struggle with artifacts and hallucinations, and plan to expand model size and training data.

    Source: §6 · p. 13

Abstract

from arXiv · show

In this report, we introduce Ovis-U1, a 3-billion-parameter unified model that integrates multimodal understanding, text-to-image generation, and image editing capabilities. Building on the foundation of the Ovis series, Ovis-U1 incorporates a diffusion-based visual decoder paired with a bidirectional token refiner, enabling image generation tasks comparable to leading models like GPT-4o. Unlike some previous models that use a frozen MLLM for generation tasks, Ovis-U1 utilizes a new unified training approach starting from a language model. Compared to training solely on understanding or generation tasks, unified training yields better performance, demonstrating the enhancement achieved by integrating these two tasks. Ovis-U1 achieves a score of 69.6 on the OpenCompass Multi-modal Academic Benchmark, surpassing recent state-of-the-art models such as Ristretto-3B and SAIL-VL-1.5-2B. In text-to-image generation, it excels with scores of 83.72 and 0.89 on the DPG-Bench and GenEval benchmarks, respectively. For image editing, it achieves 4.00 and 6.42 on the ImgEdit-Bench and GEdit-Bench-EN, respectively. As the initial version of the Ovis unified model series, Ovis-U1 pushes the boundaries of multimodal understanding, generation, and editing.

1 Introduction

Ovis-U1 is a 3-billion-parameter model that combines multimodal understanding, image generation, and editing through a diffusion-based visual decoder and bidirectional token refiner. Its unified training across diverse multimodal tasks is reported to improve both understanding and generation.

Research Questions

  • The report investigates how to equip a multimodal understanding model with image generation and how to train one model effectively on both understanding and generation tasks.The authors describe GPT-4o’s understanding performance as enhanced by integrating image-generation capabilities.Source: §1 · p. 2

Architecture Improvement

  • Ovis-U1 combines a diffusion Transformer visual decoder and bidirectional token refiner to generate images from text and edit images using textual prompts.The decoder uses MMDiT with rotary position embedding, while the refiner strengthens interaction between textual and visual features.Source: §1 · p. 2

Unified Training

  • Unified training across diverse understanding, text-to-image generation, and image-editing tasks is reported to improve both understanding and generation performance.The approach spans six training stages and draws on diverse multimodal data.Source: §1 · p. 2

Motivation

  • The authors aim to advance multimodal understanding, generation, and editing through architectural and training innovations, and to accelerate community research by open-sourcing Ovis-U1.They frame this as supporting collaborative development of general-purpose AI systems for multimodal reasoning and manipulation.Source: §1 · p. 2

2 Architecture

Ovis-U1 inherits the Ovis architecture and adds a visual decoder for image generation (Lu et al., 2024).

LLM & Text tokenizer

  • Ovis-U1 uses Qwen3-1.7B, initialized from a language model and jointly trained on visual understanding and generation data.This differs from approaches that use an unchanged multimodal language-model backbone, such as Qwen-VL (Bai et al., 2025); the Qwen3 series is cited to Yang et al. (2025).Source: §2 · p. 3

Visual Encoder & Adapter

  • The Aimv2-large-patch14-448-initialized encoder handles arbitrary-resolution images without sub-image partitioning, using interpolated positional embeddings, 2D RoPE, and packed variable-length attention.The original Ovis adapter uses probabilistic visual-vocabulary tokenization: pixel shuffle, a linear head, and softmax produce a distribution whose weighted average of learnable embeddings is fed to the LLM (Lu et al., 2024). The encoder and attention design cite Fini et al. (2025), Su et al. (2024), Dao et al. (2022), Dao (2024), and Dehghani et al. (2023).Source: §2 · pp. 3–4

Visual Decoder & VAE

  • A from-scratch 1B MMDiT diffusion decoder uses RoPE and flow matching, with 27 layers and 16 heads; a frozen 4-channel SDXL VAE supplies latents.Visual semantic embeddings are concatenated with text embeddings as generation conditions; context-image tokens encoded by the VAE, together with noise image tokens, enter the decoder’s visual stream. The design cites Labs (2024a), Esser et al. (2024), Su et al. (2024), Labs (2024b), and Labs et al. (2025).Source: §2 · p. 4

Refiner

  • A bidirectional refiner uses two modulated transformer blocks to interact with concatenated last- and second-to-last-layer LLM features and a learnable [CLS] token.The learnable token is concatenated with the LLM embedding to capture global information as an alternative to CLIP; the refined interaction provides conditional guidance. The passage cites Kong et al. (2024), Ma et al. (2024), Labs (2024a), and Radford et al. (2021).Source: §2 · p. 4

3 Data Composition and Training Procedure

Ovis-U1 combines multimodal understanding, text-to-image, and image-plus-text generation data, then trains through six stages that build and align its understanding and generation capabilities.

3.1 Data Composition

  • Ovis-U1 uses three data types: multimodal understanding, text-to-image generation, and image-plus-text-to-image generation.Understanding data includes public datasets and in-house data, with preprocessing to filter noise, improve captions, and adjust data ratios. Text-to-image data uses Laion5B samples with aesthetic scores above 6, described by Qwen to create Laion-aes6, plus JourneyDB.Source: §3.1 · p. 4
  • Image-plus-text generation data covers editing, reference-image-driven generation, pixel-level control, and additional in-house tasks.Editing uses OmniEdit, UltraEdit, and SeedEdit; reference-driven data includes Subjects200K and SynCD for subjects and StyleBooth for style. MultiGen 20M supports canny-to-image, depth-to-image, inpainting, and outpainting; in-house data adds style, content removal, translation, restoration, colorization, and text rendering tasks.Source: §3.1 · p. 4

3.2 Training Procedure

  • Unlike works that directly use pretrained MLLMs such as Qwen-VL (Bai et al., 2025), Ovis-U1 is trained from a pretrained LLM.Source: §3.2 · p. 4
  • The first three stages build generation and multimodal alignment: visual-decoder pretraining, adapter pretraining, and joint visual-encoder alignment.Stage 0 trains a randomly initialized 1B diffusion transformer with text-to-image data, alongside the refiner, to generate from LLM embeddings. Stages 1 and 2 train the adapter, then fine-tune the visual encoder and adapter across understanding, text-to-image, and editing tasks; generation aids modality alignment.Source: §3.2 · p. 5
  • The final three stages train understanding, then generation learning and fine-tuning, with understanding parameters frozen before generation learning.Stage 3 trains the visual encoder, adapter, and LLM on understanding tasks, then fixes their parameters. Stage 4 trains the refiner and visual decoder to align with the optimized embeddings and improves text-to-image performance over Stage 0; Stage 5 fine-tunes the decoder for text-to-image and image editing.Source: §3.2 · pp. 5–6

4 Evaluation

Evaluation benchmarks Ovis-U1 across image understanding, text-to-image generation, and image editing. The protocols use established academic and task-specific benchmarks, with raw prompts for GenEval and GPT-based evaluation for edited images.

4 Evaluation

  • The evaluation covers three capabilities: image understanding, text-to-image generation, and instruction-based image editing.The report benchmarks unified models on these tasks, reflecting their ability to comprehend input images, generate images from prompts, and edit images according to instructions.Source: §4 · p. 6

Understanding

Text-to-Image Generation

  • Text-to-image generation is evaluated with CLIPScore, DPG-Bench, and GenEval, using raw prompts for GenEval.CLIPScore uses the first 1K prompts; the report notes its use in DALL-E 3 and reports DPG-Bench and GenEval performance without rewriting GenEval prompts.Source: §4 · p. 6

Image Editing

  • Image editing is evaluated on GEdit-Bench and ImgEdit, which contain 606 and 811 image-instruction pairs, respectively.Both benchmarks use an advanced GPT model to evaluate edited images.Source: §4 · p. 6

5 Experiments

The experiments evaluate Ovis-U1’s understanding, generation, and editing performance, then examine refiner design, unified-training effects, classifier-free guidance, and qualitative examples.

5.1 The main result

  • Across evaluated tasks, 3.34B Ovis-U1 shows outstanding performance and surpasses leading 3B-range models on OpenCompass, with about 2B parameters devoted to understanding.The comparison includes InternVL2.5-2B, SAIL-VL-2B, InternVL3-2B, Qwen2.5-VL-3B, Ovis2-2B, SAIL-VL-1.5-2B, and Ristretto-3B.Source: Table 4 · p. 7, §5.1 · p. 8
  • On text-to-image generation across GenEval and DPG-Bench, Ovis-U1 significantly outperforms OmniGen despite a similar parameter count, while its 1B visual decoder achieves performance comparable to larger models.Source: §5.1 · p. 8
  • Ovis-U1 demonstrates strong performance on both ImgEdit-Bench and GEdit-Bench-EN against image-editing models.Previous ImgEdit-Bench results are referenced from OmniGen2 (Wu et al., 2025a), while GEdit-Bench-EN results come from its leaderboard.Source: Table 7 · p. 7, Table 8 · p. 8, §5.1 · p. 8

5.3 Enhance understanding with unified training

  • 1.14 points higher average score on OpenCompass was achieved by Ovis-U1 than by Ovis without unified training.The report attributes the gain to using text-to-image generation and image-editing tasks to align the visual encoder during training stages 1 and 2.Source: Table 10 · p. 9, §5.3 · p. 9

5.2 Ablation study on refiner

  • 83.81 DPG-Bench was achieved by 50M-data Variant V7 with CLS tokens, though its CLIPScore was slightly below the baseline.In the earlier limited-data study, the baseline trained on about 10M text-to-image data scored 32.19 CLIPScore and 82.32 DPG-Bench; clip-free variants were slightly lower than baseline, while the CLS-token approach outperformed averaged refiner outputs. These ablations used earlier model versions.Source: §5.2 · pp. 8–9, Table 9 · p. 9

5.4 Enhance generation with unified training

  • 0.77-point DPG-Bench improvement followed adding image-editing data in Stage 5, after text-to-image training in Stage 1 and visual-text embedding alignment in Stage 4.The generation-stage ablations used an earlier model version; the tables report DPG-Bench and GenEval results across stages.Source: §5.4 · p. 9, Table 11 · p. 10, Table 12 · p. 10

5.5 Instruction based image editing

  • 4.13 on ImgEdit-Bench was achieved with CFGimg set to 2 and CFGtxt set to 7.5, while score differences across CFG settings stayed within 0.2 on both editing benchmarks.Higher CFGimg preserves more input-image detail, higher CFGtxt improves instruction adherence, and optimal settings vary by benchmark; the 4.13 score exceeds the Tab. 3 result because that table uses shared settings.Source: §5.5 · pp. 9, 11, Table 13 · p. 10, Figure 4 · p. 10

5.6 More qualitative results

  • Qualitative examples show fine-grained visual reasoning, coherent high-fidelity image generation, and precise localized edits that preserve background integrity.The examples include complex visual content, diverse aesthetics and structural constraints, and content replacement or stylistic changes with minimal artifacts and adherence to instructions.Source: §5.6 · pp. 11, 13, Figure 5 · p. 11, Figure 7 · p. 12

6 Conclusion

Ovis-U1 is a 3-billion-parameter unified model for multimodal understanding, text-to-image generation, and image editing, with strong benchmark performance. The conclusion highlights unified training’s role and outlines priorities for future development.

6 Conclusion

  • With 3B parameters, Ovis-U1 performs strongly across understanding, generation, and editing benchmarks, surpassing some task-specific models.The report presents it as the initial model in the Ovis unified series.Source: §6 · p. 13
  • Unified training collaboratively aligns the visual encoder and significantly enhances both understanding and generation performance.The report also addresses visual-decoder design, its connector with large language models, and the unified training procedure.Source: §6 · p. 13
  • Future work will increase model size to reduce generation artifacts and hallucinations, and curate more diverse, high-quality training data, especially interleaved image-text content.The report expects more parameters to support higher-quality generated images.Source: §6 · p. 13
  • Future work will develop unified-model architectures and specialized visual encoder-decoder structures to preserve fine-grained input-image details during editing.The report also identifies the absence of a reinforcement-learning stage and human-preference alignment as open challenges.Source: §6 · p. 13

Related papers

Loading 2506.23044v2…