Source-linked AI summary
Ovis-U1 Technical Report
Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, Yang Li, Qing-Guo Chen
TL;DR
Ovis-U1 addresses how one multimodal model can combine understanding, image generation, and editing, and how such capabilities can be trained together. It uses a language-model-centered architecture with a diffusion decoder, bidirectional refiner, and unified training across multimodal tasks. The model reports strong performance across understanding, generation, and editing benchmarks, while future work targets larger models, better data, and specialized editing architectures.
Problem
The report asks how multimodal understanding models can gain image-generation capability and how one model can be trained effectively across understanding and generation tasks.
Method
Ovis-U1 combines a diffusion visual decoder, bidirectional token refiner, and unified multimodal training initialized from a language model.
Results
Ovis-U1 shows strong performance across multimodal understanding, text-to-image generation, and image editing evaluations.
Takeaways & Limitations
Unified training is presented as improving both understanding and generation performance within one multimodal model.
Takeaways & Limitations
Future work identifies limited model capacity, less diverse training data, and the need for specialized architectures to improve image quality and fine-grained editing.
Abstract
from arXiv · showhide
In this report, we introduce Ovis-U1, a 3-billion-parameter unified model that integrates multimodal understanding, text-to-image generation, and image editing capabilities. Building on the foundation of the Ovis series, Ovis-U1 incorporates a diffusion-based visual decoder paired with a bidirectional token refiner, enabling image generation tasks comparable to leading models like GPT-4o. Unlike some previous models that use a frozen MLLM for generation tasks, Ovis-U1 utilizes a new unified training approach starting from a language model. Compared to training solely on understanding or generation tasks, unified training yields better performance, demonstrating the enhancement achieved by integrating these two tasks. Ovis-U1 achieves a score of 69.6 on the OpenCompass Multi-modal Academic Benchmark, surpassing recent state-of-the-art models such as Ristretto-3B and SAIL-VL-1.5-2B. In text-to-image generation, it excels with scores of 83.72 and 0.89 on the DPG-Bench and GenEval benchmarks, respectively. For image editing, it achieves 4.00 and 6.42 on the ImgEdit-Bench and GEdit-Bench-EN, respectively. As the initial version of the Ovis unified model series, Ovis-U1 pushes the boundaries of multimodal understanding, generation, and editing.
1 Introduction
Ovis-U1 is a 3-billion-parameter unified model designed to address multimodal understanding, image generation, and image editing through new architecture and unified training strategies. The report frames unified training as a way to improve both understanding and generation performance.
- The report studies how to equip multimodal understanding models with image generation and how to train one model effectively on understanding and generation tasks.
- Ovis-U1 is a 3-billion-parameter unified model extending the Ovis series across multimodal understanding, text-to-image generation, and image editing.
- The architecture adds a diffusion-based visual decoder and bidirectional token refiner to improve interactions between textual and visual embeddings.
- Ovis-U1 is intended to advance multimodal capabilities and accelerate community research through open sourcing.
- Unified training jointly uses diverse multimodal data spanning understanding, text-to-image generation, and image editing.
2 Architecture
Ovis-U1 extends the Ovis architecture with a language-model-centered multimodal pipeline that connects visual inputs to the LLM and adds diffusion-based image decoding. Its visual encoder supports arbitrary-resolution images, while the refiner improves visual-textual interaction before decoding.
- Ovis-U1 adds a visual decoder to the Ovis architecture for image generation while retaining the broader multimodal language-model pipeline.
- The model starts from a pretrained language model and trains it with visual understanding and generation data rather than freezing a pretrained MLLM backbone.
- The visual encoder and adapter process arbitrary-resolution images and bridge vision features into the language model through spatial compression and probabilistic visual tokenization.
- The visual decoder uses an MMDiT diffusion Transformer with RoPE and a frozen SDXL VAE, totaling a 1B decoder trained from scratch.
- The bidirectional token refiner stacks two modulated Transformer blocks to strengthen interaction between visual and textual embeddings.
3 Data Composition and Training Procedure
Ovis-U1 is trained on multimodal understanding, text-to-image, and image-editing data through a six-stage pipeline. The stages progressively train the decoder, align modalities, preserve understanding capability, and fine-tune generation and editing.
- Data Composition: Training data covers multimodal understanding, text-to-image generation, and image-plus-text-to-image generation, including editing and controlled-generation subsets.
- Data Composition: The model uses public and in-house datasets for understanding, generation, editing, reference-image conditioning, pixel-level control, and related transformations.
- Training Procedure: The proposed pipeline adds generation stages to Ovis’s training procedures and begins from a pretrained language model rather than a pretrained frozen MLLM.
- Training Procedure: Stage 0 trains a randomly initialized 1B diffusion decoder and refiner on text-to-image data, while Stage 1 trains the visual adapter across all three task types.
- Training Procedure: Stage 2 jointly fine-tunes the visual encoder and adapter, and Stage 3 trains understanding parameters before fixing them to preserve understanding capability.
- Training Procedure: Stages 4 and 5 align the decoder with optimized embeddings, improve text-to-image performance over Stage 0, and fine-tune text-to-image and image-editing capabilities.
4 Evaluation
The evaluation benchmarks Ovis-U1 on the three capabilities targeted by the unified model: image understanding, text-to-image generation, and image editing. It uses established benchmark suites spanning multiple subskills and image-instruction pairs.
- The evaluation covers image understanding, text-to-image generation, and image editing.
- Understanding: Understanding is measured with the OpenCompass Multi-modal Academic Benchmarks, whose average aggregates eight benchmarks.
- Text-to-Image Generation: Text-to-image generation is evaluated with CLIPScore, DPG-Bench, and GenEval.
- Image Editing: Image editing is evaluated on GEdit-Bench and ImgEdit, containing 606 and 811 image-instruction pairs, respectively.
5 Experiments
Experiments evaluate Ovis-U1 across multimodal understanding, text-to-image generation, and image editing, alongside ablations of token refinement and unified training. The results show strong cross-task performance and improvements from integrating understanding and generation data.
- Overall evaluation: Ovis-U1 is evaluated across multimodal understanding, text-to-image generation, and image editing tasks.The evaluation combines benchmark results, ablation studies, and qualitative examples.
- Main results: Ovis-U1 surpasses leading models in the 3B parameter range on OpenCompass Multi-modal Academic Benchmarks.The comparison includes InternVL2.5-2B, SAIL-VL-2B, InternVL3-2B, Qwen2.5-VL-3B, Ovis2-2B, SAIL-VL-1.5-2B, and Ristretto-3B.
- Main results: Ovis-U1 achieves text-to-image performance comparable to larger models despite using a 1B visual decoder.The reported generation comparisons use GenEval and DPG-Bench, and Ovis-U1 significantly outperforms OmniGen with a similar parameter count.
- Main results: Ovis-U1 demonstrates strong image-editing performance on ImgEdit-Bench and GEdit-Bench-EN against specialized editing models.The comparison includes MagicBrush, Instruct-Pix2Pix, AnyEdit, UltraEdit, Step1X-Edit, and ICEdit.
- Refiner ablation: Token-refiner design materially affects text-image alignment and generation performance, with larger-data results favoring different designs across metrics.With 50M training data, the baseline reaches CLIPScore 32.57 and DPG-Bench 82.97, while Variant V7 reaches DPG-Bench 83.81 with a slightly lower CLIPScore.
- Unified training: In unified-training ablations, Ovis-U1 improves understanding by 1.14 points over the Ovis baseline.The report attributes this improvement to using text-to-image generation and image-editing tasks to align the visual encoder during training stages 1 and 2.
- Unified training: Including image-editing data improves text-to-image performance by 0.77 on DPG-Bench in earlier-model ablations.The training-stage study reports progressive generation improvements and notes that the results come from an earlier model version.
- Image editing: Higher CFGimg preserves more input-image detail, whereas higher CFGtxt improves adherence to editing instructions.Across ImgEdit-Bench and GEdit-Bench-EN, results remain within 0.2 across CFG settings, although optimal settings differ by benchmark.
6 Conclusion
The report presents Ovis-U1 as a unified model for understanding, generation, and editing, emphasizing unified training as central to its performance. It identifies missing reinforcement learning, larger-scale data, and architectural expansion as future directions.
- Conclusion: Ovis-U1 is a 3-billion-parameter unified model spanning multimodal understanding, text-to-image generation, and image editing.The report positions it as the first model in the Ovis unified model series.
- Conclusion: Unified training aligns the visual encoder and enhances both understanding and generation performance through collaborative training.The report evaluates these capabilities using established benchmarks for understanding, generation, and editing.
- Future directions: Future work targets more parameters, more diverse high-quality data, specialized editing architectures, and reinforcement learning.The report specifically notes that Ovis-U1 currently lacks a reinforcement learning stage for aligning unified multimodal models with human preferences.
7 Contributors
The contributor passage lists the authors associated with the report.
- Contributors: The report lists Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, Yang Li, and Qing-Guo Chen.