Source-linked AI summary
UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation
Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, Yatian Pang, Li Yuan
TL;DR
Unified models often cover understanding and text-to-image generation but rarely image perception and manipulation, despite their practical relevance. UniWorld-V1 combines multimodal understanding with high-resolution semantic visual features, and reports broad task performance using 2.7M training samples.
Problem
Most unified models address image-to-language understanding and language-to-image generation, while few cover image-to-image perception and manipulation, which require several distinct visual and semantic capabilities.
Source: §1 · p. 2Method
UniWorld-V1 combines multimodal large models that provide autoregressive understanding tokens with high-resolution contrastive semantic encoders that extract local pixel information and global concepts.
Source: §1 · p. 2Results
Using 2.7M training samples, UniWorld-V1 reports strong results across manipulation, generation, and perception, including outperforming BAGEL on ImgEdit-Bench and achieving performance comparable to GPT-4o-Image on GenEval.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that semantic encoders provide richer and more versatile visual representations than traditional VAE-based approaches.
Source: §5 · p. 10On GEdit-Bench, limited editing-data quantity and diversity and the absence of text-editing samples constrain instruction-following and text-editing performance.
Source: §4.3 · p. 9
Abstract
from arXiv · showhide
Although existing unified models achieve strong performance in vision-language understanding and text-to-image generation, they remain limited in addressing image perception and manipulation -- capabilities increasingly demanded in practical applications. Recently, OpenAI introduced the powerful GPT-4o-Image model, which showcases advanced capabilities in comprehensive image perception and manipulation, sparking widespread interest. Through carefully designed experiments, we observe that GPT-4o-Image likely relies on semantic encoders rather than VAEs for feature extraction, despite VAEs being commonly regarded as crucial for image manipulation tasks. Inspired by this insight, we propose UniWorld-V1, a unified generative framework built upon semantic features extracted from powerful multimodal large language models and contrastive semantic encoders. Using only 2.7M training data, UniWorld-V1 achieves impressive performance across diverse tasks, including image understanding, generation, manipulation, and perception. We fully open-source the UniWorld-V1 framework, including model weights, training and evaluation scripts, and datasets to promote reproducibility and further research.
1 Introduction
UniWorld-V1 addresses the limited coverage of unified models on image perception and manipulation by combining multimodal understanding with semantic visual features. It reports strong results across editing, generation, and perception using 2.7M training samples.
1 Introduction
- Most unified models focus on image-to-language understanding and language-to-image generation, while few address image-to-image perception and manipulation.The latter require unified intent understanding, pixel-level information retention, and semantic extraction.Source: §1 · p. 2
- VAE-based visual features can hinder extending image-editing models to multiple perception and manipulation tasks because they encode substantial low-frequency information.The paper identifies this as a limitation for semantic-level tasks.Source: §1 · p. 2
- Experiments lead the authors to hypothesize that GPT-4o-Image uses semantic encoders rather than VAEs for visual feature extraction.This hypothesis informs UniWorld-V1’s design.Source: §1 · p. 2
- UniWorld-V1 combines multimodal large models for autoregressive understanding tokens with high-resolution contrastive semantic encoders that extract local pixel information and global concepts.The authors release code, weights, datasets, and training and evaluation scripts.Source: §1 · p. 2
- With 2.7M training samples, UniWorld-V1 outperforms BAGEL on ImgEdit-Bench, surpasses Step1X-Edit on add, adjust, and extract, and performs competitively on generation and perception tasks.It outperforms BAGEL on WISE, achieves performance comparable to GPT-4o-Image on GenEval, and performs detection, segmentation, and depth prediction.Source: §1 · p. 2
2 Observation
Experiments on GPT-4o-Image’s edits and denoising behavior suggest that it relies more on semantic encoders than VAEs. The authors use this inference to motivate UniWorld-V1.
2 Observation
- In a local bus-ad editing example, text labels shift position, contrary to the authors’ expectation that VAE features would preserve their positions.The example asks GPT-4o-Image to turn the bus advertisement blue.Source: §2 · pp. 3–4
- GPT-4o-Image’s denoising of a heavily noised dog image as a deer, matching captions from GPT-4o and Qwen2.5-VL, suggests reliance on multimodal-model priors.At noise level 0.6×, the image is denoised as a deer; the models caption the noisy input as a deer.Source: §2 · p. 4
- The authors infer from the experiments that GPT-4o-Image more likely uses semantic encoders than VAEs because source-image low-level information is not preserved after manipulation.They use this inference to guide UniWorld-V1’s architecture.Source: §2 · p. 4
3 Method
UniWorld-V1 combines a VLM with SigLIP visual features and a diffusion transformer, then trains the model in stages and uses sharded EMA and task-specific data and loss weighting. The method also describes procedures for generating edit masks and balancing loss on edited regions.
3.1 Model Architecture
- The architecture uses VLM features for high-level semantics and historical state, SigLIP for low-level image features, and a frozen autoregressive VLM for understanding alongside flow-matching generation.The reference image passes through both Qwen2.5-VL-7B and SigLIP; their outputs are concatenated as input to FLUX’s text branch, and T5 features are optional.Source: Figure 3 · p. 4, §3.1 · p. 4
3.2 Training Recipe
- Stage 1 trains only the VLM-to-FLUX MLP to align VLM features with T5 features; stage 2 unfreezes the FLUX image branch while freezing its text branch.During stage 1, SigLIP features are excluded; in stage 2, the model begins using SigLIP as reference cues after 5,000 to 10,000 steps.Source: §3.2 · p. 5
3.3 ZeRO-3 EMA
- ZeRO-3 shards the FP32 EMA model across GPUs so each GPU stores and updates only its own shard, reducing memory overhead.The training DiT operates under ZeRO-2, and the EMA is updated every step with computation reduced to 1/N as GPU count increases.Source: Figure 4 · p. 5, §3.3 · p. 5
3.4 Training Data
- Training data includes about 1.4M perception samples, 1M manipulation samples, and 300k text-to-image samples.Perception data includes maps such as depth and segmentation; manipulation data includes common edits and style transfer.Source: §3.4 · pp. 5–6
3.5 Adaptive Editing Region Weighting Strategy
- Edit masks are generated by pixel-wise differencing, dilation, connected-component filtering, and max-pooling downsampling.Differencing yields noisy differences, dilation can leave isolated pixels, and component filtering does not remove bubbles within larger edited regions.Source: Figure 5 · p. 6, §3.5 · p. 6
- Because edited regions may be small, the method assigns them higher loss weights based on the full-image-to-edit-area ratio; the weighting returns to uniform when the whole image is edited.The authors compare four functions and select logarithmic weighting, which grows moderately and avoids instability for extremely small regions.Source: §3.5 · pp. 6–7
4 Experiment
Experiments report state-of-the-art or near-state-of-the-art results across the three benchmark categories of understanding, generation, and editing. They also assess image perception qualitatively, where UniWorld-V1 is competitive with GPT-4o-Image across several tasks.
4.1 Main Results
- UniWorld-V1 achieves state-of-the-art or near-state-of-the-art performance in every subcategory across image understanding, generation, and editing.The section identifies these as its main comparative results.Source: §4.1 · p. 7
4.2 Text-to-Image Generation
- 0.84 is UniWorld-V1’s GenEval score with prompt rewriting, compared with BAGEL’s 0.88; without rewriting, UniWorld-V1 scores 0.79.UniWorld-V1 uses 2.7M training samples, compared with BAGEL’s 2665M.Source: §4.2 · p. 7, Table 2 · p. 8
- 0.55 is UniWorld-V1’s overall WISE score, and its Space-category score of 0.73 is closest to GPT-4o-Image’s 0.89.UniWorld-V1 is reported as the top performer in Space among evaluated models other than GPT-4o-Image.Source: §4.2 · pp. 7–8, Table 3 · p. 8
4.3 Image Manipulation
- 3.26 is UniWorld-V1’s ImgEdit-Bench overall score, ahead of BAGEL (3.20) and Step1X-Edit (3.06) among open-source models.GPT-4o-Image leads with 4.20; UniWorld-V1 ranks first among open-source models in Adjust (3.64), Extract (2.27), Replace (3.47), Remove (3.24), Background (2.99), and Hybrid (2.96).Source: §4.3 · pp. 8–9, Table 4 · p. 9
- On the English GEdit-Bench dataset used to test out-of-domain generalization, UniWorld-V1 has a relatively high perceptual score (G_PQ) of 7.43 and relatively low instruction-following score (G_SC) of 4.93.Source: §4.3 · p. 9
4.4 Visual Understanding
- Freezing the multimodal language model lets UniWorld-V1 inherit Qwen2.5-VL-7B’s understanding capabilities without retraining and yields competitive benchmark performance.The comparison reports that it surpasses Janus, Show-o, and Emu3 across multiple metrics and is competitive with BAGEL.Source: Table 6 · p. 9, §4.4 · p. 9
4.5 Image Perception
- In sampled qualitative comparisons, UniWorld-V1 shows competitive or sometimes superior perception performance to GPT-4o-Image, including on edge detection, normal maps, HED, segmentation, and sketches.The authors note that no suitable benchmark comprehensively evaluates its full image-perception capabilities.Source: Figure 6 · p. 10, §4.5 · p. 10
5 Conclusion
UniWorld-V1 uses a high-resolution semantic encoder in a unified architecture for image perception and manipulation. The authors report strong performance with 2.7M training samples and release code, weights, and datasets.
5 Conclusion
- With 2.7M training samples, UniWorld-V1 achieves superior performance against much larger models across diverse benchmarks, and the authors conclude semantic encoders provide richer, more versatile representations than VAE-based approaches.Source: §5 · p. 10
- Limited training data and lack of VLM fine-tuning mean specific instruction templates are needed to outperform BAGEL.Source: §5 · p. 10
- Reference images processed at 512 × 512 lack sufficient detail for generating all details at 1024 × 1024 scale.Source: §5 · p. 10
- Authors flag benchmark validity limits: thresholded DPG/GenAI scores may not reflect human preference, while other benchmarks miss natural co-occurrence or reference-region sensitivity.Source: §5 · pp. 10–11
Future Work
- Future work includes collecting more data and jointly training with a VLM, or using higher-resolution semantic encoders or VLM techniques such as multi-scale image gridding.Source: Future Work · p. 11
Failed Attempts
- DINO V2 and RADIO V2.5 substitutions were unsuccessful; direct Qwen visual features also yielded poor reference consistency, attributed to objective mismatch and insufficient low-level control.Source: Failed Attempts · p. 11