Source-linked AI summary
InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing
Changyao Tian, Danni Yang, Guanzhou Chen, Erfei Cui, Zhaokai Wang, Yuchen Duan, Penghao Yin, Sitao Chen, Ganlin Yang, Mingxin Liu, Zirun Zhu, Ziqian Fan, Leyao Gu, Haomin Wang, Qi Wei, Jinhui Yin, Xue Yang, Zhihang Zhong, Qi Qin, Yi Xin, Bin Fu, Yihao Liu, Jiaye Ge, Qipeng Guo, Gen Luo, Hongsheng Li, Yu Qiao, Kai Chen, Hongjie Zhang
TL;DR
Unified multimodal models must balance strong semantic understanding with capable visual generation and editing. InternVL-U addresses this gap with a modular 4B-parameter architecture and reasoning-centered data synthesis, and evaluations report stronger generation and editing while retaining multimodal understanding.
Problem
Unified multimodal models face a trade-off between semantic understanding and visual generation, compounded by conflicting objectives and data regimes.
Method
InternVL-U combines unified contextual modeling, modality-specific modularity, decoupled visual representations, an MMDiT-based generation head, and Chain-of-Thought data synthesis.
Results
InternVL-U consistently outperforms unified baselines in generation and editing, particularly on text-rich and knowledge-intensive tasks, while retaining robust multimodal understanding.
Takeaways & Limitations
The model provides a unified framework combining understanding, reasoning, generation, and editing with a reported balance between performance and efficiency.
Abstract
from arXiv · showhide
Unified multimodal models (UMMs) that integrate understanding, reasoning, generation, and editing face inherent trade-offs between maintaining strong semantic comprehension and acquiring powerful generation capabilities. In this report, we present InternVL-U, a lightweight 4B-parameter UMM that democratizes these capabilities within a unified framework. Guided by the principles of unified contextual modeling and modality-specific modular design with decoupled visual representations, InternVL-U integrates a state-of-the-art Multimodal Large Language Model (MLLM) with a specialized MMDiT-based visual generation head. To further bridge the gap between aesthetic generation and high-level intelligence, we construct a comprehensive data synthesis pipeline targeting high-semantic-density tasks, such as text rendering and scientific reasoning, under a reasoning-centric paradigm that leverages Chain-of-Thought (CoT) to better align abstract user intent with fine-grained visual generation details. Extensive experiments demonstrate that InternVL-U achieves a superior performance - efficiency balance. Despite using only 4B parameters, it consistently outperforms unified baseline models with over 3x larger scales such as BAGEL (14B) on various generation and editing tasks, while retaining strong multimodal understanding and reasoning capabilities.
1. Introduction
InternVL-U addresses the tension between multimodal understanding and visual generation through modular architecture and reasoning-centered data synthesis. Evaluations report stronger generation and editing while retaining multimodal understanding.
- Motivation: Existing unified multimodal models face unresolved design trade-offs and costly training challenges when jointly acquiring understanding and generation.The field lacks consensus on modeling, architecture, and representation, while differing modality data distributions complicate training.
- Approach: InternVL-U combines unified contextual modeling, modality-specific modularity, and decoupled visual representations with an MMDiT-based generation head.The design builds on an existing strong multimodal language model and uses a unified semantic conditioning interface.
- Approach: Its data synthesis pipeline targets text rendering, scientific reasoning, spatial manipulation, and humor generation as high-semantic-density tasks.The pipeline includes automated typography and editing data, structured scientific visual-text data, and specialized task data.
- Approach: Chain-of-Thought transforms vague user instructions into executable plans with constraints for precise visual execution and intent alignment.The paradigm is applied to tasks including meme generation, geometric transformation, and logically constrained editing.
- Results: InternVL-U consistently outperforms unified baselines in generation and editing, especially for text-rich and knowledge-intensive scenarios, while retaining robust multimodal understanding.The reported results emphasize instruction following, legible text rendering, knowledge-rich generation, and complex logic-dependent editing.
2. Related Work
Related work situates unified multimodal models among MLLM-based vision-language systems and image-generation frameworks. Existing UMMs commonly use autoregressive discrete-token or other unified visual representations.
- Multimodal Language Models: Modern multimodal language models connect vision encoders to large language models through adapters for vision-language understanding.Representative open-source families include LLaVA, Qwen-VL, and InternVL, alongside proprietary GPT and Gemini models.
- Visual Generation: Visual generation has progressed from GANs to diffusion and flow-matching frameworks, while discrete-token methods generate images autoregressively through VQ-style codecs.These families differ in how they represent and generate visual content, with discrete approaches offering a unified token space with language models.
- Unified Multimodal Models: Existing unified multimodal models generally include autoregressive discrete-token approaches that treat image generation as next-token prediction.Examples include Chameleon, Emu3, and SynerGen-VL.
3. Method: InternVL-U
InternVL-U uses unified contextual modeling with modality-adaptive targets, modality-specific modules, and decoupled visual representations. Its visual generation head combines dual-stream MMDiT processing, gated attention, and unified spatial position encoding.
- Unified Contextual Modeling: InternVL-U projects visual and linguistic tokens into a shared latent space for unified autoregressive contextual modeling, while using modality-adaptive generation objectives.Text uses autoregressive modeling, whereas visual generation uses Flow Matching for continuous spatial signals.
- Structural Efficiency: Modality-specific encoding stems and a dedicated generation head improve structural efficiency by avoiding uniform processing of semantically different modalities.The backbone uses a pre-trained ViT to aggregate visual information before unified latent-space processing.
- Decoupled Visual Representations: Decoupled visual representations assign high-level ViT features to understanding and separate VAE latents to image generation.This avoids forcing one encoder to balance semantic abstraction with reconstructable pixel detail.
- Visual Generation Head: The visual generation head uses dual-stream MMDiT blocks with joint attention but disentangled QKVO projections and feed-forward networks for context and target streams.The design accounts for their distinct statistical properties while preserving token-level interaction.
- Visual Generation Head: Gated attention modulates each stream’s output through a sigmoid gating projection, adding expressivity with minimal parameter overhead.The paper identifies this as the first integration of a gating mechanism within MMDiT.
- Visual Generation Head: Unified MSRoPE applies temporal, height, and width positional embeddings to both visual context tokens and generative targets, with interpolation supporting resolution scaling.Using the full positional range with larger strides during low-resolution training aims to preserve a consistent global spatial representation.
4. Data Construction
InternVL-U constructs a broad training corpus by combining open-source data with synthesized examples for generation, editing, text rendering, scientific reasoning, and other semantically demanding tasks. The pipeline uses preprocessing, specialized synthesis branches, and task-aware verification to improve data quality and alignment.
- Corpus construction: The training corpus combines publicly available datasets with synthetic pipelines tailored to diverse image-generation and image-editing tasks.The initial pool includes many open-source datasets and specialized additions for long-tail portraits and text-rich imagery.
- General preprocessing: Filtering, expansion, and deduplication produce a high-quality source pool before parallel branches generate text-to-image pairs and instruction-guided editing data.Filtering considers aesthetics, resolution, safety, and watermarks; expansion uses retrieval and synthesis, while perceptual hashes remove near-duplicates.
- General preprocessing: Captioning uses concise, dense, and human-centric descriptions to improve concept binding, scene coverage, and portrait-specific detail.A pretrained MLLM generates captions at different levels of granularity, covering visual elements, holistic structure, style, facial attributes, expressions, poses, and clothing.
- Image editing data: Image-editing synthesis covers global, object, attribute, and compositional operations through routed instruction-generation and image-editing agents.An automated verifier retains pairs that exceed thresholds for instruction following, editing consistency, and generation quality.
- Text-centric data synthesis: Text-centric synthesis renders semantically related or pure text on images and supports Chinese and English through masks, typography controls, and adaptive layouts.The pipeline also targets fine-grained edits within existing text-rich images, while adaptive sizing and line breaks improve text arrangement.
- Reasoning-centric data synthesis: Reasoning-centric synthesis converts abstract instructions into structured reasoning traces and execution targets for controllable generation and editing, including science-image tasks.Scientific data uses intermediate conceptual analysis, layout planning, and step-by-step directives for mathematically, chemically, physically, and computationally structured images.
5. Experiments
InternVL-U performs strongly across multimodal understanding, generation, and editing evaluations, balancing compact scale with broad capabilities. Explicit reasoning particularly benefits knowledge-intensive generation, complex editing, and text rendering.
- Multimodal Understanding and Reasoning: InternVL-U remains competitive on multimodal understanding and reasoning, scoring 1607.5 on MME-P, 83.9 on OCRBench, and 54.7 on MMMU versus BAGEL’s 55.3.Its compact 2B+1.7B architecture surpasses comparable-sized Janus-Pro and Ovis-U1 while retaining performance close to the larger BAGEL model.
- General Image Generation: 0.85 overall on GenEval, InternVL-U ranks highest among unified models and surpasses most specialized generation models despite using half or fewer parameters than BAGEL.
- Text-centric Image Generation: 0.738 English and 0.860 Chinese on LongText-Bench demonstrate robust multilingual text generation, with large gains over previous unified models.Qualitative results also report more readable Chinese, English, numerical, and mathematical text with fewer artifacts.
- Knowledge-informed Image Generation: 0.46 to 0.58 on WISE shows that CoT substantially improves world-knowledge integration in text-to-image generation.InternVL-U surpasses unified baselines on cultural commonsense, spatio-temporal reasoning, and natural science.
- Knowledge-informed Image Generation: 22.9 overall on GenExam with 3.7B parameters gives InternVL-U the highest unified-model scores, especially in physics, chemistry, and biology.The results support science-centric image generation combining understanding, reasoning, and generation.
- Image Editing: 6.88 on GEdit-Bench with CoT surpasses BAGEL at 6.52 and Ovis-U1 at 6.42, while TextEdit reaches F1 0.71 and an MLLM-based score of 0.88.On RISEBench, CoT raises the score from 3.6 to 9.4, exceeding BAGEL’s 6.1 and Qwen-Image-Edit’s 8.9.
6. Conclusion
InternVL-U unifies understanding, reasoning, generation, and editing through a strong understanding backbone with modular visual generation. Its data synthesis pipeline and CoT paradigm align abstract intent with precise visual execution while retaining competitive multimodal reasoning.
- InternVL-U integrates generative capabilities into a strong understanding backbone using unified context modeling, modality-specific modularity, and decoupled visual representations.
- A comprehensive data synthesis pipeline with CoT aligns abstract user intent with precise visual execution for knowledge-intensive generation and editing.
- Empirical results show strong knowledge-intensive generation and editing alongside competitive multimodal understanding and reasoning performance.
A. TextEdit Benchmark
TextEdit is a human-curated benchmark for systematic, fine-grained evaluation of text-centric image editing. It covers diverse virtual and real-world scenarios and combines manually edited ground truth with a practical evaluation protocol.
- TextEdit addresses unreliable text editing by evaluating diverse real-world requirements such as advertising, posters, UI localization, and commercial asset updates.
- The benchmark uses a human-filtered pipeline and manually edited ground-truth images to reduce ambiguity and support precise quantitative evaluation.
- TextEdit bridges practical user needs and model assessment through systematic scenario coverage and a robust evaluation protocol.
- A scenario taxonomy organizes examples into Virtual Scenes and Real-world Scenes, spanning digital formats and physical carriers.
A.2.2. Evaluation Metrics
The evaluation framework combines objective text and image metrics with perceptual MLLM-based judgments to assess both editing correctness and visual preservation.
- A hybrid strategy combines Classic Metrics and MLLM-based Metrics because text editing must manipulate target content while preserving the background.
- Classic Metrics separately measure target-region editing success and background-region preservation using OCR-based edit distance, detection rates, CLIPScore, and aesthetics.
- MLLM-based Metrics use Gemini-3-Pro to score local realism and scene integrity, capturing ghosting, lighting inconsistencies, and partial erasure.
(a) Classic Metrics (Text-Centric)
Text-centric classic metrics evaluate target-text correctness, background-text preservation, and source-text removal using OCR similarity and ROI-aware measures.
- Target-text metrics: OCR Accuracy measures whether the target text is correctly rendered in the editing region.Detected texts overlapping the target region are matched against the ground-truth target text, with penalties for failed edits.
- Background-text metrics: OCR Precision evaluates whether detected background text matches the original, penalizing hallucinated or incorrect text.The metric compares background detections against original background texts.
- Background-text metrics: OCR Recall measures completeness of background-text preservation by penalizing missing original text.Higher recall indicates that most original background text remains without accidental removal or alteration.
- Background-text metrics: OCR F1-Score combines OCR Precision and Recall through their harmonic mean to summarize text-preservation quality.The unified metric captures both accuracy and completeness of background-text preservation.
- Target-region editing: ROI-Aware NED evaluates target-region editing by comparing predicted and target strings while penalizing residual source text.A residual similarity above 0.9 triggers a failed-erasure penalty, ensuring the region contains new text and removes the original.
(b) Classic Metrics (General)
General classic metrics assess semantic alignment between edited images and captions alongside overall aesthetic quality.
- Semantic and aesthetic metrics: CLIPScore measures semantic alignment between the predicted edited image and a caption generated for the ground-truth image.It is computed as cosine similarity between CLIP visual and text embeddings.
- Semantic and aesthetic metrics: Aesthetic Score evaluates the overall visual appeal of the generated image with a CLIP-based aesthetic predictor.The score typically ranges from 1 to 5, or 0 to 1 after normalization, with higher values indicating better visual quality and composition.
(c) MLLM-based Metrics
MLLM-based evaluation combines five Likert-scored dimensions into a weighted metric, with a cutoff that makes target-text accuracy prerequisite for secondary visual-quality scores.
- Evaluation dimensions: Target Accuracy measures spelling correctness, source-text erasure, and clarity of the replacement text.A score of 5 indicates perfect replacement without residual source-text artifacts.
- Evaluation dimensions: Text Preservation assesses whether non-target background text remains complete, accurate, and correctly positioned.The dimension penalizes unintended modifications to surrounding text and layout.
- Evaluation dimensions: Scene Integrity, Local Realism, and Visual Coherence assess structural stability, seamless local inpainting, and stylistic harmony with the scene.These dimensions cover geometry and perspective, editing-region artifacts, and matching font, lighting, shadows, and textures.
- Aggregation and cutoff: The final weighted score normalizes dimension scores and applies a cutoff when Target Accuracy s1 < 4, zeroing secondary-dimension contributions.Default weights are w1 = 0.4, w2 = 0.3, and w3 = w4 = w5 = 0.1; scores are reported separately for Virtual and Real subsets.
- Evaluation protocol: The evaluation prompt specifies three compared images and requires strict JSON output for automated parsing.The prompt compares original, ground-truth, and edited images, while Q5 focuses on aesthetic and lighting harmony.
A.3. MiniSet-500 Results
MiniSet-500 is a balanced, lightweight TextEdit evaluation subset used to report classic and MLLM-based editing metrics across virtual and real scenes.
- MiniSet-500 construction: MiniSet-500 contains 500 image-editing pairs sampled across all 18 TextEdit subcategories.Balanced sampling preserves scenario diversity while reducing evaluation cost for rapid benchmarking and ablation studies.
- MiniSet-500 construction: The full TextEdit benchmark remains the standard for comprehensive evaluation, while MiniSet-500 provides an efficient protocol for rapid experiments.The subset is intended to support lightweight open-source evaluation.
- Reported metrics: Table 23 reports classic metrics including OCR, ROI-Aware NED, CLIPScore, and Aesthetic Score, while Table 24 reports MLLM-based dimensions and overall average.Model sizes use separate understanding and generation parameter counts; results distinguish Real and Virtual scenes.
B. Data Construction Details
The data construction process filters images using image type, subject, text length, visual complexity, and subject-knowledge density. These criteria target images that are suitable for multimodal generation and reasoning tasks.
- Images are categorized by type, with some subject-specific categories removed, including microscopic images for biology and tables across subjects.
- Images from subjects such as Literature, Art, and Design are removed because they are usually natural images, paintings, or photographs.
- Images containing more than 200 characters of text are removed after extracting all image text.
- Image complexity is rated from 1 to 10, and subject-specific ranges such as 5-7 retain images of moderate complexity.
- Subject knowledge density is rated from 1 to 10, with subject-specific ranges such as 7-8 selecting knowledge-intensive images.
B.2. Chemistry Text-to-Image Data Synthesis
The construction process combines large-scale chemistry data acquisition with procedurally defined visual reasoning and editing tasks. These tasks cover trees, graphs, finite-state machines, and geometric transformations implemented through configurable rendering pipelines.
- Chemistry Text-to-Image Data Synthesis: The chemistry pipeline harvested 800,000 raw entries from PubChem, including physicochemical descriptors and unique Compound IDs.
- Task Definitions: Tree tasks include topology editing, traversal visualization, BST operations, heap operations, Huffman coding, and lowest-common-ancestor path highlighting.
- Task Definitions: Graph tasks cover k-hop neighborhoods, degree identification, cycle detection, bipartite coloring, shortest paths, and directed reachability.
- Task Definitions: Finite-state-machine tasks trace input strings, identify start and accepting states, and complete missing transitions with labels.
- Task Implementations: Geometric tasks include solids of revolution, plane and point symmetry, solid translation, and solid projection using GeoGebra or matplotlib.