InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing

Motivation

• Fully-native UMMs face unresolved design choices, costly joint training, and loss of SOTA MLLM benefits, while fully-ensemble systems trade large generator heads for fragmented conditioning pipelines.The introduction says these ensemble interfaces are difficult to align with a single MLLM's hidden-state space, limiting gains from constrained post-alignment training.Source: §1 · p. 6

Model design

• InternVL-U combines an MMDiT-based generation head with an MLLM-aligned conditioning interface, decoupled visual representations, and modality-specific modularity.Built on InternVL 3.5, it uses three-stage progressive training to acquire generation and editing while inheriting understanding and reasoning capabilities.Source: §1 · pp. 6–7

Motivation

• Generation models prioritize aesthetics and fidelity, while understanding models emphasize knowledge and reasoning, reflecting a training-data domain gap that complicates AGI-oriented unification.Natural-image generation data is described as texture-rich but relatively low in semantic density, unlike text-rich, structured understanding data.Source: §1 · p. 6

Data synthesis

• Its data synthesis pipeline targets text rendering, scientific reasoning, spatial manipulation, and humor, including bilingual typography, local editing, and structured scientific visuals.Scientific data is built with tools such as GeoGebra and SVG and academic corpora across mathematics, physics, and computer science.Source: §1 · p. 7

Reasoning-centric data

• A reasoning-centric synthesis paradigm uses explicit Chain-of-Thought to turn vague instructions into executable plans with constraints for intent-aligned visual generation and editing.The introduction describes its use in meme generation, geometric transformation, and logically constrained editing.Source: §1 · p. 7

Results

• InternVL-U outperforms unified models across general, text-centric, and knowledge-intensive generation benchmarks, retains strong multimodal understanding, and introduces GenEditEvalKit and TextEdit Benchmark.The introduction highlights legible text rendering, instruction following, knowledge-rich generation, and complex logic-dependent editing as strengths.Source: §1 · p. 7

2.1. Multimodal Large Language Models

• Open-source LLaVA, Qwen-VL [3] [4] [5] [128], and InternVL [17] [18] [19] [89] [90] [129] [163], alongside GPT [1] and Gemini [21] [115], demonstrate visual understanding; standard MLLMs connect vision encoders to LLMs through adapters.Recent extensions include interleaved image-text sequences and video understanding for long-context multimodal interaction.Source: §2.1 · p. 7

2.2. Visual Generative Models

• Visual generation has progressed from GANs [57] [58] to diffusion [51] [108], flow matching [85], and autoregressive discrete-token methods [10] [36] [107] [119], while instruction-driven editing [6] [83] requires localized changes with semantic consistency.Recent text-to-image systems emphasize instruction following and complex scenes, while data- and architecture-centric work targets complex structures, text rendering, long prompts, and multi-concept descriptions.Source: §2.2 · p. 8

2.3. Unified Multimodal Models

• Unified multimodal models integrate understanding, generation, and editing; discrete-token autoregressive methods unify modalities but may face fidelity challenges, while diffusion/hybrid methods combine LLM reasoning with high-fidelity generation.Examples include Chameleon [114], Emu3 [24], and SynerGen-VL [69] for discrete-token methods, and BLIP-3o [14], BAGEL [31], and Ovis-U1 [124] for diffusion/hybrid methods; InternVL-U builds on InternVL3.5 [129] for general-purpose and domain-specific capabilities.Source: §2.3 · p. 8

3.1.1. Overall Design Principles

• InternVL-U unifies visual and language tokens for contextual reasoning, then uses autoregressive text modeling and continuous Flow Matching for image generation.The design avoids discrete visual-token quantization bottlenecks while retaining categorical next-token prediction for text.Source: §3.1.1 · pp. 8–9
• A unified backbone combines modality-specific encoding stems with a dedicated MMDiT image-generation head conditioned on shared hidden states.The encoder-based MLLM backbone aggregates visual information before semantic reasoning, while the generation head synthesizes images in a continuous visual latent space.Source: §3.1.1 · p. 9
• Understanding uses ViT semantic features, while generation uses separate VAE latents, avoiding a single-encoder optimization trade-off and extra context-backbone computation.This decoupling preserves high-level features for reasoning and reconstructable low-level detail for synthesis.Source: §3.1.1 · pp. 9–10

3.1.2. Visual Generation Head

• The visual generation head uses independent projectors and a normalized context branch, then processes context and target streams with joint attention and disentangled parameters.An element-wise attention gate is introduced within MMDiT to improve expressivity with minimal parameter overhead and mitigate attention sinks in high-resolution, long-context settings.Source: §3.1.2 · pp. 10–11, Figure 4 · p. 10
• Unified 3D MSRoPE encodes temporal, height, and width positions for context and target tokens, while resolution interpolation supports scaling without tiling artifacts.The method defines positions using the maximum target resolution and uses a larger stride during low-resolution pre-training.Source: §3.1.2 · p. 11

3.2.1. Training Objective

• Training jointly optimizes autoregressive next-token prediction for text and velocity-based Flow Matching for continuous image latents.The two losses are combined with weights dynamically adjusted across training stages to prioritize capabilities such as visual fidelity or reasoning.Source: §3.2.1 · pp. 11–12

3.2.2. Training Pipeline

• A three-stage curriculum freezes the MLLM while grounding the generation head and expanding resolution, then unfreezes the full model for CoT-integrated generation and editing.Early training mixes text-to-image and editing data at 512px; continued training uses 512–1024px images and injects condition-image VAE latents for pixel-level consistency.Source: §3.2.2 · p. 12

4.1. Open-source Data Collection

• The corpus combines public generation and editing datasets with specialized portrait and text-rich data, then filters, expands, and deduplicates samples to build a high-quality source pool.Expansion uses search-based retrieval to capture long-tail concepts and synthesis-based variants to increase diversity; perceptual hashes remove near-duplicates.Source: §4 · p. 12, §4.1 · p. 12, §4.2.1 · pp. 13–14

4.2.2. Text-to-Image Data

• General text-to-image data use concise, dense, and human-focused captions, while targeted text-rich synthesis expands long-tail examples and supports English and Chinese.A captioning MLLM generates varied descriptions; text-rich images are synthesized from variant captions and specified content and backgrounds.Source: §4.2.2 · p. 14

4.2.3. Image Editing Data

• Image-editing data span global, object, attribute, and compositional tasks, with a multi-agent pipeline routing images to specialized instruction and editing agents.An automated task-aware verifier retains pairs only when they pass thresholds for instruction following, editing consistency, and generation quality.Source: §4.2.3 · pp. 14–15

4.3. Text-centric Data Synthesis

• Text-centric data cover text rendering on natural images and pure-color backgrounds, plus text editing within images, to strengthen visual text understanding and generation.Rendering varies masks, colors, fonts, and adaptive layouts; OCR-extracted regions and MLLM-generated instructions guide context-aware text edits.Source: §4.3 · p. 15, §4.3.1 · p. 16, §4.3.2 · p. 16

4.4. Science-centric Data Synthesis

• Science data cover multiple disciplines: general science generation uses filtered images and synthesized captions, while computer-science editing data target trees, graphs, and finite-state machines.Task-specific renderers, fixed node anchors, and overlap checks support structural consistency; editing prompts are generated by solving tasks and rewriting the solutions as CoT.Source: §4.4 · p. 16, §4.4.1 · p. 17, §4.4.3 · p. 19

4.4.2. SVG-based Physics Editing Data

• An SVG-based pipeline synthesizes paired physics-image editing data from textbooks and exams by manipulating structured SVG code, yielding resolution-independent targets; compared with directly generating output images using Nano Banana Pro, it lowers cost from $0.16 to $0.03 per sample.The pipeline filters images containing rich text formatting, physical structural errors, or annotation errors.Source: §4.4.2 · pp. 18–19

4.5. Spatial-centric Data Synthesis

• Spatial data include solid-geometry transformations, three-view CAD editing, and object rotations with filtered backgrounds and paired edits.CAD views vary object colors and materials; rotation data filter for shape, orientation, and scene quality, then use object-first or background-first synthesis strategies.Source: §4.5.1 · p. 19, §4.5.2 · pp. 19–20, §4.5.3 · pp. 20–21

4.6. Humor-centric Data Synthesis

• Meme data include generation from abstract intent and instruction-guided editing that aims to preserve the input image’s content and style.The synthesis pipeline collects memes from the Internet and open-source datasets and generates paired images with aligned instructions using CoT reasoning.Source: §4.6 · p. 22
• A five-stage humor-data pipeline detects text, generates image-grounded and user-style prompts, constructs paired images, and produces aligned editing triplets.Text-containing images are cleaned with a Text Removing Agent; other images are modified or given instructed content by a Text Adding Agent.Source: §4.6 · p. 22

4.7. Reasoning-centric Data Synthesis

• Reasoning-centric synthesis translates brief, abstract prompts into structured objectives, subtasks, constraints, and ordered operations, linking instructions and reasoning traces to execution targets across four image settings.The structured interpretations are intended to improve instruction following, robustness, and controllability in generation and editing.Source: §4.7 · pp. 22–23
• For general images, CoT expands generation prompts into visual descriptions and editing prompts into localized changes, preserved attributes, and consistency constraints without changing intent.The passage reports that CoT-enhanced instructions at inference yield more accurate results better aligned with user requirements.Source: §4.7 · p. 23
• For knowledge-infused images, reasoning makes implicit background knowledge explicit as concrete visual concepts or detailed scenes to guide logically consistent generation and editing.The process is described as using open-source models to enhance instructions with detailed reasoning.Source: §4.7 · p. 23
• For memes, reasoning-based enhancement specifies visual details, humor structure, template and typography constraints, and localized edits while preserving unchanged regions.This provides stronger supervision for controllability and alignment with user intent.Source: §4.7 · p. 24
• Science T2I and editing use intermediate concept and layout reasoning or executable directives; computer-science solutions map to 13 human-defined templates rewritten by Qwen3-Max [4].For physics editing, detailed step-by-step prompts are used before summarization.Source: §4.7 · p. 24

5.2. Multimodal Understanding and Reasoning

• InternVL-U scores 1607.5 on MME-P and 83.9 on OCRBench, surpassing comparable-sized Janus-Pro and Ovis-U1, and reaches 54.7 on MMMU versus BAGEL’s 55.3.Across seven understanding and reasoning benchmarks, its performance is described as comparable to the much larger BAGEL model on reasoning, with minimal degradation from understanding-only baselines.Source: §5.2 · p. 26, Table 4 · p. 26

5.3.1. General Image Generation

• InternVL-U scores 0.85 overall on GenEval, the highest among unified models, using half or fewer parameters than models such as BAGEL and surpassing most specialized generation models.GenEval evaluates compositional image properties including object co-occurrence, position, count, and color.Source: §5.3.1 · p. 27, Table 5 · p. 27
• On DPG-Bench, InternVL-U outperforms other unified models, especially on Global and Entity dimensions; on TIIF it performs strongly among unified models, particularly in advanced instruction following.On OneIG-Bench, it has the highest overall score among small-scale open-source unified models, demonstrating fine-grained alignment and multilingual robustness.Source: §5.3.1 · pp. 27–29, Table 6 · p. 27, Table 7 · p. 28

5.3.2. Text-centric Image Generation

• On CVTG-2k, InternVL-U achieves a 0.623 average word accuracy and state-of-the-art performance among unified models; on LongText-Bench, it scores 0.738 in English and 0.860 in Chinese.The LongText-Bench results exceed previous unified models by large margins and address their earlier difficulty rendering legible text.Source: §5.3.2 · p. 31, Table 11 · p. 31, Table 12 · p. 31

5.3.3. Knowledge-informed Image Generation

• On WISE, CoT raises InternVL-U’s overall score from 0.46 to 0.58, surpassing unified baselines BAGEL and UniWorld-V1; on GenExam, CoT yields an overall score of 22.9 with 3.7B parameters.InternVL-U achieves the highest GenExam scores among unified models, especially on physics, chemistry, and biology.Source: §5.3.3 · p. 33, Table 13 · p. 33, Table 14 · p. 33

5.4.1. General Image Editing

• InternVL-U achieves an average score of 6.66 on GEdit-Bench, exceeding BAGEL (6.52) and Ovis-U1 (6.42), while CoT raises its score to 6.88.Qwen-Image-Edit retains a lead on certain metrics.Source: §5.4.1 · p. 34, Table 16 · p. 35

5.4.2. Text-centric Image Editing

• On TextEdit, InternVL-U reaches an F1 score of 0.71, matching Nano Banano Pro and exceeding Ovis-U1 (0.35); its real-scene MLLM average is 0.88 versus BAGEL’s 0.53.TextEdit is a new benchmark with 2,148 samples spanning diverse text-editing scenarios; on real-scene images, InternVL-U also shows competitive capability against GPT-Image-1.5.Source: §5.4.2 · p. 36, Table 17 · p. 36, Table 18 · p. 36

5.4.3. Reasoning-informed Image Editing

• On RISEBench, CoT lifts InternVL-U’s overall score from 3.6 to 9.4, surpassing BAGEL (6.1) and Qwen-Image-Edit (8.9) and improving Instruction Reasoning and Appearance Consistency.Qualitative examples show handling of multi-step constraints, including temporal calculations, spatial and cultural understanding, and algorithmic rules; further examples cover computer-science, humor-centric, and mathematics-related edits.Source: §5.4.3 · p. 37, Table 19 · p. 37, Figure 25 · p. 38

• InternVL-U integrates generative capabilities into a strong understanding backbone using unified context modeling, modality-specific modularity, and decoupled visual representations.Source: §6 · p. 39
• A CoT-based data synthesis pipeline aligns abstract user intent with precise visual execution, bridging high-level intelligence and visual generation.Source: §6 · p. 39
• Empirical results show strength in knowledge-intensive generation and editing while retaining competitive performance on multimodal understanding and reasoning benchmarks.Source: §6 · p. 39

A.1. Design Motivation

• TextEdit targets unreliable text rendering, unintended changes to non-target regions, and existing benchmarks’ limited scenario coverage and evaluation of edit faithfulness and visual preservation.These limitations affect text editing in settings such as advertising, posters, UI localization, and commercial asset updates.Source: §A.1 · p. 51
• TextEdit uses human-filtered data and manually edited ground-truth images to support realistic scenarios and precise pixel-level fidelity and background-preservation evaluation.The benchmark describes these as advantages over prior benchmarks relying heavily on synthetic or automatically collected data.Source: §A.1 · p. 51
• Its taxonomy covers 18 diverse text-editing sub-classes, while its instruction paradigm uses plain text without auxiliary glyph maps or segmentation masks.The taxonomy offers finer-grained evaluation than benchmarks with limited or coarse categorizations.Source: §A.1 · p. 51
• TextEdit combines OCR, image-fidelity metrics, and multimodal LLM evaluation across target accuracy, text preservation, scene integrity, local realism, and visual coherence.The benchmark presents this as a hybrid protocol for comprehensive assessment.Source: §A.1 · p. 51

A.2. Design Details

• TextEdit integrates its scenario taxonomy and evaluation protocol to assess both technical text-editing performance and practical usability against real-world needs.The design is grounded in practical user scenarios.Source: §A.2 · p. 51

A.2.1. Scenario Taxonomy

• The taxonomy groups text editing into Virtual Scenes and Real-world Scenes, spanning digital carriers such as posters, comics, slides, and GUIs and real-world carriers such as products, buildings, transport, and paper media.The passage also identifies board-like media, personal accessories, and watermarks among the real-world examples.Source: §A.2.1 · p. 52
• TextEdit pairs curated scenario-aligned source images with rule-based editing instructions, then manually verifies ground truths for visual naturalness and instruction faithfulness.The verified images establish a gold standard for quantitative evaluation.Source: §A.2.1 · p. 52

A.2.2. Evaluation Metrics

• The hybrid evaluation strategy combines classic metrics and MLLM-based metrics to assess text editing.It addresses the dual requirement of manipulating specific text content while strictly preserving the background.Source: §A.2.2 · p. 53
• Classic metrics use OCR edit distance and detection rates for target-region editing success and background-region preservation, alongside CLIPScore [50] and aesthetic quality evaluation.CLIPScore assesses general image quality and semantic alignment.Source: §A.2.2 · p. 53
• Gemini-3-Pro provides fine-grained MLLM-based scoring of local realism and scene integrity, including visual artifacts and inconsistencies, to better align evaluation with human preference.The assessment simulates expert forensic analysis of ghosting, lighting inconsistencies, and partial erasure.Source: §A.2.2 · p. 53

(a) Classic Metrics (Text-Centric)

• OCR Accuracy scores the best similarity between detected text overlapping the edit region and the target, reducing the score when the source text remains and the target is absent.Detected text qualifies when its IoU with the target editing region exceeds 0.5; the failure penalty is 0.2 in that case and 1.0 otherwise.Source: (a) Classic Metrics (Text-Centric) · pp. 53–54
• OCR Precision measures preservation of background text by matching generated background detections to original background text, penalizing hallucinated or incorrect text.Background detections have IoU below 0.5 with the target region; higher scores indicate closer matches and fewer spurious text detections.Source: (a) Classic Metrics (Text-Centric) · p. 54
• OCR Recall measures how completely original background text is retained, while OCR F1-Score gives a harmonic-mean summary of background-text preservation.Recall penalizes missing background text; F1 balances OCR Precision and completeness.Source: (a) Classic Metrics (Text-Centric) · p. 54
• ROI-Aware NED compares OCR text cropped from the source-text bounding box with the target and penalizes residual source text similarity above 0.9.The shared string similarity is normalized using Levenshtein distance to range from 0 to 1; the residual penalty is 0.2 when triggered and 1.0 otherwise.Source: (a) Classic Metrics (Text-Centric) · pp. 53–54

(b) Classic Metrics (General)

• CLIPScore [50] compares the predicted edited image with a concise Qwen3-VL [4] caption of the labeled ground-truth image via embedding cosine similarity; higher scores indicate better semantic consistency.The metric reflects successful integration of edited text into the scene context.Source: (b) Classic Metrics (General) · p. 54
• A CLIP-based aesthetic predictor scores generated-image appeal, typically on a 1–5 scale or normalized 0–1, with higher values indicating better visual quality, composition, and perceptual appeal.The metric helps ensure text editing does not degrade overall image quality.Source: (b) Classic Metrics (General) · p. 55

(vi) Score Normalization and Cutoff Mechanism

• Gemini-3-Pro-Preview scores five dimensions on 1–5 Likert scales; normalized scores use default weights 0.4, 0.3, 0.1, 0.1, and 0.1, while raw D1 below 4 zeros secondary scores.D1 is Target Accuracy, and the secondary dimensions are D2–D5.Source: (c) MLLM-based Metrics · pp. 55–56

(i) Target Accuracy (D1)

• Target Accuracy (D1) scores spelling, complete erasure of source text, and new-text clarity and legibility in the edited region.A score of 5 indicates perfect replacement without residual source-text artifacts.Source: (c) MLLM-based Metrics · p. 55

(ii) Text Preservation (D2)

• Text Preservation (D2) measures whether background text remains complete and unchanged, including its layout and positioning.The score reflects preservation of non-target text elements.Source: (c) MLLM-based Metrics · p. 55

(iii) Scene Integrity (D3)

• Scene Integrity (D3) assesses preservation of architecture, object boundaries, spatial relationships, perspective, and depth without geometric distortion or warping.Its score indicates how well overall scene structure is maintained.Source: (c) MLLM-based Metrics · p. 55

(iv) Local Realism (D4)

• Local Realism (D4) evaluates clean, seamless edges, absence of blurring, ghosting, or seams, and natural integration of new text with its surroundings.The score reflects the edited region’s photorealistic quality.Source: (c) MLLM-based Metrics · p. 55

(v) Visual Coherence (D5)

• Visual Coherence (D5) scores how well font style, lighting, shadows, and texture match the original scene context.The score measures integration of edited text with the image’s overall visual style.Source: (c) MLLM-based Metrics · pp. 55–56

(vii) MLLM Overall Average

• The weighted five-dimension scalar, with the cutoff applied, is computed separately for Virtual synthetic scenes and Real-world scenes.This enables fine-grained analysis across the two scene subsets.Source: (c) MLLM-based Metrics · p. 56

MiniSet-500 construction

• MiniSet-500 contains 500 image-editing pairs sampled across all 18 TextEdit subcategories to preserve a balanced distribution of scenarios.The subset is randomly sampled from the full benchmark.Source: §A.3 · p. 59

Evaluation scope

• MiniSet-500 enables lower-cost rapid benchmarking and ablation studies, while the full TextEdit benchmark remains the standard for comprehensive evaluation.Source: §A.3 · p. 59
• Tables 23 and 24 report MiniSet-500 text-centric editing evaluations using classic and MLLM-based metrics, respectively.Source: Table 23 · p. 59, Table 24 · p. 59, §A.3 · p. 59

B. Data Construction Details

• Data-construction details describe a second-stage filtering process for general-science image generation.The filter is applied across five dimensions.Source: §B.1 · p. 60, B. Data Construction Details · p. 60

B.1. Filtering details for General Science Image Generation

• Images are classified into 30 MMMU image types [156], with types such as subject-specific microscopic images or tables excluded.Examples include removing microscopic images for biology and tables across all subjects.Source: §B.1 · p. 60
• Images classified in subjects such as literature, art, or design are removed because they are usually natural images, including paintings or photographs.Source: §B.1 · p. 60
• Images are removed when their extracted text exceeds 200 characters.Source: §B.1 · p. 60
• Image complexity is rated from 1 (very complex) to 10 (very simple), and subject-specific ranges such as 5–7 select images of moderate complexity.Ratings consider the number and complexity of components, objects, and text, including whether an image is too complex to draw.Source: §B.1 · p. 60
• Subject-knowledge density is rated from 1 (least knowledge) to 10 (very dense knowledge), with subject-specific ranges such as 7–8 applied.The rating reflects how much subject knowledge and reasoning generation requires.Source: §B.1 · p. 60

B.2. Chemistry Text-to-Image Data Synthesis

• The chemistry pipeline curated 600,000 instances from 800,000 PubChem entries and synthesized 600,000 visual-textual QA pairs linking chemical names, SMILES, and 2D visualizations.The curated corpus includes CIDs, chemical nomenclature, molecular formulas, SMILES representations, and structural diagrams.Source: §B.2 · p. 60

B.3. Computer Science Editing

• Computer science tree tasks cover completion, insertion, pruning, traversal-order visualization, BST insertion or repair, heap operations, Huffman construction, and LCA or path highlighting.Heap examples include 1-3 insert or extract-root steps with sift-up or sift-down and a memory-array view; Huffman tasks construct trees or merge nodes from character frequencies.Source: §B.3 · p. 60
• Graph editing tasks visualize exact k-hop neighborhoods, identify nodes of a specified degree, depict cycles, color bipartite graphs, mark shortest paths, and show directed reachability.Reachability marks the complete set of downstream nodes reachable forward from a source.Source: §B.3 · p. 61
• Finite-state-machine tasks trace input-string transition paths, identify start and accepting states, and complete missing arrowed edges and input labels.Source: §B.3 · p. 61

B.4. Solid Geometry

• Solid-geometry examples cover revolution, plane symmetry, and point symmetry, generated with GeoGebra or parallel GeoGebra and matplotlib pipelines.Revolution samples use 3-6-vertex polygons swept around an x, y, or z axis; plane symmetry varies primitives, base edges, and colors, while point symmetry varies primitive, color, rotation, and viewpoint.Source: §B.4 · p. 61
• Translation examples use configurable solids and colors, with movement along the x-, y-, or z-axis by a random integer magnitude from 4 to 10.The matplotlib pipeline keeps the original and translated solids visible in the same view.Source: §B.4 · p. 61
• Projection examples rotate configurable 3D solids by a configurable angle before orthographically projecting them onto the xoy plane.Source: §B.4 · p. 61