Source-linked AI summary
Uni-Edit: Intelligent Editing Is A General Task For Unified Model Tuning
Dian Zheng, Manyuan Zhang, Hongyu Li, Hongbo Liu, Kai Zou, Kaituo Feng, Hongsheng Li
TL;DR
Unified multimodal models face conflicting demands across understanding, generation, and editing, while existing editing instructions underuse understanding. Uni-Edit addresses this with reasoning-intensive editing data and finds that single-task tuning simultaneously improves all three capabilities without auxiliary operations.
Problem
Unified multimodal models face conflicting understanding and generation demands, while existing editing data uses simplistic instructions and mixed training requires complex pipelines and balancing.
Method
Uni-Edit transforms diverse visual-understanding queries into reasoning-intensive editing instructions with embedded questions and nested logic, producing the Uni-Edit-148k dataset.
Results
Uni-Edit tuning simultaneously improves understanding, generation, and editing performance on unified models without auxiliary modules or intricate data balancing.
Takeaways & Limitations
Intelligent editing can serve as a general task for unified-model tuning, enabling mutual reinforcement across understanding, generation, and editing.
Takeaways & Limitations
Uni-Edit’s effectiveness is partly bottlenecked by base-model capabilities, with some Caption, Math, and OCR data underutilized because of limited text-rendering performance.
Abstract
from arXiv · showhide
Currently, enhancing Unified Multimodal Models (UMMs) with image understanding, generation, and editing capabilities mainly relies on mixed multi-task training. Due to inherent task conflicts, such strategy requires complex multi-stage pipelines, massive data mixing, and balancing tricks, merely resulting in a performance trade-off rather than true mutual reinforcement. To break this paradigm, we propose Uni-Edit, an intelligent image editing task that serves as the first general task for UMM tuning. Unlike complex mixed pipelines, Uni-Edit improves performance across all three abilities at once using only one task, one training stage, and one dataset. Specifically, we first identify image editing as an inherently ideal general task, as it naturally demands both visual understanding and generation. However, existing editing data relies on simplistic instructions that severely underutilize a model's understanding capacity. To address this, we introduce the first automated and scalable data synthesis pipeline for intelligent editing, transforming diverse VQA data into complex and effective editing instructions with embedded questions and nested logic. This yields Uni-Edit-148k, pairing diverse reasoning-intensive instructions with high-quality edited images. Extensive experiments on BAGEL and Janus-Pro demonstrate that tuning solely on Uni-Edit achieves comprehensive enhancements across all three capabilities without any auxiliary operations.
1 Introduction
Uni-Edit addresses conflicting understanding and generation demands in unified multimodal models by using intelligent image editing as a single general tuning task. Its automated data synthesis pipeline produces reasoning-intensive instructions, enabling simultaneous improvements in understanding, generation, and editing without auxiliary operations.
- Motivation: UMM training faces conflicting demands because understanding needs high-level semantics while generation requires fine-grained structural details, causing performance degradation when optimized together.The passage describes a trade-off in which improving one capability can impair the other.
- Uni-Edit: Uni-Edit treats image editing as an inherently general task because editing naturally requires both visual understanding and generation.The proposed task is intended as the first general task for unified model tuning.
- Uni-Edit: Uni-Edit simultaneously improves understanding, generation, and editing using one task, one dataset, and one training stage.This avoids the complex mixed pipelines used to balance multiple tasks.
- Data construction: The data synthesis pipeline transforms diverse VQA data into reasoning-intensive editing instructions by embedding questions and nested logic before the edit.GPT-4o classifies data into eight edit types and constructs instructions that require editing based on the question answer.
- Results: Fine-tuning BAGEL with Uni-Edit improves generation, understanding, and editing without data balancing or auxiliary modules.The result supports the existence of a general tuning task for unified models.
- Data construction: Uni-Edit-148k is presented as the first editing dataset with intelligent instructions and high-quality images, covering broad knowledge and supporting scalable construction from understanding datasets.The pipeline is designed to be applicable to any understanding dataset.
2 Related Work
Unified multimodal models have progressed from world understanding toward jointly understanding and generating images, but these tasks inherently conflict. Intelligent editing is presented as a general-task candidate because it combines complex, world-knowledge-grounded instructions with precise image modification.
- Unified multimodal models: MLLM research has shifted from world understanding toward unified models that both understand and create images.This direction builds on progress in LLMs and MLLMs such as Qwen and LLaMA.
- Unified multimodal models: Understanding and generation inherently conflict, motivating the search for a task that can universally improve unified-model capabilities.
- Intelligent editing: Intelligent editing is proposed as an ideal general task because it uses complex instructions grounded in diverse world knowledge for holistic unified training.Unlike existing approaches, its instructions are not limited to simple image operations.
- Image editing: Early image-editing systems supported only basic operations such as adding, deleting, or changing objects because their text understanding was limited.Editing nevertheless requires precise, coherent instruction-following modifications to existing images.
3 Uni-Edit
Uni-Edit converts diverse visual-understanding tasks into complex image-editing instructions to jointly target understanding, generation, and editing. Built on BAGEL, its two-stage pipeline uses generation-only tuning followed optionally by language-head alignment.
- Data construction: Uni-Edit addresses the challenge of improving understanding, generation, and editing simultaneously by transforming heterogeneous understanding responses into editing instructions.The responses include multiple-choice, fill-in-the-blank, captions, and chain-of-thought reasoning, making conversion nontrivial.
- Data construction: The pipeline categorizes understanding data into seven domains to preserve broad capabilities and reduce catastrophic forgetting.The listed approach covers diverse visual-understanding categories, including chart understanding, mathematical reasoning, and OCR.
- Data construction: OCR, caption, and mathematical samples require solving or extracting image content and rendering answers onto specified media, while other categories use verification-based editing variations.These transformations simulate generation and editing workflows while enforcing precise text generation for OCR and mathematical reasoning.
- Training pipeline: Stage 1 fine-tunes BAGEL using only generation loss with VAE-feature dropout set to 1, encouraging reliance on ViT features for understanding.Only ViT features are used during image-understanding tasks.
- Training pipeline: Stage 2 optionally fine-tunes only BAGEL’s language-modeling head on 80k LLaVA-OV1.5 understanding samples to align it with the updated understanding branch.This addresses the domain gap created because the understanding head is not optimized during generation-loss-only tuning.
4 Experiment
Uni-Edit consistently improves image understanding, generation, and editing, with evidence across BAGEL, Janus-Pro, comparisons against AnyEdit, and visualizations of generalized reasoning. Ablations show that joint training, knowledge-rich editing data, discarded VAE inputs, and the original source resolution are important to this performance.
- Quantitative Results: Uni-Edit consistently improves understanding, generation, and editing on BAGEL, while MMMU and MathVista confirm stronger general knowledge and mathematical reasoning.The method avoids the comprehension trade-off observed when understanding parameters are unfrozen in approaches such as RecA.
- Visualization: Visualizations show improved prompt understanding, knowledge reasoning, spatial perception, composition, aesthetics, instruction following, logic, and spatial reasoning, including unseen Rubik’s Cube and knowledge-reasoning samples.These unseen examples are presented as evidence that intelligent editing induces generalized reasoning capabilities.
- Architecture beyond BAGEL: Uni-Edit also boosts all three tasks after tuning Janus-Pro, demonstrating generality beyond the BAGEL architecture.
- Compared with Existing Editing Dataset: Compared with BAGEL fine-tuned on AnyEdit, Uni-Edit shows that intelligent instructions grounded in general knowledge are necessary for universal unified-model tuning.Standard editing data is described as insufficient for this purpose.
- Ablation Studies: Discarding VAE inputs, jointly training the understanding branch, and retaining the original 224 × 518 source resolution produce better unified-tuning behavior than the tested alternatives.Retaining VAE inputs degrades all three tasks; freezing the understanding branch offers almost no generation or editing improvement; changing resolution reduces performance.
- The function of each type of data: Knowledge editing data improves multidisciplinary understanding benchmarks and knowledge-related generation benchmarks, whereas attribute data improves generation and editing but reduces understanding.The category comparison uses 6,000 samples per editing category, and the final data ratios are adjusted for optimal overall performance.
5 Conclusion and Limitations
Uni-Edit frames intelligent editing as a general UMM-tuning task and introduces Uni-Edit-148k to bridge the semantic gap in existing editing data. Its effectiveness is partly limited by base models’ capabilities, particularly limited text rendering.
- Conclusion: Uni-Edit-148k transforms diverse understanding queries into complex editing instructions, bridging the semantic gap that limits existing editing data.The dataset is described as high quality and supports intelligent editing as a general task for UMM tuning.
- Conclusion: Training with Uni-Edit on BAGEL simultaneously improves understanding, generation, and editing through mutual reinforcement.The supplied passage reports this result for experiments on the BAGEL framework.
- Limitations: Uni-Edit-148k data are partially bottlenecked by base-model capabilities, limiting their effectiveness during tuning.Caption, Math, and OCR data are not fully utilized in BAGEL and Janus-Pro because of limited text-rendering performance.
A System Prompt
The system prompts classify visual questions into task categories and generate editing instructions that require solving the question before modifying or augmenting the image. They enforce relevance, prevent answer leakage, and require aesthetically improved outputs in structured JSON.
- Task Type Classification: The classification prompt analyzes the original question, answer, and image path, then outputs structured JSON with a task category and processed answer.The category taxonomy includes visual question types such as shape, with further category-specific processing rules.
- QA-to-Edit Instruction: The editing prompt requires models to understand visual content and solve the question before performing image generation while improving aesthetic quality.This objective is repeated across attribute-bool, attribute-generation, and count instruction templates.
- QA-to-Edit Instruction: Instructions must remain relevant, avoid directly revealing answers, derive answers from the original image, and end with a visual-quality improvement directive.The prompt also requires direct JSON output without Markdown formatting.
- Category Guidelines: Category-specific rules use conditional branches for shape, color, and count edits, or synthesize semantically consistent objects while preserving the queried attributes.Examples include changing an attribute only when the derived answer matches, adding an object with the same queried attribute, and modifying object counts conditionally.
B Examples of Uni-Edit-148k
The examples of Uni-Edit-148k demonstrate intelligent editing instructions spanning visual question types, including shape, count, caption, location, color, OCR, and math. These prompts require object reasoning, region identification, text extraction, and geometric calculation.
- Visual reasoning examples: Shape and count examples ask models to infer an object’s shape and count same-colored objects.The prompts ask for a short shape description and the number of rubber balls matching a metallic block’s color.
- Grounding and extraction examples: Location and OCR examples require identifying a described region with coordinates and extracting text from a boxed webpage area.The location prompt requests [xmin, ymin, xmax, ymax], while the OCR prompt specifies extracting text within a red bounding box.
- Mathematical reasoning examples: Math examples require calculating the BAD angle of a parallelogram from its area and side lengths, rounded to two decimal places.The stated values are area 102 and side lengths AB = 23 and AD = 14.