PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructX: Towards Unified Visual Editing with MLLM Guidance

Chong Mou, Qichao Sun, Yanze Wu, Pengze Zhang, Xinghui Li, Fulong Ye, Songtao Zhao, Qian He

arXiv:2510.08485v1cs.CV

TL;DR

InstructX addresses the unresolved integration of MLLMs and diffusion models for instruction-driven image and video editing, particularly when video data is scarce. It studies this integration and builds a unified model with modality-specific guidance and mixed image-video training. The resulting system handles diverse editing tasks, achieves state-of-the-art performance, and gains video-editing capabilities from image training data.

  • Problem

    Integrating MLLMs with diffusion models for visual editing remains insufficiently studied, while scarce high-quality video data constrains video-editing development.

  • Method

    InstructX unifies image and video editing by connecting MLLM guidance to a diffusion transformer and using modality-specific queries with mixed image-video training.

  • Results

    Extensive experiments show state-of-the-art performance across diverse image and video editing tasks.

  • Takeaways & Limitations

    Image training can induce zero-shot video-editing capabilities, helping broaden task coverage despite limited video supervision.

  • Takeaways & Limitations

    InstructX is constrained by its pre-trained video DiT for high-resolution editing above 1080P, and image training is only a temporary response to video-data shortages.

Abstract

from arXiv · show

With recent advances in Multimodal Large Language Models (MLLMs) showing strong visual understanding and reasoning, interest is growing in using them to improve the editing performance of diffusion models. Despite rapid progress, most studies lack an in-depth analysis of MLLM design choices. Moreover, the integration of MLLMs and diffusion models remains an open challenge in some difficult tasks, such as video editing. In this paper, we present InstructX, a unified framework for image and video editing. Specifically, we conduct a comprehensive study on integrating MLLMs and diffusion models for instruction-driven editing across diverse tasks. Building on this study, we analyze the cooperation and distinction between images and videos in unified modeling. (1) We show that training on image data can lead to emergent video editing capabilities without explicit supervision, thereby alleviating the constraints imposed by scarce video training data. (2) By incorporating modality-specific MLLM features, our approach effectively unifies image and video editing tasks within a single model. Extensive experiments demonstrate that our method can handle a broad range of image and video editing tasks and achieves state-of-the-art performance.

1 INTRODUCTION

InstructX addresses the open challenge of integrating MLLMs with diffusion models for visual editing, especially under scarce video data. It unifies image and video editing and reports state-of-the-art performance across diverse tasks.

  • Scarce high-quality video data limits video-editing quality, generalizability, and dataset construction efficiency.
  • InstructX unifies image and video editing in a single framework while studying how MLLMs and diffusion models should be integrated.
  • Image training data extends zero-shot video editing to a wider range of tasks despite limited video supervision.
  • Extensive experiments show state-of-the-art performance across diverse image and video editing tasks.

2 RELATED WORK

Prior work combines multimodal understanding and visual generation through autoregressive, hybrid, and connector-based designs. InstructX studies unresolved architectural choices for connecting MLLMs to diffusion models in visual editing.

  • 2.1 INSTRUCTIONAL IMAGE AND VIDEO EDITING: Existing video-editing methods face limited quality, content drift, costly data construction, or restricted task coverage.
  • 2.2 UNIFIED UNDERSTANDING AND GENERATION MODELS: Connector-based designs bridge understanding and diffusion models, but the roles of hidden-state compression, learnable queries, connector size, and MLLM freezing remain open questions.

3 METHOD

InstructX uses an MLLM to extract instruction-relevant guidance for a diffusion transformer and trains the unified model through staged alignment, mixed image-video learning, and quality refinement. Image data induces additional video-editing capabilities, while modality-specific queries distinguish image and video information.

  • 3.1 OVERVIEW: The MLLM embeds instructions and visual inputs, while learnable queries and an MLP connector transfer editing guidance to the DiT decoder.
  • 3.2 ARCHITECTURE CHOICE: The architecture study compares diffusion-only encoding, QFormer compression, MetaQuery bridging, and alternative MLLM–DiT integration choices.
  • 3.3 TRAINING STRATEGIES: Training proceeds through feature alignment, full-data training, and quality fine-tuning to align representation spaces, improve fidelity, and remove low-quality-texture artifacts.
  • 3.3 TRAINING STRATEGIES: Mixed image-video training enables unified modeling and induces video style-transfer capabilities for tasks absent from the video data.

4 EXPERIMENT

Experiments evaluate InstructX on image and video benchmarks against open-source and closed-source systems, with ablations examining modality-specific queries and MLLM video inputs. The method achieves competitive or superior results across the reported tasks.

  • 4.3 COMPARSION RESULT: InstructX outperforms current open-source video-editing models on most metrics and remains competitive with state-of-the-art closed-source solutions.It attains the highest average scores on Style/Tone/Weather among the reported video results.
  • 4.3 COMPARSION RESULT: On ImgEdit-Bench, InstructX outperforms other open-source methods in overall score while remaining competitive across multiple image-editing subtasks.
  • 4.4 ABLATION STUDY: Separating image and video queries improves VIE-Bench performance by distinguishing modality-specific feature extraction.
  • 4.4 ABLATION STUDY: Using only the first video frame for MLLM guidance can cause collapse when edited content appears mid-video.

5 CONCLUSION

InstructX unifies image and video editing through MLLM–diffusion integration, mixed image-video training, and modality-specific queries. It achieves broad editing coverage and strong performance, but high-resolution editing remains constrained by the pre-trained video DiT.

  • 5 CONCLUSION: InstructX unifies image and video editing in one framework using mixed image-video training and separate modality-specific queries.The framework combines Learnable Query, MLLM LoRA, and an MLP Connector, while mixed training expands video-editing coverage.
  • 5 CONCLUSION: InstructX outperforms recent open-source methods and matches some closed-source video editors while supporting a broader range of tasks.
  • 5 CONCLUSION: High-resolution editing above 1080P remains difficult because InstructX is constrained by its pre-trained video DiT.

A.1 VIE-BENCHMARK DETAILS

VIE-Bench is a high-quality, instruction-based benchmark designed to address the scarcity of public video-editing benchmarks. It uses diverse 720P videos with manually curated editing instructions.

  • A.1 VIE-BENCHMARK DETAILS: VIE-Bench contains high-quality instruction-based video-editing examples drawn from public datasets and web videos.
  • A.1 VIE-BENCHMARK DETAILS: The benchmark covers 720P videos lasting 3–10 seconds across indoor, outdoor, dynamic, animated, and portrait scenes.
  • A.1 VIE-BENCHMARK DETAILS: GPT-4o generates five editing instructions per video, followed by careful manual curation.

A.2 VIDEO SYNTHESIS PAIRED DATA PIPELINE

The paired video-data pipeline synthesizes add, reference-based add, remove, swap, and reference-based swap tasks from segmented source videos. It combines automated subject analysis, segmentation, erasure, instruction generation, and reference-image creation.

  • A.2 VIDEO SYNTHESIS PAIRED DATA PIPELINE: The pipeline covers five video-editing tasks: add, reference-based add, remove, swap, and reference-based swap.
  • A.2 VIDEO SYNTHESIS PAIRED DATA PIPELINE: Source videos are partitioned into single-scene clips with PySceneDetect, which become the original videos for synthesis.
  • A.2 VIDEO SYNTHESIS PAIRED DATA PIPELINE: Figure 10 depicts the pipeline for synthesizing paired video data.
  • A.2 VIDEO SYNTHESIS PAIRED DATA PIPELINE: GPT-4o identifies target subject categories, while Grounding DINO and SAM produce masks for subsequent video erasure.
  • A.2 VIDEO SYNTHESIS PAIRED DATA PIPELINE: The original and erased videos are supplied to GPT-4o to generate remove and add instructions, while Flux-Kontext creates cross-pair reference images.

A.3 USER STUDY

The user study evaluates image- and video-editing systems through ratings from 30 professional creators. It samples 120 editing pairs and compares methods across three dimensions.

  • A.3 USER STUDY: Thirty professional image and video creators evaluated the editing methods as user-study experts.
  • A.3 USER STUDY: The study sampled 30 pairs from GEdit-Bench, 30 from ImgEdit-Bench, and 60 non-reference video-editing samples from VIE-Bench.
  • A.3 USER STUDY: Participants rated 8 image-editing methods and 4 instruction-based video-editing methods on three dimensions.

A.4 FURTHER DISCUSSION ON THE GAINS OF MLLM

The paper attributes gains in instructional editing to MLLM-based visual understanding, especially for fine-grained image details. Comparisons and user-study figures examine these gains across image and video editing settings.

  • Diffusion-only editing misses complex, tiny details such as books on a corner shelf and plants in a corner, whereas MLLM understands them better.
  • The appendix includes user-study results for both image editing and video editing.
  • MLLM+Diffusion shows a noticeable performance gap over diffusion-only instruction-based editing across tasks on ImgEdit-Bench.The comparison quantifies the advantage of combining multimodal understanding with diffusion for instructional image editing.
  • The paper uses GPT-4o as an MLLM-based judge for video-editing and reference-based video-editing tasks.The scoring prompts for these evaluations are shown in Figures 16 and 17.
  • Additional visual comparisons cover VIE-Bench, image editing, and further video-editing examples.
Loading 2510.08485v1…