PaperlayerView exact revision on arXiv

Source-linked AI summary

EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning

Xuan Ju, Tianyu Wang, Yuqian Zhou, He Zhang, Qing Liu, Nanxuan Zhao, Zhifei Zhang, Yijun Li, Yuanhao Cai, Shaoteng Liu, Daniil Pakhomov, Zhe Lin, Soo Ye Kim, Qiang Xu

arXiv:2509.20360v3cs.CV

TL;DR

Video generation and editing remain fragmented by architectural limitations and scarce training data. EditVerse unifies text, image, and video editing and generation through interleaved token sequences, cross-modal training data, and a dedicated benchmark, achieving state-of-the-art performance with emergent cross-modal abilities.

  • Problem

    Video generation and editing remain fragmented because existing architectures and available instruction-based video-editing data are limited.

  • Method

    EditVerse represents text, images, and videos as a unified interleaved token sequence and combines curated video-editing data with image and video datasets for joint training.

  • Results

    EditVerse achieves state-of-the-art performance against open-source and commercial models while demonstrating emergent editing and generation abilities across modalities.

  • Takeaways & Limitations

    The findings support unified architecture and cross-modal learning as a way to mitigate video-data limitations and enable more general multimodal foundation models.

  • Takeaways & Limitations

    Full self-attention over long unified sequences creates substantial computational overhead, especially for high-resolution or long-duration videos.

Abstract

from arXiv · show

Recent advances in foundation models highlight a clear trend toward unification and scaling, showing emergent capabilities across diverse domains. While image generation and editing have rapidly transitioned from task-specific to unified frameworks, video generation and editing remain fragmented due to architectural limitations and data scarcity. In this work, we introduce EditVerse, a unified framework for image and video generation and editing within a single model. By representing all modalities, i.e., text, image, and video, as a unified token sequence, EditVerse leverages self-attention to achieve robust in-context learning, natural cross-modal knowledge transfer, and flexible handling of inputs and outputs with arbitrary resolutions and durations. To address the lack of video editing training data, we design a scalable data pipeline that curates 232K video editing samples and combines them with large-scale image and video datasets for joint training. Furthermore, we present EditVerseBench, the first benchmark for instruction-based video editing covering diverse tasks and resolutions. Extensive experiments and user studies demonstrate that EditVerse achieves state-of-the-art performance, surpassing existing open-source and commercial models, while exhibiting emergent editing and generation abilities across modalities.

1 INTRODUCTION

EditVerse addresses the limited unification of video generation and editing with a single multimodal framework, scalable video-editing data, and a new instruction-based benchmark.

  • EditVerse unifies image and video editing and generation in one model using full self-attention and cross-modal knowledge transfer.Text, images, and videos are represented as a unified sequence for in-context learning.
  • The framework represents text, image, and video as one sequence to support flexible inputs and outputs while improving editing across modalities.The design targets arbitrary combinations of modalities and natural transfer from images to videos.
  • A scalable pipeline curates 232K video-editing samples and combines them with image and video data for unified training.The pipeline generates candidate samples with task-specific models and filters for quality before training.
  • EditVerseBench evaluates instruction-based video editing across 20 categories, mixed orientations, and diverse prompts.The benchmark contains 100 videos, evenly split between horizontal and vertical formats, with two editing prompts per video.
  • Comprehensive automated evaluations and user studies report state-of-the-art performance against open-source and commercial methods, alongside emergent cross-modal abilities.The reported findings include knowledge transfer from image to video and emergent editing and generation capabilities.

2 RELATED WORK

Prior work has advanced instruction-based image editing through increasingly capable data pipelines and self-attention architectures, while video editing remains constrained by data quality, missing paired targets, and limited task flexibility.

  • Instruction-based image-editing datasets have progressed from weak automated annotation and small manual sets toward larger pipelines using specialized editing models.The related work contrasts early approaches such as InstructPix2Pix and MagicBrush with more advanced task-specific-model pipelines.
  • Video-editing datasets remain less mature because existing resources can have poor quality, lack paired edited videos, or provide insufficient instruction coverage.The cited examples include InsV2V, VIVID-10M, and Señórita-2M-related limitations.
  • Image-editing methods increasingly concatenate conditions sequentially and use self-attention to improve in-context learning over earlier latent or attention-map manipulation.This shift reflects movement from training-free editing toward data-driven, self-attention-based approaches.
  • Recent video methods support selected tasks with sequential conditioning and task-aware positional embeddings but still fall short of flexible instruction-based video editing.UNIC is described as supporting six editing tasks while retaining broader flexibility limitations.

3 METHOD

EditVerse tokenizes text and vision inputs into an interleaved sequence, adds multidimensional positional information, and trains a shared model to generate visual outputs through flow matching.

  • EditVerse concatenates tokenized text and vision inputs into an interleaved sequence processed by full self-attention.The architecture uses a four-dimensional rotary positional embedding to encode spatial, sequential, and temporal structure.
  • 3.1 INTERLEAVED TEXT AND VISION INPUT: A VAE compresses images and videos into latent representations, while a text encoder produces retained input-text tokens before projection into the shared model space.Vision latents are patchified into visual tokens, and text tokens are retained selectively to reduce computation.
  • 3.1 INTERLEAVED TEXT AND VISION INPUT: Projected text and vision tokens preserve their original interleaved order, with learned start- and end-of-vision markers identifying each visual segment.The unified sequence has length L and hidden dimension C after modality-specific linear projection.
  • Four RoPE dimensions encode global sequence position, temporal position, and spatial height and width to distinguish modalities and their locations.Separate rotary computations are applied to the sequential, temporal, and spatial dimensions.
  • 3.3 TRAINING AND INFERENCE PARADIGM: During training, one clean image or video segment is selected as the generation target and the model learns to predict its flow velocity with mean-squared error.The target segment is sampled from the concatenated sequence, while the remaining segments provide context.
  • 3.3 TRAINING AND INFERENCE PARADIGM: At inference, an ODE solver uses discrete timesteps to generate the target visual representation from an initial noisy state.The supplied passages describe iterative generation from X0 toward X1 using N timesteps.

4 DATA PIPELINE

EditVerse combines large-scale multimodal training data with a filtering pipeline to address scarce, low-quality video editing data. The resulting dataset includes substantially more image and video generation data than video editing data.

  • Training Data: EditVerse is trained on 1.9M image-generation, 3.9M video-generation, 6.0M image-editing, and 288K video-editing samples.The video-editing datasets have fewer samples and less diversity than the image-editing datasets.
  • Video Editing Data Pipeline: The video editing pipeline generates editing pairs from video inputs and filters candidates to improve sample quality.The pipeline includes object removal and addition, while VLM-based filtering scores instruction adherence, context preservation, temporal consistency, artifacts, and other quality criteria.
  • Dataset Statistics: Table 1 summarizes the training-dataset statistics used to construct the multimodal training mixture.The table is associated with the datasets described across the data pipeline section.
  • Unified Training Mix: The unified training mix combines open-source, internal, and EditVerse datasets, including Señorita-2M after extensive filtering for its low success rate.The data sources span image generation, video generation, image editing, and video editing.

5 EXPERIMENTS

Experiments evaluate EditVerse on a diverse instruction-based benchmark, compare it with open-source and commercial methods, and analyze emergent abilities and design choices. Results indicate strong editing faithfulness, cross-modal emergent behavior, and dependence on both image and video generation data.

  • Benchmark: EditVerseBench contains 200 editing pairs across 20 categories and both horizontal and vertical orientations.The benchmark is designed for diverse instruction-based video-editing evaluation.
  • Comparison on EditVerseBench: EditVerse surpasses Runway Aleph in editing faithfulness, although it trails in generation quality because of base-model differences.Table 2 compares EditVerse with training-free, propagation, instruction-guided, and commercial methods on EditVerseBench.
  • User Study: User studies collected 3,000 pairwise ratings and found state-of-the-art performance across edited-region faithfulness, unedited-region preservation, and overall video quality.The user-study results align more closely with the VLM evaluation metric than with other automatic evaluations.
  • Emergent Ability: EditVerse performs editing tasks absent from training data and can handle multiple tasks by combining capabilities such as customization and inpainting.Examples include material changes, weather changes, added effects, and reference insertion.
  • Emergent Ability: EditVerse surpasses ground-truth training data in quality and success rate by leveraging knowledge from image editing and video generation.The paper illustrates this behavior with object-removal and object-change examples.
  • Ablation: Removing either image-generation/editing data or video-generation data reduces video-editing quality.Image data improves instruction understanding and edit diversity, while video data improves temporal consistency and motion modeling.
  • Architecture Ablation: The interleaved formulation and sequential RoPE each substantially influence text alignment and editing quality.The ablations remove the interleaved formulation or sequential dimension RoPE to measure their effects.

6 CONCLUSION

EditVerse unifies text, image, and video generation and editing through an interleaved token sequence and full self-attention. The paper reports state-of-the-art performance, cross-modal knowledge transfer, and emergent editing abilities supported by a scalable data pipeline and diverse benchmark.

  • Framework: EditVerse represents text, images, and videos as one interleaved token sequence for full self-attention, in-context learning, flexible resolutions and durations, and image-to-video knowledge transfer.The architecture is designed to address architectural and video-data limitations in universal video generation and editing.
  • Results: The paper reports state-of-the-art performance and emergent abilities across image and video generation and editing.The conclusion attributes these findings to unified architecture and cross-modal learning within the reported scope.
  • Resources: The work contributes a high-quality video-editing data pipeline and EditVerseBench, which covers diverse editing tasks.The benchmark and pipeline target limitations in available video-editing data and evaluation.

A.1 IMAGE AND VIDEO COPYRIGHTS

The paper identifies the external sources used for videos and comparison images in its figures. These include Pixabay, Adobe Stock, and ImgEdit-Bench materials.

  • Copyright Sources: Figure materials include videos from Pixabay and Adobe Stock, plus comparison images from ImgEdit-Bench.Adobe Stock videos are described as officially licensed.

A.2 EVALUATION DETAILS

EditVerse is evaluated with automated metrics spanning editing quality, video quality, text alignment, and temporal consistency, alongside human pairwise judgments.

  • Six automated metrics cover four aspects: VLM-based overall editing quality, video quality, text alignment, and temporal consistency.GPT-4o scores prompt following, edit quality, and background consistency from sampled source and edited frames.
  • GPT-4o evaluates three uniformly sampled source–edited frame pairs, scoring prompt following, edit quality, and background consistency from 0 to 3.The video-level VLM score averages the summed scores across the sampled frames.
  • Human evaluation uses pairwise comparisons of edited videos, with three independent judgments per pair to reduce noise in aggregated preferences.Participants view videos labeled “Result 1” and “Result 2” through a web-based interface.
  • Figure 1 showcases supported image and video editing tasks through a selection of example instructions.The examples include transformations such as object insertion, removal, style changes, camera changes, and effects.

A.3 ADDITIONAL EXPERIMENTS

Additional experiments show competitive or state-of-the-art performance across image editing, video generation, image generation, and video editing benchmarks.

  • EditVerse achieves highly competitive image-editing performance on ImgEdit-Bench, surpassing a wide range of existing approaches.The quantitative comparison is reported in Table 6.
  • The additional-experiment tables compare EditVerse with diverse open-source and commercial baselines across the evaluated modalities.A footnote identifies results using LLM-rewritten prompts.
  • EditVerse achieves competitive video-generation performance on VBench despite its relatively small model size, matching or surpassing several larger systems.Table 7 compares total parameter counts and benchmark performance with open-source and commercial text-to-video models.
  • EditVerse achieves state-of-the-art image-generation performance on GenEval against open-source and commercial systems.The result highlights semantically aligned generation across visual reasoning and compositional fidelity.
  • On V2VBench, EditVerse achieves the best or competitive results across most metrics, despite no square video editing samples in its training data.All V2VBench videos are square, whereas the training data contain no square video editing examples.

A.4 DETAILED TRAINING DATA

The training data are organized by image and video generation and editing tasks, with dataset statistics and mixture ratios reported for the unified training corpus.

  • Table 10 reports dataset sample counts, training ratios, and descriptions for image editing, image generation, video editing, and video generation.The overview covers data quality, coverage, and characteristics across the training mixture.
  • The training corpus combines high-quality open-source datasets, internal datasets, and EditVerse datasets for unified training.The table summarizes the composition and role of each dataset in the mixture.
  • Table 9 organizes video-editing comparison methods into network and training paradigm, attention feature injection, and diffusion latent manipulation categories.Local-best and global-best results are visually distinguished in the table.

A.5 LIMITATION AND FUTURE WORK

EditVerse’s limitations concern noisy video-editing data, computational cost from long token sequences, possible specialist-model advantages, image-editing gaps, and observed generation failures.

  • The authors identify unified image and video generation and editing as a significant step while acknowledging several limitations for future research.The limitation discussion covers data quality, model specialization, failures, efficiency, and image editing.
  • Observed Failure Cases: Observed failures include artifacts, flickering, low motion, logical flaws, incorrect editing positions, and blurred edited regions.Figure 9 illustrates incorrect object placement and blurry artifacts within edited regions.
  • Computational Cost: Full self-attention over long unified token sequences causes high computational overhead, especially for high-resolution or long-duration videos.The authors report raw, unoptimized memory usage and inference time and propose more efficient attention as future work.
  • Computational Cost: GPU memory usage grows strongly linearly with token length, while inference time increases systematically as sequences become longer.The efficiency analysis scales sequence length from 17,652 to 110,822 tokens and reports predictable resource growth.
  • Image Editing Performance: EditVerse does not currently achieve state-of-the-art image-editing performance, leaving room for improved data mixing or image-only fine-tuning.The model nevertheless performs on par with many image-editing models.
  • Dataset Quality: Automated video-editing pair generation has an estimated 65% success rate, introducing imperfect or failed edits into training data.Instructions average around 10 words and may lack detail for complex or nuanced edits.
  • Generalist vs. Specialist Models: A dedicated specialist model might outperform the unified model on specific tasks with abundant high-quality data, such as inpainting.This remains a stated possibility rather than a demonstrated failure across tasks.
Loading 2509.20360v3…