PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing

Xiaoyan Cong, Tianyu Wang, Hyeonho Jeong, Zhifei Zhang, Yuqian Zhou, Yijun Li, Shaoteng Liu, Zhe Lin, Srinath Sridhar, Soo Ye Kim

arXiv:2610.22744v1cs.CV

TL;DR

Video editing with intricate dynamic changes can be difficult to specify using text or image guidance alone, and multimodal multi-reference editing remains underdeveloped. OmniXRef unifies variable text, image, and video references in an editing framework, and its model outperforms task-specific baselines and commercial models on reported evaluations.

  • Problem

    Dynamic visual changes can be difficult to specify through text or image guidance alone, while systematic multimodal multi-reference video editing remains underdeveloped.

    Source: §1 · p. 2

  • Method

    OmniXRef organizes variable text, image, and video references in task-specific banks and uses an MLLM with a single-stream diffusion transformer to generate edited video.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    OmniXRef-Model outperforms task-specific baselines in their domains and shows better reference adherence and source preservation than strong commercial models.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    OmniXRef-Bench systematically evaluates reference adherence, visual quality, and source preservation, advancing video as a control modality for editing.

    Source: §1 · p. 3

    Camera-trajectory errors are normalized independently for each trajectory, so they capture direction, shape, and timing rather than absolute translation magnitude or focal-length changes.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Creative intent in video editing can be expressed through text instructions (e.g., "remove the people"), image references (e.g., subject appearance), video references (e.g., camera movement), or a combination of the above. We introduce OmniXRef, a unified framework for multimodal multi-reference video editing that allows users to combine variable numbers of text, image, and video references to control diverse attributes of a source video. We first devise a unified input design with a reference-bank interface and introduce $R^2$-RoPE to support generalization to more references. Then our model, OmniXRef-Model, uses a single-stream diffusion transformer to jointly process the source video and multimodal context from an MLLM in an interleaved sequence. To address the scarcity of training data for video-reference editing, we construct OmniXRef-Data using synthesis pipelines for different editing tasks and combinations of references. We use curriculum learning to progressively increase task complexity and resolution. To support systematic evaluation of video as a control modality, we introduce OmniXRef-Bench, a benchmark for video-reference attribute transfer across Camera, Motion, Content, Style, and Visual Effects (VFX). Comprehensive evaluations show that our OmniXRef-Model outperforms task-specific baselines and commercial models.

1 Introduction

OmniXRef addresses the underdeveloped setting of video editing with multiple text, image, and video references by unifying the framework, model, data, and evaluation; evaluations report advantages over task-specific baselines and commercial models.

1 Introduction

  • Intricate visual details and dynamic changes can be difficult to specify with text or image guidance alone, while multimodal multi-reference editing remains largely underdeveloped.Multiple references can contribute distinct aspects such as object motion, visual effects, and camera movement.Source: §1 · p. 2
  • OmniXRef formulates editing across reference number, modality, attribute, and composition, organizing inputs in reference banks that associate references with editing tasks and instructions.The framework supports text, image, and video references across Camera, Motion, Content, Style, and VFX attributes.Source: §1 · p. 2
  • An MLLM derives semantic context from an interleaved input, which a single-stream diffusion transformer jointly processes with source, reference, and target-video tokens.Mix-Patchify adapts patchification granularity to each reference's editing task, while R2-RoPE reserves positional slots and randomizes bank assignments during training to support more inference references than seen in training.Source: §1 · p. 2
  • OmniXRef-Model outperforms task-specific baselines in their respective domains and shows better reference adherence and source preservation than strong commercial models.The commercial comparisons include Seedance 2.5 and MiniMax H3.Source: §1 · pp. 2–3
  • ∼10M training examples combine synthesized single-video-reference and compositional editing data with text-only and text-image data.OmniXRef-Bench comprises 125 cases across Camera, Motion, Content, Style, and VFX, evaluating edit adherence, source preservation, visual quality, and temporal consistency.Source: §1 · pp. 2–3

2 Related Work

Prior systems and benchmarks cover several reference-guided editing settings, but systematic evaluation of dynamic attributes transferred from video while preserving source content remains lacking.

Video Generation and Editing

Video Editing Datasets and Benchmarks

  • Existing benchmarks cover text, image, and multimodal composition, but none systematically evaluates dynamic-attribute transfer from reference videos while preserving remaining source content.The passage names ReCo-Bench, VEFX-Bench, EditVerseBench, RefVIE-Bench, IntelligentVBench, OmniVideo-1.0, and TIDE-Bench as existing evaluations.Source: §2 · p. 3
  • OmniXRef-Data addresses scarce diverse video-reference triplets with synthesis pipelines, and OmniXRef-Bench evaluates reference adherence, visual quality, and source preservation.The data pipelines cover single-video-reference tasks and compositions of two visual references.Source: §2 · p. 3

3 OmniXRef-Model

OmniXRef-Model combines reference banks with MLLM-derived semantic context and a single-stream diffusion transformer to edit videos from multimodal inputs. Mix-Patchify reduces reference-token costs, R²-RoPE supports more references at inference, and training progressively increases task complexity and resolution.

3.1 Framework Overview

  • The model uses an MLLM to encode interleaved inputs, then a single-stream DiT jointly processes contextual text, visual tokens, and noise to generate the edited video.The DiT uses full self-attention, and reference banks provide a common interface for text, image, and video references.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Mix-Patchify

  • Mix-Patchify assigns references task-suited spatial granularity while retaining fine temporal resolution, reducing token count by up to p_t · p_h · p_w.Source and target tokens retain fine-grained patchification to preserve detailed source conditioning.Source: §3.2 · p. 5

3.3 R2-RoPE

  • R²-RoPE reserves reference slots and randomly samples them during training, keeping additional inference-time references within the RoPE range seen during training.The method reserves eight reference-bank slots in practice.Source: §3.3 · p. 5

3.4 Curriculum Learning

  • Training progresses from single-reference adaptation to two-visual-reference composition, then to fine-tuning at 480P.The first two stages train at 272P; the high-resolution stage retains the Stage 2 task mixture.Source: §3.4 · p. 5

4 Data

OmniXRef-Data combines text-only editing data with synthesized visual-reference and compositional examples across editing tasks. Its approximately 10M examples cover text-only, single visual-reference, and two-visual-reference settings.

Dataset scale and coverage

  • OmniXRef-Data contains approximately 10M examples spanning text-only, one-visual-reference, and two-visual-reference editing settings.The two-reference settings combine image and video references or two video references.Source: §4 · p. 5

Text-only editing

  • Text-only supervision contributes approximately 7M source–target pairs and is retained during training to maintain editing foundations and reduce modality-specific overfitting.The pairs contain diverse instructions and support instruction following and source-video preservation.Source: §4 · p. 6

Visual-reference data

  • Visual-reference triplets are designed so source and target differ mainly in the edited attribute, while reference and target share that attribute but depict distinct content.This design discourages copying reference content instead of transferring its intended attribute.Source: §4 · p. 6

VFX and Style

  • VFX and Style data are synthesized by removing an attribute from target-video content to form the source, then generating a different-scene reference with the same attribute.The pipeline uses first-frame editing and video propagation for the source, and generated first frames and videos for the reference.Source: §4 · p. 6

Camera

  • Camera-reference triplets use two visually distinct videos reshot under the same sampled camera path.Vista4D produces the reshot videos used to form the triplets.Source: §4 · p. 6

Motion, Content, and composition

  • Motion data pair different characters performing the same motion, while Content data includes image-reference edits and video-reference background harmonization.Compositional data are made by applying the Camera reshooting pipeline to Content and Motion pipelines, then filtering the results with an MLLM.Source: §4 · p. 6

5 Experiments

OmniXRef-Model is evaluated on video-reference attribute transfer, camera trajectories, reference conditioning, and inference cost. It leads overall on OmniXRef-Bench, while analyses examine portability, design choices, reference robustness, and the cost of conditioning on more references.

5.2 Quantitative Comparison

  • OmniXRef-Model achieves the highest overall Score, RA, and SP on OmniXRef-Bench, leads Motion, Content, and Style, and beats task-specific baselines in their categories.Table 2 reports per-category metrics and five-category averages; the remaining VQ gap relative to commercial systems may reflect pretrained backbone capacity.Source: Table 2 · p. 7, §5.2 · p. 7
  • Fine-tuned OmniXRef (MiniMax-H3) improves on MiniMax-H3 across all five categories and surpasses OmniXRef-Model on Camera and VFX.It uses the Ref2VA pretrained checkpoint without architecture changes or task-specific components, supporting portability of the formulation and practical value of OmniXRef-Data.Source: §5.2 · p. 8

5.3 Qualitative Comparison

  • In Figure 3, OmniXRef-Model transfers the reference rain streaks and droplets while preserving the window and avoiding reference leakage seen with Seedance 2.5.The comparison spans VFX, Style, Content, and Camera editing.Source: §5.3 · p. 8, Figure 3 · p. 8

5.4 Camera Trajectory Evaluation

  • OmniXRef-Model has the lowest RotErr, TransErr, and CamMC among evaluated open-source and commercial baselines on camera trajectories.Trajectories are compared relative to their first frames over 32 aligned frames; normalized translations do not measure absolute translation magnitude or focal-length changes.Source: §5.4 · pp. 8–9, Table 3 · p. 9

5.5 Ablation Studies

  • Mix-Patchify assigns fine patches to Content and coarser patches to Camera, Motion, Style, and VFX, providing a spatial prior that helps the model converge faster.Sequential asset indexing instead of R2-RoPE creates conflicting positional signals as reference counts vary, while training without curriculum learning hinders optimization and slows each 480P step.Source: §5.5 · p. 9

5.6 Analysis of Reference Conditioning

  • With the source and instruction fixed, changing the visual reference changes the transferred sunset lighting; swapping reference-bank slots still transfers camera motion and replaces the table.Randomized slot assignment exposes every reserved slot to diverse references during training, although each sample contains at most two visual references.Source: §5.6 · p. 10
  • With three video references for motion, camera, and style—more than the two-reference training maximum—OmniXRef-Model follows all three while preserving the source subject and scene.Seedance 2.5 largely fails to follow the motion and camera references and only partially reproduces the requested style.Source: §5.6 · p. 10

5.7 Inference Latency and Computational Cost

  • More references and finer patches increase inference latency and computational cost, while Mix-Patchify reduces both costs with a larger advantage as reference count grows.The measurements use DiT inference on one NVIDIA A100 GPU with 480P source and reference videos of 121 frames.Source: §5.7 · p. 11

6 Conclusion

OmniXRef unifies variable numbers of text, image, and video references for controlling source-video attributes. Its model outperforms task-specific baselines and commercial systems on OmniXRef-Bench.

6 Conclusion

  • OmniXRef-Model outperforms task-specific baselines and commercial systems on OmniXRef-Bench.Analyses also find that it uses each reference's visual content, is robust to reference ordering, and generalizes to compositions beyond the training distribution.Source: §6 · p. 11
  • The framework organizes variable numbers of text, image, and video references through a common reference-bank interface to control distinct source-video attributes.The model integrates an MLLM with a single-stream DiT, uses Mix-Patchify for multiple video references, and adopts R2-RoPE to generalize beyond training reference counts.Source: §6 · p. 11
  • OmniXRef-Data uses synthesis pipelines for five video-reference editing categories and two-reference compositions, while OmniXRef-Bench evaluates reference adherence, visual quality, and source preservation.The five categories are Camera, Motion, Content, Style, and VFX.Source: §6 · p. 11

A Interleaved Sequence Specification

The sequence specification treats each reference bank as a separate edit clause alongside the source video. Fixed markers distinguish the source from generated output, while bank instructions and optional visual references define each edit.

A.1 System Prompt

  • The system prompt directs each edit bank to use only its own reference and preserve source content that the bank does not change.It describes the input as a source video followed by one or more banks, each with a task definition, instruction, and optional image or video reference.Source: §A.1 · p. 15

A.2 Source and Output Markers

  • Tsrc precedes the source video, while Tout precedes target tokens and marks where the edited video is generated.The source marker identifies the video whose unedited content must be preserved.Source: §A.2 · p. 15

A.3 Reference Banks

  • Each reference bank combines a category-defining task, a free-form edit instruction, and an optional image or video reference.The task definition restricts which attribute to extract independently of the instruction, while the visual reference supplies that attribute's content; concatenated banks cover text-only, visual-reference, and composed edits.Source: §A.3 · p. 15

B.1 Training Objective

Training noises target-video latents and optimizes the DiT to predict their velocity. Mix-Patchify adapts reference-token granularity to the editing task to reduce computation while retaining fine temporal resolution.

B.1 Training Objective

  • The DiT is trained with flow matching to predict the velocity of noisy target-video latents at the target positions in the interleaved sequence.The clean target latent is mixed with Gaussian noise according to noise level t.Source: §B.1 · p. 16

B.2 Mix-Patchify

  • Each latent patch is flattened and mapped to a DiT token by a patch-size-specific MLP.Coarser reference patches are initialized by averaging pretrained fine-grained MLP outputs before unconstrained optimization.Source: §B.2 · p. 16
  • Mix-Patchify reduces reference-token count by up to the patch volume while retaining fine temporal resolution and reported performance.Content references use (1, 1, 1) patches; Camera, Motion, Style, and VFX use spatially coarser (1, 2, 2) patches.Source: §B.2 · p. 16

C.1 Category-Specific Evaluation

OmniXRef-Bench separately evaluates whether video-reference edits transfer the requested attribute, look convincing, and preserve unchanged source information. Its deterministic scoring rules cap failures that simple averages could overlook.

C.1 Category-Specific Evaluation

  • The benchmark evaluates video-reference editing across Camera, Motion, Content, Style, and VFX using Reference Adherence, Visual Quality, and Source Preservation criteria.Source Preservation includes whether unwanted reference content was transferred.Source: §C.1 · pp. 16–17

C.2 Deterministic Scoring

  • The composite score weights Reference Adherence at 0.45, Visual Quality at 0.25, and Source Preservation at 0.30.The scoring procedure applies category-specific caps per example before averaging; bounding Visual Quality by Reference Adherence limits high scores for attractive outputs that ignore the edit.Source: §C.2 · p. 17, Table 6 · p. 17

C.3 Camera Trajectory Errors

Camera-trajectory errors are computed from independently estimated trajectories after first-frame normalization. These measures capture motion direction, trajectory shape, and timing, but not absolute translation magnitude or focal-length changes such as zooms.

C.3 Camera Trajectory Errors

  • Each trajectory is expressed relative to its own first frame, with translations normalized by maximum displacement from the first camera.The reference and output poses are estimated independently.Source: §C.3 · p. 17
  • Using each trajectory’s own first-frame coordinates removes arbitrary reconstruction world frames; no rigid alignment or temporal warping is applied.Source: §C.3 · p. 17
  • The first frame contributes no error, so per-frame errors are summed over the remaining N − 1 = 31 frames, following CamI2V (Zheng et al., 2024).Source: §C.3 · p. 18
  • Independent trajectory normalization and un compared intrinsics mean the errors capture motion direction, trajectory shape, and timing, but not absolute translation magnitude or zoom.Source: §C.3 · p. 18

D Instruction-Based Video Editing Benchmarks

On VEFX-Bench, OmniXRef-Model outperforms open-source baselines and Wan 2.6, and performs on par with Grok Imagine and Seedance 2.0. The section also reports evaluations on ReCo-Bench across four instruction-based editing tasks.

D Instruction-Based Video Editing Benchmarks

E Limitations

This section describes structured evaluation protocols for reference adherence, source preservation, and visual quality across camera, motion, content, style, and VFX edits, using category-specific criteria and concrete observations.

F VLM Judge Prompts

  • The judge prompts specify what may change, what must be preserved, and permitted side effects; they return closed-set labels and locatable evidence, from which scores are computed deterministically.Reference-adherence and source-preservation calls are independent, use time-aligned frame grids, and share no context.Source: F VLM Judge Prompts · p. 20

F.1 Camera

  • Camera adherence compares reference and output camera movement only, while preservation allows viewpoint-driven changes but checks that source content remains and reference content does not leak in.The camera prompt compares direction, extent, timing, and framing; changes in viewpoint, crop, perspective, parallax, scale, and visibility are allowed for preservation.Source: §F.1 · pp. 20–21

F.2 Motion

  • Motion adherence separates matching the reference’s pose from preserving the source subject’s appearance, while preservation checks for lost source content or unintended reference content.The transferred pose is not a preservation failure, but changes to the source subject’s identity or appearance are, with identity changes treated as especially important.Source: §F.2 · pp. 22–23

F.3 Content

  • Content adherence judges only the requested reference element, treating its resemblance to the reference as success while checking its match, integration, extra imports, and localization.The prompt evaluates properties such as identity, appearance, and extent, with integration judged against the output scene; it also flags changes spilling beyond the intended region.Source: §F.3 · p. 24
  • Content source-preservation checks whether the output retains the source video apart from the requested transfer; importing any other reference content is a failure.The requested attribute’s arrival from the reference is correct, not a “from_reference” error.Source: §F.3 · p. 25

F.4 Style

  • Style adherence compares palette, lighting, and texture across different scenes, while a separate structure check flags distortion caused by over-stylization.For source preservation, style changes are allowed, but objects, shapes, positions, and setting should remain; judge structure apart from color and lighting.Source: §F.4 · pp. 26–27

F.5 VFX

  • VFX adherence compares appearance, color, dynamics, and integration, and labels whether the effect is correctly localized, leaks, or is misplaced, as well as its visibility over time.Duration categories are absent, brief, intermittent, or sustained; a brief effect can still be correct.Source: §F.5 · pp. 27–28
  • VFX source preservation permits the transferred effect and its physical consequences, such as cast light, reflections, shadows, and occlusion, while treating other source changes or imported reference content as failures.Source: §F.5 · p. 28
  • VFX judgments classify each attribute’s output status, explain the overall preservation verdict, and record certainty alongside concrete observations.Attribute verdicts distinguish preserved, side-effect, altered, from-reference, and absent; overall status is preserved, partly changed, or a different video.Source: §F.5 · pp. 28–29

F.6 Visual Quality

  • Visual quality is assessed from six output frames without source or reference, separating intrinsic quality from reference similarity.The shared prompt requests concrete, locatable defects and forbids an overall rating or numeric score.Source: §F.6 · p. 29
  • Defects are recorded by damaged property, concrete location, and obviousness, while an empty list is valid when none are found.Properties are appeal, plausibility, and coherence; obviousness ranges from subtle to obvious, and the summary names the worst listed defect or none.Source: §F.6 · p. 29

Related papers

Loading 2610.22744v1…