PaperlayerView exact revision on arXiv

Source-linked AI summary

LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing

Jianzong Wu, Hao Lian, Jiongfan Yang, Dachao Hao, Ye Tian, Yunhai Tong, Jingyuan Zhu, Biaolong Chen, Qiaosong Qi, Aixi Zhang, Wanggui He, Mushui Liu, Jinlong Liu, Pipei Huang, Hao Jiang

arXiv:2606.06042v2cs.CV

TL;DR

Unified video generation and editing frameworks often rely on large models and token concatenation for source-video conditions. LoomVideo pairs a 5B multimodal architecture with Scale-and-Add editing conditioning, achieving state-of-the-art or on-par benchmark performance and at least 5.41× faster inference than similar-capability concatenation-based models.

  • Problem

    Existing unified video generation and editing frameworks typically use 13B+ models and concatenate source-video tokens, doubling sequence length and quadrupling self-attention complexity.

    Source: §1 · p. 1

  • Method

    LoomVideo is a 5B unified generation-and-editing model that uses MLLM Deepstack alignment and zero-token Scale-and-Add conditioning for source-video editing.

    Source: §1 · p. 1, §3.1 · p. 5

  • Results

    LoomVideo achieves state-of-the-art or on-par performance across comprehensive benchmarks and at least 5.41× faster inference than similar-capability concatenation-based unified models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    LoomVideo offers a compact, practical unified video model, with particularly strong performance in e-commerce and fashion generation scenarios.

    Source: §1 · p. 3, §5 · p. 10

    Under highly dynamic scenes, LoomVideo shows distorted eyes and unnatural limb movements, while e-commerce-focused training data limits background reconstruction on general MI2V benchmarks.

    Source: B Limitation · p. 15

Abstract

from arXiv · show

Developing unified video generation and editing models capable of interpreting interleaved multimodal inputs is a promising yet challenging frontier field. Existing unified frameworks predominantly rely on massive models (typically 13B parameters or more) and incorporate source video conditions for editing by concatenating sequence tokens. This concatenation inevitably doubles the sequence length, quadrupling the computational complexity of the self-attention mechanism and introducing prohibitive overhead. To address these bottlenecks, we present LoomVideo, a highly efficient 5B-parameter unified architecture for both video generation and editing. LoomVideo replaces the standard text encoder with a Multimodal Large Language Model (MLLM) and employs Deepstack injection mechanism to align multi-layer MLLM features with the Diffusion Transformer (DiT). Crucially, we introduce a zero-overhead Scale-and-Add conditioning approach for video editing. By scaling and directly adding the clean source video latent to the noised target latent, this elegant design eliminates the need for token concatenation, drastically reducing computational cost while maintaining robust capabilities for complex, non-rigid edits. Furthermore, a Negative Temporal RoPE strategy is seamlessly integrated to handle multiple reference images. Extensive experiments demonstrate that our compact 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, exhibiting exceptional superiority in e-commerce and fashion generation scenarios. Benefiting from the zero-overhead conditioning mechanism, LoomVideo achieves at least a 5.41x acceleration in inference speed compared to models of similar capabilities, paving the way for highly practical and efficient video foundation models.

1 Introduction

LoomVideo unifies multimodal video generation and editing in a compact architecture designed to reduce the resource and attention costs of existing approaches.

  • Existing unified video frameworks typically use 13B-or-larger models and concatenate source-video tokens for editing, doubling sequence length and quadrupling self-attention complexity.The paper motivates unified models by the demand for controllable generation and editing from interleaved video, image, and fine-grained instruction inputs.Source: §1 · p. 1
  • LoomVideo builds on the 5B Wan 2.2 TI2V model Wan et al. (2025), replacing T5 with Qwen3-VL Bai et al. (2025b) and injecting features from every MLLM layer into corresponding DiT layers.Deepstack injection uses cross-attention to strengthen semantic alignment between multimodal controls and generation.Source: §1 · pp. 1, 3
  • Scale-and-Add scales the clean source-video latent by the current timestep and adds it to the noised target latent, avoiding extra tokens while supporting complex non-rigid edits.Negative Temporal RoPE distinguishes multiple reference images from video frames without disrupting spatiotemporal dynamics.Source: §1 · p. 3
  • A three-stage training strategy progresses from low-resolution semantic alignment to higher-resolution generation and editing, then diverse references and advanced instructions; reinforcement-learning post-training improves instruction following and fidelity.The paper says this enables the 5B model to perform tasks previously requiring 13B-scale models.Source: §1 · p. 3
  • LoomVideo achieves state-of-the-art or on-par performance across comprehensive benchmarks, with particular strength in reference-image-guided editing and controllable generation for e-commerce and fashion.Source: §1 · p. 3
  • At least 5.41× acceleration in inference speed comes from LoomVideo’s compact scale and zero-overhead Scale-and-Add conditioning versus similar-capability concatenation-based unified models.Source: §1 · p. 3

2 Related Work

The section situates LoomVideo amid diffusion-based video generation and instruction-guided editing, then contrasts its compact architecture and efficient multimodal conditioning with prior approaches.

Video Generation and Editing

  • Most unified editing models use 13B or more parameters, while LoomVideo pairs a 5B Diffusion Transformer with an 8B VLM for efficient multi-task unification, especially in e-commerce and fashion.The section also traces progress from diffusion-based text-to-video models to instruction- and reference-guided editing, which advanced beyond early zero-shot methods and rigid structural preservation.Source: §2 · p. 3

Multimodal Input for Video Generation

  • Prior multimodal video methods concatenate source-video and target tokens, increasing sequence length and creating computational bottlenecks.The passage names UniVideo (Wei et al. (2025)) and OmniWeaving (Pan et al. (2026)) in connection with this approach.Source: §2 · p. 3
  • LoomVideo injects embeddings from every VLM layer into corresponding DiT layers via cross-attention for deep semantic alignment.Figure 2 depicts this Deepstack injection alongside the model’s other conditioning designs.Source: §2 · p. 3, Figure 2 · p. 4
  • For editing, LoomVideo scales the clean source-video latent and adds it to the noised target latent, adding zero tokens while supporting complex, non-rigid edits.Examples include changing human actions or camera directions; Figure 2 specifies scaling the source latent by the current timestep.Source: §2 · pp. 3–4, Figure 2 · p. 4

3 Method

LoomVideo combines layer-wise MLLM conditioning, token-free source-video editing, and negative temporal indexing for reference images. Its training uses staged alignment, reconstruction, and multi-task fine-tuning over image, video, and editing data.

3.1 Architecture

  • Layer-wise Deepstack injection projects every MLLM layer’s hidden states through a shared MLP into corresponding DiT layers as cross-attention keys and values.This aligns fine-grained multimodal controls with the spatiotemporal generative process without heavy adapter networks.Source: §3.1 · p. 4
  • Scale-and-Add conditions editing by adding a timestep-scaled clean source latent to the noised target latent, avoiding extra tokens and achieving at least 5.41× faster inference than concatenation-based models.The source contribution grows with timestep, providing stronger guidance early in denoising and purer target latents later; the new patch embedding is zero-initialized.Source: §3.1 · pp. 4–5
  • Negative temporal RoPE assigns reference images indices −τ, −2τ, …, separating them from target frames while enabling prompt pointers to align each image with its description.The approach provides multi-image guidance with minimal extra tokens.Source: §3.1 · p. 5

3.2 Training Data

  • The training data combines text-to-image with approximately O(10M) internal image-text pairs and text-to-video with around O(10M) open-source video-text pairs, plus instruction-based and reference-guided image/video editing.Instruction-based image editing uses approximately O(10M) open-source samples and O(10M) internal samples focused on Taobao fashion; video editing uses O(3M) open-source and O(1M) internal samples. Reference-guided editing includes O(10M) internal image pairs and approximately O(0.5M) RefVIE video samples alongside O(1M) internal video samples; multi-reference prompts use pointers such as “The person in @Figure 1”.Source: §3.2 · p. 5

3.3 Training Pipeline

  • Training proceeds from 256p MLLM alignment (image:video sample ratio 4:1; batch size about 640) to 480p reconstruction and editing, then full multi-task fine-tuning biased toward challenging reference-guided tasks.Stage 2 adds reconstruction prompted by “reconstruct this image/video” as a foundation for editing.Source: §3.3 · pp. 5–6
  • The T5 encoder is discarded because later-stage training showed the model relied on T5 embeddings and largely ignored MLLM conditioning; post-training uses DiffusionNFT (Zheng et al., 2025) with PickScore (Kirstain et al., 2023).The RL reward optimizes human aesthetic preferences and perceptual quality.Source: §3.3 · p. 6

3.4 Implementation Details

  • LoomVideo initializes from Wan 2.2 TI2V 5B (Wan et al., 2025) and Qwen3-VL-8B-Instruct (Bai et al., 2025b), injecting the MLLM’s last 30 of 36 layers into the DiT’s 30 layers.The two-layer MLP projection adds RMSNorm before projection for stability.Source: §3.4 · p. 7
  • Dynamic 1D batching uses block-diagonal attention masks and token limits, while multi-resolution buckets support varied image and video sizes; timestep shifting uses Nbase = 64 and scale = 0.5.For a 480 × 832 × 97 video, the shift reaches approximately 4.5; EMA is maintained during Stages 2 and 3.Source: §3.4 · p. 7

4 Experiments

LoomVideo leads or matches strong baselines across several video-generation and editing benchmarks, performs strongly on held-out fashion tasks, and substantially accelerates inference. Its results include a slight final-model drop on pure instruction-based editing and comparable—but non-leading—performance on compositional MI2V.

4.1 Experimental Settings

  • Evaluation covers T2V, instruction-based editing, reference-guided editing, and multi-reference generation, with unified and task-specialized baselines.The settings use VBench, OpenVE-Bench, RefVIE-Bench, and IntelligentVBench; FashionVideoBench adds held-out e-commerce cases. Most compared state-of-the-art models have 13B or more parameters, versus LoomVideo's 5B.Source: §4.1 · pp. 7–8

4.2 Quantitative Comparison

  • On VBench, LoomVideo beats Wan 2.2 on average and leads Imaging Quality and Overall Consistency; Stage 2 leads OpenVE-Bench overall.The final model has a slight drop on pure instruction-based editing after integrating more diverse multimodal tasks, but remains competitive with larger-parameter baselines; it also excels on OpenVE-Bench's Creative Edit metric.Source: Table 2 · p. 8, Table 3 · p. 8, §4.2 · p. 8
  • LoomVideo leads open-source RefVIE-Bench by 7% and TIV2V by 8%, while matching SOTA UniVideo (query) on compositional MI2V.The RefVIE-Bench margin is over second-place VINO, and the TIV2V margin is over second-best OmniWeaving. The paper attributes MI2V's non-leading result partly to the 5B model's capacity and training-data domain shift.Source: §4.2 · p. 8, Table 4 · p. 9, Table 5 · p. 9
  • RL post-training improves results across all reported benchmarks except FashionVideoBench, where editing quality had limited room for further enhancement.The comparison is between the Stage 3 model and the final model.Source: §4.2 · p. 9

4.3 Qualitative Comparison

  • Qualitative comparisons show stronger instruction following and editing quality, more accurate reference restoration, and precise MI2V reference-image restoration with favorable video quality.The comparisons cover OpenVE-Bench editing, RefVIE-Bench reference-guided editing, and FashionVideoBench MI2V.Source: §4.3 · p. 9

4.4 Efficiency Comparison

  • 6.24× on T2V and 5.41× on editing are LoomVideo's speedups over fastest baseline OmniWeaving.For UniVideo and VINO, which use token concatenation, editing inference takes approximately four times their respective T2V generation time.Source: §4.4 · pp. 9–10

4.2 Quantitative Comparison

  • On FashionVideoBench, LoomVideo achieves the highest overall scores across all six held-out e-commerce sub-tasks.The benchmark contains 300 samples: 50 for each of six tasks, evaluated by Gemini 2.5 Pro for Subject Consistency, Prompt Following, and Video Quality.Source: §4.1 · p. 7, §4.2 · p. 9, Table 7 · p. 10

5 Conclusion and Future Work

LoomVideo combines multimodal video generation and editing with efficient source-video conditioning, achieving strong benchmark performance and identifying scale, resolution, and duration as future directions.

5 Conclusion and Future Work

  • The compact 5B-parameter LoomVideo achieves state-of-the-art or highly competitive performance across open-domain benchmarks, with over 5.4× inference speedup and particular strength in e-commerce and fashion.The paper also reports robust capability for complex, non-rigid edits.Source: §5 · p. 10
  • LoomVideo replaces the standard text encoder with an MLLM, uses Deepstack injection for deep multimodal alignment, and applies Scale-and-Add conditioning and Negative Temporal RoPE for editing and multi-reference guidance.Scale-and-Add bypasses token concatenation to reduce editing computation.Source: §5 · p. 10
  • Future work will scale the diffusion transformer and extend multi-resolution training to support 720p or 1080p and longer-duration video generation.The stated aim is to advance physical-world simulation and visual fidelity.Source: §5 · p. 10

6 Contributions and Acknowledgments

The authors are listed by affiliation, with symbols marking equal contribution, corresponding authorship, and project leadership; the paper also acknowledges project support and discussions.

Contributions and Acknowledgments

  • Peking University’s listed authors include project leader Jianzong Wu and corresponding author Yunhai Tong.The roster also includes equal contributor Hao Lian, Jiongfan Yang, Dachao Hao, and Ye Tian.Source: §6 · p. 10
  • Alibaba Group’s listed authors include corresponding author Hao Jiang.The roster also names Jingyuan Zhu, Biaolong Chen, Qiaosong Qi, Aixi Zhang, Wanggui He, Mushui Liu, Jinlong Liu, and Pipei Huang.Source: §6 · p. 10
  • The authors thank MSALab at Peking University and Alibaba Group for support and discussions throughout the project.Source: §6 · p. 10

Appendix · A More Experimental Results

Qualitative results across diverse tasks show that LoomVideo unifies video-generation capabilities in one compact 5B-parameter model while maintaining high efficiency.

A More Experimental Results

  • Across diverse tasks, LoomVideo unifies video-generation capabilities in one compact 5B-parameter model while maintaining high efficiency.The appendix points to qualitative results in Figures 5–8.Source: A More Experimental Results · p. 15

B Limitation

LoomVideo still shows visual artifacts in highly dynamic scenarios and struggles with general-scene background reconstruction because its e-commerce-focused training data lacks diversity; future work targets these limitations.

B Limitation

  • Distorted eyes and unnatural limb movements appear in highly dynamic scenarios, which the authors attribute to the 5B model’s limited capacity to learn fine-grained temporal dynamics.The authors propose enhancing model capacity to address this limitation.Source: B Limitation · p. 15
  • E-commerce-focused training data lacks sufficient real-world scene diversity, hindering accurate background reconstruction on general MI2V benchmarks such as IntelligentVBench.The authors plan to expand training-data coverage.Source: B Limitation · p. 15
  • The authors identify greater model capacity and broader training-data coverage as future-work directions for addressing these limitations.Source: B Limitation · p. 15

C FashionVideoBench Evaluation Prompts

FashionVideoBench provides task-specific evaluation prompt templates for three video-editing tasks and three reference-guided generation or editing tasks, using 5-point ratings across three dimensions.

Evaluation Prompt Template for Product Editing

Evaluation Prompt Template for Video Model Replacement Editing

Evaluation Prompt Template for Freeform Video Editing

Evaluation Prompt Template for Video Item Replacement with Reference Image

  • Reference-image item-replacement evaluation gives raters the original video, reference image, edited video, and instruction, with performance scored on three dimensions.The prompt specifies a 5-point scale.Source: C FashionVideoBench Evaluation Prompts · p. 23

Evaluation Prompt Template for Motion-Transfer Video Generation

  • Motion-transfer generation raters assess whether the generated video combines motion or pose from a source video with appearance and background from a reference image.The prompt uses a 5-point scale across three dimensions.Source: C FashionVideoBench Evaluation Prompts · p. 24

Evaluation Prompt Template for Multi-Reference-Image-to-Video Generation

  • Multi-reference-image generation raters judge whether the video reflects visual elements from multiple references and composes them according to a text instruction.The prompt uses a 5-point scale across three dimensions.Source: C FashionVideoBench Evaluation Prompts · p. 25

Related papers

Loading 2606.06042v2…