Source-linked AI summary
LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing
Jianzong Wu, Hao Lian, Jiongfan Yang, Dachao Hao, Ye Tian, Yunhai Tong, Jingyuan Zhu, Biaolong Chen, Qiaosong Qi, Aixi Zhang, Wanggui He, Mushui Liu, Jinlong Liu, Pipei Huang, Hao Jiang
TL;DR
Existing unified video models are often large and computationally expensive because editing concatenates source and target video tokens. LoomVideo introduces a 5B-parameter multimodal architecture with zero-overhead Scale-and-Add conditioning, achieving state-of-the-art or competitive benchmark performance and at least 5.41× faster inference than similar concatenation-based models.
Problem
Existing unified video frameworks typically use at least 13B parameters and concatenate source-video tokens, doubling sequence length and quadrupling self-attention computation.
Method
LoomVideo replaces the text encoder with an MLLM, aligns its features through Deepstack injection, and uses Scale-and-Add conditioning for unified video generation and editing.
Results
At least 5.41× acceleration accompanies state-of-the-art or competitive performance across comprehensive benchmarks, with particularly strong e-commerce and fashion results.
Takeaways & Limitations
LoomVideo demonstrates that a compact unified model can support multimodal video generation and editing with substantially lower inference overhead.
Takeaways & Limitations
The 5B model produces visual artifacts in highly dynamic scenarios and struggles with background reconstruction because its training data lacks sufficient real-world diversity.
Abstract
from arXiv · showhide
Developing unified video generation and editing models capable of interpreting interleaved multimodal inputs is a promising yet challenging frontier field. Existing unified frameworks predominantly rely on massive models (typically 13B parameters or more) and incorporate source video conditions for editing by concatenating sequence tokens. This concatenation inevitably doubles the sequence length, quadrupling the computational complexity of the self-attention mechanism and introducing prohibitive overhead. To address these bottlenecks, we present LoomVideo, a highly efficient 5B-parameter unified architecture for both video generation and editing. LoomVideo replaces the standard text encoder with a Multimodal Large Language Model (MLLM) and employs Deepstack injection mechanism to align multi-layer MLLM features with the Diffusion Transformer (DiT). Crucially, we introduce a zero-overhead Scale-and-Add conditioning approach for video editing. By scaling and directly adding the clean source video latent to the noised target latent, this elegant design eliminates the need for token concatenation, drastically reducing computational cost while maintaining robust capabilities for complex, non-rigid edits. Furthermore, a Negative Temporal RoPE strategy is seamlessly integrated to handle multiple reference images. Extensive experiments demonstrate that our compact 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, exhibiting exceptional superiority in e-commerce and fashion generation scenarios. Benefiting from the zero-overhead conditioning mechanism, LoomVideo achieves at least a 5.41x acceleration in inference speed compared to models of similar capabilities, paving the way for highly practical and efficient video foundation models.
1 Introduction
LoomVideo is a unified 5B-parameter architecture for controllable video generation and editing with interleaved multimodal inputs. It combines MLLM-based control, deep feature injection, and zero-overhead conditioning to improve efficiency while maintaining strong benchmark performance.
- Architecture: Qwen3-VL features from every layer are injected into corresponding DiT layers through cross-attention for deeper multimodal semantic alignment.This replaces the standard T5 text encoder and aligns multimodal controls with the generative process.
- Efficient conditioning: Scale-and-Add conditioning avoids appending source tokens by scaling and directly adding the clean source video latent during editing.The approach targets the efficiency bottleneck caused by token concatenation in video editing tasks.
- Architecture: LoomVideo provides a unified 5B-parameter architecture for visual-language multimodal control across video generation and editing.It is built on the 5B-parameter Wan 2.2 TI2V model and replaces its T5 text encoder with the Qwen3-VL MLLM.
- Results: LoomVideo achieves state-of-the-art or on-par performance across comprehensive benchmarks, with particularly strong reference-image-guided editing and controllable generation for e-commerce and fashion.These results highlight the model’s effectiveness in practical product-focused scenarios.
- Efficiency: 5.41× acceleration in inference speed is achieved versus existing concatenation-based unified models with similar capabilities.The acceleration benefits from LoomVideo’s compact parameter scale and zero-overhead Scale-and-Add conditioning mechanism.
2 Related Work
Video generation has evolved from diffusion-based text-to-video models toward multimodal conditioning for precise generation and editing control. LoomVideo contributes an MLLM-based architecture with Deepstack injection and zero-overhead Scale-and-Add conditioning for complex non-rigid edits.
- Video Generation and Editing: Diffusion models have reshaped text-to-video generation, enabling influential foundational models including CogVideoX, Wan, and the HunyuanVideo series.The related work highlights these models’ popularity, power, and extensive open-source community support.
- Multimodal Input for Video Generation: Recent multimodal video frameworks move beyond pure text-to-video generation by exploring interleaved multimodal inputs for more precise generation and editing control.Representative systems include VINO, VACE, OmniWeaving, UniVideo, and Omni-Video.
- LoomVideo Architecture: LoomVideo processes interleaved multimodal inputs with an MLLM and injects every-layer MLLM features into corresponding DiT layers through cross-attention.This Deepstack mechanism aligns multimodal features with the diffusion transformer.
- LoomVideo Architecture: LoomVideo scales the clean source video latent by the current timestep and directly adds it to the noised target latent for zero-overhead video-editing conditioning.The approach supports complex non-rigid edits such as changing human actions or camera directions.
3 Method
LoomVideo unifies multimodal video generation and editing by replacing T5 with an MLLM, aligning hierarchical features through Deepstack injection, and conditioning edits with Scale-and-Add. Negative temporal RoPE separates reference images from target frames, while staged training progressively develops alignment, reconstruction, editing, and multi-task control.
- Architecture: LoomVideo replaces the T5 text encoder with an MLLM and injects hidden states from multiple MLLM layers into corresponding DiT layers for deep semantic alignment.An MLP projects MLLM hidden states into condition features used as keys and values in cross-attention, with parameters shared across layers.
- Video Editing Conditioning: Scale-and-Add directly combines the clean source-video latent with the noised target latent, avoiding token concatenation and its doubled sequence length.The timestep t serves as the scaling factor, strengthening source guidance early in denoising while preserving purer target latents later.
- Reference-Image Conditioning: Negative temporal RoPE assigns target frames positive indices and reference images negative indices to distinguish them and provide multi-image guidance.Reference images receive indices −τ, −2τ, …, −N_refτ, while target frames use 0, 1, 2, …, T.
- Training Strategy: Training proceeds through three stages: MLLM alignment, 480p reconstruction and editing, and multi-task fine-tuning emphasizing reference-guided and multi-reference tasks.The first stage discards T5 after observing that the model otherwise relied on T5 embeddings and ignored MLLM conditionings.
- Post-training: Post-training applies DiffusionNFT with PickScore rewards to optimize human aesthetic preferences and perceptual quality.The reinforcement-learning data is curated from an in-house dataset.
4 Experiments
Experiments evaluate LoomVideo across general video generation, instruction- and reference-guided editing, multimodal conditioning, and a held-out e-commerce benchmark. The 5B model achieves leading or competitive quality while delivering substantial inference acceleration.
- Benchmarks: LoomVideo is evaluated on VBench, OpenVE-Bench, RefVIE-Bench, IntelligentVBench, and the held-out FashionVideoBench.FashionVideoBench contains six core tasks and 300 evaluation samples.
- General Generation: On VBench, fully trained LoomVideo surpasses Wan 2.2 in average score and leads Imaging Quality and Overall Consistency.The result supports replacing T5 with Qwen3-VL for hierarchical multimodal alignment.
- Instruction-Based Editing: On OpenVE-Bench, the Stage 2 model achieves the highest overall score, while the final model remains competitive with larger-parameter baselines.The model particularly excels on Creative Edit, indicating strong semantic alignment and editing-intent execution.
- Reference-Guided Editing: 7%: LoomVideo surpasses second-place VINO on RefVIE-Bench and achieves the highest overall score among open-source baselines.This benchmark evaluates instruction-and-reference-guided editing with a single subject or background reference image.
- Multimodal and E-Commerce Evaluation: 8%: LoomVideo leads TIV2V on IntelligentVBench over second-best OmniWeaving, while achieving comparable performance on compositional MI2V.FashionVideoBench results show the highest overall scores across all six core sub-tasks against open-source counterparts.
- Efficiency: 6.24× and 5.41×: LoomVideo accelerates T2V generation and video editing, respectively, compared with the fastest baseline, OmniWeaving.Token-concatenation models such as UniVideo and VINO require approximately four times longer to edit one video.
5 Conclusion and Future Work
LoomVideo is presented as an efficient unified framework for diverse video generation and editing with interleaved multimodal inputs, enabled by MLLM-based semantic understanding and Deepstack alignment. Future work will scale the model and training pipeline toward higher-definition, longer-duration video generation.
- Conclusion: LoomVideo handles diverse video generation and editing tasks through interleaved multimodal inputs.The framework is described as highly efficient and unified.
- Conclusion: Replacing the standard text encoder with an MLLM provides rich, hierarchical semantic understanding.The MLLM is paired with Deepstack injection to leverage multimodal information.
- Conclusion: Deepstack injection aligns multimodal controls deeply with the spatiotemporal generative prior.This mechanism is proposed to fully leverage the MLLM’s multimodal representations.
- Future Work: Future work will scale the diffusion transformer and extend multi-resolution training for 720p or 1080p and longer-duration video generation.The stated goal is to improve physical world simulation and visual fidelity.
6 Contributions and Acknowledgments
The paper lists contributors by affiliation, marking equal contributors, corresponding authors, and the project leader with dedicated symbols. It also acknowledges MSALab at Peking University and Alibaba Group for project support and discussions.
- Contributions and Acknowledgments: Author notation defines ∗ for equal contribution, § for corresponding authors, and † for the project leader.
- Contributions and Acknowledgments: Peking University contributors are Jianzong Wu, Hao Lian, Jiongfan Yang, Dachao Hao, Ye Tian, and Yunhai Tong.Jianzong Wu is marked as equal-contributing and project leader; Hao Lian is marked as equal-contributing; Yunhai Tong is a corresponding author.
- Contributions and Acknowledgments: Alibaba Group contributors are Jingyuan Zhu, Biaolong Chen, Qiaosong Qi, Aixi Zhang, Wanggui He, Mushui Liu, Jinlong Liu, Pipei Huang, and Hao Jiang.Hao Jiang is identified as a corresponding author.
- Contributions and Acknowledgments: The authors thank MSALab at Peking University and Alibaba Group for support and discussions throughout the project.
Appendix · A More Experimental Results
LoomVideo unifies diverse video-generation capabilities within one compact 5B-parameter model, while maintaining high efficiency. Qualitative results across multiple tasks support this unified and efficient design.
- A More Experimental Results: LoomVideo unifies diverse video-generation capabilities within a single model.Qualitative results are presented across figures 5–8.
- A More Experimental Results: The appendix reports qualitative results across various video-generation tasks.These results appear in fig. 5, fig. 6, fig. 7, and fig. 8.
- A More Experimental Results: The reported figures demonstrate LoomVideo’s versatility across diverse capabilities.The evidence is qualitative and spans four figures.
- A More Experimental Results: LoomVideo achieves this versatility while maintaining high efficiency.The passage attributes this efficiency to the model’s compact architecture.
- A More Experimental Results: The model uses a compact 5B parameter architecture.This compact architecture is identified as powering LoomVideo’s efficiency.
- A More Experimental Results: The qualitative appendix results jointly support LoomVideo as a unified and efficient video-generation model.This synthesis follows the passage’s claims about unified capabilities, versatility, compact size, and efficiency.
B Limitation
LoomVideo remains limited in highly dynamic scenarios, where its compact 5B-parameter scale constrains fine-grained temporal-dynamics learning and produces visible motion artifacts. Its training data is also primarily curated, indicating another area for future exploration.
- Model limitations: Highly dynamic scenarios produce distorted eyes and unnatural limb movements in LoomVideo outputs.The paper attributes these visual artifacts to the model’s limited ability to learn fine-grained temporal dynamics.
- Model limitations: The compact 5B-parameter model size constrains learning of fine-grained temporal dynamics.This limitation is identified as a cause of artifacts under highly dynamic conditions.
- Data limitations: The training data is primarily curated, leaving data curation as an area for future exploration.The supplied passage introduces this as a second limitation but does not specify its downstream effects.
C FashionVideoBench Evaluation Prompts
This section defines FashionVideoBench evaluation prompt templates for product, model-replacement, and freeform editing, plus reference-guided video editing and generation tasks. Each template evaluates performance on a 5-point scale across three key dimensions.
- Editing Evaluation Templates: FashionVideoBench provides dedicated evaluation prompt templates for product editing, model replacement editing, and freeform video editing.The product, model-replacement, and freeform templates are detailed in Figures 9, 10, and 11, respectively.
- Editing Evaluation Templates: Each editing template supplies an original video, edited video, and instruction, then rates performance on a 5-point scale across three dimensions.The instructions target product changes, model or person replacement, or general video edits, depending on the task.
- Reference-Guided Editing: Reference-guided evaluation covers video item replacement using a reference image, with scoring focused on matching the replaced item and maintaining consistency.The evaluator receives the original video, reference image, edited video, and editing instruction.
- Reference-Guided Generation: Reference-guided generation templates evaluate motion transfer and multi-reference-image-to-video generation using 5-point scoring across three dimensions.Motion-transfer evaluation combines source-video motion with reference-image appearance, while multi-reference evaluation composes visual elements from multiple images according to a text instruction.