Source-linked AI summary
Seedance 1.0: Exploring the Boundaries of Video Generation Models
Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, Xunsong Li, Yifu Li, Shanchuan Lin, Zhijie Lin, Jiawei Liu, Shu Liu, Xiaonan Nie, Zhiwu Qing, Yuxi Ren, Li Sun, Zhi Tian, Rui Wang, Sen Wang, Guoqiang Wei, Guohong Wu, Jie Wu, Ruiqi Xia, Fei Xiao, Xuefeng Xiao, Jiangqiao Yan, Ceyuan Yang, Jianchao Yang, Runkai Yang, Tao Yang, Yihang Yang, Zilyu Ye, Xuejiao Zeng, Yan Zeng, Heng Zhang, Yang Zhao, Xiaozheng Zheng, Peihao Zhu, Jiaxin Zou, Feilong Zuo
TL;DR
Video generation models still struggle to balance prompt following, motion plausibility, and visual quality. Seedance 1.0 addresses this with curated captioned data, an efficient unified architecture, post-training optimization, and acceleration, reporting strong generation capabilities and 41.4-second 1080p inference on NVIDIA-L20.
Problem
Current video generation foundation models face challenges in simultaneously balancing prompt following, motion plausibility, and visual fidelity.
Method
Seedance 1.0 combines multi-source captioned data, an efficient architecture for text-to-video and image-to-video, video-specific RLHF, and multi-stage inference acceleration.
Results
Seedance 1.0 demonstrates precise instruction following, coherent multi-shot generation, strong spatiotemporal quality, and fast inference, generating 5-second 1080p video in 41.4 seconds on NVIDIA-L20.
Takeaways & Limitations
The unified model delivers bilingual text-to-video and image-to-video generation with reported capabilities spanning complex scenarios, multi-shot generation, and long-range temporal coherence.
Abstract
from arXiv · showhide
Notable breakthroughs in diffusion modeling have propelled rapid improvements in video generation, yet current foundational model still face critical challenges in simultaneously balancing prompt following, motion plausibility, and visual quality. In this report, we introduce Seedance 1.0, a high-performance and inference-efficient video foundation generation model that integrates several core technical improvements: (i) multi-source data curation augmented with precision and meaningful video captioning, enabling comprehensive learning across diverse scenarios; (ii) an efficient architecture design with proposed training paradigm, which allows for natively supporting multi-shot generation and jointly learning of both text-to-video and image-to-video tasks. (iii) carefully-optimized post-training approaches leveraging fine-grained supervised fine-tuning, and video-specific RLHF with multi-dimensional reward mechanisms for comprehensive performance improvements; (iv) excellent model acceleration achieving ~10x inference speedup through multi-stage distillation strategies and system-level optimizations. Seedance 1.0 can generate a 5-second video at 1080p resolution only with 41.4 seconds (NVIDIA-L20). Compared to state-of-the-art video generation models, Seedance 1.0 stands out with high-quality and fast video generation having superior spatiotemporal fluidity with structural stability, precise instruction adherence in complex multi-subject contexts, native multi-shot narrative coherence with consistent subject representation.
1 Introduction
Seedance 1.0 addresses the challenge of balancing prompt following, motion plausibility, and visual quality through data, architecture, post-training, and acceleration improvements. It reports precise instruction following, coherent multi-shot generation, strong spatiotemporal quality, and fast inference.
- Seedance 1.0 targets the joint challenge of prompt following, motion plausibility, and visual fidelity in video generation.
- Multi-source curation and precision video captioning support learning across diverse scenarios, entities, actions, and complex narratives.
- Decoupled spatial-temporal layers and interleaved multimodal positional encoding jointly support text-to-video, image-to-video, and native multi-shot generation.
- Fine-grained supervised fine-tuning and video-specific RLHF with multiple reward models improve motion naturalness, structural coherence, and visual fidelity.
- Over 10× end-to-end speedup is achieved through multi-stage distillation and inference infrastructure optimization without performance degradation.
- 41.4 seconds produces a 5-second 1080p video on an NVIDIA-L20, while reported strengths include precise instruction following, multi-shot coherence, and state-of-the-art generation quality.
2 Model Design
Seedance 1.0 combines a temporally causal VAE, diffusion-transformer design, unified task formulation, prompt engineering, and cascaded high-resolution generation. These components connect compressed video representations and dense captions to efficient multi-task video synthesis.
- 2.1 Variational Autoencoder: The temporally causal VAE compresses RGB images and videos into continuous latent representations while supporting image input and output when temporal dimensions are zero.
- 2.2 Diffusion Transformer: The diffusion backbone concatenates VAE visual tokens with text-encoder tokens before processing them through transformer blocks.
- 2.2 Diffusion Transformer: Decoupled spatial and temporal layers allocate within-frame attention to spatial processing and across-frame attention to temporal processing, improving training and inference efficiency.
- 2.2 Diffusion Transformer: Multimodal positional encoding supports interleaved visual-textual sequences and multi-shot training with temporally ordered shots and individual captions.
- 2.3 Unified Task Formulation: A unified formulation mixes text-to-image, text-to-video, and image-to-video tasks by concatenating noisy inputs with clean or padded frames and using binary masks.
- 2.4 Prompt Engineering (PE): Prompt engineering uses supervised fine-tuning and reinforcement learning to convert user prompts into dense video-caption formats aligned with DiT training.
3 Data
The data system transforms heterogeneous raw videos into diverse, safe, high-quality training data through staged sourcing, curation, captioning, and infrastructure support. Captioning emphasizes both dynamic actions and camera movements and static visual characteristics.
- 3.1 Data Pre-Processing: The processing workflow sources diverse content, curates raw videos into clips, and packs captions and VAE embeddings for training.
- 3.1 Data Pre-Processing: Diversity-oriented sourcing targets variation in duration, resolution, subjects, scenes, actions, genres, styles, camera motion, and cinematography.
- 3.1 Data Pre-Processing: Shot-aware segmentation limits clips to 12 seconds while preserving natural transitions and local narrative flow.
- 3.1 Data Pre-Processing: Overlay rectification, quality filtering, semantic deduplication, and distribution rebalancing remove noise and redundancy while improving dataset coverage.
- 3.2 Video Captioning: Dense captions describe dynamic actions and camera movements alongside static properties of core characters and scenes.
- 3.3 Efficient Engineering Infrastructure: The engineering infrastructure combines a unified platform layer, heterogeneous CPU/GPU/NPU computation, and cloud resources for data processing.
4 Model Training
Seedance 1.0 uses staged training, curated data, efficient architecture, and multi-dimensional feedback learning to improve video quality, task coverage, and alignment.
- Training Pipeline: The training pipeline proceeds through pre-training, continue training, supervised fine-tuning, and RLHF, with a refiner receiving its own training stages.Visualization results are reported for the different stages, which progressively improve outputs.
- Continue Training: Continue training raises the image-to-video ratio from 20% to 40% to strengthen image-to-video generation and multitask performance.The phase also refines the training dataset.
- Training Outcomes: The resulting model supports text-to-video and image-to-video generation with more natural, smoother videos and improved visual fidelity.These improvements are attributed to richer motion dynamics, diverse captions, and higher-aesthetic-quality training data.
- Data and SFT: Curated, category-balanced data and manually verified captions improve coverage of styles, motions, scenarios, aesthetics, and text controllability.Separate models trained on curated subsets are merged to combine their strengths while limiting overfitting.
- Feedback Learning: RLHF combines foundational, motion, and aesthetic reward models to optimize alignment, structural stability, motion quality, and visual fidelity.The method directly maximizes composite rewards and uses multi-round iterative learning between the diffusion model and reward models.
5 Inference Optimizations
Inference optimization reduces diffusion, decoding, communication, and serving costs through distillation and system-level techniques while preserving generation quality.
- Model Acceleration: Distillation reduces the number of function evaluations while preserving prompt alignment, motion quality, visual fidelity, and source-image consistency.The final distilled model achieves comparable results to the original model across four expert-evaluated dimensions.
- VAE Optimization: A thin VAE decoder provides a 2× speedup with no loss in end-to-end visual quality.The design narrows channel widths in decoder stages that dominate latency and retrains the decoder with a fixed encoder.
- Inference Infrastructure: Kernel fusion improves inference throughput by 15%.The improvement is cumulative across extensive fusion efforts in core model modules.
- System Optimization: Quantization, structured sparsity tuning, and hybrid parallelism balance efficiency, generation quality, memory use, and communication overhead.The customized parallel strategy reduces communication overhead to one quarter of the level observed in Ulysses.
- System Optimization: Adaptive asynchronous offloading enables deployment on memory-limited devices with a performance drop of less than 2%.The strategy addresses attention and large-model memory demands.
- Pipeline Optimization: Pipeline optimizations including kernel fusion, quantization, parallelization, continuous batching, and caching improve end-to-end inference efficiency.The passage describes these techniques as improving overall prompt-engineering throughput and accelerating long-video encoding.
6 Training Infrastructure
The training infrastructure combines scalable parallelism, memory management, workload balancing, and fault tolerance to support efficient long-context video training and post-training.
- Infrastructure Overview: The infrastructure integrates kernel fusion, hybrid parallelism, activation checkpointing, workload balancing, and fault tolerance for stable, high-throughput training.These components target hardware efficiency, scalability, and robustness across workloads and hardware scales.
- Kernel Optimization: Fused CUDA kernels reduce global memory traffic by over 90% for memory-bound operations.The system combines torch.compile with handcrafted kernels and stores intermediate results in registers or shared memory.
- Parallelism: Hybrid sharded data and sequence parallelism distribute long-context model computation across thousands of GPUs while reducing scaling degradation.HSDP provides memory-efficient weight sharding, while sequence parallelism shards tokens across GPUs.
- Memory Management: Multi-level activation checkpointing reduces GPU memory pressure with negligible recomputation overhead while overlapping transfers and computation.It prioritizes offloading tensors from operators with the highest recomputation cost.
- Workload Balance: Runtime-aware workload balancing distributes heterogeneous video workloads evenly across GPUs within each batch.The strategy addresses imbalance from different video lengths and resolutions using an additional all-to-all communication step.
- Post-Training Infrastructure: Dynamic memory management combines CPU offloading, recomputation, FSDP, and sequence parallelism for efficient multi-node post-training.These techniques address memory contention, complex training modes, and variable workload demands.
- Post-Training Infrastructure: The post-training optimizations ensure stable and efficient performance across multiple model components and diverse video workloads.The framework uses hooks for CPU offloading and localized static planning to mitigate memory fragmentation.
7 Model Performance
Seedance 1.0 is evaluated through public leaderboards and the SeedVideoBench-1.0 benchmark across text-to-video and image-to-video tasks. It achieves top public rankings and demonstrates strengths in instruction following, motion quality, multi-shot coherence, and style preservation.
- Artificial Analysis Arena: Seedance 1.0 tops both the text-to-video and image-to-video Artificial Analysis leaderboards.The evaluation section reports a substantial advantage over competing models, including more than 100 points over Veo 3 and Kling 2.0 in image-to-video.
- SeedVideoBench-1.0: SeedVideoBench-1.0 uses 300 prompts each for text-to-video and image-to-video, spanning diverse real-world application scenarios.The benchmark was designed from real-world user prompts and evaluated with criteria developed in collaboration with film director experts.
- SeedVideoBench-1.0: The benchmark taxonomy evaluates subjects, subject descriptions, actions, action descriptions, camera language, and aesthetic description.Its categories cover entity attributes and positioning, physical and creative motion, shot transitions, style consistency, composition, and lighting.
- Video Evaluation Metrics: Human evaluation uses motion quality and aesthetic dimensions to assess structural accuracy, motion plausibility, stability, vividness, and visual quality.The framework was developed with film directors and is intended to provide finer-grained assessment than public preference evaluations.
- Human Evaluation: Seedance 1.0 and Veo 3 show stronger prompt following in text-to-video, while Seedance 1.0 and Kling 2.1 perform strongly in image-to-video.The text-to-video comparison identifies prompt adherence as important for adoption, while the image-to-video comparison emphasizes preserving reference-image characters and backgrounds.
- Multi-Shot and Multi-Style Generation: Seedance 1.0 generates multiple consecutive shots with subject continuity and stylistic coherence, while supporting fine-grained styles and reference-image preservation.Reported examples include shot-reverse shot, cut-in, and cut-away sequences, alongside cinematic, animated, and fantasy styles.
8 Conclusion
The conclusion presents Seedance 1.0 as a bilingual foundation model that unifies text-to-video and image-to-video generation. It combines strong instruction following, motion stability, visual quality, complex-scenario handling, multi-shot generation, long-range temporal coherence, and fast photorealistic generation.
- Conclusion: Seedance 1.0 unifies text-to-video and image-to-video generation in a native bilingual foundation model.The conclusion attributes its performance to integrated improvements in dataset construction, architecture and training, post-training optimization, and inference acceleration.
- Conclusion: The model handles complex scenarios, multi-shot generation, and long-range temporal coherence while delivering fast, photorealistic videos.The conclusion also highlights instruction following, motion stability, and visual quality.
A Contributions and Acknowledgments
The contributions and acknowledgments section states that all Seedance contributors are listed alphabetically by last name.
- Acknowledgments: All contributors of Seedance are listed in alphabetical order by their last names.
Yu Gao
The supplied section lists contributor names, including Yu Gao and a longer alphabetical sequence of additional contributors.
- Yu Gao: The section lists Yu Gao, Tuyen Hoang, and Weilin Huang among the Seedance contributors.
- Yu Gao: Additional listed contributors include Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, and many others.