Seedance 1.0: Exploring the Boundaries of Video Generation Models

Multi-Source Data with Comprehensive Video Captioning

• A multi-source, multi-stage curation and balancing process builds a diverse high-quality video dataset, while precision captioning supports accurate instruction interpretation and complex narratives.Source: §1 · p. 3

Efficient Architecture Design

• Decoupled spatial and temporal layers with interleaved multimodal positional encoding enable joint text-to-video and image-to-video learning and native multi-shot generation; window attention improves efficiency.Source: §1 · p. 3

Enhanced Post-Training Optimization

• Small-set supervised fine-tuning followed by video-tailored RLHF with multiple reward models improves motion naturalness, structural coherence, and visual fidelity for text-to-video and image-to-video.Source: §1 · p. 3

Inference Acceleration

• Multi-stage distillation and inference infrastructure optimization achieve over 10× end-to-end speedup with no degradation in model performance.Source: §1 · p. 3

Comprehensive Generation Capabilities

• Seedance 1.0 delivers spatiotemporal coherence, structural stability, fluid and physically plausible motion, photorealistic visuals, and state-of-the-art performance across proprietary and third-party evaluations.The introduction also highlights nuanced textures and compositional richness.Source: §1 · p. 3

Precision Instruction Following

• The model precisely follows complex specifications, handling multi-agent interactions, adaptive camera control, and stylistic variation while maintaining narrative continuity.Source: §1 · p. 3

Multi-Shot Narrative Capability

• Seedance 1.0 natively generates coherent multi-shot stories with stable view transitions and consistent subject representation across temporal-spatial transformations.Source: §1 · p. 3

Ultra-Fast Generation Experience

• 41.4 seconds on NVIDIA-L20 generates a 5-second video at 1080p, which the paper describes as substantially faster than other commercial counterparts.Source: §1 · p. 3

2.1 Variational Autoencoder

• The VAE uses temporally causal convolutions to compress images and videos into continuous spatiotemporal latents, and handles image input and output when T = T′ = 0.The chosen downsampling ratios are (rt, rh, rw) = (4, 16, 16), with C = 48; patchification is removed on the DiT side.Source: §2.1 · p. 4
• The VAE is trained with L1 reconstruction, KL, LPIPS perceptual, and adversarial losses, using a hybrid PatchGAN-like discriminator for appearance and motion.Adversarial supervision targets local textures and detailed structures in the reconstruction.Source: §2.1 · p. 4

2.2 Diffusion Transformer

• A fine-tuned decoder-only LLM supplies text tokens, while decoupled spatial and temporal transformer layers process within-frame and across-frame attention, respectively.Text and visual tokens are concatenated; text interacts cross-modally in spatial layers, while temporal layers use within-frame windows for a global temporal receptive field. MMDiT uses separate modality weights in spatial layers and normalizes Q and K embeddings before attention.Source: §2.2 · p. 4
• 3D MM-RoPE adds 1D positional encoding for text to 3D visual RoPE, supports interleaved visual-text sequences, and extends to temporally ordered multi-shot training with a detailed caption per shot.The unified task formulation concatenates noisy inputs with cleaned or zero-padded frames and uses binary masks to identify instruction frames; training mixes text-to-image, text-to-video, and image-to-video tasks.Source: §2.2 · pp. 4–5

2.3 Diffusion Refiner

• The diffusion refiner upsamples base-model 480p videos to 720p or 1080p and enhances visual details and textures.Initialized from the pretrained base model, it is conditioned on low-resolution video upsampled and concatenated with diffusion noise.Source: §2.3 · p. 5

2.4 Prompt Engineering (PE)

• Prompt engineering initializes from Qwen2.5-14B and uses supervised fine-tuning followed by reinforcement learning to rewrite user prompts as dense video captions.SFT uses manually annotated prompt-caption pairs with separate image-to-video and text-to-video prompt styles; DPO on correct and incorrect rewrites uses LoRA fine-tuning of the SFT model to address hallucinations. The resulting model produces precise, high-quality caption-format rewrites consistent with DiT training.Source: §2.4 · pp. 5–6

3 Data

• The data section presents an automated, high-throughput workflow for converting heterogeneous raw video collections into refined, high-quality, diverse, and safe training data.The broader training corpus also includes image data, prepared using methods similar to Seedream [8].Source: §3 · p. 6

3.1 Data Pre-Processing

• The automated preprocessing pipeline curates diverse public and licensed video, segments long footage into clips of at most 12 seconds, and removes visual defects and unsafe content.It detects shot boundaries, rectifies overlays through adaptive cropping, and filters for visual quality and safety.Source: §3.1 · pp. 6–7
• Semantic deduplication retains the highest-quality clip in each near-duplicate cluster, while distribution rebalancing downsamples overrepresented categories and boosts underrepresented ones.Rebalancing analyzes subject, scene, action, genre, style, duration, resolution, and motion attributes, with targeted acquisition for tail categories.Source: §3.1 · p. 7

3.2 Video Captioning

• Dense bilingual video captions describe changing actions and camera movements alongside stable subject or scene traits, supporting instruction-following and prompt rephrasing aligned with training captions.The caption model is trained on manually annotated data using Tarsier2 [32], with its visual encoder frozen and language model fully fine-tuned.Source: §3.2 · p. 7

3.3 Efficient Engineering Infrastructure

• The three-layer processing infrastructure combines workflow automation, heterogeneous computation, and cloud resources to support large-scale data processing.Customized BMF and Ray [19] dynamically assign operations to suitable hardware, use asynchronous communication, and provide auto-scaling and retries for near-linear scalability and high throughput.Source: §3.3 · pp. 7–8, Figure 4 · p. 8

4.1 Pre-Training

• Pre-training uses flow matching with velocity prediction and progressively introduces video at higher resolution and frame rate, while a resolution-aware timestep shift adds more noise to higher-resolution, longer videos.Training starts with 256px text-to-image, then joint 256px image-video training, 640px video training, and finally 24 fps video; text-to-image is retained and image-to-video makes up 20%.Source: §4.1 · p. 9

4.2 Continue Training (CT)

• Continued training raises the image-to-video data share from 20% to 40% and, with richer motion data and diverse captions, improves naturalness, smoothness, and text-to-video visual fidelity.It uses higher-aesthetic-quality data selected with aesthetic and optical-flow motion evaluators, plus long captions and motion-only short captions; the final model supports both text-to-video and image-to-video.Source: §4.2 · p. 9

4.3 Supervised Fine-Tuning (SFT)

• SFT uses manually verified captions and a category-balanced, human-curated video dataset, then merges models trained on curated subsets to improve visual fidelity and motion quality.The dataset spans several hundred categories; lower learning rates, limited GPUs, and early stopping help preserve text controllability and prevent overfitting.Source: §4.3 · p. 9

4.4.1 Feedback Data Infrastructure

• Feedback data combines balanced, filtered prompts and human preference labels on high-quality video pairs, with multidimensional comparisons designed to improve reward-model capacity and generalization.Pairs include synthetic videos from different model stages; annotators select best and worst videos for a given dimension while ensuring the best is not inferior on other dimensions.Source: §4.4.1 · p. 11

4.4.2 Reward Model

• Three specialized reward models target foundational capabilities, motion, and aesthetics, with video-tailored RLHF strategies supporting improvements across model capabilities.The foundational model uses a vision-language architecture for image-text alignment and structural stability; the motion model targets artifacts, amplitude, and vividness; the aesthetic model evaluates video keyframes in image space, inspired by Seedream [6] [8].Source: §4.4.2 · p. 11

4.4.3 Base Model Feedback Learning

• Seedance 1.0 simulates video inference during training, predicts clean video x0 when the reward model adequately assesses quality, and maximizes composite rewards from multiple RMs, reported most efficient and effective versus DPO/PPO/GRPO.It performs multi-round iterative learning between the diffusion model and reward models, which is reported as more stable and controllable than dynamically updating the reward model.Source: §4.4.3 · p. 12

4.4.4 Super-Resolution RLHF Framework

• RLHF on the accelerated diffusion refiner maximizes a linear combination of reward signals, improving motion quality and visual fidelity in low-NFE scenarios while maintaining computational efficiency.Low-resolution VAE latents condition super-resolution, and multiple reward models evaluate the generated high-resolution videos.Source: §4.4.4 · p. 12

5.1 Model Acceleration

• TSCD provides 4x acceleration, while Score Distillation aligns student and teacher noise; human-preference adversarial training mitigates artifacts and improves perceptual realism.TSCD segments the denoising trajectory to reduce function evaluations; Score Distillation supports trajectory-level optimization at low NFEs.Source: §5.1 · p. 12
• The final distilled model achieves results comparable to the original across prompt alignment, motion quality, visual fidelity, and consistency with the source image.Expert evaluations covered all four dimensions.Source: §5.1 · p. 12
• Narrowing channels in pixel-proximal VAE decoder stages yields a 2× speedup with no loss in end-to-end video visual quality.The thin decoder is retrained with a fixed pre-trained encoder.Source: §5.1 · p. 12

5.2 Inference Infrastructure

• Kernel fusion across core modules improves model inference throughput by 15%.Source: §5.2 · p. 12
• Building on Seedream [8], fine-grained mixed-precision quantization targets Attention and Gemm; AdaSpa [27]-based streamlined tuning minimizes search-stage overhead, integrates optimized Attention Quantization, and balances performance and efficiency.The work also addresses the impact of full quantization and sparsity on pixel-level generation quality.Source: §5.2 · pp. 12–13
• Adaptive hybrid sequence parallelism reduces communication overhead to a quarter of Ulysses [12], while FP8 communication further reduces end-to-end communication overhead.AsyncOffloading enables deployment on memory-limited devices with a performance drop of less than 2%; hybrid spatial-temporal parallelism for the VAE decoder improves parallel scaling.Source: §5.2 · pp. 13–14
• Kernel fusion, quantization, parallelization, continuous batching, prefix caching, and video-encoding acceleration boost end-to-end inference-pipeline efficiency.The encoding acceleration addresses low encoding efficiency in long videos.Source: §5.2 · p. 14

6.1 Pre-Training Optimization

• Pre-training combines HSDP and Ulysses sequence parallelism with MLAC and runtime-aware workload balancing to support long-context training across thousands of GPUs.MLAC uses asynchronous caching and prefetching to reduce GPU memory pressure; workload balancing distributes heterogeneous batch workloads across GPUs to reduce idle time.Source: §6.1 · p. 14
• Fused CUDA kernels for memory-bound operations reduced global memory traffic by over 90%.The system combined torch.compile with handcrafted kernels, fusing operations such as rotary position encoding and normalization.Source: §6.1 · p. 14
• Multi-level fault tolerance supports reliable prolonged distributed training through state checkpointing, machine health checks, and reduced model initialization overhead.Checkpointing includes model, optimizer, and dataloader states to support bitwise-exact resumption.Source: §6.1 · pp. 14–15

6.2 Post-Training Optimization

• Post-training memory use is challenged by dynamically shared model components, mixed trainable and frozen modes, and variable-length video workloads.The three phases are supervised fine-tuning, reinforcement learning, and distillation.Source: §6.2 · p. 15
• A dynamic memory-management framework combines CPU offloading, recomputation, localized static memory planning, FSDP, and sequence parallelism for post-training.PyTorch hooks implement CPU offloading, and sequence-parallelism degrees are configured per model; the optimizations support stable, efficient performance across multiple components and diverse workloads.Source: §6.2 · p. 15

7 Model Performance

• The performance section combines external public evaluation with internal benchmark scoring and comparative analysis, then highlights multi-shot transitions and multi-style generation.The section overview identifies Figure 1 as presenting the overall results.Source: §7 · p. 15

7.1 Artificial Analysis Arena

• Over 100 points separate Seedance 1.0 from the second- and third-best models, Veo 3 and Kling 2.0, in image-to-video; Seedance tops both Arena tracks with one unified model.The paper contrasts this with prior models that typically excelled in one domain while underperforming in the other.Source: Figure 9 · p. 16, §7.1 · p. 16

7.2.1 SeedVideoBench 1.0

• SeedVideoBench-1.0 uses systematic analysis of real-world prompts across scenarios such as special effects, e-commerce, and professional-generated content; it includes 300 prompts per task, with first-frame labels added for image-to-video.Prompts are distributed across categories to support discriminative and statistically confident evaluation.Source: §7.2.1 · pp. 16, 18

7.2.3 Human Evaluation

• Seedance 1.0, Kling 2.1, and Veo 3 substantially outperform other text-to-video models; Seedance and Veo lead on prompt following, while Kling’s weaker prompt following and Veo’s weaker motion quality constrain them.The paper identifies prompt following as critical for adoption of generated content.Source: Figure 10 · p. 17, Figure 11 · p. 17, §7.2.3 · p. 19
• Human evaluation compared leading models on text-to-video and image-to-video using five-point absolute scores and pairwise Good-Same-Bad judgments.Text-to-video comparisons included Kling 2.1, Veo 3, Wan 2.1, and Sora; image-to-video substituted Runway Gen4 for Sora.Source: §7.2.3 · pp. 18–19
• Seedance 1.0 and Kling 2.1 perform strongly in image-to-video; Seedance matches Kling’s motion quality and offers better prompt following for complex transitions or detailed instructions.The paper notes that image conditioning makes character and background preservation challenging, and reports occasional detail degradation for Kling.Source: Figure 12 · p. 19, Figure 13 · p. 19, §7.2.3 · p. 20

7.3 Multi-Shot Generation

• Seedance 1.0 generates consecutive shots from one prompt while maintaining subject continuity and stylistic coherence across frames.This supports complex cinematic storytelling and temporal coherence.Source: §7.3 · p. 21
• The model supports shot-reverse-shot dialogue, cut-ins, and cut-aways to enrich narrative pacing and contextual layering.Source: §7.3 · p. 21
• Seedance 1.0 supports match cuts and action cuts for seamless transitions that preserve visual continuity.Source: §7.3 · p. 21
• Figure 14’s continuous-shot example is described as more coherent and fluid in cinematic storytelling than examples from other models.Source: §7.3 · p. 21

7.4 Multi-Style Alignment

• In T2V, Seedance generates fine-grained stylistic videos; in I2V, it reliably preserves the reference image’s visual characteristics.Source: §7.4 · p. 21
• Its supported styles range from silent-film, Hong Kong cinema, and retro Hollywood aesthetics to anime, cyberpunk, and ink-wash animation.It can transition between realism and fantasy without extensive task-specific tuning, supporting versatility and controllability for filmmaking and AIGC creation.Source: §7.4 · p. 21

7.5 Visualization

• The visualization section presents Seedance 1.0 outcomes in Figures 14–16 and directs readers to its official website for additional examples.Source: §7.3 · p. 21, Figure 16 · p. 21, §7.5 · p. 22

• Seedance 1.0 is a native bilingual foundation model unifying text-to-video and image-to-video generation.Source: §8 · p. 22
• Its dataset construction, efficient architecture and training paradigm, post-training optimization, and inference acceleration combine to deliver high performance with fast inference.Source: §8 · p. 22
• The model demonstrates instruction following, motion stability, visual quality, complex-scenario handling, multi-shot generation, and long-range temporal coherence.Source: §8 · p. 22

Yu Gao

• The Yu Gao entry lists Haoyuan Guo, Tuyen Hoang, and Weilin Huang.Source: Yu Gao · p. 25