Source-linked AI summary
Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, Feng Cheng, Feilong Zuo, Xuejiao Zeng, Ziyan Yang, Fangyuan Kong, Meng Wei, Zhiwu Qing, Fei Xiao, Tuyen Hoang, Siyu Zhang, Peihao Zhu, Qi Zhao, Jiangqiao Yan, Liangke Gui, Sheng Bi, Jiashi Li, Yuxi Ren, Rui Wang, Huixia Li, Xuefeng Xiao, Shu Liu, Feng Ling, Heng Zhang, Houmin Wei, Huafeng Kuang, Jerry Duncan, Junda Zhang, Junru Zheng, Li Sun, Manlin Zhang, Renfei Sun, Xiaobin Zhuang, Xiaojie Li, Xin Xia, Xuyan Chi, Yanghua Peng, Yuping Wang, Yuxuan Wang, Zhongkai Zhao, Zhuo Chen, Zuquan Song, Zhenheng Yang, Jiashi Feng, Jianchao Yang, Lu Jiang
TL;DR
Video generation foundation models are costly to train and serve, motivating investigation of more efficient medium-sized alternatives. The report trains Seaweed-7B, a roughly 7-billion-parameter DiT, from scratch with 665,000 H100 GPU hours and studies design choices for this constrained setting. Seaweed-7B matches or exceeds larger models trained with substantially more GPU resources and generalizes across diverse video-generation tasks.
Problem
Video generation requires substantial training and inference compute, while few studies have shown that smaller models can approach contemporary state-of-the-art quality.
Method
The report trains Seaweed-7B, an approximately 7-billion-parameter DiT, from scratch using cost-conscious data, model, and training strategies.
Results
Seaweed-7B matches or exceeds larger models trained with significantly more GPU resources and demonstrates strong generalization across diverse video-generation tasks.
Takeaways & Limitations
Medium-sized models remain promising efficient video foundation models when supported by appropriate design choices and training strategies.
Takeaways & Limitations
The model still has limited fine-grained detail generation, including small faces and delicate patterns, because of constrained computational capacity.
Abstract
from arXiv · showhide
This technical report presents a cost-efficient strategy for training a video generation foundation model. We present a mid-sized research model with approximately 7 billion parameters (7B) called Seaweed-7B trained from scratch using 665,000 H100 GPU hours. Despite being trained with moderate computational resources, Seaweed-7B demonstrates highly competitive performance compared to contemporary video generation models of much larger size. Design choices are especially crucial in a resource-constrained setting. This technical report highlights the key design decisions that enhance the performance of the medium-sized diffusion model. Empirically, we make two observations: (1) Seaweed-7B achieves performance comparable to, or even surpasses, larger models trained on substantially greater GPU resources, and (2) our model, which exhibits strong generalization ability, can be effectively adapted across a wide range of downstream applications either by lightweight fine-tuning or continue training. See the project page at https://seaweed.video/
1 Introduction
Seaweed-7B investigates whether careful design and training strategies can make a medium-sized video generation foundation model cost-effective without sacrificing competitive performance. Trained from scratch with approximately 7 billion parameters, it is evaluated for generation quality and generalization across video tasks.
- Motivation: Scaling DiT models improves performance but requires massive GPU resources, with MovieGen using more than 6,000 NVIDIA H100 GPUs.These costs can impede innovation in video generation.
- Motivation: Video generation has especially high inference costs, favoring small to medium-sized models for better training and serving efficiency.Inference can be orders of magnitude more expensive than language, image, or audio generation and may be constrained by GPU memory.
- Motivation: Few video-generation studies have demonstrated scaling efficiency, while earlier small models remained separated from contemporary state-of-the-art quality.The report addresses this gap with a medium-sized model and resource-conscious design choices.
- Approach: Seaweed-7B is a roughly 7-billion-parameter DiT trained from scratch using 665,000 H100 GPU hours.The model is designed with FLOPs optimized for deployment on a single GPU, while its training examines data curation, model design, and optimization strategy.
- Evaluation: Seaweed-7B evaluates text-to-video and image-to-video generation for fidelity, aesthetics, motion quality, prompt alignment, and inference efficiency.The experiments also assess generic generation capability and downstream task generalization as foundation-model capabilities.
- Contributions: The report contributes VAE reconstruction designs, DiT training insights, and empirical evidence for competitive medium-sized performance across multiple video-generation tasks.The contributions emphasize compression–fidelity–generation trade-offs, cost-effective training, and cross-task evaluation.
2 Data
The data strategy prioritizes quality and diversity through large-scale curation, caption enrichment, synthetic augmentation, and scalable processing. The pipeline combines filtering, balancing, deduplication, and structured prompts to produce training data suited to video generation.
- Data Curation: In constrained computing settings, data quality and diversity take precedence over quantity, supported by a pipeline processing about 100 million clips averaging 8 seconds.The infrastructure is designed for scalable collection and processing of high-quality video data.
- Data Curation: The curation pipeline applies temporal splitting, spatial cropping, quality filtering, multi-aspect balancing, video deduplication, and video captioning.These processors transform diverse raw sources into higher-quality training data.
- Data Curation: Temporal splitting detects shot boundaries using HSV 3D color-histogram similarity and can merge clips with ImageBind features into multi-shot sequences.The method performs comparably to pyscenedetect and supports long-video generation data.
- Data Curation: Quality filtering removes unsuitable clips using duration, resolution, aspect-ratio, aesthetics, clarity, and motion criteria.The process combines attribute filtering, visual-quality evaluation, and spatial-temporal motion filtering.
- Data Curation: Visual and semantic clustering into more than 10,000 groups supports duplicate removal and balancing of long-tailed subjects, scenes, and actions.Visual features come from a CLIP-like model, while semantic features use LLM-generated caption labels.
- Video Captioning: The captioning system generates both action-centric short captions and richer detailed captions describing scenes, objects, and attributes.A 72B teacher distills knowledge into a 7B student, while detailed-then-short captioning raises test accuracy from 84.81% to 90.84%.
3 Design and Discussions
Seaweed’s design combines a high-quality, high-compression VAE with efficient DiT architecture and staged training strategies. These choices target reconstruction quality, convergence, long-context modeling, and task generalization under constrained compute.
- Variational Autoencoder: The VAE compresses pixel data into latent space and reconstructs it, making compression ratio and reconstruction quality central to generation fidelity.The architecture encodes images and videos across space and time, with the first frame represented as a dedicated latent.
- Variational Autoencoder: VAE compression within the latent encoder significantly outperforms equivalent sequence compression through DiT patchification.The comparison holds sequence length and attention cost constant while changing where compression occurs.
- Variational Autoencoder: 64× VAE converges faster and reaches a better stationary point without noticeable high-resolution visual artifacts, including at 720p.Its higher spatial compression ratio is compared against a 48× VAE followed by DiT patchification.
- Diffusion Transformer Model: The hybrid-stream DiT achieves faster convergence and lower loss than the dual-stream architecture under the same parameters, steps, and compute budget.The resulting 7B model uses a hidden size of 3584 and 32 layers, while shared deeper-layer FFN parameters improve parameter efficiency.
- Diffusion Transformer Model: Full attention scales better with compute and improves motion consistency in image-to-video, while its text-to-video perceptual benefit is marginal.Long video contexts create a trade-off between attention capacity and sequence length; a 720×1280, five-second video at 24 fps exceeds 100,000 tokens.
- Multi-stage, Multi-task Learning: A 20% image-to-video ratio during pre-training benefits both tasks, whereas excessive image-to-video mixing is detrimental.A dedicated post-training image-to-video branch increases that ratio to 50–75%.
- Multi-stage, Multi-task Learning: Prolonged supervised fine-tuning can rapidly overfit, degrading prompt following and motion quality.The reported effect constrains how long the post-training stage should continue.
4 Evaluation
Seaweed-7B is evaluated on text-to-video and image-to-video generation, inference efficiency, and VAE reconstruction. It achieves competitive rankings against larger models, with strong reconstruction and substantially faster inference, while image-to-video visual fidelity is limited by lower output resolution.
- Generation evaluation: Human evaluation uses MagicArena’s Elo system to compare overall fidelity, with more than 500 raters and at least 7,000 pairwise trials per model.The evaluation uses official model APIs available in early March 2025.
- Generation evaluation: Image-to-video Seaweed-7B ranks second, outperforming Sora and Veo 2.0 despite having 7B parameters and substantially fewer training resources than larger models.It was trained with resources equivalent to 1,000 H100 GPUs over 27.7 days and surpassed Wan 2.1’s 14B and HunyuanVideo’s 13B models.
- Generation evaluation: Seaweed-7B outperforms Sora, Wan-2.1, and HunyuanVideo across image-to-video metrics, while remaining competitive with Kling 1.6 in prompt following and motion quality.It trails Kling 1.6 in visual quality, which lowers its overall ranking.
- Generation evaluation: Text-to-video Seaweed-7B ranks among the top 2 models, following Veo 2 and outperforming Wan 2.1-14B and Kling 1.6.The model used 665,000 H100 GPU hours, compared with larger models trained using substantially greater computational resources.
- Inference efficiency: Seaweed-7B requires 12 NFEs and operates 62 times faster than Wan-2.1, whose default classifier-free-guidance configuration requires 100 NFEs.Timing includes end-to-end generation on a single H100 GPU, including the text encoder and VAE decoder.
- VAE reconstruction: The VAE achieves state-of-the-art reconstruction across rFVD, LPIPS, PSNR, and SSIM, including the lowest LPIPS on high-resolution, long-duration MCL-JCV videos.It retains this performance at a higher compression ratio than the previous state-of-the-art VAE.
5 Applications
Seaweed-7B supports diverse downstream video applications through native conditioning, lightweight fine-tuning, or continued adaptation. Demonstrated uses include restoration, controllable generation, subject consistency, audiovisual generation, storytelling, and real-time synthesis.
- Applications: Seaweed-7B is positioned as a foundation model supporting downstream video tasks through zero-shot generation or lightweight fine-tuning such as LoRA.
- Applications: The model natively supports image-to-video generation from an image and text prompt, plus transitions conditioned on first and last frames.
- Applications: Fine-tuning enables subject-consistent generation for faces, objects, clothing, animals, and virtual characters, including realistic multi-subject interactions.
- Applications: A video-conditioned audiovisual system uses contrastive audio-visual pretraining and latent diffusion with flow matching for audio generation.The design targets cross-modal understanding and temporal coherence across scenes.
- Applications: Long-context tuning adapts single-shot Seaweed generation to scene-level video, enabling autoregressive rollout, shot extension, and composable generation.Interleaved video and text generation is explored for narrative script creation.
- Applications: Seaweed-APT enables one-step generation at 1280×720 resolution and 24fps, demonstrating real-time video synthesis.
- Applications: Seaweed supports high-resolution video restoration and super-resolution generation, including arbitrary-length restoration applications.
- Applications: Camera-controlled generation supports categories such as Dolly-out, Pan-left, and Spin-shot, with CameraCtrl II providing precise controllability.The broader system also supports image-to-video, long-video, and real-time generation.
6 Related Work
Video generation has advanced rapidly but remains unusually resource-intensive because it must model motion and temporal coherence. Existing scaling approaches improve quality at high computational cost, leaving efficient small-model scaling comparatively underexplored.
- Motivation: Video generation integrates text, image, audio, and video modalities into applications including text-to-video, image-to-video, and audio-driven synthesis.
- Motivation: Unlike images, videos require motion dynamics and temporal coherence across long sequences, greatly increasing training and inference complexity.
- Prior approaches: Earlier diffusion approaches adapted text-to-image models with temporal layers or modified architectures, while later systems trained jointly on images and videos from scratch.
- Prior approaches: Sora combined DiTs, 3D VAEs, joint image-video training, and sequence packing while scaling model and dataset size to achieve substantially higher video quality.
- Resource constraints: Scaling video generation models typically requires thousands of GPUs, concentrating development in industrial research labs and increasing computational barriers.
- Resource constraints: Seaweed-7B addresses this resource gap with a video generation model designed for competitive performance at significantly lower computational cost.
7 Conclusion
Seaweed-7B is presented as a cost-effective 7-billion-parameter video generation foundation model whose performance and generalization remain competitive despite moderate resources. The authors also identify limitations in fine-grained detail generation, broader video-model quality, and responsible generation.
- Seaweed-7B is a cost-effective video generation foundation model with 7 billion parameters.
- Despite moderate computational resources, Seaweed-7B matches or exceeds larger models trained with significantly more GPU resources.
- Seaweed-7B demonstrates strong generalization across diverse video generation tasks, highlighting the potential of medium-sized video foundation models.
- Limited computational capacity constrains the model's ability to generate fine-grained details such as small faces and delicate patterns.
- The authors identify data curation, model design, post-training, safety, fairness, and ethics as areas requiring further work.
Contributors and Acknowledgments
The project involved researchers and contributors spanning research leadership, engineering, infrastructure, modeling, VAE, DiT, post-training, processing, acquisition, and evaluation. The acknowledgments identify individuals associated with each work stream.
- Contributors are distinguished from full-time researchers, and an asterisk marks alphabetically ordered lists.
- Named technical work streams include modeling, VAE, DiT, post-training, processing algorithms, and the processing pipeline.
- The acknowledgments also credit acquisition and evaluation and annotation contributors.
- The project credits research leads, code and engineering staff, and infrastructure personnel among its researchers and contributors.