PaperlayerView exact revision on arXiv

Source-linked AI summary

Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model

Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu, Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie, Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jiahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang, Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang, Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang, Qiaohui Chen, Qinglin He, Qiuyan Liang, Quan Sun, Ran Sun, Rui Wang, Shaoliang Pang, Shiliang Yang, Sitong Liu, Siqi Liu, Shuli Gao, Tiancheng Cao, Tianyu Wang, Weipeng Ming, Wenqing He, Xu Zhao, Xuelin Zhang, Xianfang Zeng, Xiaojia Liu, Xuan Yang, Yaqi Dai, Yanbo Yu, Yang Li, Yineng Deng, Yingming Wang, Yilei Wang, Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng, Yuxiang Yang, Zecheng Tang, Zekai Zhang, Zidong Yang, Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Heung-Yeung Shum, Daxin Jiang

arXiv:2502.10248v3cs.CVcs.CL

TL;DR

Existing video generation systems still struggle to represent dynamic reality and to handle complex actions and physical interactions. This report presents Step-Video-T2V, a 30B-parameter text-to-video model with compressed video representation, bilingual prompting, staged training, and video preference optimization. It reports high-quality generation, introduces a 128-prompt benchmark, and identifies limitations of current diffusion-based paradigms.

  • Problem

    Existing language abstractions poorly capture dynamic motion and spatial-temporal relationships, while diffusion-based video models struggle with complex actions and physical laws.

  • Method

    Step-Video-T2V combines a 30B-parameter DiT, deep-compression Video-VAE, bilingual text encoders, Flow Matching, cascaded training, and video-based DPO.

  • Results

    Step-Video-T2V generates high-quality text-conditioned videos with strong motion dynamics, high aesthetics, and consistent content, and is evaluated with Step-Video-T2V-Eval.

  • Takeaways & Limitations

    The released model and benchmark provide a baseline for studying video foundation models and accelerating research on their development and application.

  • Takeaways & Limitations

    Step-Video-T2V has limited final-stage 540P training data and less high-quality post-training data than two commercial engines, while long-video training is more challenging.

Abstract

from arXiv · show

We present Step-Video-T2V, a state-of-the-art text-to-video pre-trained model with 30B parameters and the ability to generate videos up to 204 frames in length. A deep compression Variational Autoencoder, Video-VAE, is designed for video generation tasks, achieving 16x16 spatial and 8x temporal compression ratios, while maintaining exceptional video reconstruction quality. User prompts are encoded using two bilingual text encoders to handle both English and Chinese. A DiT with 3D full attention is trained using Flow Matching and is employed to denoise input noise into latent frames. A video-based DPO approach, Video-DPO, is applied to reduce artifacts and improve the visual quality of the generated videos. We also detail our training strategies and share key observations and insights. Step-Video-T2V's performance is evaluated on a novel video generation benchmark, Step-Video-T2V-Eval, demonstrating its state-of-the-art text-to-video quality when compared with both open-source and commercial engines. Additionally, we discuss the limitations of current diffusion-based model paradigm and outline future directions for video foundation models. We make both Step-Video-T2V and Step-Video-T2V-Eval available at https://github.com/stepfun-ai/Step-Video-T2V. The online version can be accessed from https://yuewen.cn/videos as well. Our goal is to accelerate the innovation of video foundation models and empower video content creators.

1 Preface

Video foundation models generate videos from text, visual, or multimodal inputs, but current diffusion-based systems remain limited to cross-modal translation and struggle with complex actions, physical laws, and causal reasoning. Step-Video-T2V addresses the Level-1 practice while identifying challenges for advancing toward predictive video foundation models.

  • Video foundation models are pre-trained on large video datasets to generate videos from text, visual, or multimodal user inputs.They support downstream generation, understanding, editing, conversion, question answering, and task completion.
  • Level-1 models perform cross-modal translation, whereas Level-2 models would forecast future events and support multimodal reasoning or real-world simulation.
  • Current diffusion-based text-to-video models produce high-quality videos but often fail on complex action sequences, physical laws, causal tasks, and logical tasks.
  • Autoregressive text-to-video models add causal modeling by predicting the next token, frame, or clip, but do not yet match diffusion-based models on text-to-video generation.
  • The report presents Step-Video-T2V as a Level-1 model and uses experimental challenges to identify problems for developing Level-2 video foundation models.

2 Introduction

Video generation is important for representing dynamic reality and lowering barriers to video creation. The report presents and open-sources Step-Video-T2V, develops its training and data pipeline, introduces Step-Video-T2V-Eval, and analyzes remaining challenges.

  • 2 Introduction: Video generation is positioned as an important AGI frontier because language abstractions poorly capture dynamic motion and spatial-temporal relationships.
  • 2 Introduction: Step-Video-T2V is a 30B-parameter diffusion Transformer model that generates high-quality text-conditioned videos with strong motion dynamics, aesthetics, and content consistency.
  • 2 Introduction: The model uses a deep-compression Video-VAE, bilingual text encoders, Flow Matching, and cascaded text-to-image, text-to-video, SFT, and DPO training.
  • 2 Introduction: The report identifies training and generation challenges involving caption hallucinations, rare concept composition, computational cost, and complex physical actions.
  • 2 Introduction: 16x16 spatial and 8x temporal compression ratios are achieved by Video-VAE while maintaining exceptional video reconstruction quality.
  • 2 Introduction: The report releases Step-Video-T2V-Eval with 128 diverse prompts across 11 categories and comparison results from open-source and commercial engines.

3 Related Work

Step-Video-T2V builds on recent open-source and commercial text-to-video systems while targeting more efficient video modeling through stronger compression. Its architecture combines high-compression Video-VAE, bilingual prompt encoding, 3D full-attention DiT denoising, and video-based DPO.

  • Video Generation Models: Recent video-generation engines support text-to-video and related applications, while open-source models provide implementation transparency for researchers and creators.Commercial systems include Sora, Gen-3, Kling, and Hailuo; open-source examples include HunyuanVideo, CogVideoX, Open-Sora, and Open-Sora-Plan.
  • Comparison with Movie Gen Video: Compared with Movie Gen Video, Step-Video-T2V adds high-compression VAE processing, bilingual English-Chinese prompt understanding, and an additional DPO training stage.The cited passage also identifies Step-Video-T2V as having four distinguishing features, but the supplied text truncates the fourth feature.
  • Efficient Video Modeling: Videos require efficient modeling because their spatial and temporal information creates substantially larger data volumes than images.The paper discusses 3D causal convolution, wavelet transforms, and related approaches for reducing video-modeling complexity.
  • Step-Video-T2V Architecture: Step-Video-T2V uses a high-compression Video-VAE, bilingual text encoders, a 3D full-attention DiT trained with Flow Matching, and video-based DPO.The pipeline denoises input noise into latent frames conditioned on text embeddings and timesteps, while DPO reduces artifacts and improves smoothness and realism.

4 Model

Step-Video-T2V combines unified latent compression with bilingual conditioning and a 3D full-attention diffusion transformer. Its Video-VAE is designed to reduce token and computation costs while preserving video detail and supporting joint image-video modeling.

  • Overall Model: Step-Video-T2V represents videos with a high-compression Video-VAE, bilingual text encoders, a Flow Matching-trained 3D full-attention DiT, and Video-DPO.The DiT denoises input noise into latent frames using text embeddings and timesteps as conditioning factors.
  • Latent Space Compression: Existing video models commonly use spatial-temporal downscaling and patchification, but their two-stage compression and tokenization can add complexity and degrade later diffusion stages.Attention computation scales quadratically with token count, motivating more compact latent representations.
  • Latent Space Compression: 16×16 spatial and 8× temporal compression are achieved through unified 3D convolutions and optimized pixel unshuffling in the Video-VAE.For input X, the encoder produces latent Z with temporal, height, and width dimensions reduced by factors of 8, 16, and 16.
  • Causal 3D Convolutional Modules: Causal 3D convolution makes each frame depend only on previous frames, supporting temporal causality in the encoder.The architecture uses causal Res3DBlocks and downsampling stages, followed by a MidBlock combining convolution and attention.
  • Dual-Path Latent Fusion: Dual-Path Latent Fusion preserves high-frequency detail through convolution while retaining low-frequency structure through channel averaging.The unified image-video structure uses parameters more efficiently and addresses blurring artifacts associated with traditional VAEs.

1. Conv Path: Combines causal 3D convolutions with pixel unshuffling,

Step-Video-T2V combines a staged Video-VAE, bilingual prompt encoding, and a DiT with 3D full attention. Human-feedback training further targets visual quality and faster convergence.

  • Conv Path: 3D pixel unshuffle and causal 3D convolution jointly compress video into spatial-temporal latent representations.The architecture combines a convolutional path with a shortcut path, then fuses their outputs through residual summation.
  • Conv Path: The decoder reverses compression with 3D pixel shuffle, grouped channel repetition, and spatial group normalization to reduce temporal flickering.These operations efficiently unfold compressed information back into spatial-temporal dimensions.
  • Video-VAE: The Video-VAE uses a staged training process to achieve efficient, high-quality video data modeling.Training progresses from low-level reconstruction learning to dual-path refinement, using reconstruction, perceptual, KL, and later GAN losses.
  • Bilingual Text Encoder: Two bilingual text encoders, Hunyuan-CLIP and Step-LLM, handle prompts of varying lengths and provide representations that guide latent-space generation.Hunyuan-CLIP aligns text with visual representations but is limited to 77 tokens, whereas Step-LLM has no input-length restriction.
  • DiT with 3D Full Attention: The DiT uses 3D full attention and cross-attention to incorporate text prompts while modeling spatial and temporal video information.The architecture also includes optimized AdaLN, RoPE-3D, and QK-Norm components for efficiency, variable video inputs, and stable attention training.
  • Video-DPO: Reducing β in Video-DPO and increasing the learning rate produces much faster convergence, while human-feedback gains saturate as outdated preference data becomes less aligned.The authors address this mismatch by dynamically evaluating new samples with a reward model periodically fine-tuned on fresh annotations.

5 Distillation for Step-Video-T2V Turbo

Step-Video-T2V Turbo distills the base rectified-flow model to reduce inference steps. A diminishing CFG schedule enables comparable sample quality with substantially fewer evaluations.

  • Motivation: More than 50 ODE-integration steps are typically required for video diffusion inference, motivating reduction of the number of function evaluations.The authors report that a large-scale Video DiT can reduce NFE to as few as 8 steps with negligible performance degradation.
  • Distillation Objective: The distillation objective trains a 2-rectified-flow model from a base 1-rectified-flow model to create more direct ODE paths.The reducible error is emphasized at difficult timesteps, where the 2-rectified-flow loss is largest near the interval endpoints.
  • Distillation Data: Approximately 95,000 samples form the distillation dataset, using curated prompts, 50 NFE generation, and a U-shaped timestep-sampling distribution with a = 5.Positive and negative prompts are carefully designed for the distillation data.
  • Inference: Up to 10 times fewer steps yield comparable sample quality when sampling time shifts are combined with a linearly diminishing CFG schedule.The turbo model generates 204-frame videos with 10 NFE.

6 System

Step-Video-T2V’s system combines large-scale distributed training infrastructure with parallelism, communication, monitoring, and hybrid load balancing to improve robustness and utilization.

  • System workflow: The training system uses an offline emulator to estimate resource allocation and parallelism before deploying jobs across training and inference GPU clusters.The workflow also includes large-scale monitoring and operational management.
  • Reliability: 99% effective GPU training time was achieved over more than one month through infrastructure reliability improvements.The system dynamically assigns GPUs to training and inference clusters and supports large jobs spanning multiple clusters.
  • Parallelism strategy: 8-way tensor parallelism combined with sequence parallelism and Zero1 is adopted as a robustness-oriented trade-off against more complex parallelism combinations.Combining tensor, context, pipeline, and virtual pipeline parallelism achieves the best MFU, but pipeline parallelism is avoided for flexibility and simplicity.
  • Communication optimization: StepCCL overlaps communication with computation through DMA-based collective operations, allowing communication and GEMM execution to proceed concurrently on the same GPU.The design targets lower tensor-parallel overhead and higher hardware utilization.
  • Load balancing: Hybrid-grained load balancing aligns FLOPs across mixed-resolution samples by adjusting batch sizes according to per-sample computational cost.For resolution r, batch sizes B_r are computed from estimated FLOPs F_r, a target batch FLOPs, and a normalization factor α.

6.3 StepRPC

StepRPC and StepTelemetry address communication, observability, and infrastructure reliability challenges in large-scale Step-Video-T2V training, while StepMind sustains high effective utilization.

  • StepRPC: StepRPC is a distributed communication framework using named pipes and spraying to distribute data across training servers.It supports simultaneous data generation for large-scale pre-training and small-scale ablation workloads.
  • StepRPC: Tensor-native RDMA and TCP communication transfers tensor bits directly in memory, eliminating serialization and deserialization overheads.The approach supports direct transfer of GPU and CPU tensors.
  • StepTelemetry: StepTelemetry provides asynchronous data collection, database-backed statistics, and near-zero-overhead anomaly detection using CUDA events.It supports both batch and streaming writes and enables visualization and drill-down across training ranks.
  • StepTelemetry: StepTelemetry identifies iteration bottlenecks and data-parallel imbalance; mixed image-video input retained iteration time while increasing data throughput after imbalance correction.Its metrics cover stage timing and dataloader throughput.
  • StepMind: StepMind exceeds 99.0% effective GPU utilization through full-coverage monitoring, node-quality assessment, and automated server recovery procedures.The system selects reliable machines, detects faults rapidly, and validates repaired servers before reuse.
  • StepMind: 99% effective training time was maintained over more than one month, with fewer restarts contributing to this outcome.The report also records only seven fatal hardware failures over a month.

7 Data

The data pipeline converts raw media into large-scale video-text training data through segmentation, quality and motion assessment, captioning, concept balancing, and alignment filtering.

  • Dataset construction: The dataset contains 2B video-text pairs and 3.8B image-text pairs produced through a multi-stage preprocessing pipeline.The stages include video segmentation, quality assessment, motion assessment, captioning, concept balancing, and video-text alignment.
  • Video segmentation: Scene-change detection and FFmpeg splitting convert raw videos into single-shot clips while preventing artifacts in certain high-resolution inputs.The process adjusts crop start times to preserve necessary reference frames.
  • Motion assessment: Motion assessment averages optical-flow magnitudes across frame pairs and uses Motion_Mean, Motion_Max, and Motion_Min tags to characterize clip dynamics.These tags help identify static, excessively jittery, or abruptly paused content.
  • Video captioning: An in-house VLM generates short, dense, and original-title captions to improve caption precision, richness, and style diversity.Dense captions include subjects, actions, environments, visual style, and camera movements.
  • Concept balancing: VideoCLIP embeddings and K-means clustering over 120,000 clusters support concept balancing and removal of outlier clips.Cluster_Cnt and Center_Sim tags describe cluster size and distance to the cluster center.
  • Alignment and quality: CLIP Score measures video-text alignment by averaging cosine similarities between caption embeddings and embeddings from eight sampled frames.Quality filtering also evaluates aesthetics, NSFW content, watermarks, subtitles, saturation, and blur.

8 Training Strategy

Step-Video-T2V uses cascaded training that progresses from spatial knowledge to motion learning, video specialization, and preference optimization, with data quality shaping training behavior.

  • Training stages: The training recipe has four stages: T2I pre-training, T2VI pre-training, T2V fine-tuning, and DPO training.Variable-duration and aspect-ratio buckets accommodate videos with different lengths and shapes.
  • T2I pre-training: T2I pre-training is performed first because direct T2V training from scratch slowed convergence and hindered concept learning in early experiments.The initial stage establishes spatial knowledge before video training.
  • T2VI pre-training: T2VI pre-training begins with 192x320 low-resolution videos to emphasize motion learning, then increases resolution to 544x992 for later training.The stage combines image and video training.
  • T2V fine-tuning: T2V fine-tuning uses a small text-video dataset without T2I to adapt the pretrained model specifically to text-to-video generation.This stage addresses artifacts and style variation introduced by diverse pre-training data.
  • Checkpoint selection: Averaging checkpoints fine-tuned on different SFT datasets improves video quality and stability and outperforms EMA; same-source averaging also reduces distortions.Checkpoints are selected after gradient norm peaks, when gradient norm and loss have decreased.
  • Data filtering: Progressively stricter filtering creates six pre-training subsets, while manual filtering constructs the final SFT dataset.Figure 11 distinguishes removed data from data retained at each filtering stage.
  • Training observations: Training loss decreases as data quality improves, with a sudden drop following human-oriented filtering based on measures such as CLIP and aesthetic scores.The observation is attributed to dataset-quality changes rather than direct supervision through the training loss.

9 Experiments

The experiments evaluate Step-Video-T2V with a new human-assessed benchmark spanning 11 categories and compare it with open-source and commercial systems. Results show strong overall performance, especially for motion dynamics, while revealing weaknesses in aesthetics, resolution, text generation, and training scale.

  • Benchmark and metrics: Step-Video-T2V-Eval uses 128 Chinese user prompts across 11 categories to assess text-to-video quality.The categories include Sports, Food, Scenery, Animals, Festivals, Combined Concepts, Surreal, People, 3D Animation, Cinematography, and Style.
  • Benchmark and metrics: Metric-1 compares masked video pairs with Win/Tie/Loss labels, while Metric-2 scores instruction following, motion smoothness, physical plausibility, and aesthetic appeal.Metric-1 is pairwise; Metric-2 evaluates four dimensions for each generated video.
  • Open-source comparisons: Step-Video-T2V is reported as the strongest open-source text-to-video model, though HunyuanVideo performs better in Animals because of aesthetic issues.The report attributes its open-source result to structural design and pre-training and post-training strategies.
  • Commercial comparisons: Step-Video-T2V ranks below T2VTopA but above T2VTopB overall, while outperforming both commercial systems in Sports and motion-related dimensions.The report associates the commercial models’ aesthetic advantage with higher resolution and high-quality aesthetic post-training data.
  • Commercial comparisons: Step-Video-T2V outperforms Gen-3 Alpha overall, performs comparably to Movie Gen Video, and improves across all categories over HunyuanVideo.Gen-3 Alpha is stronger in Cinematography; Movie Gen Video benefits from more high-resolution training, more labeled data, and 720P output.
  • Ablations and analysis: Text generation remains inaccurate, Chinese-character generation is limited, and Video-DPO still faces underused dynamic conditions and sparse, imprecise feedback.These limitations constrain both textual fidelity and the effectiveness of preference optimization for large video outputs.
  • Ablations and analysis: Video-VAE maintains state-of-the-art reconstruction quality despite an eight-times-larger compression ratio than most baselines, and Video-DPO raises preference from 45% to 55%.The reconstruction evaluation uses 1,000 videos, while human feedback improves plausibility and consistency in the DPO comparison.

10 Discussion

The discussion examines architectural choices and persistent challenges in instruction following, physical realism, data curation, and preference-based post-training. It argues that current diffusion models remain limited by attention failures, weak physical simulation, expensive curation, and difficult-to-define video objectives.

  • Architecture: DiT and MMDiT show similar early training performance, so the report selects DiT because it disentangles text and video representations.The comparison evaluates alternative text-conditioning architectures before the final design choice.
  • Architecture: 3D full attention outperforms spatial-temporal attention, particularly for high-motion videos, but incurs high training and inference costs.The chosen mechanism jointly captures spatial and temporal information, while efficiency remains an open concern.
  • Instruction following: Even with 30B parameters, Step-Video-T2V struggles with complex action sequences and low-frequency combinations of multiple concepts.The report treats both cases as instruction-following problems.
  • Instruction following: Concentrated cross-attention can cause missing objects, incorrect details, or incomplete actions, while repeating missing objects in prompts can improve some cases.The proposed heuristic motivates future work on balancing attention across prompt elements.
  • Physical realism: Leading text-to-video models struggle to simulate real-world interactions and obey physical laws, and isolated successes may overfit specific annotations.Examples include balls bouncing and water falling into cups.
  • Data and post-training: Small amounts of high-quality human-labeled SFT data substantially improve video quality, but curation is expensive and time-consuming.The report emphasizes quality, diversity, style, and motion dynamics over dataset scale alone.
  • Data and post-training: Video-DPO still requires early human labeling and becomes time-consuming to extend broadly, while video-generation objectives remain difficult to define for RL-based optimization.Unlike mathematical or coding tasks, video generation lacks similarly clear task answers.

11 Conclusion and Future Work

The report concludes by open-sourcing Step-Video-T2V and its evaluation benchmark as a foundation for further video-generation research. It identifies data quality, instruction following, physical realism, and RL-based optimization as continuing challenges while aiming to support video creators.

  • Contributions: Step-Video-T2V provides a 30B-parameter text-to-video model with Video-VAE compression, DPO post-training, and generation up to 204 frames.The report also open-sources the model and introduces Step-Video-T2V-Eval.
  • Contributions: Step-Video-T2V-Eval is introduced as a benchmark for evaluating text-to-video generation models.The benchmark accompanies the open-sourced model and training overview.
  • Challenges: The report highlights high-quality labeled data, instruction following, physical-law adherence, and RL-based optimization as major challenges for current text-to-video models.These challenges span data, generation behavior, physical realism, and post-training mechanisms.
  • Future work: The authors plan further open-source projects focused on video foundation models to drive innovation and empower video content creators.Step-Video-T2V is described as the starting point for this series.

Contributors and Acknowledgments

The report distinguishes core contributors, who participated throughout development, from contributors involved in early versions or part-time work. Contributors are organized by development role and listed alphabetically by first name.

  • Contributor roles: Core contributors are defined as participants involved throughout the entire Step-Video-T2V development process.They are listed across Model & Training, Infrastructure, and Data & Evaluation roles.
  • Contributor roles: Contributors are defined as people who worked on early versions or contributed part-time.The report lists these contributors separately from the core contributors.
  • Acknowledgments: All contributors are listed in alphabetical order by first name.The ordering applies to both core contributors and contributors.
Loading 2502.10248v3…