Source-linked AI summary
Helios: Real Real-Time Long Video Generation Model
Shenghai Yuan, Yuanyang Yin, Zongjian Li, Xinwei Huang, Xiao Yang, Li Yuan
TL;DR
Helios addresses the gap between demand for real-time, minute-scale video generation and existing systems that are slow and short-lived. It combines drift-aware training, compressed context, fewer sampling steps, and infrastructure optimizations, achieving 19.5 FPS on one H100 while supporting minute-scale generation.
Problem
Mainstream video models typically generate only 5–10 seconds, and even these clips can require tens of minutes to synthesize, leaving real-time, effectively infinite generation largely unsolved.
Method
Helios uses a unified input representation for T2V, I2V, and V2V, drift-simulating training strategies, compressed historical and noisy context, fewer sampling steps, and infrastructure optimizations.
Results
19.5 FPS on a single H100: Helios supports minute-scale generation without conventional anti-drifting or acceleration techniques and significantly outperforms existing methods in quality.
Takeaways & Limitations
Helios demonstrates that a 14B video generation model can combine minute-scale generation, real-time inference, and unified support for T2V, I2V, and V2V.
Takeaways & Limitations
Helios prioritizes real-time long-video generation without a dedicated design for long-video memory, leaving that capability for future work.
Abstract
from arXiv · showhide
We introduce Helios, the first 14B video generation model that runs at 19.5 FPS on a single NVIDIA H100 GPU and supports minute-scale generation while matching the quality of a strong baseline. We make breakthroughs along three key dimensions: (1) robustness to long-video drifting without commonly used anti-drifting heuristics such as self-forcing, error-banks, or keyframe sampling; (2) real-time generation without standard acceleration techniques such as KV-cache, sparse/linear attention, or quantization; and (3) training without parallelism or sharding frameworks, enabling image-diffusion-scale batch sizes while fitting up to four 14B models within 80 GB of GPU memory. Specifically, Helios is a 14B autoregressive diffusion model with a unified input representation that natively supports T2V, I2V, and V2V tasks. To mitigate drifting in long-video generation, we characterize typical failure modes and propose simple yet effective training strategies that explicitly simulate drifting during training, while eliminating repetitive motion at its source. For efficiency, we heavily compress the historical and noisy context and reduce the number of sampling steps, yielding computational costs comparable to -- or lower than -- those of 1.3B video generative models. Moreover, we introduce infrastructure-level optimizations that accelerate both inference and training while reducing memory consumption. Extensive experiments demonstrate that Helios consistently outperforms prior methods on both short- and long-video generation. We plan to release the code, base model, and distilled model to support further development by the community.
1 Introduction
Helios targets real-time, minute-scale video generation, addressing limited duration, drifting, quality, and efficiency in existing systems. It combines a 14B architecture and training, sampling, and infrastructure strategies to achieve strong quality and real-time performance without common acceleration or anti-drifting techniques.
- Existing models typically generate only 5–10 seconds, and even short clips can require tens of minutes to synthesize.
- Helios simulates position, color, and restoration drifting during training, generating minute-scale videos without self-forcing, error-banks, keyframe sampling, or inverted sampling.
- 19.5 FPS enables end-to-end inference for a 14B model on one H100 without standard acceleration techniques such as KV-cache, causal masking, sparse or linear attention, or quantization.
- Helios improves training and inference throughput while reducing memory use, allowing 14B training without parallelism or sharding and with image-model-scale batch sizes.
- HeliosBench evaluates real-time long-video generation, and experiments show Helios outperforms existing methods in quality while surpassing some 1.3B distilled models in inference speed.
2 Related Work
Prior long-video methods address duration and efficiency through noise rescheduling, context corruption, larger models, distillation, attention reductions, caching, and quantization. However, real-time quality and robustness remain constrained by drifting, costly generation, and altered inference regimes.
- 2 Related Work: Long-video methods commonly simulate inference-time context corruption during training to enable autoregressive diffusion for longer synthesis.
- 2 Related Work: Producing a 5-second Wan2.1 14B video can take roughly 50 minutes on one A100, motivating parallelism, distillation, efficient attention, caching, and quantization.
- 2 Related Work: Krea reports 11 FPS on a B200 but drops to 6.7 FPS on an H100, while its results suffer from severe drifting; some claimed real-time systems require 8 GPUs.
- 2.2 Real-Time Video Generation: Helios compresses historical and noisy context through Multi-Term Memory Patchification and Pyramid Unified Predictor Corrector while unifying T2V, I2V, and V2V with Representation Control.
3 Helios
Helios unifies long-video generation across T2V, I2V, and V2V while addressing drifting, efficiency, and few-step distillation. Its design combines history control, drift-aware training, compressed context, multiscale sampling, and hierarchical distillation.
- 3.1 Helios: Unified History Injection and Representation Control convert a bidirectional model into an autoregressive generator for arbitrarily long T2V, I2V, and V2V videos.Zero, single-frame, and video history representations select T2V, I2V, and V2V respectively.
- 3.2 Easy Anti-Drifting: Relative RoPE stabilizes arbitrary-length generation by constraining temporal indices separately for historical and noisy contexts, reducing repetitive motion from positional periodicity.The method assigns historical indices within 0:THist and noisy indices after THist.
- 3.2 Easy Anti-Drifting: Frame-Aware Corrupt improves robustness to imperfect autoregressive history by simulating realistic history drift during training.This targets restoration shifts such as blur and noise caused by accumulated errors in generated history.
- 3.3 Deep Compression Flow: Multi-Term Memory Patchification keeps the token budget constant while retaining longer history, and training-time context masking supports T2V, I2V, and V2V inference conditions.The hierarchical history representation reduces computation and memory while extending context length.
- 3.3 Deep Compression Flow: Pyramid Unified Predictor Corrector uses coarse-to-fine multiscale sampling, while Adversarial Hierarchical Distillation and staged backward simulation support stable few-step generation.The sampling schedule prioritizes global structure early and fine details later; the distillation framework adds multistage and teacher-independent training components.
4 Infrastructure
Helios addresses the memory and compute costs of 14B video generation through context compression, memory-management strategies, and custom GPU optimizations. These techniques enable single-GPU training stages while improving throughput and reducing peak memory.
- Memory and workload challenges: 14B DiT video generators face quadratic 3D-attention costs and typically require parallelism or parameter and activation sharding.The primary bottleneck is attention over temporal and spatial tokens.
- Memory and workload challenges: Deep Compression Flow enables full forward and backward passes for the first two training stages on a single GPU without parallelism or sharding.The approach compresses both historical and noisy contexts.
- Context compression: 8× and 2.29× token reductions for historical and noisy contexts yield roughly 64× and 5.2× lower attention FLOPs, respectively.The reductions come from Multi-Term Memory Patchification and Pyramid Unified Predictor Corrector with K = 3.
- Stage-3 memory management: Sharded EMA, asynchronous VRAM freeing, and Cache Grad reduce stage-3 memory pressure when training four 14B models and multiple GAN heads.Cache Grad frees estimator activations early and reuses cached input gradients during backpropagation.
- Stage-3 memory management: Cache Grad reduces peak memory to that of a single 14B model by decoupling discriminator gradients from the default backward pass.The estimator’s intermediate activations are released after its forward pass.
- GPU optimizations: Custom forward and backward implementations, including fused normalization and rotary-position kernels, improve computational efficiency and reduce activation-memory overhead.Flash Normalization reduces intermediate activation complexity from O(BℓD) to O(Bℓ), while fused RoPE removes copying and fragmentation overheads.
- GPU optimizations: Removing causal masking lets Helios use high-efficiency attention backends such as FlashAttention, improving throughput and reducing latency.
5 Experiments
Experiments evaluate Helios across training stages, short and long videos, benchmark construction, and component ablations. Helios achieves real-time speed, strong quality, lower long-horizon drifting, and comparable robustness without long rollout-based anti-drifting training.
- Evaluation setup: HeliosBench evaluates real-time long-video generation using 240 LLM-refined prompts across four duration tiers.
- Short-video comparison: 19.53 FPS on a single H100 GPU, while Helios matches or surpasses strong models on semantic quality and naturalness.Helios also balances dynamic and motion smoothness without the static motion or temporal jitter associated with some accelerated models.
- Long-video comparison: 7.08 total score versus 6.88 for Reward Forcing, with higher naturalness and lower drifting across aesthetic, semantic, and naturalness dimensions.Helios preserves content identity and scene layout across hundreds or thousands of frames.
- Long-video comparison: Helios preserves visual quality over time and mitigates long-horizon drifting without Self-Forcing or Error-Banks in its first two stages.
- Component ablations: Removing the First Frame Anchor degrades color and subject consistency as early as frame 720, with errors compounding over longer sequences.
- Component ablations: Removing Frame-Aware Corrupt causes severe drifting at 240 frames and sharper degradation for minute-scale generation.Aesthetic, semantic, and naturalness scores all decline.
- Distillation ablations: Pure Teacher Forcing achieves long-video drift robustness comparable to Self-Forcing while avoiding long rollouts and substantially reducing training overhead.
- Distillation ablations: Extending one flow trajectory into multiple multi-scale trajectories nearly doubles throughput with only a modest performance drop.The gap narrows in Stage 3 through Adversarial Hierarchical Distillation.
6 Application
Randomly zeroing historical context during training allows Helios to generalize across text-to-video, image-to-video, and video-to-video generation. The paper reports satisfactory quality for these task modes.
- Unified task support: Randomly zeroed historical context enables Helios to support T2V, I2V, and V2V tasks within the conventional text-to-video training pipeline.
- Unified task support: The paper reports satisfactory quality for the supported T2V, I2V, and V2V showcases.
7 Conclusion
Helios is presented as a 14B model for minute-scale video generation at 19.5 FPS on one H100 GPU, without conventional anti-drifting or acceleration techniques. It supports T2V, I2V, and V2V and consistently outperforms existing methods across short and long videos.
- 19.5 FPS on a single NVIDIA H100 GPU supports minute-scale generation without conventional anti-drifting or acceleration techniques.Helios uses a unified representation for text-to-video, image-to-video, and video-to-video tasks.
- Explicit drifting simulation and compressed historical, noisy, and sampling contexts reduce the computational cost to image-diffusion-model levels.
- Helios consistently outperforms existing methods across multiple dimensions for both short and long videos.
8 Limitations and Future Work
The paper identifies limitations in evaluation, temporal consistency, and experimental scope. It calls for perceptually aligned metrics, smoother stitching, higher-resolution studies, and dedicated long-video memory design.
- Existing video-generation metrics insufficiently reflect human judgment, with Aesthetic and Smoothness showing only marginal differences from prior approaches.The paper proposes developing perceptually aligned metrics.
- Generated segments may suffer from flickering artifacts at stitching boundaries, motivating reinforcement learning for smoothness-related objectives.
- 384×640 is the highest experimental resolution because resource constraints leave higher-resolution settings unexplored.
- The work does not include a dedicated design for long-video memory, which remains an important direction for future research.
9 Declaration of LLM Usage
The authors used GPT to assist with language editing, data organization, selection, preliminary figure design, and alternative phrasing. They state that the academic content, analyses, and conclusions were independently developed and validated by the authors.
- GPT assisted with grammar correction, spell-checking, expression refinement, data organization, figure design, and drafting alternative phrasing.
- The authors independently developed, validated, and interpreted the academic content, analyses, and conclusions.
10 Ethics Statement
Helios can generate high-quality, realistic videos, but the technology could also be misused to create deceptive content for fraud.
- Helios can generate high-quality, realistic videos that could be exploited to create deceptive content for fraudulent purposes.
A Calculation of Helios-Bench
Helios-Bench converts noisy raw metrics into normalized, discretized ratings and combines them with duration-aware metric weights. Its weighting scheme differs between short- and long-video evaluation, with long-video scores explicitly incorporating drifting metrics.
- Score normalization and discretization: Raw metric scores are clipped and normalized to [0, 1] before being converted into discrete ratings.The normalization uses predefined minimum and maximum bounds for each metric.
- Score normalization and discretization: A threshold-based step function assigns 10-point ratings, using opposite threshold directions for higher-is-better and lower-is-better metrics.Lower-is-better metrics include the drifting variants used for long-video evaluation.
- Weighted aggregation: The overall score is a duration-aware weighted sum of discretized metric ratings.Each metric contributes according to its metric-specific weight.
- Metric weighting: Short-video evaluation weights Semantic and Naturalness at 0.35 each, while Aesthetic, Motion Amplitude, and Motion Smoothness receive 0.10 each.For long videos, four drifting metrics are introduced with weight 0.099 each to penalize degradation over time.
- Metric weighting: Table 7 lists empirical discretization thresholds, with the final two Aesthetic thresholds marked as potential anomalies.The asterisks indicate a possible anomaly in the empirical data or source configuration.
B Latent Re-encode.
Helios re-encodes concatenated video sections into one continuous latent sequence before applying standard re-noising. This removes repeated first-frame distributions, improving alignment with the real and fake score-estimator inputs and enhancing temporal coherence.
- Distributional mismatch: Generating sections separately satisfies the estimator’s requirement for 21 latent frames but creates distributional mismatch when sections are concatenated.The first frame of each section differs statistically from later frames, producing multiple first-frame occurrences.
- Latent re-encoding: VAE decoding and re-encoding turns concatenated sections into a single continuous latent sequence with one first-frame distribution.The standard re-noising operation is applied after re-encoding.
- Latent re-encoding: The re-encoded sequence improves alignment with preal and pfake, reducing score-estimation bias and enhancing temporal coherence.