Source-linked AI summary
Stream-T1: Test-Time Scaling for Streaming Video Generation
Yijing Tu, Shaojin Wu, Mengqi Huang, Wenchuan Wang, Yuxin Wang, Chunxiao Liu, Zhendong Mao
TL;DR
Existing video Test-Time Scaling methods face costly candidate exploration and limited temporal guidance. Stream-T1 adapts Test-Time Scaling to streaming video generation and significantly improves temporal consistency, motion smoothness, and frame-level visual quality on 5s and 30s benchmarks.
Problem
Existing video Test-Time Scaling methods face costly candidate exploration, limited temporal guidance, and unresolved long-term semantic, motion, and temporal consistency challenges.
Method
Stream-T1 applies beam search with noise propagation, reward-guided candidate pruning, and adaptive memory sinking to streaming video generation.
Results
Stream-T1 significantly improves temporal consistency, motion smoothness, and frame-level visual quality across 5s and 30s video benchmarks.
Takeaways & Limitations
Stream-T1 provides a test-time scaling framework for streaming generation with strong long-term stability, spatiotemporal coherence, and visual fidelity.
Abstract
from arXiv · showhide
While Test-Time Scaling (TTS) offers a promising direction to enhance video generation without the surging costs of training, current test-time video generation methods based on diffusion models suffer from exorbitant candidate exploration costs and lack temporal guidance. To address these structural bottlenecks, we propose shifting the focus to streaming video generation. We identify that its chunk-level synthesis and few denoising steps are intrinsically suited for TTS, significantly lowering computational overhead while enabling fine-grained temporal control. Driven by this insight, we introduced Stream-T1, a pioneering comprehensive TTS framework exclusively tailored for streaming video generation. Specifically, Stream-T1 is composed of three units: (1) Stream -Scaled Noise Propagation, which actively refines the initial latent noise of the generating chunk using historically proven, high-quality previous chunk noise, effectively establishes temporal dependency and utilizing the historical Gaussian prior to guide the current generation; (2) Stream -Scaled Reward Pruning, which comprehensively evaluates generated candidates to strike an optimal balance between local spatial aesthetics and global temporal coherence by integrating immediate short-term assessments with sliding-window-based long-term evaluations; (3) Stream-Scaled Memory Sinking, which dynamically routes the context evicted from KV-cache into distinct updating pathways guided by the reward feedback, ensuring that previously generated visual information effectively anchors and guides the subsequent video stream. Evaluated on both 5s and 30s comprehensive video benchmarks, Stream-T1 demonstrates profound superiority, significantly improving temporal consistency, motion smoothness, and frame-level visual quality.
1 Introduction
Stream-T1 introduces a comprehensive Test-Time Scaling framework for streaming video generation, exploiting chunk-by-chunk synthesis with minimal denoising steps to address the costs and temporal-guidance limitations of existing video TTS methods. Its three components improve temporal consistency, motion smoothness, and frame-level visual quality on 5s and 30s benchmarks.
- Concept: Stream-T1 pioneers Test-Time Scaling for streaming video generation by combining candidate-space expansion with active generation refinement.Streaming generation uses chunk-by-chunk autoregression and minimal denoising steps, such as 4 steps per chunk.
- Performance: Stream-T1 establishes new state-of-the-art performance on 5s and 30s video-generation benchmarks, improving temporal consistency, motion smoothness, and frame-level visual quality.Evaluations report superior long-term stability and visual fidelity relative to existing state-of-the-art baselines.
- Technology: Stream-Scaled Noise Propagation refines each chunk’s initial latent noise using historically proven, high-quality noise from previous chunks.This component establishes temporal dependency by leveraging historical generation information.
- Technology: Stream-Scaled Reward Pruning evaluates candidates to balance local spatial aesthetics with global temporal coherence, while Memory Sinking routes KV-cache updates using reward feedback.Memory Sinking preserves long-term semantics and guides the subsequent video stream.
2 Related Work
Related work frames test-time scaling as inference-time computation that searches over candidate outputs using feedback, while streaming video generation faces context overload and computational bottlenecks as duration increases. Existing streaming methods mitigate memory growth with heuristic context management but often encounter spatial-temporal trade-offs.
- Test-Time Scaling: Test-time scaling boosts pretrained-model performance by directly increasing the computational budget during inference.Existing methods primarily formulate this process as search over multiple candidates, using feedback to select optimal samples.
- Test-Time Scaling: ImagerySearch dynamically adjusts the inference search space and reward function according to semantic relationships in the prompt.
- Streaming Generation: Longer streaming videos require continuous accumulation of historical information, causing severe context overload and computational bottlenecks.Existing approaches rely heavily on heuristic context management strategies to mitigate memory explosion.
- Streaming Generation: Heuristic context management in streaming generation frequently incurs a severe spatial-temporal trade-off.
3 Methodology
Stream-T1 applies test-time scaling to streaming video generation through three sequential stages: conditional noise initialization, reward-guided candidate pruning, and adaptive memory sinking. These mechanisms connect current chunks to historical noise and context while jointly evaluating spatial quality and temporal coherence.
- Overall Pipeline: Stream-T1 uses three sequential stages for each autoregressive chunk: noise propagation before synthesis, reward pruning after synthesis, and adaptive memory sinking after pruning.The framework is built upon LongLive and employs beam search to expand the candidate space.
- Stream-Scaled Reward Pruning: Beam search maintains K candidates, expands each into M alternative next chunks, and evaluates the resulting K × M candidates with complementary short and long rewards.The short score assesses frame-level image quality, while the long score uses a sliding window and video reward model to assess text alignment, visual quality, and motion coherence.
- Stream-Scaled Memory Sinking: Stream-Scaled Memory Sinking dynamically updates historical context through reward-guided memory pathways to preserve semantic alignment and temporal coherence across the video stream.The mechanism alternates among discarding, EMA, and appending; reward thresholds govern whether stored KV-cache quality is sufficient and whether a long-score drop signals a scene or action change.
4 Experiments · 4.1 Experiments Settings
Stream-T1 is evaluated on LongLive, built on Wan2.1-T2V-1.3B, across short 5s and long 30s video-generation benchmarks. The experiments use dual-level reward evaluation for frame-level aesthetics and long-sequence video quality.
- 4.1.1 Implementation Details: Stream-T1 is evaluated on LongLive, which is built on Wan2.1-T2V-1.3B.The evaluation follows LongLive’s setup.
- 4.1.1 Implementation Details: The initial KV-cache configuration uses an attention window size of 9 and a sink size of 3.This is the initial KV-cache strategy used in the pipeline.
- 4.1.1 Implementation Details: The feedback pipeline uses dual-level evaluation to assess generated video candidates.Short-sequence and long-sequence evaluations use different reward-model families.
- 4.1.1 Implementation Details: Short-sequence evaluation uses HPSv3, ImageReward, and MHP to assess frame-level spatial aesthetics and visual fidelity.These are established image reward models used for short sequences.
- 4.1.2 Evaluation Benchmark: The experiments cover 5s short-video generation and 30s long-video generation.This tests the method on both short and long streaming-video durations.
- 4.1.2 Evaluation Benchmark: The 5s benchmark uses 946 VBench prompts, while the 30s benchmark uses the first 128 MovieGen prompts.The resulting videos are evaluated with VBench and VBench-long, respectively.
- 4.1.2 Evaluation Benchmark: Evaluation measures Subject Consistency, Background Consistency, Motion Smoothness, and Aesthetic Quality.VBench evaluates the 5s videos, and VBench-long evaluates the 30s videos.
4.2 Comparison with state-of-the-art methods
Stream-T1 outperforms representative open-source baselines on 5s and 30s video generation, achieving strong quality across spatial, temporal, motion, and human-aligned metrics. It also maintains long-term spatiotemporal coherence as video length increases, avoiding the severe degradation seen in existing methods.
- 5s video generation: On 5s clips, Stream-T1 ranks first on six quality metrics and second on Imaging Quality and VQ.The first-place metrics are Subject Consistency, Background Consistency, Motion Smoothness, Aesthetic Quality, MQ, and TA.
- 30s video generation: On 30s videos, Stream-T1 achieves the highest scores on five metrics, best VQ and TA, and second-place MQ.The five highest-scoring metrics are Subject Consistency, Background Consistency, Motion Smoothness, Imaging Quality, and Aesthetic Quality.
- Long-term stability: As video length increases, CausVid and Self-Forcing suffer severe frame-level distortion, while LongLive experiences a drastic drop in temporal consistency.LongLive mitigates spatial distortion to some extent but loses temporal consistency in long-sequence generation.
- Long-term stability: In contrast, Stream-T1 consistently maintains high spatiotemporal coherence and superior visual quality during long-sequence generation.The comparison describes Stream-T1 as demonstrating remarkable long-term stability despite increasing video length.
4.3 Comparison with Test-Time Scaling
Stream-T1 contrasts with Best-of-N and Beam Search by replacing passive candidate selection with active optimization of video generation. On 30-second video generation, it achieves state-of-the-art performance across all evaluative metrics.
- Comparison with Test-Time Scaling: Stream-T1 is compared against standard test-time scaling algorithms, specifically Best-of-N and Beam Search.These methods address generation errors through redundant candidate sampling or iterative exploration.
- Comparison with Test-Time Scaling: Best-of-N and Beam Search use a passive selection paradigm that chooses among outcomes from a fixed generated pool.Their strategy relies on redundant candidate sampling or iterative exploration rather than directly modifying generation.
- Comparison with Test-Time Scaling: Stream-T1 adopts active optimization by dynamically refining both latent noise and context memory during video generation.The framework actively guides generation instead of merely expanding the search space.
- Comparison with Test-Time Scaling: Stream-T1 achieves state-of-the-art performance across all evaluative metrics for 30s video generation.This result is reported in Table 3’s quantitative comparison with test-time scaling methods.
4.4 Ablations
Ablation studies on 30s video generation using 128 MovieGen prompts show that all three Stream-T1 components are necessary for high-quality, stable, and temporally coherent videos. Removing individual modules produces distinct degradations in background stability, structural fidelity, semantic alignment, or overall metrics.
- Experimental setup: Ablations on 30s generation with the first 128 MovieGen prompts quantitatively and qualitatively validate the necessity of all proposed components.The study reports comprehensive quantitative metrics and extensive qualitative visual comparisons.
- Component ablations: Removing Stream-Scaled Memory Sinking degrades background stability and severely compromises Subject and Background Consistency despite a noticeable Imaging Quality gain.This deterioration validates dynamic context management as crucial for long-term temporal coherence over extended sequences.
- Component ablations: Removing Stream-Scaled Noise Propagation introduces local structural artifacts and causes a uniform performance drop across all metrics.The qualitative example includes artifacts on the subject’s tail, supporting its role in overall stability.
- Component ablations: Removing Stream-Scaled Reward Pruning causes semantic misalignment and deteriorated aesthetic quality while marginally increasing Imaging Quality and drastically declining all other metrics.The results explicitly emphasize the necessity of reward pruning for high-quality video generation.
5 Conclusion
Stream-T1 is a novel test-time scaling framework tailored for streaming video generation. It combines historical noise refinement with reward-based evaluation of short-term spatial and long-term temporal quality.
- Stream-T1 is a novel TTS framework tailored for streaming video generation.
- Stream-Scaled Noise Propagation refines each generating chunk’s initial latent noise using historically proven, high-quality noise from previous chunks.
- Stream-Scaled Reward Pruning comprehensively evaluates generated outputs by fusing short-term spatial and long-term temporal metrics.