Source-linked AI summary
One-Minute Video Generation with Test-Time Training
Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, Tatsunori Hashimoto, Sanmi Koyejo, Yejin Choi, Yu Sun, Xiaolong Wang
TL;DR
Long-context video generation remains difficult because self-attention is inefficient and recurrent alternatives have less expressive hidden states. The paper adds Test-Time Training layers to a pre-trained Transformer and finds more coherent multi-scene videos than several baselines, while reporting efficiency and artifact limitations.
Problem
One-minute, complex multi-scene video generation requires long contexts that make self-attention inefficient, while Mamba- and DeltaNet-like alternatives struggle to retain distant relationships.
Method
The paper adds TTT layers with neural-network hidden states to pre-trained CogVideo-X 5B, fine-tuning the model on a Tom and Jerry storyboard dataset.
Results
TTT layers generate more coherent complex-story videos than Mamba 2, Gated DeltaNet, and sliding-window attention, leading by 34 Elo points in human evaluation.
Takeaways & Limitations
The results support TTT layers as a promising proof-of-concept approach for long-context, multi-scene video generation within the tested domain.
Takeaways & Limitations
TTT-MLP inference and training are 1.4× and 2.1× slower than Gated DeltaNet, and generated 63-second videos still contain notable artifacts.
Abstract
from arXiv · showhide
Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: https://test-time-training.github.io/video-dit
1. Introduction
The paper targets one-minute, multi-scene video generation, where self-attention is costly for long contexts and existing recurrent alternatives struggle to retain distant relationships. It adds expressive Test-Time Training layers to a pre-trained video Transformer and reports more coherent complex stories than linear-complexity baselines.
- Motivation: Public video-generation APIs produce clips of at most 20 seconds and cannot autonomously generate complex multi-scene stories.The cited APIs include Sora at 20 seconds, MovieGen at 16 seconds, Ray 2 at 10 seconds, and Veo 2 at 8 seconds.
- Motivation: Over 300k tokens are required for each one-minute video, making self-attention substantially more expensive as context grows.Self-attention scales quadratically with context length; generating one minute would take 11× longer than generating twenty 3-second videos, with training taking 12× longer.
- Motivation: Mamba- and DeltaNet-like recurrent layers may lose distant relationships because their fixed-size matrix hidden states compress hundreds of thousands of vectors into limited rank.Their linear context cost comes with less expressive hidden states, motivating alternatives that can store richer representations.
- Approach: TTT layers use neural-network hidden states updated by training on test sequences, providing a more expressive recurrent alternative.The implementation uses two-layer MLP hidden states with 2× more hidden cells and richer nonlinearities than linear hidden states.
- Approach: The system adds scratch-initialized TTT layers to CogVideo-X 5B, fine-tunes it on text storyboards, and limits self-attention to 3-second segments.The starting model generated 3-second clips at 16 fps or 6-second clips at 8 fps; the training run used the equivalent of 50 hours on 256 H100s.
- Results: 34 Elo points separate TTT layers from Mamba 2, Gated DeltaNet, and sliding-window attention in human evaluations of 100 videos per method.The Tom and Jerry dataset contains approximately 7 hours of cartoons with human-annotated storyboards and emphasizes long, dynamic, multi-scene stories.
2. Test-Time Training Layers
TTT layers compress historical context into weights of a model trained online with a self-supervised loss. The paper instantiates this mechanism with projections, an MLP-based inner model, and outer-loop optimization of the task parameters.
- TTT as Updating a Hidden State: RNN layers map an input sequence to outputs while maintaining a fixed-size hidden state, trading efficient per-token processing against limited long-context storage.TTT aims to make that hidden state more expressive so it can compress massive context more effectively.
- TTT as Updating a Hidden State: TTT represents the hidden state W_t as the weights of a model f and updates W_t by gradient descent on a self-supervised loss.Historical tokens are treated as an unlabeled dataset, while the model weights serve as the state that accumulates context.
- Learning a Self-Supervised Task: The self-supervised loss can reconstruct a corrupted input, requiring f to learn correlations among dimensions from partial information.This denoising-autoencoder-like objective uses a corrupted input ˜x_t rather than directly reconstructing the unmodified token.
- TTT as Updating a Hidden State: TTT trains distinct weights W_1, ..., W_T for each input sequence even during inference, while exposing the same interface as RNN and self-attention layers.Optimization of W inside the layer is the inner loop; optimization of the larger network and task parameters is the outer loop.
- Learning a Self-Supervised Task: The self-supervised task is learned end-to-end using low-rank projections θ_K, θ_V, and θ_Q rather than being fully hand-designed.θ_K projects inputs for reconstruction, θ_V can define lower-dimensional reconstruction targets, and θ_Q supports the output rule; these projections are outer-loop parameters.
- TTT-MLP Instantiation: TTT-MLP wraps a two-layer Transformer-like MLP with Layer Norm and a residual connection, whereas TTT-Linear uses a linear inner model as a baseline.The default TTT-MLP inner model has hidden dimension 4× the input dimension and uses GELU activation.
3. Approach
The approach adds gated TTT layers to a pre-trained Diffusion Transformer and combines local self-attention over 3-second segments with global TTT processing. It extends generation through progressively longer Tom and Jerry videos while introducing GPU-oriented parallelization and memory optimizations.
- Architecture: The method adds TTT layers to a pre-trained Diffusion Transformer and fine-tunes it on long videos with text annotations.The starting checkpoint is CogVideo-X 5B.
- Sequence construction: The pipeline uses 3-second segments, applying self-attention locally within segments and TTT layers globally across the full sequence.Videos contain multiple scenes, with each scene composed of one or more 3-second segments.
- Architecture: A learnable gate controls each TTT layer, initially keeping its contribution small so randomly initialized layers do not significantly overwrite the pretrained representation.The gate uses tanh(α), with α initialized to 0.1.
- Architecture: Bidirectional TTT applies the layer to reversed sequences and reverses the result back, allowing TTT to operate within the non-causal Diffusion Transformer.The twice-reversed output remains in chronological order while the internal scan runs in reverse chronological order.
- Training: Context length is extended in five stages from 3-second segments to 9, 18, 30, and 63 seconds, while later stages update only selected model components.The later stages fine-tune TTT layers, gates, and self-attention layers with a lower learning rate.
- On-chip tensor parallelism: The implementation shards TTT-MLP hidden states across streaming multiprocessors and keeps updates on-chip, reducing transfers between HBM and SMEM.Hidden states and activations are transferred to and from HBM only during initial loading and final output.
4. Evaluation
The evaluation compares TTT-MLP with linear-complexity and local-attention baselines using blind pairwise human preferences on long, multi-scene videos. TTT-MLP leads overall, while efficiency and artifacts remain important limitations.
- Results: TTT-MLP preserves details across scene transitions and depicts complex actions more naturally than Gated DeltaNet and sliding-window attention.The accompanying examples compare TTT-MLP with the leading baselines from the human evaluation.
- Efficiency: 11× longer inference and 12× longer training would be required by full attention than local attention for 63-second videos, versus 2.5× and 3.8× for TTT-MLP.Gated DeltaNet is faster than TTT-MLP at 1.8× local-attention time for both inference and training.
- Evaluation protocol: The benchmark evaluates TTT-MLP and five baselines using blind pairwise preferences across four video-quality axes.The axes include text following, motion naturalness, aesthetics, and temporal consistency, with temporal consistency covering both within- and across-scene consistency.
- Limitations: 18-second videos favor Gated DeltaNet, which leads Mamba 2 by 27 Elo points and TTT-MLP by 28, showing that shorter contexts remain a limitation for TTT-MLP.Results for both 18- and 63-second videos also indicate that Gated DeltaNet improves meaningfully on Mamba 2.
- Limitations: TTT-MLP videos still contain artifacts including object morphing, unnatural floating, inconsistent lighting, and inaccurate parallax.These artifacts are common among all methods and may reflect limitations of the pre-trained CogVideo-X 5B model.
5. Related Work
Prior work uses recurrent and attention-based approaches for sequence and long-video modeling, while TTT builds more expressive hidden states by combining scalable recurrent processing with neural-network representations.
- Recurrent sequence models: Mamba and DeltaNet are modern recurrent or linear-attention variants that have demonstrated strong performance in natural-language tasks.TTT is inspired by their success and by Fast Weight Programmers.
- Long video modeling: Long-video methods include next-frame GANs, autoregressive and diffusion approaches, and Transformer sliding-window attention.Phenaki and TATS are cited as examples of autoregressive and sliding-window strategies for videos longer than the training length.
6. Future Work
Future work targets faster TTT-MLP kernels and better integration of TTT layers into pre-trained models.
- Faster implementation: TTT-MLP efficiency could improve by reducing register pressure and making asynchronous operations more compiler-aware.The current kernel is bottlenecked by register spills and suboptimal asynchronous-instruction ordering.
- Better integration: Bidirectional processing and learned gates are only one possible strategy for integrating TTT layers into a pre-trained model.The paper identifies better integration as a separate future-work direction.
A. Experiment Details
The experiments use CogVideoX-style diffusion training and sampling configurations, with specified optimization, precision, and TTT inner-loop settings.
- Diffusion schedule: Fine-tuning uses v-prediction with a 1000-step diffusion schedule and Zero-SNR enforced at the final step.These settings follow CogVideoX.
- Training configuration: Training uses AdamW, linear warmup over 2% of steps, batch size 64, gradient clipping 0.1, weight decay 10^-4, and mixed precision with PyTorch FSDP2.Text-prompt dropout is 0.1 and the VAE scale factor is 1.0.
- TTT configuration: The TTT inner-loop learning rate is η = 1.0 for TTT-Linear and η = 0.1 for TTT-MLP.The paper identifies this learning rate as a key TTT hyperparameter.
- Sampling configuration: Sampling uses DDIM with 50 steps, dynamic classifier-free guidance increasing from 1 to 4, and negative prompts.These choices are intended to further enhance video quality.
B. On-Chip Tensor Parallel Details
The implementation combines a fused TTT-MLP kernel with tensor-parallel hidden-state sharding and GPU latency optimizations. The surrounding setup also considers multiple storyboard prompt formats.
- ThunderKittens implements the TTT-MLP kernel used in the system.
- Hidden state sharding: Tensor parallelism shards the first layer column-wise and the second layer row-wise, requiring one reduction for the inner-loss update.The single reduction is enabled by the elementwise GeLU nonlinearity.
- The study discusses three storyboard prompt formats, progressing from plot summaries to segment descriptions and detailed storyboards.The formats increase in detail and are illustrated in Figure 8.