Temporally Consistent Transformers for Video Generation

1. Introduction

• Short-context models can generate long videos with sliding windows, but revisited locations may receive inconsistent content instead of reproducing what was previously observed.A suitable model must also keep newly imagined, unobserved regions consistent in future predictions.Source: §1 · p. 1
• Existing methods struggle to scale to long-range dependencies because recurrence increases training time and attention over long latent sequences has quadratic complexity.Long videos can contain tens of thousands of tokens, making full attention expensive.Source: §1 · pp. 1–2
• Prior evaluations often use short-term settings or FVD, which emphasizes image fidelity rather than long-range temporal dependencies.This leaves temporal consistency insufficiently assessed.Source: §1 · p. 2
• The paper proposes three video datasets with long-range dependencies, including rendered walks through DMLab, Minecraft, and Habitat scenes.These datasets are designed to support evaluation of long-horizon temporal consistency.Source: §1 · p. 2
• TECO accurately incorporates long-range context and outperforms state-of-the-art baselines across diverse datasets while maintaining fast sampling.The evaluation targets long-horizon video prediction and temporal consistency.Source: §1 · p. 2
• TECO learns compressed representations for efficient training and generation, enabling high-quality, temporally consistent predictions with long-term context.The paper benchmarks current models and analyzes their ability to learn long-horizon dependencies.Source: §1 · p. 2

2.1. VQ-GAN

• VQ-GAN compresses images into discrete latent representations using an encoder, decoder, codebook, and discriminator.The encoder quantizes representations through nearest-neighbor lookup, and the decoder reconstructs the input.Source: §2.1 · pp. 2–3
• Prior video models apply a single spatio-temporal transformer over all VQ codes, whose quadratic attention cost becomes prohibitive for long sequences.TECO instead downsamples spatially before temporal attention and expands representations with a per-frame spatial MaskGit.Source: Figure 3 · p. 3
• VQ-GAN uses perceptual reconstruction loss and a patch-level discriminator to encourage higher-fidelity reconstructions.Its objective replaces the original ℓ2 reconstruction loss with LPIPS-based perceptual loss.Source: §2.1 · p. 3

2.2. MaskGit

• MaskGit models distributions over discrete tokens with a masked-token prediction objective during training.During inference, it can sample arbitrary token subsets iteratively, including through confidence-based sampling.Source: §2.2 · p. 3

3. TECO

• TECO efficiently scales video generation to longer-horizon sequences through a compressed representation and temporal modeling architecture.The model is intended to retain high-capacity scaling while making training and inference more efficient.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1. Architectural Overview

• The encoder conditions each frame on the previous frame, quantizes its representation, and uses temporal aggregation to learn compressed codes.The first frame is concatenated with zeros and is not quantized to prevent information loss.Source: §3.1 · p. 4
• A strided convolution downsamples discrete latents before a temporal transformer models dependencies, then a transposed convolution restores their original resolution.Downsampling is adjusted according to dataset visual complexity.Source: §3.1 · p. 4
• The decoder reconstructs frames from latent codes and temporal summaries, while a spatial MaskGit models the latent prior for faster and higher-quality sampling.During generation, frames are produced in chunks of 8 followed by two revise rounds that regenerate half the tokens each time.Source: §3.1 · p. 4

3.2. DropLoss

• DropLoss computes prior and reconstruction losses on only 10% of randomly selected timesteps, omitting decoder and dynamics-prior computation for the others.Random resampling across iterations still exposes all timesteps during training and reduces computational cost significantly.Source: §3.2 · p. 4, Figure 4 · p. 4
• TECO remains more temporally consistent than state-of-the-art models over hundreds of predicted timesteps, according to LPIPS comparisons against ground-truth frames.The comparison excludes conditioning frames and begins at the first predicted frame.Source: Figure 5 · p. 5

4.1. Datasets

• The experiments use three long-range 3D-environment datasets—DMLab, Minecraft, and Habitat—where models must reproduce observed scenes and generate unobserved regions.Kinetics-600 is additionally used to test scaling to complex natural video without actions.Source: §4.1 · p. 5

4.2. Baselines

• Baselines span variational, autoregressive, and diffusion families, with models trained on latent codes under matched compute budgets.Space-time transformer comparisons include VideoGPT, Phenaki, MaskViT, Hourglass transformers, and Perceiver AR; Latent FDM provides the diffusion baseline.Source: §4.2 · pp. 5–6, §4.3 · p. 6

Evaluation

• The evaluation modifies PSNR, SSIM, and LPIPS to measure temporal consistency using sufficient conditioning, while FVD assesses stochastic video quality separately.For DMLab, Minecraft, and Habitat, models condition on 144 past frames and actions before predicting 156 future ground-truth frames.Source: Evaluation · pp. 6–7

4.4. Benchmark Results

• TECO performs best across all metrics on DMLab and Minecraft, while achieving significantly better FVD on Habitat than the evaluated baselines.On DMLab, TECO also produces more consistent 3D mazes; Habitat models perform similarly poorly on per-frame metrics because of scene complexity.Source: §4.4 · pp. 7–8
• TECO produces more stable Kinetics-600 generations that degrade more slowly than Perceiver AR, with Latent FDM performing between them.Kinetics-600 tests scaling to complex natural video despite containing relatively few long-range dependencies.Source: §4.4 · p. 8
• TECO samples 20–60x faster than Perceiver AR and Latent FDM while retaining comparably fast sampling and high sample quality.FitVid and CW-VAE are faster but have poor sample quality, whereas Perceiver AR and Latent FDM produce high-quality samples more slowly.Source: §4.4 · p. 8

4.5. Ablations

• Ablations favor VQ-latent dynamics with a MaskGit prior and conditional encodings, while DropLoss enables up to 60% faster training with minimal FVD increase.The DropLoss benefit is greater for longer sequences and helps models account for long-horizon context with little performance cost.Source: §4.5 · p. 8

4.6. Further Insights

• A representation bottleneck prioritizes long-horizon information, and longer training improves long-term consistency after short-horizon metrics saturate.TECO’s compact representations improve the fidelity–temporal-consistency trade-off, while long-term metrics continue improving through training.Source: §4.6 · p. 8

5. Discussion

• PSNR, SSIM, and LPIPS remain useful for conditioned consistency, but better metrics are needed as prediction horizons grow and newly generated regions diverge from ground truth.The paper identifies this as an evaluation limitation because unobserved scene content is unlikely to correlate with ground truth.Source: §5 · p. 8
• Using a full-attention transformer limits further scaling, and efficient sequence models are identified as a likely route to larger systems.The limitation concerns the sequence model rather than the compressed-representation or dynamics components.Source: §5 · p. 9
• VQ-GAN compression enables longer training sequences but introduces reconstruction artifacts, including corrupted text and incoherent faces in Kinetics-600.Direct pixel training avoids that compression step but produces slightly blurry predictions under an ℓ2 loss.Source: §5 · p. 9

A. Sampling Process

• TECO samples autoregressively in latent space: a temporal transformer produces representations, MaskGit predicts future latents, and VQ-GAN decodes the completed trajectory to RGB.RGB rendering can be performed in parallel after all latent timesteps have been computed.Source: A. Sampling Process · p. 14

B.1. DMLab

• The supplementary figures show 156 frames generated conditioned on 144 and 264 frames conditioned on 36, including action-conditioned and no-action-conditioning examples, plus 3D visualizations of generated DMLab mazes.Source: Figure B.1 · p. 15, Figure B.2 · p. 15, Figure B.3 · p. 16

B.4. Kinetics-600

• The supplementary Kinetics-600 figures compare 80-frame generations conditioned on 20 frames with and without top-k sampling.The captions specify the sampling condition but do not state an outcome comparison.Source: Figure B.8 · p. 19, Figure B.9 · p. 19

C. Performance versus Horizon

• TECO shows much better full-trajectory temporal consistency than competing models on DMLab and Minecraft, while all methods struggle on Habitat.Latent FDM is generally second on DMLab and close to TECO on Minecraft; Habitat remains difficult for every method.Source: C. Performance versus Horizon · p. 20

D. Performance versus Training Sequence Length

• TECO scales better to longer training sequences on DMLab, whereas Latent FDM struggles on full sequences.The paper links TECO’s improvement to its ability to use longer temporal context under the same compute budget.Source: Figure D.1 · p. 21, D. Performance versus Training… · p. 22
• Under fixed compute and batch size, shorter training videos support larger models and higher per-image quality but reduce temporal consistency through shorter context.The result is framed as a trade-off between model capacity or FLOPs per image and available temporal context.Source: D. Performance versus Training… · p. 22
• Minecraft achieves its best balance near 50–100 training frames, combining higher-fidelity image predictions with sufficient temporal context.The reported optimum is specific to the Minecraft experiments.Source: D. Performance versus Training… · p. 22, Figure D.2 · p. 22

E. Sampling Time

• The section includes sampling-time analysis, with the reported measure labeled sampling time per frame in milliseconds.The supplied passage identifies the metric but does not provide comparative values.Source: E. Sampling Time · p. 23

F. Ablations

• The ablation tables cover alternative prior, posterior, and codebook designs, as well as scaling different parts of TECO.The supplied table captions specify these scopes but not the corresponding numerical outcomes.Source: Table F.1 · p. 23, Table F.2 · p. 24
• Removing latent dynamics produces worse temporal consistency and overall sample quality than TECO in the DMLab comparisons.The comparison includes architectures that directly predict VQ-GAN latents, including MaskGit and autoregressive variants.Source: F. Ablations · p. 24, Table F.3 · p. 24
• TECO’s latent-dynamics design is hypothesized to improve consistency by avoiding excessive modeling of imperceptible image and video statistics.MaskGit also exhibits very high FVD because it tends to collapse in later prediction frames.Source: F. Ablations · p. 24

G. Metrics During Training

• FVD saturates around 200k iterations, while LPIPS continues improving through 1M iterations, indicating that long-horizon consistency develops after frame fidelity.The paper hypothesizes that models first learn locally temporal patterns before learning longer-horizon dependencies.Source: G. Metrics During Training · p. 25, Figure G.1 · p. 25
• The training curves evaluate FVD on generated 16-frame chunks and LPIPS at checkpoints every 50k iterations over 1M iterations.This setup distinguishes the early-saturating fidelity metric from the longer-improving consistency metric.Source: G. Metrics During Training · p. 25

H. High Quality Spatio-Temporal Compression

• At the same compression rate, TECO learns substantially better spatio-temporal codes than TATS, especially on visually complex Minecraft scenes.The comparison is reported through reconstruction FVD.Source: H. High Quality Spatio-Temporal… · p. 25, Table H.1 · p. 25

I. Trade-off Between Fidelity and Learning Long-Range Dependencies

• Video prediction architectures exhibit a trade-off between frame or image fidelity and temporal consistency across long-range dependencies.The supplied passages identify this comparison for TECO and Latent FDM without reporting the table values.Source: §I · p. 26, Table I.1 · p. 26, Table I.2 · p. 26

• Tables J.1–J.4 organize the full results by DMLab, Minecraft, Habitat, and Kinetics-600.Source: Table J.1 · p. 27, Table J.2 · p. 27, Table J.3 · p. 27
• Figure J.1 compares FVD on Kinetics-600 across different top-k values for Perceiver-AR and TECO.Source: Figure J.1 · p. 28

K. Scaling Results

• Tables K.1 and K.2 report scaling results for DMLab and Minecraft.Source: Table K.1 · p. 29, Table K.2 · p. 29

L. Related Work

• Prior video-generation work includes variational, likelihood-based, diffusion, autoregressive, flow, and GAN-based models.Source: L. Related Work · p. 30
• The paper focuses on video prediction as a conditional interpretation of video generation, with autoregressive methods naturally extending to prediction.Source: L. Related Work · p. 30
• Long-horizon generation methods use hierarchical latents, sliding windows, strided sampling, continuous-time representations, diffusion conditioning, or hierarchical prediction.Source: L. Related Work · p. 30
• Long-horizon video-understanding work introduces architectures for modeling extended dependencies in videos.Source: L. Related Work · p. 30

M.1. DMLab

• DMLab uses 40k noisy 300-frame trajectories from random 7 × 7 mazes with discrete turning and forward actions.The dataset also records depth, camera intrinsics, and camera extrinsics for 3D visualization, while RGB-only evaluation uses learned depth and pose estimators.Source: §M.1 · p. 31
• GQN Mazes are difficult for measuring temporal consistency because their scenes are simple and lack actions that reduce metric stochasticity.The paper therefore uses its dataset for 3D visualizations that GQN Mazes cannot provide.Source: §M.1 · p. 31

M.2. Minecraft

• Minecraft contains 200k trajectories of 300 128 × 128 frames from distinct marsh worlds traversed with randomized movement and constant jumping.The marsh biome was selected for landmark-rich hilly turns, unlike denser, flatter, or harder-to-traverse alternatives.Source: §M.2 · p. 31

M.3. Habitat

• Habitat supplies 200k trajectories of 300 128 × 128 frames rendered from roughly 1400 scans across HM3D, MatterPort3D, and Gibson.Agents move between randomly sampled scene locations using left-turn, right-turn, and forward actions.Source: §M.3 · p. 31

N.2. TECO

• TECO hyperparameters are tabulated for the main setup and separate DMLab and Minecraft scaling experiments.Source: §N.2 · p. 32, Table N.1 · p. 33, Table N.2 · p. 34
• The listed training configuration uses cosine learning-rate schedules, 10k warmup steps, 1M total steps, and a 0.9 DropLoss rate.Source: §N.2 · p. 32

N.3. Latent FDM

• Latent FDM has a general hyperparameter table plus separate DMLab and Minecraft scaling tables.Source: Table N.3 · p. 35, Table N.4 · p. 35, Table N.5 · p. 36

N.4. CW-VAE

• CW-VAE and FitVid each have dedicated hyperparameter tables.Source: Table N.6 · p. 37, Table N.7 · p. 37