PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Long Videos of Dynamic Scenes

Brooks, Tim, Hellsten, Janne, Aittala, Miika, Wang, Ting-Chun, Aila, Timo, Lehtinen, Jaakko, Liu, Ming-Yu, Efros, Alexei A., Karras, Tero

arXiv:2206.03429v2cs.CVcs.AIcs.LGcs.NE

TL;DR

Long-video generation must produce new content over time while preserving physically plausible motion and object consistency. This paper introduces a hierarchical, time-prioritized generator trained across long low-resolution and short high-resolution videos, achieving state-of-the-art long-video generation with realistic motion and content changes.

  • Problem

    Realistic long-video generation requires producing new content over time while maintaining physically plausible motion, viewpoint changes, and object persistence.

  • Method

    The model uses a hierarchical generator with a temporal embedding and large receptive field, training long low-resolution videos before high-resolution super-resolution refinement.

  • Results

    The model achieves state-of-the-art performance in generating long videos with realistic motion and changes in content.

  • Takeaways & Limitations

    The approach produces realistic new content over time rather than repeatedly returning to the same scene.

  • Takeaways & Limitations

    The simple super-resolution network is adequate but could be replaced by more advanced video super-resolution models.

Abstract

from arXiv · show

We present a video generation model that accurately reproduces object motion, changes in camera viewpoint, and new content that arises over time. Existing video generation methods often fail to produce new content as a function of time while maintaining consistencies expected in real environments, such as plausible dynamics and object persistence. A common failure case is for content to never change due to over-reliance on inductive biases to provide temporal consistency, such as a single latent code that dictates content for the entire video. On the other extreme, without long-term consistency, generated videos may morph unrealistically between different scenes. To address these limitations, we prioritize the time axis by redesigning the temporal latent representation and learning long-term consistency from data by training on longer videos. To this end, we leverage a two-phase training strategy, where we separately train using longer videos at a low resolution and shorter videos at a high resolution. To evaluate the capabilities of our model, we introduce two new benchmark datasets with explicit focus on long-term temporal dynamics.

1 Introduction

The paper targets long-video generation with realistic motion, viewpoint changes, and newly appearing content while preserving long-term consistency. It introduces datasets and a hierarchical, multi-resolution generator with a temporal embedding, achieving state-of-the-art performance on these goals.

  • 1 Introduction: Real-world videos require unbounded new content to evolve over time while remaining consistent with feasible physical dynamics.Relevant constraints include smooth camera movement and persistent environmental structure.
  • 1 Introduction: Existing video generators often limit temporal change, producing little new content despite nominally supporting infinite videos.Short-clip datasets may reinforce this limitation by containing little new content over time.
  • 1 Introduction: The paper introduces two datasets featuring motion, changing camera viewpoints, and objects or scenery entering and leaving over time.The datasets support learning long-term consistency from videos containing richer temporal dynamics.
  • 1 Introduction: Its hierarchical generator uses a large temporal receptive field, a novel temporal embedding, and separate low-resolution generation and super-resolution refinement.The multi-resolution strategy makes long-video training practical because major video content persists at low spatial resolution.
  • 1 Introduction: The method demonstrates state-of-the-art performance in generating long videos with realistic motion and changes in content.The authors compare against several recent video generative models.

2 Prior work

Prior video-generation methods commonly rely on stationary-content assumptions, short videos, or autoregressive dynamics that accumulate errors over time. Recent approaches span GANs, transformers, conditional prediction, and multi-resolution training, while this work emphasizes temporal representation and arbitrarily long generation.

  • Unconditional video generation: Earlier texture-based and subsequent video-synthesis methods often generate little or no new objects entering the frame, reflecting limitations from stationary assumptions and short training videos.Classic approaches model videos as textures, while later methods retain related limitations despite dropping the texture terminology.
  • Unconditional video generation: GAN-based methods include fixed-length generators and recurrent latent-code models, while StyleGAN-based approaches struggle to produce realistic motion.The paper compares with MoCoGAN-HD and StyleVideoGAN, and also considers DIGAN’s implicit pixel-wise video representation.
  • Unconditional video generation: Transformer video generators such as TATS and VideoGPT represent videos as token sequences, but can accumulate errors, diverge, or change too rapidly over time.These models are also described as expensive to train and deploy.
  • Conditional video prediction: Conditional video prediction forecasts frames from observed frames, actions, camera trajectories, or depth, whereas this work targets a different generation setting.Camera-conditioned methods specifically generate infinite scenery by simulating a virtual camera through a 3D scene.
  • Multi-resolution training: Multi-resolution video training has precedent, but this work designs its approach to enable arbitrarily long videos rather than fixed-length outputs.Prior multi-scale methods increase spatial resolution and sequence length during training to produce fixed-length videos.

3 Our method

The method prioritizes long-term temporal modeling through a two-stage generator: a low-resolution network captures motion and scene composition over long sequences, while a frame-based super-resolution network adds spatial detail. Its temporal representation and architecture provide broad receptive fields and flexible video duration control.

  • 3 Our method: The low-resolution generator is fully convolutional over time, so shifting and reshaping temporal noise control the generated video’s time offset and duration.The super-resolution network instead processes a short temporal neighborhood frame by frame, receiving nine consecutive low-resolution frames to produce one high-resolution frame.
  • 3 Our method: The generator separates long-term motion and scene composition from fine spatial detail using a low-resolution generator followed by a super-resolution network.The low-resolution stage produces 64^2 frames, which the super-resolution stage refines to 256^2 output.
  • 3.1 Low-resolution generator: Training long sequences increases overfitting risk, so strong discriminator augmentation is required to stabilize low-resolution generator training.The authors attribute the difficulty to simultaneously modeling dynamics at multiple time scales while making discriminator errors easier to detect.
  • 3.1 Low-resolution generator: The low-resolution generator uses temporal noise, long training sequences, and a 5k-frame receptive field to model correlations across multiple time scales.It generates 128-frame sequences, corresponding to 4.3 seconds at 30 fps, and its architecture combines temporal latent design, spatiotemporal convolutions, and temporal upsampling.
  • 3.1 Low-resolution generator: Lowpass filtering enriches per-frame temporal noise with context spanning 100–5000 frames, enabling intermediate latents to capture long-term changes and persistent objects.The enriched representation is formed from 128 lowpass-filtered channels before frame-wise mapping.
  • 3.2 Super-resolution network: The super-resolution network is a straightforward StyleGAN3 extension that generates one frame using the current low-resolution frame and four neighboring frames as temporal input.It is trained separately from the low-resolution generator, and the authors note that more advanced video super-resolution models remain future work.

4 Datasets

The paper introduces two challenging first-person video datasets featuring motion, moving viewpoints, and new content over time, complementing existing datasets with limited temporal change. It also evaluates on ACID and SkyTimelapse, which isolate camera motion or evolving content but have important limitations.

  • Dataset limitations: Existing datasets often provide limited new content over time, with talking-head videos retaining the same person and UCF101 offering short videos with little camera motion or object entry.These limitations motivate datasets designed to evaluate long-term temporal dynamics.
  • New datasets: Two new first-person mountain-biking and horseback-riding datasets feature subject motion, moving camera viewpoints, and new scenery and objects over time.The high-definition videos were manually trimmed to remove problematic segments, including scene cuts, text overlays, and obstructed views.
  • Existing evaluation datasets: ACID supplies significant camera motion without other motion, while SkyTimelapse supplies evolving cloud content but remains relatively homogeneous with a fixed camera.Both datasets provide complementary evaluation settings for specific aspects of temporal change.

5 Results

The model improves long-term video realism by generating evolving content and plausible motion, outperforming StyleGAN-V on complex-motion datasets despite limitations of standard FVD. Analyses show that long-video training and temporal filter design are important, while super-resolution generally preserves quality.

  • 5.1 Qualitative results: Our model generates realistic new content over time, whereas StyleGAN-V repeatedly returns to the same scene; users preferred our method over 80% of the time.The preference study used 50 video pairs per dataset across four datasets, with 10 participants judging each pair.
  • 5.3 FVD: FVD128 favors our model over StyleGAN-V on horseback riding and mountain biking, but StyleGAN-V performs better on ACID and slightly better on SkyTimelapse.The authors attribute this discrepancy to StyleGAN-V’s stronger individual-frame realism and note that FVD misses long-term repetition.
  • 5.4 Ablations: Training on longer videos consistently improves long-video generation by helping the model learn long-term consistency, provided its architecture has a suitable temporal receptive field.The benefit became evident only after the generator was designed to use the richer long-video training signal.
  • 5.4 Ablations: The temporal lowpass-filter footprint is important for learning correlations at appropriate time scales, while the super-resolution network generally performs well and is not the main quality bottleneck.Changing the filter footprint by an order of magnitude hurts performance, and real low-resolution inputs yield strong super-resolution FVD.

6 Conclusions · A Additional results · A.1 User study

The paper identifies multiscale temporal representations and longer-sequence training as useful for complex motion and new content, while highlighting unresolved challenges in video quality, evaluation, and societal impact. A user study provides an additional human realism signal comparing the method with StyleGAN-V across four datasets.

  • 6 Conclusions: Multiscale temporal representations help model complex motions and introduce new content over time, while the most suitable latent space for video remains unresolved.The authors specifically identify persistent objects that exit and later re-enter while retaining identity as an open challenge.
  • 6 Conclusions: Longer-sequence training motivates exploring even longer inputs, larger frame spacings, and augmentations that reduce overfitting.The current training uses adjacent input frames, with larger spacing proposed to cover longer temporal contexts.
  • 6 Conclusions: Separating low- and super-resolution networks improves computational feasibility but may compromise high-resolution quality through RGB-bottleneck artifacts.The authors attribute some visible “swirly” artifacts to this bottleneck and suggest integrating advanced video super-resolution or additional low-resolution-generator features.
  • 6 Conclusions: FVD is insufficient for long-term dynamics because it can overlook repetitive and implausible results, motivating targeted evaluation beyond color and LPIPS trajectories.The authors consider their color- and LPIPS-over-time tests only a partial bridge toward more comprehensive evaluation.
  • 6 Conclusions: The work highlights misuse risks and high energy requirements for data-driven video generation, consuming 300MWh on an in-house V100 and A100 GPU cluster.The authors frame these concerns as potential negative societal impacts accompanying increasing generative quality.
  • A.1 User study: A user study compares motion realism between the proposed method and StyleGAN-V, providing human feedback not captured by FVD, which does not favor the method on every dataset.The study is described as relatively small and as measuring only one aspect of video quality.
  • A.1 User study: The study presents 50 video pairs per dataset, each containing one random video from each method, and asks participants to choose the video with more realistic motions.The forced-choice prompt instructed participants to view both videos before selecting the more realistic motion.
  • A.1 User study: Each pair received judgments from 10 participants, producing 500 responses per dataset from highly approved, experienced Mechanical Turk workers.Workers had approval ratings above 95% and more than 1000 completed jobs.

A.2 Qualitative results

Qualitative comparisons show that the proposed method generates evolving content, realistic motion, and smooth camera changes, while baseline methods often repeat content, move implausibly, or deteriorate over time.

  • A.2 Qualitative results: StyleGAN-V repeatedly replays the same content instead of generating new content, while the proposed model generates new clouds and weather over time.On SkyTimelapse, clouds change between 1s and 2s but return to the same clouds at 5s under StyleGAN-V.
  • A.2 Qualitative results: The proposed method generates realistic motion and scenery changes, including a biker leaving the woods over 10 seconds, unlike StyleGAN-V.The mountain-biking example requires both complex motion and changing environmental context.
  • A.2 Qualitative results: The model learns smooth camera-viewpoint trajectories, whereas StyleGAN-V produces pulsating motion and cannot create a smooth trajectory.The example depicts rotation while moving forward through 3D space.
  • A.2 Qualitative results: On SkyTimelapse, MoCoGAN-HD quickly explodes, TATS changes too rapidly, and StyleGAN-V repeatedly replays content over long durations.MoCoGAN-HD starts with a realistic frame, while TATS produces plausible short segments before changing too rapidly.

A.3 Analyzing change over time in feature spaces · A.4 Image quality tradeoff

The analysis finds that our method generally matches real-video rates of change better than StyleGAN-V, while StyleGAN-V achieves better per-frame image quality on most datasets. Together, the evaluations expose a tradeoff between image quality and faithful long-term dynamics.

  • A.3 Analyzing change over time in feature spaces: Color-histogram intersection measures similarity across increasing frame spacings as a spatially agnostic probe of how quickly generated videos change.The authors favor it as a simple approximation rather than a standalone metric.
  • A.3 Analyzing change over time in feature spaces: Our method generally matches real-video rates of change better than StyleGAN-V, which changes too slowly on horseback, ACID, and SkyTimelapse.Mountain biking is an exception under perceptual similarity: both methods change too much, and StyleGAN-V is closer to the dataset curve.
  • A.4 Image quality tradeoff: StyleGAN-V outperforms our model on per-frame image quality on three of four datasets, whereas our model prioritizes accurate change over time.Table 4 reports video-balanced FIDV, for which lower is better.
  • A.4 Image quality tradeoff: Per-frame image quality and realistic long-duration motion involve a practical tradeoff, motivating separate evaluation of image quality and temporal change.The model prioritizes accurate changes over long durations, while many video models prioritize frame quality.
  • A.4 Image quality tradeoff: The evaluation combines qualitative results, user studies, color-change analysis, FVD, and FID to reveal complementary strengths and weaknesses of video-generation methods.No single evaluation captures every aspect of generated-video quality.
  • A.4 Image quality tradeoff: Video-balanced FIDV evaluates per-frame image quality while preventing very long dataset videos from dominating the result.This is especially important for SkyTimelapse, which contains an extremely long outlier video.
  • A.4 Image quality tradeoff: FIDV weights each clip’s covariance and mean inversely by its frame count, valuing videos equally while retaining contributions from all frames.The weighting uses the Wasserstein-2 distance between feature sets.

B Dataset details … C.2 Temporal lowpass filters

The paper evaluates on four datasets, including newly introduced long-form horseback-riding and mountain-biking collections, and details low-resolution training augmentations and temporal filters. These choices support learning long-term dynamics while addressing discriminator overfitting and temporal variation.

  • B Dataset details: The evaluation uses ACID, SkyTimelapse, and new horseback-riding and mountain-biking datasets, with dataset-specific cropping, resizing, and split-selection procedures.ACID combines train and test splits, while SkyTimelapse uses only its training split for comparability with prior work.
  • B Dataset details: New datasets have median durations of 6504 and 330 frames, compared with 91 and 81 frames for existing datasets, enabling evaluation of long-term dynamics.Figure 15 reports duration distributions for all four datasets.
  • B.1 Horseback riding: The horseback-riding dataset contains first-person helmet-camera footage and is released for non-commercial research after filtering 194 source videos to 44 videos and extracting 66 clips.The footage is high quality, originally recorded at 1080p and 60fps, then subsampled to 30fps.
  • B.2 Mountain biking: The mountain-biking dataset provides first-person footage released under a CC BY 4.0 license, with extensive filtering needed because source videos contain cuts and abrupt changes.The collection was created from Brian Kennedy’s videos and underwent curation to remove problematic footage.
  • C.1 Augmentation: DiffAug applies color, cutout, and translation augmentations to real and generated videos to reduce severe discriminator overfitting on long sequences.Maximum translations are 32 pixels for square SkyTimelapse videos and 16 pixels for the non-square datasets; ADA caused augmentation leakage.
  • C.1 Augmentation: Fractional time stretching resizes the temporal axis by s = 2^a, with a ∼ U(−1, 1), using interpolation and zero padding within the original 128-frame window.Videos are randomly cropped after upsampling and randomly padded before and after downsampling.
  • C.2 Temporal lowpass filters: Each of 8 input temporal-noise channels receives N = 128 exponentially spaced Kaiser lowpass filters to capture long-term correlations in intermediate latent codes.The filter footprints range between k_min = 500 and k_max = 10000.

C.3 Discriminator architecture

The low-resolution discriminator adapts StyleGAN-style processing with spatiotemporal components to model realistic motion and temporal change. It combines residual downsampling blocks, temporal convolutions, and fully connected layers to produce logits.

  • C.3 Discriminator architecture: The discriminator combines a 1×1 input convolution, four residual blocks, four temporal 1D convolutions, and two linear layers that produce output logits.The architecture reshapes features between residual processing and temporal convolutions before flattening them for the final linear layers.
  • C.3 Discriminator architecture: Temporal processing is omitted from the first residual block to reduce the cost of full-resolution 3D convolutions, then applied in the remaining blocks.The remaining blocks downsample spatially and temporally using spatiotemporal convolutions.
  • C.3 Discriminator architecture: The residual blocks double channel counts until reaching 512 while incorporating spatial and temporal downsampling for motion modeling.The first block operates only spatially, whereas the remaining three blocks use spatiotemporal processing.
  • C.3 Discriminator architecture: Four temporal convolutions use kernel size 5 and LeakyReLU nonlinearities, followed by flattened features and two StyleGAN-style linear layers.The temporal convolutions further prioritize learning accurate motions and changes over time.

C.4 Training … D.2 Prefiltering of low-res conditioning

The implementation trains the low-resolution generator with long video batches and applies augmentation and prefiltering strategies to improve super-resolution robustness and reduce aliasing. The super-resolution network uses conditioning augmentation, discriminator regularization, and a nine-frame low-resolution context.

  • C.4 Training: The low-resolution generator trains on batches of 64 videos, each 128 frames long, for up to 100,000 steps, stopping early if FVD increases.Maximum low-resolution training takes 1.7 days on 4× nodes with 8× NVIDIA A100 GPUs each.
  • D.1 Augmentation: The super-resolution network augments real and generated videos before discrimination and augments conditional real low-resolution videos to improve inference-time generalization.These two augmentation forms target discriminator overfitting and the domain gap between real training and generated inference conditioning.
  • D.1 Augmentation: ADA applies identical augmentations across all frames, while dropout with p = 0.9 can zero the entire conditional low-resolution video to limit discriminator overfitting and conditioning reliance.The dropout augmentation is applied alongside ADA with default settings.
  • D.1 Augmentation: Real low-resolution conditioning videos are augmented during training because generated low-resolution videos are used at inference, creating a domain gap.The augmentation is intended to improve super-resolution robustness to this training–inference mismatch.
  • D.2 Prefiltering of low-res conditioning: Each upsampled frame is conditioned on four preceding and four following low-resolution frames, forming a stack of nine frames.The stack is resized and concatenated with features at every StyleGAN3 generator layer.
  • D.2 Prefiltering of low-res conditioning: Strong prefiltering during resizing removes aliasing in the final video.The authors found stronger prefiltering more effective among the tested prefiltering strengths.

D.3 Training

The model is trained with batches of 32 videos for up to 275,000 steps, using short discriminator clips and temporally contextualized super-resolution inputs.

  • D.3 Training: Training uses batches of 32 videos for a maximum of 275,000 steps, taking 6.8 days on one node with eight 16GB NVIDIA GPUs.The discriminator receives real and generated four-frame videos, while the super-resolution network receives nine frames per generated frame, including four neighboring frames on each side.
Loading 2206.03429v2…