PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2

Ivan Skorokhodov, Sergey Tulyakov, Mohamed Elhoseiny

arXiv:2112.14683v4cs.CVcs.AIcs.LG

TL;DR

Video synthesis methods commonly treat videos as discrete sequences, limiting efficient modeling of continuous, long-duration content. StyleGAN-V instead builds a continuous-time generator with motion-aware positional representations, sparse-frame training, and feature-concatenating discrimination; it retains StyleGAN2-like efficiency and quality while improving benchmark performance and enabling arbitrary-length, high-frame-rate generation.

  • Problem

    Video synthesis frameworks commonly treat videos as discrete sequences, creating challenges for complex, long, and high-resolution video data.

  • Method

    StyleGAN-V extends neural representations to continuous-time video generation with motion-dependent positional embeddings, sparse-frame training, and concatenated temporal features in its discriminator.

  • Results

    ≈30% better average performance than the closest runner-up is reported across five video-synthesis benchmarks.

  • Takeaways & Limitations

    The model generates arbitrarily long, high-frame-rate videos while preserving StyleGAN2-like efficiency, image quality, and editable latent-space properties.

  • Takeaways & Limitations

    Training uses a maximum time location T and limits sampled frame distances to 32; interpolation-based motion codes also produce poor results in practice.

Abstract

from arXiv · show

Videos show continuous events, yet most $-$ if not all $-$ video synthesis frameworks treat them discretely in time. In this work, we think of videos of what they should be $-$ time-continuous signals, and extend the paradigm of neural representations to build a continuous-time video generator. For this, we first design continuous motion representations through the lens of positional embeddings. Then, we explore the question of training on very sparse videos and demonstrate that a good generator can be learned by using as few as 2 frames per clip. After that, we rethink the traditional image + video discriminators pair and design a holistic discriminator that aggregates temporal information by simply concatenating frames' features. This decreases the training cost and provides richer learning signal to the generator, making it possible to train directly on 1024$^2$ videos for the first time. We build our model on top of StyleGAN2 and it is just ${\approx}5\%$ more expensive to train at the same resolution while achieving almost the same image quality. Moreover, our latent space features similar properties, enabling spatial manipulations that our method can propagate in time. We can generate arbitrarily long videos at arbitrary high frame rate, while prior work struggles to generate even 64 frames at a fixed rate. Our model is tested on four modern 256$^2$ and one 1024$^2$-resolution video synthesis benchmarks. In terms of sheer metrics, it performs on average ${\approx}30\%$ better than the closest runner-up. Project website: https://universome.github.io.

1. Introduction

StyleGAN-V treats videos as continuous-time signals rather than discrete frame sequences, combining motion-aware representations, sparse training, and a redesigned discriminator. The resulting generator preserves StyleGAN2-like efficiency and image quality while supporting long, high-frame-rate videos.

  • Video synthesis remains difficult because discrete sequence representations demand expensive architectures for long, high-resolution videos.
  • StyleGAN-V represents each video as a continuous signal x(t) and extends neural representations to GAN-based video generation.
  • Time-varying motion-dependent positional embeddings and a padding-less conv1d motion model avoid periodic motion and recurrent instability during long unrolls.
  • As few as 2 frames per clip can suffice to learn a meaningful video generator, according to the paper’s theoretical exposition and experiments.
  • The conv2d-based discriminator concatenates sparse frame features and conditions on time distances, improving efficiency and generator learning signals.
  • ≈5% higher training cost than StyleGAN2 accompanies direct training at 1024^2 resolution, while image FID is ≈10% worse and benchmark performance is ≈30% better on average.The benchmark comparison is reported across the paper’s evaluation protocol and five video-synthesis benchmarks.

2. Related work

Related work spans video prediction, interpolation, GAN-based generation, high-resolution synthesis, and neural representations. StyleGAN-V extends continuous neural representations to video generation while replacing the common expensive 3D-convolutional discriminator design with temporal feature aggregation.

  • Video prediction generates future frames from observed sequences, whereas video generation synthesizes videos from scratch without conditioning on previous frames.
  • GAN-based video generators commonly separate content and motion codes, a strategy introduced by MoCoGAN and TGAN and adopted by later methods.
  • High-resolution video synthesis has often relied on training in the latent space of a pretrained image generator, while StyleGAN-V trains on extremely sparse videos.
  • Modern video synthesis approaches frequently use expensive conv3d blocks and separate image- and video-level discriminators.
  • Neural representations model continuous signals with neural networks, and StyleGAN-V extends this paradigm from images, 3D objects, and multimodal signals to video generation.
  • DIGAN is a concurrent continuous-video approach using a different motion parameterization and dual discriminator, while StyleGAN-V aggregates temporal features through concatenation.

3. Model

StyleGAN-V models videos as continuous-time signals by combining continuous motion codes with a minimally modified StyleGAN2 generator and a holistic sparse-frame discriminator. Its design addresses acyclic motion, sparse training, and efficient temporal aggregation while preserving non-autoregressive generation.

  • Continuous-time formulation: StyleGAN-V represents each video as a continuous function of time and trains on subsampled frames, allowing videos with varying lengths and frame positions.The dataset contains subsampled signals, with each frame associated with a time coordinate.
  • Generator structure: The generator separates content and motion, sharing a StyleGAN2-derived content code across time while producing each frame non-autoregressively from a continuous motion code.Motion codes are concatenated to StyleGAN2’s constant input tensor, and only the noise vectors on which a requested code depends need to be sampled.
  • Motion representation: Acyclic positional embeddings derive wave parameters from motion information, preventing periodic temporal behavior and allowing different videos to exhibit different motions.The motion representation uses time-varying amplitudes, periods, and phases rather than fixed periodic embeddings.
  • Motion representation: Directly interpolated motion codes can produce sharp transitions or lose high-frequency motion, motivating the paper’s acyclic positional-code construction.Small token spacing causes unnatural transitions, whereas large spacing can prevent modeling motions such as blinking.
  • Discriminator structure: The holistic discriminator replaces separate image and video discrimination with independently extracted frame features that are concatenated and conditioned on inter-frame time distances.This conv2d-based design is motivated by sparse sampling and avoids expensive conv3d blocks designed for long equidistant sequences.
  • Sparse training: Training can use extremely sparse videos, with as few as 2 frames per clip, under conditions where the sampled marginals provide the dependencies needed to represent the joint distribution.The sparse-training analysis frames optimization as learning marginal distributions and gives a condition involving preceding-coordinate subsets.

4. Experiments

The experiments evaluate StyleGAN-V across five video benchmarks, ablations, long-duration generation, latent manipulations, efficiency, image quality, and high-resolution synthesis. StyleGAN-V generally outperforms existing methods while retaining strong efficiency and scalability.

  • Evaluation: StyleGAN-V is evaluated on five benchmarks using FVD16, FVD128, and, for UCF-101, Inception Score under a unified protocol.The benchmarks are FaceForensics, SkyTimelapse, UCF101, RainbowJelly, and MEAD; FVD is the primary metric.
  • Main results: StyleGAN-V significantly outperforms existing methods on almost all benchmarks in FVD16 and FVD128.It can generate hour-long plausible videos, although motion diversity and global motion coherence are limited at that duration.
  • Ablations: Removing time conditioning from the discriminator hurts performance because the discriminator cannot identify the temporal scale it processes.Replacing continuous motion codes with LSTM-produced codes also hurts performance, especially when motion-code distance is small.
  • Ablations: Training with 2 frames per video performs better than using larger numbers of sampled frames in the reported experiments.The tested values were k = 2, 3, 4, 8, and 16; performance decreased for larger k, potentially because of weaker temporal aggregation.
  • Properties: StyleGAN-V generates arbitrarily long and high-frame-rate videos, and its latent space supports embedding, editing, and animating images over time.The model preserves StyleGAN2-like appearance and motion manipulation properties, including extrapolation from an edited first frame.
  • Efficiency and resolution: StyleGAN-V reaches FID 9.44 in 8 GPU-days versus StyleGAN2’s FID 8.42 in 7.72 GPU-days, making it approximately 10% worse in image quality.The model is directly trainable at 1024^2 resolution; on MEAD 1024^2 it obtains FID 24.12 and FVD16 156.1 after 7 days on 4 V100 GPUs.
  • Discriminator analysis: The redesigned discriminator provides more informative gradients than the conv3d-based MoCoGAN-HD discriminator.StyleGAN-V’s gradients reveal fine-grained facial structure, whereas MoCoGAN-HD’s gradients are largely redundant and lack structural information.

5. Conclusion

The paper presents StyleGAN-V as a continuous-time video generator built on neural representations and StyleGAN2. It combines continuous motion representations, sparse-video training, and a redesigned discriminator while retaining efficient training, image quality, and editable latent-space properties.

  • Conclusion: StyleGAN-V treats videos as continuous-time signals and develops motion representations, sparse training, and a redesigned video discriminator.The model is built on StyleGAN2 and preserves several of its practical properties.

A.1. Limitations

The authors identify limitations involving sparse training, dataset quality, periodic motion, new content, and hyperparameter sensitivity. They also note potential misuse for deepfakes and weaker outcomes than image-based counterparts on limited video datasets.

  • Sparse training prevents the discriminator from capturing complex dependencies between frames, although it achieves state-of-the-art results on current datasets.The authors infer that these datasets may not contain highly sophisticated motion.
  • Dataset quality constrains performance: FaceForensics embedding and manipulation results are inferior to StyleGAN2 because it has only 700 identities and greater quality variation than FFHQ.
  • Periodic motions still occur despite acyclic positional embeddings, leaving periodicity artifacts unresolved.
  • The generator tends to reuse global-latent content, causing poor handling of new content appearing in SkyTimelapse and RainbowJelly videos.The authors suggest ALIS-like ideas as a possible remedy.
  • The generator is sensitive to the minimum initial period length: SkyTimelapse required increasing it from 16 to 256 to avoid unnatural sharp transitions.
  • The method may facilitate realistic deepfakes, while limited high-quality video datasets leave its results below those of image-based counterparts trained on datasets such as FFHQ.

B. Implementation and training details

StyleGAN-V uses StyleGAN2 with continuous motion representations, sparse-frame training, video-consistent augmentation, and a bounded positional-embedding design. Implementation choices prioritize training speed and temporal stability.

  • StyleGAN-V is built on StyleGAN2 because mixed-precision training makes it approximately twice as fast as INR-GAN.The authors report that mixed precision severely decreases INR-GAN quality.
  • Training uses non-saturating loss, R1 regularization with coefficient 0.2, and fmaps 0.5 to reduce channel dimensionalities and speed training.
  • The motion mapping network is padding-less Conv1D with kernel size 11, stride 1, no dilation, and 512-dimensional latent variables.
  • Padding-less convolutions require sampling a slightly longer motion-noise sequence because two kernel-11 layers reduce the sequence by 20 positions.
  • Video-consistent adaptive augmentations apply the same transformation to every frame; otherwise discriminator performance and overall quality decrease.
  • Unbounded wave periods can create temporal aliasing, so tanh(ωt)+1 bounds them to (0, 2) with mean 1.0.

C. Evaluation details

The evaluation emphasizes reproducible FVD computation because implementation choices substantially affect scores. The paper documents dataset protocols, metric caveats, and the costs reported for competing methods.

  • Figure 9 compares positional-embedding period initializations using cycle length, with linear scaling chosen for a larger global cycle and guaranteed frequency coverage.
  • FVD does not capture motion collapse: LSTM-motion variants score 196.1 or 165.8 on FVD128 versus 197.0 for StyleGAN-V despite severe collapse in samples.
  • FID uses 2048 generated 16-frame videos, yielding approximately 33k frames with lower diversity than a typical 50k-image dataset.
  • FVD varies with the number of clips per video, clip offsets, and frame rate, complicating comparisons across papers.For example, selecting all clips can bias statistics for long SkyTimelapse videos, while zero-offset clips can overrepresent FaceForensics introductions.
  • The authors use a defined protocol for computing FVD_n after identifying discrepancies in I3D weights and evaluation implementations.
  • Inception Score on UCF101 is computed after resizing videos to 112×112, preventing the metric from capturing high-resolution details.

D. Failed experiments

Several attempted modifications failed to improve evaluation or introduced undesirable behavior. These experiments reveal difficulties in representing motion coherently and balancing discriminator strength.

  • Hierarchical motion codes neither improved scores nor disentangled motion information, while the authors found motion codes difficult to keep temporally coherent.
  • Entropy maximization failed to alleviate motion collapse because the generator reserved coordination dimensions to synchronize motions.
  • Progressively introducing positional-embedding frequencies made high-frequency motion unnaturally slow because low frequencies represented it instead.
  • Adding spatial coordinates produced no qualitative or quantitative improvement and slowed training by approximately 10%.
  • Feature differences made the discriminator too powerful, causing it to outpace the generator and decrease final scores.
  • An auxiliary class-prediction head for frame time distances noticeably decreased scores.

E. Datasets details

The paper introduces RainbowJelly as an intermediate-difficulty video benchmark and documents its construction, motion structure, quality, and clip duration. MEAD is evaluated using only front-camera shots.

  • E.1. Dataset preprocessing: For MEAD, the experiments use only front-camera shots, although the original dataset contains several camera positions.
  • E.2. Rainbow Jelly: RainbowJelly fills a gap between datasets that are too simple or too difficult in content and motion.It is designed as a challenging but approachable test bed for modern video generators.
  • E.2. Rainbow Jelly: RainbowJelly is built from an 8-hour, 4K, 30 FPS movie and sliced into 1686 clips of 512 frames at 256^2 resolution.The source begins at frame 150 to remove the loading screen, then clips are center-cropped and resized.
  • E.2. Rainbow Jelly: RainbowJelly contains hierarchical motion spanning global direction, medium-frequency arm movement, and local body-and-tentacle perturbations.Its content is simple, keeping the benchmark focused on motion.
  • E.2. Rainbow Jelly: RainbowJelly provides high-quality 4K source material and long videos for evaluation.

F. Implicit assumptions of sparse training

The sparse-training analysis asks when fitting random k-frame marginals can recover a full video distribution. It identifies a representative-moment condition under which sparse training is viable and explains the corresponding reverse implication.

  • F.1. Sparse-training formulation: Sparse training selects k random coordinates per sample, making optimization equivalent to learning all length-k marginals instead of the full joint distribution.The setup treats videos as finite discretizations for practical analysis.
  • F.2. Sufficient condition: If each conditional p(x_i|x_<i) depends only on a preceding representative set J_i of fewer than k indices, the full distribution factorizes through sparse marginals.The sets J_i contain earlier indices and encode the limited history needed for each coordinate.
  • F.2. Sufficient condition: The forward statement shows that representative moments make k-sparse training a viable alternative for obtaining the full distribution.The fitted q_θ need not equal p exactly, provided the relevant marginals support the factorization.
  • F.3. Reverse implication: The reverse claim states that if p(x) is a product of n conditionals using sets of at most k−1 earlier indices, corresponding sets J_i exist for every i.The proof cancels chain-rule terms one by one, starting from the end.
  • F.4. Implication for video synthesis: For video synthesis, using very small k assumes that randomly sampled frames contain enough information to model the relevant k-marginals rather than the full distribution directly.The paper notes that multiple explainable frame sets may exist for a given frame.

G. Additional samples

The supplementary material provides additional samples and compares StyleGAN-V with DIGAN in sparse training, continuous generation, motion representation, discriminator design, backbone, sampling, and training cost. It also notes dataset frame-rate differences and a potential length-distribution bias.

  • G.1. Comparison with DIGAN: StyleGAN-V and DIGAN both use sparse training, but StyleGAN-V explores the optimal number of frames and uses k = 3 rather than DIGAN’s k = 2.StyleGAN-V samples frames uniformly at random, whereas DIGAN uses beta-distributed sampling with T = 16.
  • G.2. Motion representation: StyleGAN-V uses an infinite sequence of temporally interpolated motion codes, while DIGAN uses one global motion code that can eventually repeat.StyleGAN-V derives wave parameters and transforms them into motion codes.
  • G.4. Generator backbone: StyleGAN-V is built on StyleGAN2, while DIGAN uses INR-GAN and consequently inherits slower training and reduced stability described for that backbone.
  • G.3. Discriminator structure: StyleGAN-V concatenates frame features in one discriminator, whereas DIGAN retains separate image- and video-level discriminators.StyleGAN-V conditions on time differences after feature concatenation.
  • G.5. Training cost: StyleGAN-V training reaches about 25k real frames in 2 days, while DIGAN stopped improving after about 5 days and about 20k real frames in the reported experiment.The supplementary discussion says the authors’ updated 8 GPU-day figure is consistent after accounting for resolution.
  • G.6. Dataset statistics: RainbowJelly and MEAD use 30 FPS, whereas the other listed datasets use 25 FPS; SkyTimelapse includes very long videos that may bias its length distribution.
Loading 2112.14683v4…