Hierarchical Video Generation from Orthogonal Information: Optical Flow and Texture

• Video generation must produce realistic frames, a consistent scene and foreground, and plausible motion over time.Prior GAN approaches had not focused on these difficulties, according to the authors.Source: Introduction · p. 1
• Optical flow provides motion information without dataset annotations, allowing the model to be trained unsupervised.Source: Introduction · p. 1
• FTGAN first generates optical flow with FlowGAN, then uses TextureGAN to add texture while maintaining scene and foreground consistency.FlowGAN provides motion and rough edges; TextureGAN adds appearance, yielding videos with more plausible motion.Source: Introduction · p. 1
• Separately learning motion and appearance lets FTGAN generate combinations absent from training data, such as a person doing sit-ups on a baseball ground.Source: Introduction · p. 2
• On UCF101, FTGAN achieves significantly improved motion-recognition accuracy over Vondrick, Pirsiavash, and Torralba (2016) and Saito and Matsumoto (2016).Source: Introduction · p. 2

• Image GAN examples include Pix2Pix's same-edge image conversion and S2GAN's surface-normal-to-RGB generation.The paper builds on these generative models.Source: Related Work · p. 2
• Unlike future-frame prediction, which conditions on current or past frames and targets ground truth, this work generates videos from Gaussian noise and discriminates real from fake videos.The authors characterize their task as different and more difficult than future-frame prediction.Source: Related Work · p. 2
• Two-stream action recognition separates RGB texture from optical-flow motion, and later methods also use this split; the paper applies the principle to video generation.Source: Related Work · p. 2
• VGAN generates static backgrounds and moving foregrounds separately, but its single-stream 3D convolutions can yield plausible frames without movement or movement that is implausible.Source: Related Work · p. 2
• TGAN separates appearance and motion using 2D spatial then 1D temporal convolutions, but may lose spatial change after downsampling and fail to assure movement or scene consistency.The cited UCF101 comparison reports that FSTCN's TCL path did not improve over Two-stream's RGB stream.Source: Related Work · p. 2
• The proposed approach generates optical flow first to maintain motion consistency, then adds texture to maintain appearance consistency.Source: Related Work · p. 2

Generative Adversarial Networks

• A GAN pairs a generator that maps a random latent variable to data with a discriminator that distinguishes real data from generated fakes.The networks are updated simultaneously using an objective function.Source: Generative Adversarial Networks · pp. 2–3

Generative Adversarial Network for Video

• TextureGAN takes an optical-flow video and ztex as input and generates RGB video, using 3D convolutions for foreground and 2D convolutions for background.The illustrated input and output are 64x64 pixels and 32 frames.Source: Figure 2 · p. 3
• VGAN uses a mask to select moving foreground or static background at each pixel, with L1 regularization on the mask to encourage background.Source: Generative Adversarial Network for Video · p. 3

Method

• FTGAN splits video generation between FlowGAN and TextureGAN, trains them separately, then joins them for joint learning.FlowGAN generates optical flow, which TextureGAN converts into RGB video.Source: Method · p. 3

FlowGAN: Optical Flow Generation Model

• For fixed-camera scenes, FlowGAN omits a background stream and uses a zero background matrix, equivalent to VGAN's foreground stream alone.Source: FlowGAN: Optical Flow Generation Model · p. 3

TextureGAN: Optical Flow Conditional Video Generation Model

• TextureGAN takes optical flow and ztex as inputs and generates video with a VGAN/Pix2Pix-based U-net; its discriminator is conditioned on ground-truth flow.Source: TextureGAN: Optical Flow Conditional… · p. 3
• Figure 3 conditions TextureGAN on ground-truth optical flow not observed in training and shows input flow, generated frames, and corresponding ground-truth frames.Source: Figure 3 · p. 4

FTGAN: Both Optical Flow and Video Generation with Joint-Learning

• During joint learning, FlowGAN receives loss from its own discriminator and supplemental loss from TextureGAN's discriminator, with λ = 0.1.The two latent variables ztex and zflow are independent.Source: FTGAN: Both Optical Flow and Video… · p. 4

Network Configuration

• Separate training uses Adam at α = 0.0002 with batch size 32; joint learning uses α = 1e-7 for FlowGAN and α = 1e-6 for TextureGAN.Source: Network Configuration · pp. 4–5

FTGAN: Both Optical Flow and Video Generation with Joint-Learning

• FTGAN samples from zflow and ztex show plausible optical-flow motion and recognizable actions, though their frames are less clear than videos conditioned on ground-truth flow.Penn Action examples include jumping jacks, pitching, and swinging a baseball bat.Source: Figure 4 · p. 5

Experiments

• The experiments evaluate video generation and unsupervised video representation through action classification, following prior work because generative-model evaluation lacks appropriate metrics.The action-classification experiment compares FTGAN with other GAN-based methods.Source: Experiments · p. 5, Unsupervised Action Classification · p. 7

Dataset and Settings

• Video-generation experiments use Penn Action and SURREAL, with original Penn Action splits and SURREAL test videos for training plus 1,659 original training videos for testing.The models train for 60,000 iterations individually and 10,000 joint iterations per dataset; videos shorter than 32 frames are removed.Source: Dataset and Settings · p. 5

Qualitative Comparison to Previous Study: Visualized Results

• In Figure 5, VGAN videos lack realistic motion and persistent human contours, while the proposed methods show plausible motion and humans consistently across frames.The figure presents examples used in the AMT comparison.Source: Figure 5 · p. 6, Qualitative Comparison to Previous… · p. 7

Dataset and Settings

• Given ground-truth optical flow, TextureGAN generates plausible motion on Penn Action and SURREAL, including moving objects and a sit-up video set in a baseball ground, a combination absent from Penn Action.The reported moving objects include a barbell and a bowling ball.Source: TextureGAN: Video Generation Results… · p. 6
• FTGAN’s latent-variable videos are less clear than those conditioned on ground-truth flow, but show plausible movement; examples include pitching in Penn Action and more human-like frames with consistent motion on SURREAL than VGAN.The comparison reports VGAN’s SURREAL movement disappearing, while the proposed methods maintain consistent motion.Source: TextureGAN: Video Generation Results… · p. 6

Quantitative Comparison to Previous Study: Human Evaluation

• Although TextureGAN is slightly inferior to VGAN on SURREAL, the proposed methods outperform VGAN overall, especially on Penn Action.The authors infer that separating motion and appearance matters as dataset motion complexity increases; the human evaluation collected 8,000 opinions from 160 workers comparing 50 videos.Source: Table 1 · p. 6, Quantitative Comparison to Previous… · p. 7

Unsupervised Action Classification

• Both FTGAN discriminators outperform VGAN and TGAN on UCF101 action classification, and their fusion achieves significantly improved results.The authors interpret the fusion result as evidence that the discriminators capture complementary motion and appearance information; the pretrained VGAN results are not directly comparable because those models used 5,000 hours of video.Source: Unsupervised Action Classification · p. 7, Table 2 · p. 7

Conclusion and Future Work

• FTGAN generates low-resolution videos only a few seconds long, yet produces more plausible motion than VGAN.The authors note that optical-flow-specific architectures remain an important direction for future work.Source: Conclusion and Future Work · p. 7
• FTGAN significantly improves unsupervised video representations, with its discriminators learning complementary information.The authors argue that video recognition and generation should focus on video’s basic underlying principles.Source: Conclusion and Future Work · p. 8
• The authors identify finding architectures specific to optical flow as important future work.The conclusion also raises whether synthesized-video length affects comparisons with previous works.Source: Conclusion and Future Work · p. 8
• Separating optical flow and texture may reduce the generation space and yield better results on longer videos.The authors present these as potential benefits, not established outcomes.Source: Conclusion and Future Work · p. 8