Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architectures

1 INTRODUCTION

• The authors identify automated video generation as a limited area of work, with caption-to-video generation presented as a first-of-its-kind capability.Earlier efforts had focused primarily on image generation and spatial characteristics.Source: §1 · p. 1
• Sync-DRAW combines a recurrent VAE with local attention synchronized across individual frames to generate videos.The paper names the architecture Synchronized Deep Recurrent Attentive Writer.Source: §1 · p. 1
• The proposed attention mechanism is reported to help generated videos maintain object structure, and Sync-DRAW is adapted to generate videos from captions.The paper describes caption-based video generation as the first effort of its kind to the authors’ knowledge.Source: §1 · p. 1
• The paper’s stated contributions also include an architecture for preserving frame-level structural integrity and an experimental study spanning datasets of varying complexity.The introduction lists architecture design, caption-based generation, and experimental study as its three contributions.Source: §1 · p. 1

2 BACKGROUND AND RELATED WORK

• Deep belief networks and deep Boltzmann machines relied heavily on MCMC sampling, which the paper describes as making them difficult to scale; GANs and VAEs are presented as scalable alternatives for generative tasks.The passage describes these newer approaches as effective on generative tasks and scalable to large datasets.Source: §2 · p. 2
• DRAW combines a recurrent VAE with attention to progressively refine an image canvas, whereas Sync-DRAW addresses the limited use of spatiotemporal latent representations for video generation.The paper also identifies text-to-video generation as lacking prior work to the authors’ knowledge.Source: §2 · p. 2
• VGAN uses a spatiotemporal GAN with separate foreground and static-background streams, but its generated videos are described as noisy and lacking objectness.Sync-DRAW instead uses recurrent VAEs and attention, and the paper reports that its generated videos maintain object structure on visual inspection.Source: §2 · p. 2
• Earlier LSTM video-prediction work learns representations to predict future frames, rather than generating complete videos from scratch.The paper distinguishes that objective from its own video-generation task.Source: §2 · p. 2

3 SYNC-DRAW METHODOLOGY

• Sync-DRAW models video generation as producing sequences, individual objects, and their motion, using recurrent frame-wise attention with a global R-VAE.A single R-VAE models temporal relationships across frames; separate frame-wise attention mechanisms operate simultaneously across them.Source: §3 · p. 2

3.1 Sync-DRAW Architecture

• Figure 2 depicts the read-write mechanisms and R-VAE, caption conditioning, and canvases produced over time.Its final row shows the final generated video.Source: Figure 2 · p. 3

3.1.1 Read Mechanism.

• At each timestep, Sync-DRAW reads a region of interest from every frame using learned Gaussian-filter grids with adjustable centers, stride, and variance.Learned per-frame parameters and frame weights let the model vary its regions of interest and selectively emphasize frames.Source: §3.1.1 · pp. 3–4

3.1.2 R-VAE.

• The R-VAE samples from the video’s learned latent distribution and generates synchronized canvases for all frames at each timestep.The paper defines a video-level error from the input video and the current generated canvas for the encoder.Source: §3.1.2 · p. 4

3.1.3 Write Mechanism.

• For each frame, the write mechanism computes what to write and where, then uses learned attention parameters and filterbanks to create its canvas.Canvas updates accumulate over timesteps, and the final video is obtained by applying the sigmoid function to the accumulated canvas.Source: §3.1.3 · p. 4

3.1.4 Sync-DRAW with captions.

• For caption-conditioned generation, a pretrained skip-thought model maps the caption to a latent representation that conditions the R-VAE’s encoder and decoder.At test time, the caption is provided to the decoder after the encoder and read mechanisms are removed.Source: §3.1.4 · p. 4

3.1.5 Loss function.

• Sync-DRAW’s loss combines video-level pixelwise cross-entropy reconstruction loss with a KL-divergence loss summed over timesteps.The KL term compares the learned latent posterior with a standard normal prior.Source: §3.1.5 · p. 4

3.1.6 Testing.

• At test time, Sync-DRAW removes the encoder and samples from the learned latent distribution, scaling and shifting samples by the learned μ_t and (σ_t)^2.The sampled latents generate the final videos.Source: §3.1.6 · p. 4

4 EXPERIMENTS AND RESULTS

• Sync-DRAW is evaluated on Single-Digit and Two-Digit Bouncing MNIST and the KTH Human Action Dataset, selected for varying complexity and caption compatibility.The authors manually created captions because no text-to-video dataset was available for this work.Source: §4 · p. 4
• Figure 3 shows sample videos from Single-Digit Bouncing MNIST, Two-Digit Bouncing MNIST, and KTH.The datasets are presented as evaluation examples.Source: Figure 3 · p. 5

4.1 Baseline Methodology

• The baseline extends an image-generation methodology with spatio-temporal cuboid attention and an added temporal filterbank, and the paper reports its best tried parameter setting.It is designed for comparison and to highlight the role of separate attention parameters for each video frame.Source: §4.1 · p. 5

• Each video contains 10 frames of size 64 × 64, with one 28 × 28 digit moving horizontally or vertically.Source: §4.2 · p. 5
• Sync-DRAW progressively forms the final video as a sequence of canvases, with generated digits showing clearer structure and boundaries than the baseline.The comparison is described alongside the Two-Digit results, while the passage also refers to the Single-Digit results in Figures 2(c) and 4(b).Source: Figure 5 · p. 5
• The same digit region receives attention at each timestep even as the digit appears at different pixel locations across frames.Source: Figure 5 · p. 5

• Sync-DRAW produces two digits with defined structure and boundaries, though they are less clear than the single-digit results.Source: §4.3 · p. 5
• At each timestep, Sync-DRAW attends to both digits simultaneously; an initially unified structure then separates into two digits.Source: §4.3 · p. 5
• The authors infer that attention needs a stimulus to recognize two digits, and say caption alignment supplies that stimulus in later experiments.Source: §4.3 · p. 5

• Figure 7 presents test videos generated for KTH without captions.Source: Figure 7 · p. 6
• KTH contains over 2,000 videos of 25 people performing six actions, with frames resized to 120 × 120 and generation over 10 frames.Source: §4.4 · p. 6
• The clarity of people in generated KTH scenes suggests Sync-DRAW performs equally well on a real-world dataset.Source: §4.4 · p. 6

4.5 Sync-DRAW with captions

• Sync-DRAW can be used to generate videos from captions.Source: §4.5 · p. 6

4.5.1 Single-Digit Bouncing MNIST.

• Figure 8 shows single-digit caption-generated results produced by the trained model at test time.Source: Figure 8 · p. 6
• For Single-Digit Bouncing MNIST, test captions describe motions withheld during training, and Figure 8 shows videos generated from those captions.Source: §4.5.1 · p. 6, Figure 8 · p. 6

4.5.2 Two-Digit Bouncing MNIST.

• For Two-Digit Bouncing MNIST, caption conditioning improves digit quality and may help attention distinguish objects instead of treating the frame as one entity.Source: §4.5.2 · p. 7

4.5.3 KTH Dataset.

• KTH captions derived from metadata describe a person and action, with walking or running direction; generated videos are reported as high quality.Source: §4.5.3 · p. 7

4.6.1 Reconstruction Loss.

• Sync-DRAW performs significantly better in negative log likelihood at convergence than the baseline and Global approaches.The comparison uses lower-is-better NLL; the supplied passages do not provide the table’s individual values.Source: §4.6.1 · p. 7, Table 1 · p. 7

4.6.2 Psychophysical Analysis.

• 37 subjects rated 10 generated videos per Sync-DRAW experiment on a 1–10 visual-quality scale, with original videos scored as 10.The analysis covered multiple datasets and caption settings.Source: §4.6.2 · p. 7
• Ratings were very high for single-digit Bouncing MNIST with or without captions, and for captioned two-digit Bouncing MNIST and KTH.Uncaptioned two-digit Bouncing MNIST and KTH received slightly lower, though not unacceptable, scores; caption stimuli improved generations on those datasets.Source: §4.6.2 · p. 7

4.7 Qalitative comparison with VGAN [25]

• The authors trained VGAN on modified Bouncing MNIST in RGB and ran Sync-DRAW on the same dataset; on the dataset from VGAN’s work, Sync-DRAW was configured to generate 32-frame videos.They also ran both codes on UCF-101, where each training video had 32 colored frames; VGAN had used it for action-classification representation learning but reported no generation result there.Source: §4.7 · p. 8
• VGAN appears sharper overall, but closer inspection finds noisy generations with poor objectness.The authors report similar issues in many results from VGAN’s own work.Source: §4.7 · p. 8
• Sync-DRAW’s generations are smoother and fairly noise-free compared with VGAN’s, but are somewhat blurry.The authors identify sharpening real-world-dataset generations while retaining smoothness as future work.Source: §4.7 · p. 8
• Human psychophysical scores for VGAN were comparable to Sync-DRAW’s, despite the reported visual differences.Figures 11–13 show the first 10 generated frames for the MNIST, VGAN-dataset, and UCF-101 comparisons, respectively.Source: §4.7 · p. 8, Figure 11 · p. 8, Figure 12 · p. 8

5 CONCLUSION AND FUTURE WORK

• Sync-DRAW combines a Variational Auto-Encoder with a Recurrent Attention Mechanism to generate videos.The conclusion describes experiments ranging from single-digit Bouncing MNIST to KTH and UCF-101.Source: §5 · p. 8
• The authors report that attention focuses on objects and that generated videos maintain their structural integrity.Source: §5 · p. 8
• Sync-DRAW generates videos from captions, which the authors describe as a first-of-its-kind capability.Source: §5 · p. 8
• Future work aims to reduce generation blurriness and generate videos from more complex captions.Source: §5 · p. 8