Source-linked AI summary
To Create What You Tell: Generating Videos from Captions
Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, Tao Mei
TL;DR
Generating videos from captions requires models to capture both visual-temporal coherence and semantic alignment. TGANs-C combines caption-conditioned generation with 3D spatio-temporal convolutions and video, frame, and motion discriminators. The model generates plausible caption-conditioned videos across synthetic and real-world datasets and is evaluated quantitatively on MSVD.
Problem
Video generation is difficult because videos contain visually coherent, semantically dependent frames, and caption-conditioned models must preserve temporal coherence and semantic alignment.
Method
TGANs-C concatenates caption embeddings with latent noise, uses 3D spatio-temporal convolutions, and employs video, frame, and motion discriminators.
Results
TGANs-C generates plausible caption-conditioned videos on SBMG, TBMG, and MSVD, with quantitative MSVD evaluation using a Generative Adversarial Metric and human study.
Takeaways & Limitations
The framework demonstrates a GAN-based approach that jointly models caption-video alignment and temporal coherence for video generation.
Takeaways & Limitations
The authors identify higher-resolution synthesis, open-vocabulary captions, and extension to audio as future challenges.
Abstract
from arXiv · showhide
We are creating multimedia contents everyday and everywhere. While automatic content generation has played a fundamental challenge to multimedia community for decades, recent advances of deep learning have made this problem feasible. For example, the Generative Adversarial Networks (GANs) is a rewarding approach to synthesize images. Nevertheless, it is not trivial when capitalizing on GANs to generate videos. The difficulty originates from the intrinsic structure where a video is a sequence of visually coherent and semantically dependent frames. This motivates us to explore semantic and temporal coherence in designing GANs to generate videos. In this paper, we present a novel Temporal GANs conditioning on Captions, namely TGANs-C, in which the input to the generator network is a concatenation of a latent noise vector and caption embedding, and then is transformed into a frame sequence with 3D spatio-temporal convolutions. Unlike the naive discriminator which only judges pairs as fake or real, our discriminator additionally notes whether the video matches the correct caption. In particular, the discriminator network consists of three discriminators: video discriminator classifying realistic videos from generated ones and optimizes video-caption matching, frame discriminator discriminating between real and fake frames and aligning frames with the conditioning caption, and motion discriminator emphasizing the philosophy that the adjacent frames in the generated videos should be smoothly connected as in real ones. We qualitatively demonstrate the capability of our TGANs-C to generate plausible videos conditioning on the given captions on two synthetic datasets (SBMG and TBMG) and one real-world dataset (MSVD). Moreover, quantitative experiments on MSVD are performed to validate our proposal via Generative Adversarial Metric and human study.
1 INTRODUCTION
Generating videos from captions requires both temporal coherence across frames and semantic alignment with the caption. TGANs-C addresses these challenges with a caption-conditioned GAN architecture evaluated on three benchmarks.
- Video generation from captions must preserve temporal coherence across frames and semantic match between the caption and generated video.These properties require adjacent frames to remain visually and semantically connected while the generated content remains relevant to the description.
- TGANs-C combines temporal coherence and semantic matching in a caption-conditioned GAN framework for video generation.The framework is presented as one of the first efforts toward generating videos conditioned on captions.
- The model uses caption information in its generator and discriminator design to produce videos aligned with the input sentence.Its discriminator evaluates realism while also learning whether videos match the conditioning caption.
- TGANs-C is validated through quantitative and qualitative experiments on three benchmarks.The experiments cover two synthetic datasets and one real-world video-caption dataset.
2 RELATED WORK
Prior work studies image synthesis and video generation with deep generative models, but video generation remains difficult because it must model spatial and temporal variation. TGANs-C differs from earlier GAN-based approaches by conditioning video synthesis on captions while adding frame-level and temporal modeling.
- Related work includes deep generative approaches for synthesizing realistic images and generating image sequences or videos from scratch.
- VAEs generate samples through latent-variable modeling but can impose restrictive posterior assumptions and produce overly smoothed samples.
- Caption-conditioned image synthesis has used VAEs with spatial attention to iteratively draw relevant patches from descriptions.
- Video generation is challenging because models must characterize complex spatial and temporal variations, with prior work using recurrent VAEs and attention for caption-conditioned sequences.
- TGANs-C differs from earlier GAN-based video synthesis by conditioning generation on captions and incorporating frame-level discrimination and stronger temporal connections.
3 VIDEO GENERATION FROM CAPTIONS
TGANs-C generates caption-conditioned video by combining sentence representations with noise and explicitly modeling video realism, semantic alignment, and temporal coherence. Its adversarial training uses video-, frame-, and motion-related objectives to coordinate these requirements.
- Optimization: TGANs-C optimizes video-level matching-aware, frame-level matching-aware, and temporal-coherence objectives in a two-player minimax game.The discriminator and generator are trained to classify sources, align videos and frames with captions, and model temporal dynamics under the proposed schemes.
- Caption Encoding: TGANs-C encodes input sentences with bi-LSTM and LSTM networks before using the resulting representation to condition video generation.The sentence encoder is pretrained with a sequence auto-encoder, after which only the bi-LSTM and LSTM encoder are retained.
- Generator Network: The generator concatenates sentence information with random noise and synthesizes a frame sequence through 3D deconvolutional layers.The 3D operations model spatial information while providing temporal structure across frames.
- Discriminator Network: The discriminator distinguishes real from synthetic videos while aligning video content with the conditioning caption.Training uses matched and mismatched real-video examples alongside synthetic videos to evaluate source and semantic correspondence.
- Discriminator Network: A frame discriminator evaluates frame realism and caption matching, while temporal modeling uses differences between consecutive frame-level tensors.The temporal representation is formed by subtracting the previous frame-level tensor from the current one.
4 EXPERIMENTS
Experiments evaluate TGANs-C across synthetic bouncing-digit datasets and the MSVD cooking-video subset, using qualitative comparisons, human rankings, and GAM-based quantitative evaluation. Results indicate that temporal-coherence modeling and frame-level matching improve generated-video quality, while TGANs-C performs strongly against several baselines.
- Datasets: Experiments use SBMG, TBMG, and an MSVD cooking subset of 518 videos, with 363 for training and 155 for testing.SBMG contains single moving digits, TBMG contains two independently moving digits, and MSVD supplies real cooking videos.
- Evaluation metrics: GAM compares TGANs-C with adversarial baselines using test and sample ratios to determine which model wins or ties.Sync-DRAW is excluded from GAM because it uses a VAE-based rather than adversarial architecture.
- Optimization analysis: TGANs-C-A consistently synthesizes plausible TBMG videos that reproduce handwritten-digit appearance and caption-conditioned temporal dynamics.The optimization analysis fixes the noise vector and caption while examining outputs across training iterations.
- Qualitative evaluation: On SBMG and TBMG, modeling temporal smoothness improves video quality, with TGANs-C-A outperforming TGANs-C-C in the reported qualitative comparison.The comparison also reports that the video-level-only TGANs-C1 performs worst and tends to produce blurry frames.
- Qualitative evaluation: On MSVD, TGANs-C produces higher-quality videos than the compared approaches, while VGAN tends toward sharp frames but stationary backgrounds.Compared with GAN-CLS, TGANs-C additionally uses frame-level matching and temporal coherence.
- Human evaluation: In a 30-evaluator MSVD study, TGANs-C is the overall winner across reality, caption relevance, and temporal coherence.Evaluators ranked four generated videos from each approach using the three criteria.
5 CONCLUSIONS
TGANs-C generates videos conditioned on input captions by modeling spatio-temporal connections and caption alignment. Experiments across three datasets and a 30-person user study validate the proposal, while higher-resolution generation remains future work.
- TGANs-C extends adversarial learning with 3D generation, caption matching for videos and frames, and motion information between consecutive frames.These components target spatio-temporal connections, semantic alignment, and temporal coherence.
- Extensive quantitative and qualitative experiments across three datasets validate the proposed architecture and analysis.
- A user study with 30 human subjects reports better video quality for the proposed approach.
- Future work targets higher-resolution videos, open-vocabulary caption conditioning, and extension of the framework to audio.The paper specifically identifies improving visual discriminability through higher-resolution synthesis as an initial focus.