PaperlayerView exact revision on arXiv

Source-linked AI summary

Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning

Ligong Han, Jian Ren, Hsin-Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dimitris Metaxas, Sergey Tulyakov

arXiv:2203.02573v1cs.CVcs.AIcs.LG

TL;DR

Single-modality video conditioning leaves a gap between specifying visual content and specifying motion. MMVID combines visual controls and language in a quantized, bidirectional-transformer framework, and the paper reports support for diverse controls and longer sequences. The authors also identify limitations in slow-motion extrapolation and generation diversity.

  • Problem

    Single-modality conditioning makes it difficult to specify both video content and a desired motion trajectory.

    Source: §1 · p. 1

  • Method

    MMVID uses a quantized two-stage video-generation framework with a bidirectional transformer, multimodal controls, a VID consistency task, improved mask-predict sampling, and text augmentation.

    Source: §1 · pp. 1–2, §3 · p. 3, §3.2 · p. 4

  • Results

    The paper reports independent and dependent multimodal generation with varied visual inputs, and generation of sequences longer than the training sequences.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    Visual controls can specify content that language may not precisely describe, while language can suggest attributes such as motion for those controls.

    Source: §1 · pp. 1–2

    For slow-motion extrapolation, sampled training frames may not cover diverse motion patterns; autoregressive models may produce more diverse videos, though many have low quality.

    Source: D. Limitation and Future Work · p. 22

Abstract

from arXiv · show

Most methods for conditional video synthesis use a single modality as the condition. This comes with major limitations. For example, it is problematic for a model conditioned on an image to generate a specific motion trajectory desired by the user since there is no means to provide motion information. Conversely, language information can describe the desired motion, while not precisely defining the content of the video. This work presents a multimodal video generation framework that benefits from text and images provided jointly or separately. We leverage the recent progress in quantized representations for videos and apply a bidirectional transformer with multiple modalities as inputs to predict a discrete video representation. To improve video quality and consistency, we propose a new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens. We introduce text augmentation to improve the robustness of the textual representation and diversity of generated videos. Our framework can incorporate various visual modalities, such as segmentation masks, drawings, and partially occluded images. It can generate much longer sequences than the one used for training. In addition, our model can extract visual information as suggested by the text prompt, e.g., "an object in image one is moving northeast", and generate corresponding videos. We run evaluations on three public datasets and a newly collected dataset labeled with facial attributes, achieving state-of-the-art generation results on all four.

1. Introduction

The paper addresses the limits of conditioning video generation on only one modality, proposing MMVID to use visual inputs and language jointly or separately. It introduces techniques for consistency, text robustness, and diverse generation, and explores multimodal controls and a new facial-attribute dataset.

1. Introduction

  • Single-modality conditioning limits flexibility: images specify content but not desired motion, while language can describe motion without precisely defining video content.The authors ask whether a model can combine visual inputs with language instructions such as “moving in a zig-zag way.”Source: §1 · p. 1
  • MMVID uses a two-stage framework: a quantized autoencoder obtains discrete image representations, then a bidirectional transformer predicts video representations from multimodal inputs.The predicted representations are decoded into videos using the first stage’s decoder.Source: §1 · p. 1
  • The framework introduces the VID token for temporal consistency, improves mask-predict sampling, and uses text augmentation to support diverse videos correlated with prompts.The paper also explores generating sequences much longer than those used in training.Source: §1 · p. 2
  • MMVID supports independent or dependent text and visual controls, including segmentation masks, drawings, and partially observed images.In dependent generation, text requests attributes from visual controls.Source: §1 · p. 2
  • The evaluation spans three public datasets and Multimodal VoxCeleb, a collected dataset of 19,522 VoxCeleb videos with 36 manually labeled facial attributes.Source: §1 · p. 2

2. Related Works

Prior video-generation work is described as unconditional or conditional, with conditional methods using visual or textual signals. This paper instead studies multimodal controls and a non-autoregressive, bidirectional-transformer pipeline.

Video Generation

  • Prior conditional video-generation methods use visual or textual information, while this work studies multiple modalities within one framework for independent and dependent controls.Source: §2 · p. 2

Transformers for Video Generation

  • Unlike prior transformer video-generation works focused on autoregressive training, this work applies a non-autoregressive pipeline with a bidirectional transformer.The approach builds on M6-UFC’s non-autoregressive multimodal image generation and adds training techniques for video synthesis.Source: §2 · p. 2

3. Methods

MMVID first quantizes image frames and then trains a bidirectional transformer on multimodal controls and video tokens. Its training combines masked sequence modeling, relevance estimation, and video consistency estimation; inference uses improved mask-predict sampling, text augmentation, and preservation controls for longer sequences.

3. Methods

  • A quantized autoencoder maps each video frame to a discrete representation, and the resulting video tokens are reshaped and concatenated with multimodal input tokens for transformer training.The first stage uses an encoder, quantizer, and decoder; the transformer models correlations between controls and video representations.Source: §3 · p. 2

3.1. Masked Sequence Modeling with Relevance

  • Masked sequence modeling trains bidirectional representations with masking strategies for mask-predict sampling and preservation control, including partial-image inputs and long-sequence generation.The strategies include random token masking, full masking, spatio-temporal block masking and its negation, and optionally keeping frames.Source: §3.1 · p. 3
  • Relevance estimation prepends a REL token and classifies positive sequences against sequences whose control signals are swapped across the batch.The authors note that swapped controls do not guarantee strictly negative examples, but report that this is sufficient in practice.Source: §3.1 · p. 3

3.2. Video Consistency Estimation

  • Video consistency estimation places VID between control and target tokens and masks its attention to controls, then uses negative video augmentations to identify temporally inconsistent sequences.Negative augmentations include frame swapping, shuffling, color jittering, and affine transformation, all applied in image space.Source: §3.2 · p. 4

3.3. Improved Mask-Predict for Video Generation

  • Improved mask-predict sampling adds noise-annealed multinomial sampling and piecewise-linear mask annealing to reduce motion washout after many iterations.The algorithm also applies beam search.Source: §3.3 · p. 4

3.4. Text Augmentation

  • Text augmentation drops input sentences and uses pretrained RoBERTa embeddings to improve robustness to textual inputs, including out-of-distribution prompts.The text features pass through an MLP and are combined as a weighted sum.Source: §3.4 · p. 4

3.5. Long Sequence Generation

  • Preservation controls support interpolation between given frames and iterative extrapolation from previous frames to generate longer videos.During interpolation, the given frames remain fixed through mask-predict iterations; extrapolation conditions on previous frames to generate subsequent ones.Source: §3.5 · p. 4

4. Experiments

Experiments span three public datasets and a new manually labeled Multimodal VoxCeleb dataset, evaluating text-to-video and multimodal controls. Reported comparisons favor MMVID, alongside long-sequence and augmentation analyses.

4. Experiments

  • The evaluation uses three public datasets and Multimodal VoxCeleb, a new set of 19,522 videos with 36 manually labeled facial attributes.The paper also describes baselines and dataset-specific evaluation metrics.Source: §4 · p. 5

4.1. Text-to-Video Generation

  • On Shapes MMVID outperforms ART-V across categories; it achieves best MUG performance and beats ART-V on every Multimodal VoxCeleb metric, with iPER results also favoring it.The Shapes table caption identifies the method as best-performing; the text reports advantages on iPER without specifying metric values.Source: Table 1 · p. 5, §4.1 · pp. 5–6

4.2. Multimodal Video Generation

  • With independent text-image controls, MMVID outperforms ART-V on all Shapes and MUG metrics; VoxCeleb examples show generated hair color changing with the text while image control stays the same.On Shapes, ART-V fails to satisfy partial-image visual constraints well; the VoxCeleb examples also cover segmentation masks and drawings.Source: §4.2 · pp. 6–7

4.3. Long Sequence Generation and Ablation

  • Long-sequence extrapolation generates two frames from the previous six, while interpolation synthesizes a frame between two real frames; uniformly sampled full augmentation gives the highest Shapes average accuracy.The language-model analysis reports greater robustness to varied text inputs with RoBERTa than without it.Source: §4.3 · p. 8

5. Limitation and Conclusion

MMVID uses quantized image and video representations with a non-autoregressive transformer to predict video tokens from multimodal inputs. The conclusion notes limits in high-resolution temporal consistency and long-sequence motion diversity, while the appendix outlines additional method details and masking settings.

5. Limitation and Conclusion

  • The framework combines an image-and-video quantized autoencoder with a non-autoregressive transformer that predicts video tokens from multimodal inputs.The paper identifies the VID token, textual embeddings, and improved mask prediction as techniques for temporally consistent videos.Source: §5 · p. 8
  • With a pretrained language model, generated frames are more correlated with out-of-distribution rewordings of the text inputs.Figure 6 compares first frames generated from equivalent descriptions absent from training, with and without RoBERTa.Source: Figure 6 · p. 8
  • The method remains limited by temporal consistency in high-resolution videos, diverse motion patterns in longer sequences, and diversity of non-autoregressive transformers.The paper names audio and higher resolution as possible future directions.Source: §5 · p. 8

A. More Details and Ablation for Methods

  • The appendix provides additional masking-strategy, relevance-estimation, VID-augmentation, mask-prediction, and text-augmentation details and ablations.For MSM, the default probabilities for strategies I–IV are [0.7, 0.1, 0.1, 0.1]; strategy V has probability 0.2 and keeps k = T/2 frames.Source: A. More Details and Ablation for Methods · p. 12, §A.1 · p. 12

A.2. Training Methods for REL Task

The appendix compares training approaches for relevance estimation and describes the video augmentations used to train the VID task. Negative sampling performs better early, but REL=swap is retained because negative sampling's FVD and F8 score become inferior at convergence.

A.2. Training Methods for REL Task

  • Negative sampling performs better early, but its FVD and F8 score become inferior at convergence, so REL=swap is the default.REL=negative samples a negative example with a different annotation from the positive; conditional swapping does not guarantee a truly negative sequence.Source: §A.2 · p. 12

A.3. Video Augmentation on VID Task

  • VID self-learning uses color jittering, affine transforms, frame swapping, and frame shuffling, sampled uniformly by default.Each augmentation strategy has probability 0.25.Source: §A.3 · p. 12

A.4. More Details on Improved Mask-Predict

The appendix details mask-predict sampling and annealing choices, including token and mask sampling, and compares schedules and noise settings. Its examples report that non-learning mask annealing preserves vivid motion and reduces artifacts relative to linear annealing, while noise annealing improves diversity with a quality tradeoff.

A.4. More Details on Improved Mask-Predict

  • Figure 7 illustrates VID-task augmentations through color jittering, affine transforms, frame swapping, and frame shuffling.The examples show the original frame, altered image frames, and temporal rearrangements.Source: Figure 7 · p. 13
  • Mask-predict samples tokens from logits perturbed by Gumbel noise, then uses sampled token probabilities to select locations for preservation.SampleMask samples n locations without replacement according to probabilities, while locations marked for preservation are always selected.Source: §A.4 · pp. 12–13
  • The piecewise mask-annealing schedule uses three phases, with default L1 = 10, L2 = 10, α1 = 0.9, β1 = 0.1, α2 = 0.125, and α3 = 0.0625.The scheme is used in Algorithm 1; the total number of mask-predict steps is L, with L1 + L2 ≤ L.Source: §A.4 · p. 13
  • Non-learning mask annealing produces vivid motion and fewer artifacts than linear annealing; at L = 20, the linear schedule also yields static videos with little visible motion.The comparison uses examples from models trained on Multimodal VoxCeleb; linear settings L = 20 and L = 50 show washed-out motion.Source: §A.4 · p. 13, Figure 8 · p. 14, Figure 9 · p. 15
  • Noise annealing improves video diversity, but too much noise can reduce sample quality.The authors suggest unconfident tokens that cannot be remasked may explain the quality tradeoff; Figure 10 compares nine videos per setting under the same text input.Source: §A.4 · p. 13, Figure 10 · p. 16
  • Early-stop does not improve FVD at B = 1, 3, or 5, and artifacts remain at the score-selected stopping steps.SREL peaks at step 3, SVID at step 15, and their average at step 9; the authors therefore do not use these special-token scores for early stopping.Source: §A.4 · p. 16

A.5. Analysis on Text Augmentation

The authors use human evaluation to compare MMVID, MMVID-TA, and ART-V on text-to-video generation using Multimodal VoxCeleb videos. In that evaluation, 61.2% of users preferred MMVID-TA over ART-V.

A.5. Analysis on Text Augmentation

  • 61.2% of users preferred MMVID-TA over ART-V in the human evaluation of text-to-video videos on Multimodal VoxCeleb.The comparison presented 600 synthesized videos to Amazon Mechanical Turk users; MMVID-TA uses text augmentation.Source: §A.5 · p. 16

A.6. Analysis on Conditioning Partially Occluded Images

The experiments test whether videos retain features from partially occluded visual inputs across three conditioning settings. About 70% of videos were judged to contain features from the occluded modalities, while the reported REL, VID, and average scores peak at different sampling steps.

A.6. Analysis on Conditioning Partially Occluded Images

  • REL is highest at step 3, VID at step 15, and the average score at step 9.At step 1, mask-predict starts from a fully masked sequence; the images beneath red masks are real video frames.Source: Figure 11 · p. 17
  • The three settings tested text paired with a partially occluded image, segmentation, or drawing, using human judgments of whether input features appeared in the video.Each setting contained 200 videos, half with conditions that did not describe the same identity, and each video was analyzed by 10 workers.Source: §A.6 · pp. 16–17
  • About 70% of generated videos were judged to contain features from the occluded visual modalities.The evaluation covered text with a partially occluded image, segmentation, and drawing; each setting included 200 videos, half with conditions describing a different identity, assessed by 10 workers per video.Source: §A.6 · p. 17

B. More Experimental Details

The additional materials document implementation and dataset choices, then provide generated examples across Shapes, MUG, and iPER. The figures illustrate text-to-video, multimodal control, long-sequence extrapolation, and interpolation; table captions identify human-preference evaluations without reporting their values.

B. More Experimental Details

  • The supplement adds human-preference evaluation tables for text-to-video and text-plus-occluded-image generation on Multimodal VoxCeleb.The supplied captions identify the evaluation settings but do not report table values.Source: Table 7 · p. 18, Table 8 · p. 18

B.1. More Implementation Details

  • The image autoencoder is finetuned from an ImageNet-pretrained VQGAN, using f = 16 and a 1,024-entry codebook; video CLIP score uses each video's maximum frame similarity.Shapes evaluation uses a classifier following TFGAN instructions, and TFGAN is retrained for a fair text-to-video comparison.Source: §B.1 · p. 18

B.2. Dataset Statistics and Textual Controls

  • The dataset descriptions cover Shapes, MUG, iPER, and Multimodal VoxCeleb; VoxCeleb has 13,706 unique attribute combinations among 19,522 samples, including 11,259 singletons.Text dropout is motivated as a way to discourage memorization of attribute combinations represented by one sample.Source: §B.2 · pp. 18–19, Figure 12 · p. 18

C. More Generated Videos

  • The supplement presents additional generated-video examples, with thumbnails shown in the figures and videos available on the project webpage.The examples span the tasks and datasets described in the following subsections.Source: C. More Generated Videos · p. 19

C.1. Results on the Shapes Dataset

  • Shapes examples cover text-to-video, independent text-plus-partially-observed-image control, and dependent text-plus-image control.The partially observed image has its center masked out; the figures show three, two, and one synthesized videos per input condition, respectively.Source: §C.1 · p. 19, Figure 13 · p. 19, Figure 14 · p. 19

C.2. Results on the MUG Dataset

  • The MUG examples show text-to-video generation.The figure presents three synthesized videos for each input text condition.Source: §C.2 · p. 19, Figure 16 · p. 19

C.3. Results on the iPER Dataset

  • On iPER, the examples include long-sequence extrapolation to 107 frames and interpolation by generating one frame between contiguous frames.Extrapolation is repeated 100 times per sequence, and text controls motion speed; the interpolation figures place real sequences beside generated interpolations.Source: §C.3 · p. 19

C.4. Results on the Multimodal VoxCeleb Dataset

The Multimodal VoxCeleb examples illustrate text-to-video and independent or dependent multimodal generation. The showcased controls include segmentation masks, drawings, partially observed images, and combinations of images with masks or motion-guiding video.

C.4. Results on the Multimodal VoxCeleb Dataset

  • The examples include text-to-video generation and independent controls pairing text with segmentation masks, drawings, or partially observed images.The figures show three synthesized videos per text condition and two per multimodal condition.Source: §C.4 · p. 19, Figure 19 · p. 20, Figure 20 · p. 20
  • Dependent multimodal examples combine text with an image and segmentation mask, or with a drawing and segmentation mask.Each figure shows two synthesized videos per input condition.Source: §C.4 · p. 20, Figure 23 · p. 21, Figure 24 · p. 21
  • Another dependent setting uses an image for appearance and a video for motion guidance.The motion guidance is more readily observed in the supplementary video; two synthesized videos are shown per condition.Source: §C.4 · p. 20, Figure 25 · p. 21
  • The supplement shows RoBERTa text augmentation alongside TFGAN and ART-V text-to-video examples, and ART-V independent generation with text and a segmentation mask.The examples are generated on Multimodal VoxCeleb.Source: §C.4 · p. 21, Figure 26 · p. 21, Figure 27 · p. 21
  • Additional examples test partially occluded-face inputs with a test-time occlusion pattern different from training and show nearest neighbors for generated samples.The nearest-neighbor examples include a text-image contradiction—text suggests a female while the image shows a beard—and conditioning variants found using face-similarity scores.Source: §C.4 · p. 21

D. Limitation and Future Work

The discussion reports artifacts and weaker temporal consistency at 256×256, as well as limited motion diversity in some slow extrapolations. It also describes a diversity-versus-consistency trade-off and raises misuse concerns, pointing to synthetic-video detection as a mitigation.

D. Limitation and Future Work

  • The supplement also shows independent multimodal examples with segmentation masks and partially observed faces, plus nearest-neighbor analyses of generated samples.The occlusion examples use a test-time pattern different from the training pattern.Source: Figure 29 · p. 22, Figure 31 · p. 22, Figure 32 · p. 22
  • At 256×256, some generated videos are blurry or show unusual colors, and temporal consistency is worse than at 128×128.The authors suggest longer token sequences per high-resolution frame as one possible reason and leave image-level temporal regularization for future work.Source: D. Limitation and Future Work · pp. 21–22, Figure 30 · p. 22
  • Long-sequence extrapolation may produce unreasonable motion, with slow-motion prompts sometimes yielding insufficiently diverse motion patterns.The authors suggest that short temporal sampling steps during training may fail to cover enough patterns and propose balancing training data as future work.Source: D. Limitation and Future Work · p. 22
  • The non-autoregressive transformer is more temporally consistent, while the autoregressive transformer may produce more diverse videos, often at low quality.Text augmentation and improved mask-predict are applied to improve non-autoregressive diversity; unifying the training approaches is left for future work.Source: D. Limitation and Future Work · p. 22

E. Ethical Implications

  • The authors flag potential nefarious use of realistic synthetic videos and point to synthetic-video detection as a way to help alleviate the concern.They note that artifacts may not be readily noticed by humans.Source: E. Ethical Implications · p. 22
Loading 2203.02573v1…