Source-linked AI summary
Few-shot Video-to-Video Synthesis
Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, Bryan Catanzaro
TL;DR
Existing vid2vid systems are data-hungry and limited to training subjects or scene domains, motivating more scalable unseen-domain synthesis. The paper introduces few-shot vid2vid, which uses test-time example images to generate adaptive network weights, and reports effective results across dance, talking-head, and street-scene videos. Its scope remains constrained when test domains differ greatly from training domains or semantic input estimations fail.
Problem
Existing vid2vid approaches require substantial target-domain data and have limited generalization to unseen subjects or scenes.
Method
Few-shot vid2vid uses a semantic video plus a few target-domain example images to dynamically configure synthesis through generated network weights.
Results
Experiments across dance, talking-head, and street-scene datasets show that the approach effectively addresses existing vid2vid limitations.
Takeaways & Limitations
The framework synthesizes videos of unseen subjects or street-scene styles at test time and performs favorably against competing methods.
Takeaways & Limitations
Performance degrades when test domains differ substantially from training domains, and failures in pose or segmentation estimates can also cause failures.
Abstract
from arXiv · showhide
Video-to-video synthesis (vid2vid) aims at converting an input semantic video, such as videos of human poses or segmentation masks, to an output photorealistic video. While the state-of-the-art of vid2vid has advanced significantly, existing approaches share two major limitations. First, they are data-hungry. Numerous images of a target human subject or a scene are required for training. Second, a learned model has limited generalization capability. A pose-to-human vid2vid model can only synthesize poses of the single person in the training set. It does not generalize to other humans that are not in the training set. To address the limitations, we propose a few-shot vid2vid framework, which learns to synthesize videos of previously unseen subjects or scenes by leveraging few example images of the target at test time. Our model achieves this few-shot generalization capability via a novel network weight generation module utilizing an attention mechanism. We conduct extensive experimental validations with comparisons to strong baselines using several large-scale video datasets including human-dancing videos, talking-head videos, and street-scene videos. The experimental results verify the effectiveness of the proposed framework in addressing the two limitations of existing vid2vid approaches.
1 Introduction
Vid2vid converts semantic videos into photorealistic videos, but conventional models require substantial target-domain data and do not generalize to unseen people or scene styles. Few-shot vid2vid addresses this by using test-time example images to configure synthesis, with experiments across multiple video domains supporting its effectiveness.
- Existing vid2vid: Vid2vid converts semantic input videos, such as poses or segmentation masks, into corresponding photorealistic videos.Applications include human-dancing and driving videos.
- Existing vid2vid: Existing approaches generally require separate target-domain training and cannot synthesize unseen people or scene styles directly.A new person or scene may require collecting a dataset and training another model.
- Few-shot vid2vid: Few-shot vid2vid adds a few target-domain example images at test time alongside the semantic video input.The examples are used to dynamically configure the video synthesis mechanism.
- Few-shot vid2vid: The framework generates network weights from example images and uses a learning objective designed to facilitate this weight-generation module.This dynamically configures synthesis for the target domain.
- Experiments: Experiments on dance, talking-head, and street-scene datasets show effective handling of existing vid2vid limitations, with quality improving as training diversity and test-time examples increase.The reported trend links broader training-domain diversity and more test-time examples with better generalization or synthesis quality.
2 Related Work
Vid2vid builds on image-to-image synthesis while adding the challenge of temporal consistency. This work targets unseen-domain video synthesis with few test-time examples, using adaptive network weights rather than task-specific human-body priors.
- GANs and video generation: Conditional GANs can generate photorealistic videos from user-provided semantic video data rather than random noise.Vid2vid is the video-conditional category of generative models.
- Image-to-image synthesis: Image-to-image synthesis provides the foundation for vid2vid, while video synthesis must maintain temporal consistency across frames.Individual frame realism is insufficient when the sequence is inconsistent.
- Few-shot vid2vid: Unlike prior vid2vid methods, the proposed approach synthesizes unseen domains by leveraging a few example images at test time.This is the paper’s stated distinction from prior works in the related category.
- Adaptive networks: The method applies adaptive networks to vid2vid, dynamically computing part of the network weights from input data.This differs from FUNIT’s image-focused adaptive instance normalization approach.
- Human pose transfer: Unlike human pose transfer methods that use specific body priors, this method uses no specific human-body priors beyond the input semantic video.The stated goal is a more general approach.
3 Few-shot Video-to-Video Synthesis
Few-shot vid2vid extends sequential semantic-to-photorealistic video synthesis with target-domain example images available at test time. A weight-generation module extracts and attentively aggregates appearance patterns to dynamically configure the synthesis network while retaining shared warping components.
- Video-to-video synthesis: Vid2vid learns to convert semantic image sequences into photorealistic output image sequences under a sequential generative model.The sequential generator predicts each output frame from observed semantic frames and previously generated outputs.
- Few-shot vid2vid: Few-shot vid2vid adds target-domain example images and their corresponding semantic images as inputs available at test time.This lets the generator extract patterns useful for synthesizing videos of unseen domains.
- Network weight generation: The network weight generation module E extracts appearance representations and computes weights θH for the intermediate image synthesis network H.For one example, EF extracts representation q and EP maps it to synthesis-network weights across corresponding layers.
- Shared warping components: The flow prediction network W and soft occlusion map network M remain shared across domains because they warp previously generated images rather than model domain-specific appearance.The framework reuses these components while adapting the intermediate image synthesis network H.
- Network weight generation: E generates weights only for SPADE spatial modulation branches, reducing generated parameters and avoiding a shortcut from example images directly to output images.The shared main synthesis branch remains separate from the dynamically generated modulation weights.
- Attention-based aggregation: For K > 1, an attention network aggregates example representations using semantic-image-conditioned attention before EP generates the synthesis weights.Weighted aggregation can align relevant patterns across examples showing different parts or views of a target subject.
4 Experiments
Experiments evaluate few-shot video synthesis across dancing, street-scene, and face videos using quantitative metrics, human judgments, visualizations, and strong baselines. Results show effective motion and style transfer, consistent metric improvements, and benefits from more training domains and test-time examples.
- Datasets: The evaluation uses YouTube dancing, street-scene, and FaceForensics video datasets, with example images and semantic inputs unavailable during training for relevant test settings.The street-scene data spans Germany, Boston, and NYC; the face dataset contains news-briefing videos from different reporters.
- Baselines: Three constructed baselines—Encoder, ConcatStyle, and AdaIN—encode example images through alternative style-conditioning mechanisms, while comparisons also include PoseWarp and MonkeyNet.All baseline comparisons and figures use one example image.
- Human synthesis: The method successfully transfers human motion across example images, while competing methods show visible artifacts or fail to transfer motion faithfully.The human synthesis visualizations use the same pose video with different example images.
- Street-scene synthesis: Street-scene outputs vary with different example images despite identical input segmentation maps, reflecting target-specific visual styles.The method is reported to synthesize realistic street-scene videos even when the style is not included in training.
- Results and analysis: The method consistently outperforms other methods on all reported performance metrics and preserves face identity while capturing input motion.Additional experiments find better synthesis quality with more training videos and more test-time example images.
5 Conclusion
The paper presents a few-shot vid2vid framework for synthesizing videos of unseen subjects or street-scene styles at test time. Its adaptive network weight generation scheme dynamically determines weights from example images and performs favorably against competing methods.
- The framework synthesizes videos of unseen subjects or street-scene styles at test time.
- Its adaptive network weight generation scheme dynamically determines synthesis-network weights from example images.
- The experimental results show favorable performance against competing methods.
A Comparisons to Baselines for Street Scenes
On street-scene sequences, the proposed method is the only compared approach that realistically reproduces the style shown in the example images. Other baselines generate artifacts or fail to capture that style.
- The proposed method is the only compared approach that realistically reproduces example-image style in street-scene sequences.
- Other baseline methods either generate artifacts or fail to capture the example-image style.
- The comparison evaluates the proposed method against baseline methods on street-scene sequences.
B Details of Our Baseline Methods
The baselines encode one example image into style information and inject it into different parts of a SPADE-based synthesis pipeline. They differ in whether the style code enters the branch head, the semantic input, or normalization layers.
- Evaluation setting: The baselines are designed for one example image and are compared with the proposed method in that setting.
- Encoder baseline: The Encoder baseline feeds an example image’s encoded style latent code into the head of SPADE’s main image-synthesis branch.
- ConcatStyle baseline: The ConcatStyle baseline concatenates the encoded style code with the input semantic image before feeding it to SPADE modules.
- AdaIN baseline: The AdaIN baseline converts an encoded example-image vector into mean and variance vectors for AdaIN operations.
C Discussion with AdaIN
The proposed weight-generation scheme is more expressive than AdaIN and can be combined with SPADE. Experiments examine both this design difference and how dataset size affects AdaIN’s performance.
- Method comparison: AdaIN represents example-image information as scaling and biased vectors, whereas the proposed scheme generates broader convolutional weights.
- Method comparison: The proposed weight-generation scheme can generate SPADE-layer weights that produce spatially adaptive demodulation parameters.
- Ablation comparison: The comparison tests AdaIN against weighted averaging and the proposed attention module to assess weight generation’s importance.
- Dataset-size discussion: As dataset size grows, AdaIN starts to fail because its expressibility is limited.
D Comparison with vid2vid
Compared with vid2vid, the proposed method substantially reduces adaptation requirements while maintaining comparable synthesis quality for a specific person, even with one example image.
- The method requires only one image and negligible time for weight generation, whereas vid2vid typically requires minutes of training data and days of training time.
- With K = 1, the model renders results comparable to vid2vid for synthesizing a specific person.
- Finetuning on the example images can achieve comparable or even better performance than vid2vid.