Source-linked AI summary
Video-to-Video Synthesis
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, Bryan Catanzaro
TL;DR
Video-to-video synthesis seeks to map input videos to photorealistic outputs while preserving content, a less-explored problem than image-to-image translation. The paper proposes a conditional GAN framework with specialized generators, discriminators, and spatio-temporal objectives, achieving coherent high-resolution videos and favorable benchmark results. It also reports limitations in handling ambiguous motion, consistent object appearance, and some semantic manipulations.
Problem
Video-to-video synthesis is less explored than image-to-image translation and requires mapping source videos to photorealistic outputs while preserving temporal dynamics.
Method
The paper uses conditional generative adversarial learning with carefully designed generators and discriminators plus a spatio-temporal objective.
Results
The method synthesizes high-resolution, photorealistic, temporally consistent videos, including 2K street-scene videos up to 30 seconds long, and compares favorably with competing methods.
Takeaways & Limitations
The framework supports video generation from segmentation masks, face sketches, and body poses, as well as multimodal outputs for the same input.
Takeaways & Limitations
The model struggles with turning cars, consistent object appearance, and artifacts during some semantic manipulations.
Abstract
from arXiv · showhide
We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video. While its image counterpart, the image-to-image synthesis problem, is a popular topic, the video-to-video synthesis problem is less explored in the literature. Without understanding temporal dynamics, directly applying existing image synthesis approaches to an input video often results in temporally incoherent videos of low visual quality. In this paper, we propose a novel video-to-video synthesis approach under the generative adversarial learning framework. Through carefully-designed generator and discriminator architectures, coupled with a spatio-temporal adversarial objective, we achieve high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats including segmentation masks, sketches, and poses. Experiments on multiple benchmarks show the advantage of our method compared to strong baselines. In particular, our model is capable of synthesizing 2K resolution videos of street scenes up to 30 seconds long, which significantly advances the state-of-the-art of video synthesis. Finally, we apply our approach to future video prediction, outperforming several state-of-the-art competing systems.
1 Introduction
The paper introduces video-to-video synthesis as learning a mapping from input videos to corresponding photorealistic output videos, addressing a less-explored problem than image-to-image translation. It uses conditional adversarial learning with spatio-temporal modeling to produce high-resolution, photorealistic, temporally coherent videos.
- Problem: Video-to-video synthesis learns a mapping from an input video to an output video depicting the source content.The paper studies this form separately from future prediction and unconditional video synthesis.
- Approach: The approach casts synthesis as conditional distribution matching using a conditional generative adversarial model trained on paired input and output videos.The model is designed to make synthesized conditional video distributions resemble those of real videos.
- Approach: Carefully designed generators, discriminators, and a spatio-temporal objective target high-resolution, photorealistic, temporally coherent videos.The framework also supports diverse appearances for the same input through multimodal synthesis.
- Results: Experiments report more photorealistic footage than strong baselines and demonstrate 2K videos lasting up to 30 seconds.The method also supports segmentation masks, face sketches, and body poses, enabling controls such as replacing buildings with trees.
2 Related Work
Related work covers image translation, unconditional video synthesis, future video prediction, and application-specific video transformations. The paper positions video-to-video synthesis as requiring both per-frame photorealism and temporal coherence while seeking broader applicability than specialized methods.
- Generative Adversarial Networks: Conditional GANs extend adversarial generation by allowing inputs such as images, categorical labels, or textual descriptions to control outputs.The paper builds its model on this conditional GAN framework.
- Image-to-Image Translation: Image-to-image methods translate individual images between domains, whereas video synthesis must additionally maintain temporal coherence across frames.Long-duration videos make temporal consistency especially challenging.
- Unconditional Video Synthesis: Unconditional video synthesis converts a random vector into a video, but existing methods often produce low-resolution, short-length results.Examples include spatio-temporal convolutional, latent-code, and motion-content disentangling approaches.
- Future Video Prediction: Future video prediction conditions on observed frames to predict future ones, yet reconstruction objectives can produce blurry videos and adversarial models may fail on long durations.Video-to-video synthesis differs because it maps an input video rather than predicting only its future.
- Video-to-Video Synthesis: Existing video super-resolution, matting, blending, inpainting, and style-transfer methods rely on problem-specific constraints and designs.Those constraints make them difficult to apply across other applications.
3 Video-to-Video Synthesis
The method formulates video-to-video synthesis as conditional distribution matching and combines sequential generation with spatio-temporal adversarial learning. Its architecture uses flow-guided warping, foreground-background modeling, and image/video discriminators to produce photorealistic, temporally coherent videos.
- Results and extensions: The approach matches conditional video distributions to generate photorealistic, temporally coherent sequences and supports multimodal outputs with diverse appearances from the same input.Multimodal synthesis uses instance-level feature embeddings for the source video.
- Conditional video synthesis: The model maps source video frames, such as semantic segmentation masks or edge maps, to corresponding output video frames through a conditional GAN.Training matches the conditional distribution of synthesized videos to that of real videos.
- Sequential generator: The sequential generator assumes each generated frame depends on the current source frame, past L source frames, and past generated frames.The generator recursively applies a feed-forward network F; experiments use L = 2.
- Flow-guided generation: Flow-guided synthesis warps pixels from the previous frame and hallucinates new pixels for occluded regions, using a soft occlusion mask.The soft mask also supports gradual blending when newly synthesized texture details are needed.
- Spatio-temporal objective: The objective combines conditional image and video GAN losses with flow estimation loss, plus feature-matching losses that improve convergence speed and training stability.The video discriminator evaluates temporal dynamics across consecutive frames, while the image discriminator evaluates individual source-output pairs; λ_W is set to 10.
- Foreground-background prior: Foreground-background modeling uses segmentation-derived masks to warp background content while synthesizing foreground content, improving visual quality with minor flickering artifacts.A user study reports that most people prefer results using foreground-background modeling.
4 Experiments
Experiments evaluate the approach across multiple datasets and video-to-video tasks, using quantitative metrics, human preferences, qualitative comparisons, ablations, and applications to multimodal synthesis, semantic manipulation, face swapping, pose transfer, and future prediction.
- Implementation: The coarse-to-fine generator progresses through 512 × 256, 1024 × 512, and 2048 × 1024 resolutions while training from few frames to 30 or more.Training 2K resolution takes approximately 10 days on eight V100 GPUs.
- Datasets and evaluation: The experiments use Cityscapes, Apolloscape, face videos, and dance videos for segmentation-, sketch-, and pose-to-video synthesis.The baselines are trained on the same data, and evaluation combines subjective and objective metrics.
- Main comparisons: The proposed method achieves smaller FID values and higher human preference than strong baselines on Cityscapes and future video prediction.Human studies assess realism using temporal coherence and image quality, while the video FID variant measures visual quality and temporal consistency.
- Main comparisons: 30-second, high-resolution videos remain photorealistic and temporally consistent, whereas pix2pixHD and COVST exhibit temporal inconsistencies.The comparison highlights inconsistent lane markings and building appearances in baseline outputs.
- Ablation study: Removing the foreground-background prior, conditional video discriminator, or flow warping significantly degrades output visual quality.Using ground-truth flows instead of estimated flows produces visually similar results, indicating robustness to estimated-flow errors.
- Additional capabilities: The model generates temporally smooth videos with different road appearances when conditioned on the same input.Multimodal variation is produced by changing sampled feature vectors for the road instance while keeping other object-instance vectors fixed.
- Additional capabilities: Users can replace trees with buildings or buildings with trees in segmentation masks and synthesize corresponding videos.The approach also converts face sketches to photorealistic videos and human poses to high-resolution dance videos with unseen body shapes and motions.
- Additional capabilities: Pose-to-video synthesis supports changing clothing for the same dancer and transferring visual appearance across dancers.These results extend the method beyond semantic segmentation inputs to sketches and poses.
5 Discussion
The framework achieves high-resolution, photorealistic, temporally consistent video synthesis and compares favorably with state-of-the-art methods, but remains limited in several scenarios.
- The framework synthesizes high-resolution, photorealistic, and temporally consistent videos using conditional GANs, designed generators and discriminators, and a spatio-temporal adversarial objective.
- Extensive experiments report results significantly better than state-of-the-art methods and favorable comparisons with competing video prediction methods.
- The model struggles with turning cars because label maps provide insufficient information, suggesting a need for additional 3D cues such as depth maps.
- Object appearance is not guaranteed to remain consistent across a video, with cars occasionally changing color gradually.
- Semantic manipulations can produce artifacts when buildings and trees have different label shapes.
A.1 Generators
The generator uses a coarse-to-fine architecture that processes semantic label maps and previously generated images to produce hallucinated images, flow maps, and masks across resolutions.
- The lowest-resolution network processes several semantic label maps and previously generated images to form intermediate features.
- The generator combines label-map and image features before separate residual networks output the hallucinated image, flow map, and mask.
- Higher-resolution networks build on lower-resolution results by adding features from the preceding network to intermediate features before producing higher-resolution images.
A.2 Discriminators
The discriminator system combines a multi-scale image discriminator with a temporally multi-scale video discriminator to enforce both short-term and long-term consistency.
- The image discriminator uses a multi-scale PatchGAN architecture.
- The video discriminator downsamples frame rates and evaluates K consecutive frames at multiple temporal scales.
- Using up to three temporal scales helps enforce both short-term and long-term consistency, while the video discriminator is also spatially multi-scale.
A.3 Feature matching loss
The objective adds VGG and discriminator feature matching losses to improve training stability and match feature statistics between real and generated outputs.
- VGG feature matching minimizes L1 distances between features extracted from real and generated images.
- Discriminator feature matching matches statistics of features extracted by the GAN discriminators.
- The method uses both image and video discriminators for feature matching.
B Evaluation for the Apolloscape Dataset
On Apolloscape, the proposed method is evaluated against competing video-to-video synthesis approaches using FID and human preference scores, outperforming the baselines on both measures.
- Our method outperforms the other baselines on both FID and human preference score.These metrics combine an automated distributional measure with human judgments of video quality.
- The Apolloscape comparison therefore reports both quantitative and human-evaluated evidence for the method’s synthesized videos.
- The evaluation compares competing video-to-video synthesis approaches on the Apolloscape dataset.