Source-linked AI summary
Animating Landscape: Self-Supervised Learning of Decoupled Motion and Appearance for Single-Image Video Synthesis
Yuki Endo, Yoshihiro Kanamori, Shigeru Kuriyama
TL;DR
Creating natural, high-quality video from one scenery image remains challenging because future motion and appearance are uncertain. This paper uses self-supervised, decoupled prediction with latent codes to generate controllable, high-resolution landscape animations. The authors report competitive or superior subjective ratings against prior methods and commercial software, while noting scope and long-term prediction limits.
Problem
Generating natural animation from one scenery image remains challenging, and single-image generative methods struggle with video quality and uncertain future frames.
Source: §1 · pp. 1–2Method
Results
The user study found the results subjectively competitive or superior to previous methods or commercial software.
Source: §1 · p. 2Takeaways & Limitations
The method produces high-resolution, longer-term animations with separate motion and appearance control, primarily for landscapes such as skies and waters.
Source: §1 · p. 2, §6.6 · p. 13, §7 · p. 14The method focuses on skies and waters, and very long-term predictions can distort because frames are reconstructed from the input image.
Source: §6.6 · p. 13
Abstract
from arXiv · showhide
Automatic generation of a high-quality video from a single image remains a challenging task despite the recent advances in deep generative models. This paper proposes a method that can create a high-resolution, long-term animation using convolutional neural networks (CNNs) from a single landscape image where we mainly focus on skies and waters. Our key observation is that the motion (e.g., moving clouds) and appearance (e.g., time-varying colors in the sky) in natural scenes have different time scales. We thus learn them separately and predict them with decoupled control while handling future uncertainty in both predictions by introducing latent codes. Unlike previous methods that infer output frames directly, our CNNs predict spatially-smooth intermediate data, i.e., for motion, flow fields for warping, and for appearance, color transfer maps, via self-supervised learning, i.e., without explicitly-provided ground truth. These intermediate data are applied not to each previous output frame, but to the input image only once for each output frame. This design is crucial to alleviate error accumulation in long-term predictions, which is the essential problem in previous recurrent approaches. The output frames can be looped like cinemagraph, and also be controlled directly by specifying latent codes or indirectly via visual annotations. We demonstrate the effectiveness of our method through comparisons with the state-of-the-arts on video prediction as well as appearance manipulation.
1 INTRODUCTION
Animating a natural video from one scenery image remains challenging: earlier single-image generative methods struggle with video quality and future uncertainty. The paper separates motion and appearance prediction, using self-supervised intermediate maps and latent codes to generate controllable, high-resolution animations.
1 INTRODUCTION
- Example-based animation relies on reference videos matching the target scene, while learning-based single-image video synthesis still faces quality and uncertainty challenges.The paper positions its approach against both exemplar transfer and DNN-based methods.Source: §1 · pp. 1–2
- Single-image video methods produce lower-resolution, lower-quality results than image-generation methods, and uncertainty makes a unique future-frame prediction difficult.Prior VAE-based approaches remain insufficient for realistic, diverse results.Source: §1 · p. 2
- The method separately learns motion and appearance because they vary at different time scales and spatial patterns, such as fast cloud movement versus slowly changing sky color.It uses self-supervised training on time-lapse videos.Source: §1 · p. 2
- Motion prediction uses backward flow fields and appearance prediction uses spatially smooth color-transfer functions, with latent codes representing uncertain future variations.The paper trains these intermediate predictions without explicit ground-truth flow or color-transfer data.Source: §1 · p. 2
- The framework uses fully convolutional CNN models and reports higher-resolution, longer-term generation from hundreds to thousands of time-lapse videos, with separate latent-code control of motion and appearance.Its 2D convolutional networks recurrently predict frames without a fixed predicted-frame limit.Source: §1 · p. 2
- A user study found the results subjectively competitive or superior to previous methods or commercial software.The authors also demonstrate control applications for motion and appearance.Source: §1 · p. 2
2 RELATED WORK
The related work spans exemplar transfer, learned flow and color manipulation, and video prediction from still images. The paper distinguishes its approach through self-supervision, 2D convolutions, and decoupled appearance control.
2.1 Optical Flow Prediction
- Optical-flow prediction has used synthetic ground-truth flow, but those methods yield poor predictions for time-lapse videos; self-supervised methods estimate flow from consecutive frames, while this method predicts from one image with latent codes.The single-image inference setting is used to handle prediction uncertainty.Source: §2.1 · p. 2
2.2 Appearance Manipulation
- Color transfer changes overall image color while retaining scene structure, whereas style transfer can overfit and produce unnatural results; the paper instead assigns texture transfer to motion prediction and appearance changes to color-transfer functions.Earlier work also explored local color distributions, semantic correspondences, and sky-specific transfer.Source: §2.2 · pp. 2–3
2.3 Video Generation from a Still Image
- Prior scene-animation methods include parameterized stochastic motion textures and exemplar transfer, which can require a matching reference video, larger databases, or manual specification.The paper contrasts those requirements with automatic generation and latent-code control.Source: §2.3 · p. 3
- Earlier DNN video predictors were limited to 16 or 32 frames at 128 × 128, while this method predicts higher-resolution, unlimited-length videos using self-supervised flow, 2D CNNs, and decoupled appearance control.Compared with Li et al. [2018], it does not require ground-truth flow and provides direct appearance-transition control.Source: §2.3 · p. 4
3 METHOD OVERVIEW
The inference pipeline predicts motion and appearance separately from the input image. Latent codes control uncertainty, with a single code for motion and a sequence of codes for appearance.
3 METHOD OVERVIEW
- The motion predictor recurrently infers backward flows, but reconstructs each frame from the input image; appearance maps are predicted directly from the input and applied without recurrent feeding.Sampling input-image RGB values once per output frame avoids RGB error accumulation and blur from repeated color sampling.Source: §3 · p. 4
- Latent codes handle future uncertainty: one code controls motion across an animation, while appearance uses a code sequence matching the output-frame length.Codes can be selected automatically or manually from the training codebook.Source: §3 · p. 4
4 MODELS
The models separately predict motion and appearance from an input image, using latent codes to represent future variation. Motion is predicted recurrently through backward flows, while appearance is applied through color-transfer maps.
4 MODELS
- Both predictors are fully convolutional encoder-decoders, and their latent codes represent uncertainty in future motion or appearance.The motion code controls a whole sequence, whereas each appearance code controls one frame.Source: §4 · p. 4
4.1 Motion Predictor
- The motion predictor learns future flow fields self-supervised from time-lapse videos without ground-truth flows; without restriction (β = 1), flows are inconsistent and reconstructed images corrupted, while β = 64 yields frames closer to ground truth.Source: §4.1 · p. 5
4.2 Appearance Predictor
- Appearance prediction infers a color-transfer map for each frame from an input image and a latent code, enabling variation without recurrently feeding output frames.Training uses pairs of frames with different appearances and combines style, spatial-color, content, and regularization losses.Source: §4.2 · pp. 5–6
4.3 Implementation
- Training samples frame pairs randomly per video and epoch; motion uses consecutive frames, whereas appearance can use any pair.The predictors are trained at 256 × 256 and the encoders at 128 × 128; latent codes have 8 dimensions.Source: §4.3 · pp. 6–7
5 SINGLE-IMAGE VIDEO GENERATION
Video generation combines recurrent motion prediction with per-frame appearance prediction. The motion sequence can optionally be looped, and latent codes and speed adjustments control its motion and appearance variations.
5 SINGLE-IMAGE VIDEO GENERATION
- The motion predictor generates a frame sequence that can be made periodic by cross-fading, then appearance prediction produces the final video.Cross-fading worked relatively well for plausible animations without significant discontinuities among the tested loop-making methods.Source: §5 · p. 7
- Predictors process fixed-resolution inputs, but their inferred maps are resized and applied to the original image, so output resolution is not limited.Reflection padding handles sampling outside previous flow fields during reconstruction.Source: §5 · p. 7
- Motion and appearance speeds can be adjusted separately, while latent codes can be selected randomly or manually from a codebook.Motion speed scales flow fields; appearance speed can be controlled by its latent-code sequence or by repeating the motion loop during an appearance cycle.Source: §5 · p. 7
6 EXPERIMENTS
The experiments compare the method with video-prediction and appearance-manipulation approaches, evaluate latent-code and annotation-based controls, and examine loss functions, regularization, and limitations. They report clearer, higher-resolution video and smoother appearance transitions, while identifying boundaries in very long-term and non-uniform motion generation.
6.1 Dataset Generation
- The experiments use separate time-lapse datasets for motion and appearance, with 1,825 motion clips for training and 125 one-day clips for appearance modeling.The motion set has 224 test clips; generated 1,010-frame cinemagraphs took 98 seconds overall in the reported setup.Source: §6.1 · pp. 7–8, §6 · p. 7
6.2 Comparisons with Video Prediction Models
- Our method generates clearer, higher-resolution frames and looped animations with appearance variation compared with prior video-prediction methods.The comparison also reports more plausible scene-structured flow fields; quantitative evaluation across 224 test clips finds better perceptual similarity despite increasing difficulty over time.Source: §6.2 · p. 8, Figure 9 · p. 9, Figure 10 · p. 9
6.3 Comparisons on Appearance Manipulation
- Latent-space interpolation produces natural, smooth appearance transitions without requiring additional inputs beyond latent codes from target images.Compared methods either require target videos or semantic segmentation maps, and frame-by-frame optimization can cause flickering.Source: §6.3 · p. 9
6.4.1 Effects of latent codes.
- Similar motion latent codes yield similar motions while preserving variety, and consecutive appearance codes produce smooth transitions.Different appearance-code sequences also generate varied looks, including sunset, twilight, and night, from the same input image.Source: §6.4.1 · p. 10
6.4.2 Motion control using arrow annotations.
- Sparse arrow annotations can guide motion latent-code optimization to generate plausible full flow fields aligned with the specified directions.Each user edit took about seven seconds to optimize.Source: §6.4.2 · pp. 10–11
6.4.3 Appearance control using image patches.
- Appearance patches let users specify looks at selected positions and frames, combine multiple patches, and create cyclic animations through interpolation.The examples include interpolation between the final and input images across frames 300–400.Source: §6.4.3 · pp. 11–12
6.5 Ablation Study
- Removing motion or appearance regularization causes scene-structure inconsistencies or artifacts, while removing spatial color-distribution loss weakens localized sunset coloration.Direct frame inference without color-transfer functions is less natural than the alternatives.Source: §6.5 · p. 12, Figure 18 · p. 13
6.6 Discussion
- VAE and WAE regularizers support plausible outputs from Gaussian-sampled latent codes, but stored codebooks remain the default and are needed to select appearance sequences.Without regularization, Gaussian-sampled codes failed to produce plausible results; WAE yielded more varied outputs than VAE. Very long-term predictions distort, non-uniform motions are difficult, and the study focuses on landscape scenes, especially skies and waters.Source: §6.6 · pp. 12–13
7 CONCLUSION
The method synthesizes plausible high-resolution animations from a single scenery image through self-supervised, decoupled learning and latent-space interpolation. It produces higher-resolution, longer-term sequences than previous methods and supports control through latent codes or simple annotations.
7 CONCLUSION
- The method produces higher-resolution, longer-term sequences than previous methods by predicting regularized intermediate maps instead of output frames directly.Latent codes extracted during training enable direct codebook control and indirect control through simple annotations.Source: §7 · p. 14
- Self-supervised decoupled learning and latent-space interpolation provide separated control over motion and appearance variations.Source: §7 · p. 14
- The authors identify higher-quality animation as a future direction and suggest semantic segmentation or explicit occlusion information as possible additions.They avoid segmentation to prevent segmentation errors from influencing results.Source: §7 · p. 14
A.1 Network Architectures
The appendix identifies separate network-architecture tables for the motion and appearance predictors and encoders.
A.1 Network Architectures
- Table 1 summarizes the motion and appearance predictors, while Table 2 summarizes their motion and appearance encoders.Source: §A.1 · p. 15
A.2 Training Algorithms
Training uses time-lapse clips for the motion and appearance predictors, with motion minibatches restricted to selected clips within each epoch. An LSTM can recurrently predict appearance latent codes from an input-image code.
A.2 Training Algorithms
- Motion and appearance predictors are trained on separate time-lapse video datasets, and each motion minibatch draws frames from clips selected for that epoch.A latent code is learned and saved for each training video.Source: §A.2 · p. 15
B LATENT CODE PREDICTION USING LSTM
- At inference, an LSTM can predict successive appearance latent codes recurrently from the code encoded from the input image.The LSTM is trained in advance using latent-code sequences in the codebook.Source: B LATENT CODE PREDICTION USING LSTM · p. 15
- Table 1 presents the network architectures of the motion and appearance predictors, while Table 2 presents those of the motion and appearance encoders; Table 2 uses the same notations as Table 1.Source: Table 1 · p. 16, Table 2 · p. 16
C GENERALIZABILITY
The paper tests generalizability beyond the cloud-like motion and one-day appearance transitions common in its time-lapse datasets. Results include plausible KTH gait motion and broader seasonal appearance variation than Photoshop Match Color.
C GENERALIZABILITY
- The evaluation extends beyond the cloud-like motion and one-day appearance transitions that dominate the available time-lapse datasets.Source: C GENERALIZABILITY · p. 16
- For KTH gait, results are more plausible than Denton and Birodkar [2017] and Xue et al. [2016], with first-half quality comparable to Li et al. [2018].Source: C GENERALIZABILITY · p. 16
- The latter-half gait frames leave room to improve prediction of which leg follows a leg-crossed pose, a long-term dependency left for future work.Source: C GENERALIZABILITY · p. 16
- Season-transition sequences show wider spatially local appearance variation and greater faithfulness to target images than Photoshop Match Color, including across times of day.Source: C GENERALIZABILITY · p. 16
D USER STUDY
In a user study, the method significantly outperformed previous methods on motion and appearance plausibility, and compared favorably with commercial software. It also reproduced target styles more faithfully and handled wider appearance variations than commercial software.
D USER STUDY
- The study used 20 scenes, 11 subjects, and a 1-to-4 rating scale, with separate criteria for motion plausibility and appearance faithfulness.Ten scenes were compared with previous methods and ten with commercial software.Source: D USER STUDY · pp. 16–17
- The method significantly outperformed previous methods in motion and appearance plausibility.The reported significance was based on paired t-tests.Source: D USER STUDY · p. 17, Figure 24 · p. 18
- Motion scores were slightly better than those for manually created commercial-software animations.Source: D USER STUDY · p. 18
- The method reproduced target styles more faithfully and handled wider appearance variations than commercial software.Source: D USER STUDY · pp. 18–19