PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Animation of Fluid Elements in Still Images

Aniruddha Mahapatra, Kuldeep Kulkarni

arXiv:2112.03051v3cs.CV

TL;DR

Animating still images with user control is challenging because prior approaches offer limited motion input or are not suited to fluid elements. The paper converts a mask and user-specified directions and speeds into a refined dense flow map for frame generation, and reports better baseline performance and animations in unseen directions.

  • Problem

    The paper addresses controllable animation of fluid elements in still images, where existing methods offer limited user input or produce brittle flow estimates.

    Source: §1 · p. 1

  • Method

    It converts a user mask and sparse direction-and-speed hints into an approximate dense flow map, refines that map with a network, and uses it to generate frames.

    Source: §1 · p. 2, §3.3 · p. 3, §3.5 · p. 4

  • Results

    The method beats previous and proposed baselines qualitatively and quantitatively on a publicly available dataset, and produces animations in directions absent from training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method supports user-directed fluid animations from still images, including directions not represented in training.

    Source: §1 · pp. 1–2

    The method is restricted to movements of fluid elements in an image.

    Source: §5 · p. 8

Abstract

from arXiv · show

We propose a method to interactively control the animation of fluid elements in still images to generate cinemagraphs. Specifically, we focus on the animation of fluid elements like water, smoke, fire, which have the properties of repeating textures and continuous fluid motion. Taking inspiration from prior works, we represent the motion of such fluid elements in the image in the form of a constant 2D optical flow map. To this end, we allow the user to provide any number of arrow directions and their associated speeds along with a mask of the regions the user wants to animate. The user-provided input arrow directions, their corresponding speed values, and the mask are then converted into a dense flow map representing a constant optical flow map (FD). We observe that FD, obtained using simple exponential operations can closely approximate the plausible motion of elements in the image. We further refine computed dense optical flow map FD using a generative-adversarial network (GAN) to obtain a more realistic flow map. We devise a novel UNet based architecture to autoregressively generate future frames using the refined optical flow map by forward-warping the input image features at different resolutions. We conduct extensive experiments on a publicly available dataset and show that our method is superior to the baselines in terms of qualitative and quantitative metrics. In addition, we show the qualitative animations of the objects in directions that did not exist in the training set and provide a way to synthesize videos that otherwise would not exist in the real world.

1. Introduction

The paper targets controllable animation of fluid elements in still images, using a mask and sparse motion hints to create cinemagraphs. It approximates and refines a dense flow map, and reports strong baseline comparisons and animations in directions absent from training.

1. Introduction

  • Users specify an animation mask and sparse flow hints, with directions and speeds at arbitrary locations in the masked region.The target is fluid elements such as water, smoke, and fire, which have repeating textures and continuous motion.Source: §1 · pp. 1–2
  • Earlier controllable methods offer limited input, while self-similarity-based displacement is unsuitable for fluid elements and estimated dense flow can produce brittle results [1] [2] [5] [9].The cited methods include single-pixel or single-direction controls, and a sparse-trajectory method whose dense flow is described as brittle.Source: §1 · p. 2
  • The paper presents its approach as a two-stage method for interactively controlling fluid animation from still images.Its reported qualitative examples include fluid motion in directions absent from the training set.Source: §1 · p. 2
  • The approach first approximates a constant dense flow map from user-provided directions and speeds using exponential operations, then refines it with a GAN-based network.The refined flow map and input image are used to generate future frames.Source: §1 · p. 2
  • The method beats previous and proposed baselines qualitatively and quantitatively on a publicly available fluid-motion dataset.It also shows animations in directions that did not exist in the training set.Source: §1 · p. 2

2. Related Work

Prior single-image video synthesis includes both automatic methods and approaches that let users control motion. This work differs from the closest controllable approach in how it estimates flow and generates frames.

2. Related Work

  • Several single-image-to-video methods generate videos automatically and do not let users control the animation [11] [14] [30].The cited approaches include layer-based animation, GAN-based frame generation, and optical-flow-based methods.Source: §2 · p. 2
  • Earlier controllable methods often allow only one motion direction or a single poke, unlike this work’s sparse directions and speeds at arbitrary locations [1] [2] [5].The paper identifies sparse input directions and speeds at arbitrary locations as necessary for its target problem.Source: §2 · p. 2
  • Compared with Hao et al., this method approximates dense flow from user inputs with exponential functions, refines it with a network, and uses symmetric feature splatting to generate frames.Hao et al. directly estimate dense flow from sparse trajectories and bilinearly warp the input image.Source: §2 · pp. 2–3

3. Methodology

The method turns user-marked regions and motion hints into a constant dense flow, refines it, and uses it to warp image features into future frames. It builds on Eulerian flow and introduces a dense-flow construction to address the limits of directly refining sparse hints.

3. Methodology

  • A UNet frame generator warps input-image features at multiple resolutions using the refined flow to produce video frames.The pipeline first converts arrow directions and speeds into sparse flow, then dense flow, and refines it before frame generation.Source: §3 · p. 3

3.1. Preliminary: Eulerian Flow Fields

  • A constant, time-invariant Eulerian flow field models fluid-element motion between consecutive frames, and Euler integration yields flow from the first frame to any later frame.The method adopts this representation for fluid elements such as water, smoke, and fire.Source: §3.1 · p. 3

3.2. Baseline: Sparse Input Flow FS →Sparse Refined Flow F R

  • Sparse input flow FS places each arrow’s speed-scaled displacement at its start location and is zero elsewhere; the baseline refines it directly with SPADE using the image and mask as cues.This baseline produces constant dense flow, unlike Hao et al.’s time-varying flow maps.Source: §3.2 · p. 3

3.3. Sparse Input Flow FS →Dense Flow FD

  • Because directly refining FS yields flows very different from ground-truth Eulerian flows, the method first forms dense FD using exponential distances from pixels to arrow starts.FD is a weighted average of flow magnitudes at nonzero mask locations; the image, mask, and FD then guide flow refinement.Source: §3.3 · p. 3

3.4. Dense Flow FD →Dense Refined Flow F R

The method refines a simple dense flow map to address its boundary and within-region errors, then generates frames by splatting multi-scale image features. Symmetric splatting combines features from both endpoints to address complementary void regions.

3.4. Dense Flow FD →Dense Refined Flow F R

  • The simple dense flow can mix motion across object boundaries and assign inconsistent flow within one region, so the refinement network uses the image, mask, and flow as cues.In the waterfall example, the unrefined flow assigns horizontal motion to most of the waterfall and lake, although the waterfall would move downward.Source: §3.4 · p. 4

3.5. Video Frame Generation

  • Forward flow maps the input toward future frames, and softmax-splatting weights source pixels that collide at a destination using the predicted importance metric Z.Source: §3.5 · pp. 4–5
  • The refined dense flow best resembles ground-truth flow, while refined sparse flow is poor; across one and five hints, dense refinement is closer to ground truth.Source: Figure 3 · p. 5, Figure 4 · p. 6

3.5.1 Multi-scale Deep Feature Warping

  • A UNet warps image features at multiple resolutions with softmax-splatting, then uses its decoder and skip connections to generate the RGB frame.Source: §3.5.1 · p. 5

3.5.2 Symmetric Splatting

  • Symmetric splatting combines features warped from the first and last frames because their void regions are expected to be complementary.Source: §3.5.2 · p. 5

3.5.3 Training and Inference

  • Training separates flow refinement and frame generation before fine-tuning; at test time, the single input image serves as both endpoints for symmetric splatting.Source: §3.5.3 · p. 5

4. Experiments

The experiments compare video and flow generation against several baselines, using a dataset with heuristically generated masks and flow hints. The final method leads on FVD, while refined dense flows improve flow-map fidelity and respect arbitrary arrow directions.

4.1. Dataset preparation

  • The dataset is adapted from an uncontrollable video dataset, with heuristic masks and flow hints derived from ground-truth average flow.Masks retain pixels above a threshold of 10 times the mean-squared flow; k-means cluster centers provide 1, 3, or 5 flow hints.Source: §4.1 · p. 6

4.2. Experimental Setup

  • The flow-refinement and frame-generation networks are trained separately for 200 epochs, followed by 40 fine-tuning epochs with the flow generator fixed.Frames and flows are resized to 288x512 before training, and inference generates 60 frames.Source: §4.2 · pp. 6–7

4.3. Baselines

  • Baselines include Endo et al., Hao et al. modified with a frame generator, an exponential-flow method, and the automatic vanilla Eulerian method.The modified Hao baseline uses the paper's sparse hints and a GAN-based dense-flow network before frame generation.Source: §4.3 · p. 7

4.5. Results

  • The refined-flow frame generator performs best on FVD across hint counts, and FVD improves as hint counts increase.PSNR is comparable across methods, except Endo et al. is lowest; the table reports FVD and PSNR for methods and hint counts.Source: Table 1 · p. 7, §4.5 · p. 8

4.4. Metrics

  • FVD measures video fidelity using I3D features, while PSNR measures mean pixel accuracy and can favor blurrier outputs.Source: §4.4 · p. 7

4.5. Results

  • Refined dense flows outperform refined sparse flows on flow PSNR and resemble ground-truth flows more closely than the modified Hao et al. baseline.Refinement also corrects inappropriate flow regions; in the waterfall example, the refined-flow video restores downward motion instead of rightward drift.Source: §4.5 · pp. 7–8, Table 2 · p. 8
  • The flow generator respects arbitrary arrow directions, including directions absent from the training set.Figure 6 shows dense and refined flows for four directions on the same input image.Source: §4.5 · p. 8, Figure 6 · p. 8

5. Conclusions and Limitations

The method animates fluid elements from masks and direction-and-speed hints using a constant flow field. Its scope excludes rigid or definite structures and may not accommodate adjacent streams belonging to different objects.

5. Conclusions and Limitations

  • The method targets fluid elements such as water, fire, and smoke, given a mask and flow hints specifying speed and direction.Source: §5 · p. 8
  • The method approximates a constant flow field with exponential operations and refines the dense field rather than only sparse hints.Source: §5 · p. 8
  • The method is restricted to fluid motion; rigid objects, building designs, and adjacent streams from different objects are stated limitations.Source: §5 · p. 8

A. DTFVD scores:

The appendix reports DTFVD scores for generated videos and says the method performs best in video quality.

A. DTFVD scores:

B. Human Evaluation

The human evaluation compares generated videos with ground truth and arrow-indicated motion across two hint counts. Workers overwhelmingly favor the paper's method for both questions and both counts.

B. Human Evaluation

  • Workers compared three generated-video methods with ground truth, judging both representativeness and agreement with arrow-indicated motion.Source: B. Human Evaluation · p. 11
  • The evaluation sampled 50 test examples each for 3 and 5 hints, yielding 100 video comparisons answered by five workers each.Source: B. Human Evaluation · p. 11
  • The method was overwhelmingly favored on both questions at both hint counts, based on 250 responses per question and hint count.Source: B. Human Evaluation · p. 11

C. Video results:

The project page presents additional flow and video results, including controllable variations for the same input image using different arrow directions or flow hints.

C. Video results:

  • The project page shows videos for the same input image animated with different arrow directions or flow hints.It also links to additional flow and video results.Source: C. Video results · p. 11

D. Negative societal impact of the work:

The framework is described as enabling users to animate fluids in arbitrary directions from a single image; user preferences favor the method over alternatives.

D. Negative societal impact of the work:

  • The framework is intended to let users generate videos from a single image with fluids moving in arbitrary directions.Source: D. Negative societal impact of the work · p. 11
  • Across 250 responses per question and hint count, the aggregate preferences favor the method over other methods by a significant margin.Table 4 reports aggregated preferences; the supplied passage gives no preference counts.Source: Table 4 · p. 12
  • Table 4 aggregates preferences across questions and numbers of hints, rather than reporting a single preference result.Source: Table 4 · p. 12
Loading 2112.03051v3…