PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Ken Burns Effect from a Single Image

Simon Niklaus, Long Mai, Jimei Yang, Feng Liu

arXiv:1909.05483v1cs.CVcs.GR

TL;DR

Creating a compelling 3D Ken Burns effect from one image is difficult because manual editing is laborious and existing automatic methods require multiple viewpoints. The paper combines semantic-aware depth estimation with context-aware depth-based view synthesis, producing realistic results and enabling users to achieve better effects with less effort than existing solutions.

  • Problem

    The paper addresses the challenge of synthesizing realistic moving-camera 3D Ken Burns effects from a single image, where depth recovery and disocclusion must be handled.

  • Method

    The system predicts and refines scene depth, maps the image to a point cloud, and uses context-aware color- and depth-inpainting for coherent novel-view synthesis.

  • Results

    Experiments on varied imagery show realistic synthesis, while user studies report results rated on-par with professional artists.

  • Takeaways & Limitations

    The system supports both fully automatic generation and interactive camera control, allowing users to create effects with little effort and fine-grained artistic control.

  • Takeaways & Limitations

    Depth and segmentation remain unreliable for challenging cases such as reflective surfaces, thin structures, and erroneous object masks.

Abstract

from arXiv · show

The Ken Burns effect allows animating still images with a virtual camera scan and zoom. Adding parallax, which results in the 3D Ken Burns effect, enables significantly more compelling results. Creating such effects manually is time-consuming and demands sophisticated editing skills. Existing automatic methods, however, require multiple input images from varying viewpoints. In this paper, we introduce a framework that synthesizes the 3D Ken Burns effect from a single image, supporting both a fully automatic mode and an interactive mode with the user controlling the camera. Our framework first leverages a depth prediction pipeline, which estimates scene depth that is suitable for view synthesis tasks. To address the limitations of existing depth estimation methods such as geometric distortions, semantic distortions, and inaccurate depth boundaries, we develop a semantic-aware neural network for depth prediction, couple its estimate with a segmentation-based depth adjustment process, and employ a refinement neural network that facilitates accurate depth predictions at object boundaries. According to this depth estimate, our framework then maps the input image to a point cloud and synthesizes the resulting video frames by rendering the point cloud from the corresponding camera positions. To address disocclusions while maintaining geometrically and temporally coherent synthesis results, we utilize context-aware color- and depth-inpainting to fill in the missing information in the extreme views of the camera path, thus extending the scene geometry of the point cloud. Experiments with a wide variety of image content show that our method enables realistic synthesis results. Our study demonstrates that our system allows users to achieve better results while requiring little effort compared to existing solutions for the 3D Ken Burns effect creation.

1 INTRODUCTION

The paper targets realistic 3D Ken Burns synthesis from a single image, addressing depth-recovery and disocclusion challenges while supporting automatic and user-controlled camera paths.

  • Motivation: Single-image 3D Ken Burns synthesis is difficult because it requires accurate scene geometry and temporally consistent novel views despite disocclusions.Existing manual workflows require segmentation, 3D arrangement, and inpainting, while prior automatic methods require multiple viewpoints.
  • Depth Estimation: The depth pipeline addresses geometric distortions, semantic distortions, and inaccurate depth boundaries that limit monocular depth estimates for view synthesis.The paper identifies these as critical issues in adapting existing depth prediction methods to 3D Ken Burns synthesis.
  • View Synthesis: The view-synthesis process must fill disocclusions with geometrically consistent content, preserve temporal consistency, and operate in real time for interactive use.Independent frame-by-frame filling is described as computationally inefficient and temporally unstable.
  • System Modes: The complete system combines depth prediction and novel-view synthesis, with automatic camera endpoints chosen to minimize disocclusion or user-specified endpoints for finer control.The interactive mode provides real-time feedback while users manipulate start- and end-view windows.
  • Evaluation: Experiments on diverse real-world imagery report convincing effects and better user results with less effort than existing creation solutions.The system includes a fully automatic mode and a view-control mode with simple interactions.

2.4 Creative Effect Synthesis

Creative-effect synthesis uses inferred 3D scene information to generate camera-based image effects, but prior single-image fly-through systems often remain manual and limited in scene complexity.

  • 2.4 Creative Effect Synthesis: Depth or scene-layout information supports creative effects including depth-of-field synthesis, 2D-to-3D conversion, and photo pop-up.The paper focuses specifically on synthesizing a moving-camera 3D Ken Burns video from a single image.
  • 2.4 Creative Effect Synthesis: Earlier single-image camera fly-through methods use simplified scene representations and manual segmentation, limiting generality and increasing user effort.The paper notes that realistic effects are achievable for certain image types, but simplified geometry is often too simplistic for general imagery.

2.5 Image-to-Video Generation

The paper distinguishes its camera-motion video synthesis task from prior image-to-video work, and frames the proposed system as a two-stage depth and novel-view pipeline.

  • 2.5 Image-to-Video Generation: Image-to-video generation commonly predicts object motion from one or a few starting frames, often with static backgrounds, rather than realistic camera-motion effects.The paper states that such methods are generally unsuitable for its target camera-motion problem.
  • 2.5 Image-to-Video Generation: The framework consists of a depth-estimation pipeline and a novel-view-synthesis pipeline for generating the 3D Ken Burns effect.These are presented as the two main components of the system.

3.1 Semantic-aware Depth Estimation

The pipeline targets three depth-estimation problems that degrade single-image view synthesis: geometric distortions, semantic inconsistencies, and inaccurate object boundaries. It combines semantic-aware coarse prediction, segmentation-based adjustment, and image-guided refinement to produce depth suitable for point-cloud rendering.

  • Limitations of Existing Depth Estimation: Existing monocular depth methods can bend planar structures, producing geometric distortions in synthesized novel views.State-of-the-art methods may capture depth ordering while missing geometric relations such as planarity.
  • Limitations of Existing Depth Estimation: Depth predictions may assign inconsistent values within one object, causing parts to appear attached to the ground or torn apart.These semantic distortions arise when object semantics are not explicitly considered.
  • Limitations of Existing Depth Estimation: Low-resolution processing followed by interpolation yields inaccurate depth boundaries and artifacts in novel-view renderings.Boundary accuracy is especially important for 3D Ken Burns synthesis.
  • Semantic-aware Depth Pipeline: The proposed depth estimator uses semantic VGG-19 features, segmentation-based depth adjustment, and image-guided refinement of boundaries.Coarse depth is first predicted, salient objects are assigned coherent depth planes using Mask R-CNN segmentation, and refinement upsamples the estimate.
  • Rendering Implications: Refinement improves object-boundary geometry, while z-filtering prevents erroneous visibility of occluded background points through cracks during forward camera motion.Intermediate results show each pipeline stage addressing semantic distortion, boundary detail, or point-cloud rendering artifacts.

3.2 Context-aware Inpainting for View Synthesis

The view-synthesis pipeline extends a depth-derived point cloud to fill disocclusions, using joint color- and depth-inpainting with context information for coherent novel views.

  • Pipeline: The method renders novel views from a point cloud built from the input image and estimated depth, but disocclusions create holes.The point cloud represents only the input viewpoint’s partial world geometry.
  • Design requirements: Applying off-the-shelf inpainting independently frame by frame can produce temporal inconsistencies and is too computationally expensive for interactive use.The pipeline instead performs inpainting at extreme camera views and renders the extended point cloud in real time.
  • Comparison: Compared with DeepFill and EdgeConnect, the proposed approach produces temporally consistent results with clear object boundaries.DeepFill fails on non-rectangular regions, while EdgeConnect does not preserve temporal consistency or object boundaries.
  • Point-cloud extension: The inpainted depth maps the filled color into new points, extending the point cloud so subsequent views remain complete and temporally consistent.This extension is repeated until the point cloud is sufficiently enlarged.
  • Inpainting: Joint color- and depth-inpainting uses color, depth, and context information to fill missing regions while preserving scene geometry.Depth separates foreground objects from the background during inpainting.

3.3 User Interface

The user interface parameterizes the 3D Ken Burns effect with start- and end-view cropping windows, supporting automatic selection and interactive camera control.

  • Camera-path parameterization: The system samples novel views along a linear camera path between start- and end-positions to generate the output video.Camera positions can be derived from cropping windows placed on the input image.
  • Automatic mode: The automatic mode selects an end-view crop that minimizes disocclusion while emphasizing the dominant salient foreground object.The entire input image serves as the start-view, and a uniform grid searches candidate end-view windows.
  • Interactive mode: The interactive mode lets users specify both cropping windows and receive real-time feedback while manipulating them.This supports fine-grained control over the synthesized effect.

3.4 Training Data

The study uses synthetic multi-view data for training and evaluates usability with non-expert participants against two existing systems.

  • Data limitations: Existing NYU v2, MegaDepth, and KITTI data are limited by sparse annotations, inaccurate depth boundaries, or sparse multi-view information.These limitations motivate constructing a dedicated synthetic dataset.
  • Training Data: The custom dataset contains 134041 scenes from 32 realistic virtual environments, with four views per scene and color, depth, and normal maps.Each view has a resolution of 512 × 512 pixels.
  • Usability Study: The usability study uses 12 representative examples spanning landscape, portrait, indoor, and man-made outdoor environments.The examples were selected from artist-created tutorial videos after excluding additional artistic effects.
  • Usability Study: Eight participants create comparable effects with the proposed system, Photo Motion Pro, and Viewmee, then rate usability and result quality.Usability and quality are scored from one to five.
  • Usability results: Participants obtained better results with much less effort using the proposed system than the comparison systems.Viewmee is limited for scenes without a distinct foreground object, while Photo Motion Pro requires substantial manual scene arrangement.

4.2 Automatic Mode Evaluation

The automatic-mode evaluation compares the system with 2D Ken Burns outputs on varied images and reports favorable depth-prediction quality and user preference for 3D results.

  • Evaluation setup: The automatic evaluation uses 12 Flickr images covering indoor, landscape, outdoor, and portrait scenes with varying complexity.The system generates one automatic result per image and a corresponding 2D result using the same camera-path crops.
  • Depth evaluation: The depth-prediction method compares favorably with state-of-the-art methods across all reported depth-quality metrics.The supplied table caption states the cross-metric conclusion without giving metric values.
  • User evaluation: Users preferred the 3D Ken Burns synthesis in a majority of cases over the 2D baseline across image categories.The 3D outputs provide realistic motion parallax and stronger depth perception, whereas the 2D outputs show typical zooming without parallax.

4.3 Depth Prediction Quality

The depth prediction pipeline is evaluated against established monocular depth methods and benchmarks, showing favorable performance across quality metrics, especially for planarity and depth boundaries.

  • Benchmark evaluation: The iBims-1 benchmark contains 100 indoor images with high-quality ground-truth depth maps and metrics covering boundary quality, planarity, consistency, and distance accuracy.It is designed to provide a holistic evaluation of depth prediction quality.
  • Benchmark evaluation: Our method compares favorably with state-of-the-art depth predictors across all reported depth-quality metrics.The evaluation uses two public single-image depth benchmarks and metrics defined by each benchmark.
  • Benchmark evaluation: On iBims-1, the proposed pipeline significantly improves Planarity Error and Depth Boundary Error over off-the-shelf methods.These metrics assess planar structure and depth-boundary preservation, both important for view synthesis.
  • Ablation analysis: Removing the proposed depth architecture or computer-generated training data produces significantly worse benchmark depth metrics.Both variants nevertheless remain favorable compared with state-of-the-art depth prediction models.
  • Visual comparison: Three-dimensional renderings show better preservation of scene structure, including planarity, than renderings based on off-the-shelf depth predictions.The comparison is illustrated in Figure 13.

4.4 Discussion

The system’s results are rated on par with professional artists overall, with preferences varying by scene category and artistic control over parallax.

  • Artist comparison: User evaluations rate the system’s 3D Ken Burns results on par with those created by professional artists.The comparison used 12 artist-generated results, matched system results, and 21 participants performing pairwise tests.
  • Category differences: The system is slightly preferred for indoor scenes, where complicated depth distributions make manual 3D Ken Burns creation tedious.The method can handle these scenes through depth prediction.
  • Category differences: Artist-created results are more preferred for portraits because artists can intentionally exaggerate parallax beyond physically correct depth.Portrait scenes often have simpler layouts, making manual editing easier and enabling stronger artistic emphasis.
  • Artistic control: Physically correct depth prediction can be suboptimal for an artistic effect when viewers prefer exaggerated parallax.The paper identifies artistic manipulation of parallax as a direction for future extension.

4.5 Limitations

The method produces plausible results across scene types but remains vulnerable to difficult depth, segmentation, and inpainting cases.

  • Depth and segmentation failures: Reflective surfaces and thin structures can cause inaccurate depth estimates, while erroneous segmentation masks can damage salient depth adjustment and object boundaries.Examples include a glossy poster reflection, a flagpole, and a deer’s nose cut off by inaccurate Mask R-CNN segmentation.
  • Inpainting limitation: The joint color- and depth-inpainting approach may generate imperfect results because it was supervised only on synthetic data.The limitation applies to extending estimated scene geometry in disoccluded regions.

5 CONCLUSION

The paper presents a single-image 3D Ken Burns system combining view-synthesis-oriented depth prediction with context-aware depth-based synthesis. Experiments across varied imagery show realistic results and reduced user effort relative to existing solutions.

  • System: The system predicts scene depth from one image and uses context-aware depth-based view synthesis to generate video results.Its depth pipeline combines semantic guidance, synthetic training data, semantic depth adjustment, and boundary-focused refinement.
  • System: The synthesis model jointly predicts the target-view image and depth map, then uses extreme camera views to efficiently generate intermediate frames.This produces the final 3D Ken Burns video from the input image and predicted depth map.
  • Findings: Experiments with a wide variety of image content show realistic synthesis results and enable better outcomes with little user effort.The conclusion reports this comparison with existing 3D Ken Burns creation solutions.
Loading 1909.05483v1…