PaperlayerView exact revision on arXiv

Source-linked AI summary

DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling

Kevin Karsch, Ce Liu, Sing Bing Kang

arXiv:2001.00987v1cs.CV

TL;DR

Conventional video depth methods depend on camera motion and static scenes, limiting them on dynamic or non-translating videos. DepthTransfer uses non-parametric depth transfer with motion and optical-flow information to estimate coherent depth for videos and single images, and reports strong benchmark and 2D-to-3D results.

  • Problem

    Existing video depth methods typically assume moving cameras and static scenes, failing on dynamic, stationary, rotating, or variable-focal-length sequences.

  • Method

    The method transfers depth non-parametrically, uses optical-flow-derived features and temporal information for video, and optimizes the objective with iteratively reweighted least squares.

  • Results

    The technique works across diverse indoor and outdoor videos, can use static RGBD images for video training, and quantitatively outperforms existing methods for single-image depth estimation.

  • Takeaways & Limitations

    The inferred depth enables automatic, time-coherent stereoscopic video generation from conventional 2D footage, including visually pleasing sequences and feature-film examples.

  • Takeaways & Limitations

    Motion-segmentation errors can produce inaccurate depth and 3D estimation, while fixed interocular distance limits control over the 3D pop-out effect in ground-truth recordings.

Abstract

from arXiv · show

We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large dataset containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.

1 INTRODUCTION

The paper addresses automatic depth extraction for arbitrary 2D videos, including settings where conventional motion-based methods fail. It introduces non-parametric depth transfer for temporally consistent video depth, supports single images, and presents a ground-truth RGBD video dataset.

  • Scene depth supports 3D visualization, spatial reasoning, robot navigation, and conversion of 2D movies into stereoscopic counterparts.
  • Conventional depth-reconstruction techniques typically assume moving cameras and static scenes, failing on dynamic scenes and stationary, rotating, or variable-focal-length sequences.
  • The paper presents a depth-map generation method for ordinary 2D videos that also applies to single images and handles cases where conventional methods fail.
  • Non-parametric depth transfer infers temporally consistent depth without imposing video requirements and includes improved estimates for moving objects.
  • The authors introduce a ground-truth stereo RGBD video dataset and describe synthesizing stereo videos from ordinary 2D videos.

2 RELATED WORK

Related work spans interactive and automatic single-image or video 2D-to-3D methods, conventional depth cues, and non-parametric transfer. DepthTransfer extends transfer-based approaches by estimating depth continuously and incorporating temporal information for video coherence.

  • Earlier single-image 2D-to-3D systems were often semi-automatic, requiring users to add planes, depth properties, surface normals, silhouettes, or creases.
  • Automatic single-image depth methods use planar assumptions, Bayesian reconstruction, supervised learning, semantic labels, or image cues such as repetition, haze, and shading.
  • DepthTransfer differs from single-image approaches by using temporal information to improve depth and enforce coherence across multiple frames.
  • Compared with contemporaneous non-parametric sampling, the method evaluates candidate depths and gradients per pixel and adds temporal video processing.
  • Interactive video conversion systems propagate user-provided depth cues, templates, or structure-from-motion information with optical flow.
  • Conventional structure-from-motion and motion-stereo systems can produce dense depth when camera motion induces parallax, but rely substantially on that motion.
  • The paper builds on semantic-label transfer by transferring depth and embedding the transfer in a continuous optimization framework rather than discrete MRFs.

3 NON-PARAMETRIC DEPTH ESTIMATION BY CONTINUOUS LABEL TRANSFER

The method transfers depth from semantically similar RGBD candidates by aligning them with the input and refining their warped depths through continuous global optimization. It uses appearance and motion features for candidate selection, spatially aware regularization, and robust iterative optimization, while requiring substantial memory for video sequences.

  • Depth transfer pipeline: The three-stage pipeline selects similar RGBD candidates, aligns their images and depths with SIFT flow, then optimizes an interpolated depth map.Candidate matching uses RGB and, for videos, optical-flow features; SIFT flow provides dense alignment before optimization.
  • Depth transfer pipeline: The approach assumes semantically similar, densely aligned scenes have comparable depth distributions, while combining multiple candidates to reduce individual inaccuracies.The global optimization considers absolute depth, gradients, spatial smoothness, and a database prior rather than relying on a single candidate.
  • Candidate matching and warping: Candidate matching combines GIST and optical-flow features, with videos represented by blockwise flow statistics totaling 8b^2 features per image.The implementation uses b = 4 and sets ω = 0.5 in the matching score.
  • Depth optimization: The objective compares inferred depth with warped candidate depths and their spatial gradients, using confidence weights and γ = 10 to enforce gradient similarity.Depth gradients are warped before differentiation to avoid artificial discontinuities introduced by warping depth values directly.
  • Numerical optimization details: IRLS solves the unconstrained nonlinear objective iteratively, using a linearized approximation and repeated least-squares updates until convergence.The per-pixel median of candidate depths initializes the optimization, while the final estimate remains influenced by all warped candidates.
  • Numerical optimization details: Video optimization can require several GBs of concurrent memory for a few seconds of standard-definition footage, while efficient time- and space-solving remains outside the paper’s scope.The authors identify this storage and computational burden as an unresolved issue of the optimization.

4 IMPROVED DEPTH ESTIMATION FOR VIDEO

The video framework jointly estimates depth across entire sequences, using optical flow for temporal coherence and motion segmentation to improve moving-object depth. It targets arbitrary videos, including rotating or zooming views and dynamic scenes where conventional methods fail.

  • 4 IMPROVED DEPTH ESTIMATION FOR VIDEO: Global optimization infers depth for the entire sequence while incorporating temporal information from all frames.Each frame still uses candidate matching and warping, but sequence-level optimization adds temporal constraints.
  • 4 IMPROVED DEPTH ESTIMATION FOR VIDEO: Optical flow differences encourage temporally coherent depth estimates between consecutive frames.Flow confidence weights emphasize smoothing where optical-flow estimates are accurate.
  • 4 IMPROVED DEPTH ESTIMATION FOR VIDEO: Motion cues constrain moving objects to have depth consistent with their ground contact points.A binary motion mask identifies moving objects, while connected components provide the floor-contact depth.
  • 4.1 Detecting moving objects: The motion-segmentation algorithm is used for static, rotational, and variable-focal-length videos, but not videos with parallax or fast motion.For excluded videos, the motion-segmentation weight is set to zero (η = 0).
  • 4.1 Detecting moving objects: Motion segmentation stabilizes non-translating video with homographies, estimates a median background, thresholds motion, and maps the mask back to original frames.The resulting segmentation improves inferred depth and applies to sequences without camera-motion-induced parallax.

5 MSR-V3D DATASET

MSR-V3D is a stereo video and depth dataset covering varied indoor and outdoor scenes, viewpoints, and moving objects. Kinect limitations provide indoor depth for training and testing, while outdoor sequences are reserved for evaluation.

  • 5 MSR-V3D DATASET: Most clips use static viewpoints with moving people or cars, while other clips use static or slowly rotated camera viewpoints.The dataset was collected across varied scenes and viewpoints for training and testing.
  • 5 MSR-V3D DATASET: Two vertically mounted Kinects captured color from both stereo views and depth only from the left Kinect.Indoor clips include view-aligned left-view depth; right-view depth and outdoor Kinect depth were unavailable because of IR interference and sunlight.
  • 5 MSR-V3D DATASET: Outdoor sequences were used for testing and evaluation but not training because sunlight prevented Kinect depth capture and stereo disparity estimates were inadequate.The authors attempted to recover ground-truth disparity, but Kinect image quality and resolution were too low.
  • 5 MSR-V3D DATASET: Kinect interference required masking the rightmost sensor and collecting depth corresponding only to left views.Training discards pixels containing depth holes, while visualization fills holes using horizontal dilation.

6 EXPERIMENTS

Experiments evaluate single-image and video depth extraction across benchmark and newly collected RGBD data. The method matches or exceeds state-of-the-art single-image results, generalizes across buildings and videos, and benefits from content-selective training while remaining sensitive to training diversity and appearance.

  • 6.1.1 Make3D Range Image Dataset: The method achieves state-of-the-art results on every reported Make3D error metric.Thin structures are usually recovered well, although spatial regularization can miss some fine structures.
  • 6.1.2 NYU Depth Dataset: On NYU, training with hold-one-out NYU examples significantly outperforms average-depth baselines across relative, log10, and RMS error.Training on MSR-V3D increases error but still outperforms the baselines on several metrics.
  • 6.2 Video results: Video results are best for the training building but remain reasonable for the three other buildings with different colors, surfaces, and structures.The video dataset contains ground-truth depth and stereo sequences for four buildings, with Building 1 used for training.
  • 6.2 Video results: The technique extracts a moving runner’s depth throughout a sequence, unlike Make3D, partly because Make3D does not use temporal information.This comparison highlights the value of temporal information for video depth estimation.
  • 6.2 Video results: Video inference does not require video training data; static RGBD images from another location can produce accurate depth and stereo views for outdoor videos.The authors report successful results even when training data come from a different location.
  • 6.1.1 Make3D Range Image Dataset: K = 7 is optimal on Make3D, while K ≥7 gives comparable results; increasing K empirically acts as a smoothing parameter.Larger candidate sets are expected to increase candidate diversity and produce smoother inferred depth.

7 APPLICATION: 2D-TO-3D

The paper applies depth-based view synthesis to automatically convert ordinary 2D videos into stereoscopic formats. Its saliency-preserving, temporally coherent warping produces compelling results, including feature-film clips, while keeping processing practical but not real-time.

  • Motivation: Automatic 2D-to-3D conversion is motivated by the high cost and manual effort of converting legacy films.Converting at most 20 minutes of Superman Returns was reported to cost $10 million.
  • Method: The system uses estimated depth to warp each input frame into stereoscopic views while reducing disocclusion holes and preserving salient regions.It extends Wang et al.’s saliency-preserving image-warping method for video view synthesis.
  • Method: Disparity is derived by inverting and scaling depth, with Wmax controlling the perceived pop-out effect and excessive values risking eye strain or fusion problems.The method sets ϵ = 0.01 and uses Wmax to regulate stereoscopic strength.
  • Method: The optimization constrains highly salient regions and nearby objects to remain stable while enforcing spatial and temporal coherence during warping.Saliency-dependent weights and smoothness terms guide the quadratic disparity optimization.
  • Output: Two synthesized views can be converted into formats such as anaglyph or interlaced stereo; the paper uses anaglyph because cyan/red glasses are widespread.The left and right images are shifted so the nearest object appears at the display surface.
  • Results: The method produces plausible and convincing 3D sequences, including clips from Charade, although more diverse training data is needed for commercial-quality conversion.The authors also report a subtler 3D effect in YouTube’s results and processing of roughly one minute per 640×480 frame.

8 DISCUSSION

The discussion finds that depth transfer works across varied indoor and outdoor sequences with practical training data, but performance depends on relevant training examples and has several operational limitations.

  • Performance: The algorithm works for diverse indoor and outdoor sequences with practical training data and tolerates large depth datasets with little quality degradation.More data mainly increases the number of candidate-search comparisons.
  • Training-data scope: Training on visually mismatched data can fail, whereas selecting similar data improves results for the queried scene.The paper specifically warns that outdoor training data is likely to fail for an indoor query.
  • Candidate retrieval: GIST matching retrieves candidates sharing structures such as building facades, sky, shrubbery, and horizon location, whose depths contribute to corresponding scene regions.Building-facade depth mainly comes from a similarly oriented facade, while ground and shrubbery rely on matching candidates.
  • Limitations: Motion-segmentation errors can produce inaccurate depth and 3D estimation, although spatio-temporal regularization partly mitigates them.The limitation concerns both missed and falsely identified moving pixels.
  • Limitations: The method assumes moving objects contact the ground and may therefore fail for airborne objects.This assumption is illustrated in Fig. 20.
  • Limitations: Because depth estimation precedes view synthesis serially, errors in estimated depth can propagate into visually implausible synthesized views.The authors suggest using synthesized-view appearance to correct depth errors as a future improvement.

9 CONCLUDING REMARKS

The paper concludes that non-parametric depth transfer automatically estimates depth for videos and single images, including cases where motion-parallax methods fail. It also enables time-coherent stereoscopic video generation from conventional 2D footage.

  • Conclusion: The fully automatic technique applies to single images and dynamic or otherwise difficult videos where motion-parallax and structure-from-motion methods can fail.The conclusion specifically includes static or rotating cameras and dynamic scenes among its target cases.
  • Conclusion: The non-parametric approach produces qualitatively good depth results and quantitatively outperforms existing single-image methods.The conclusion presents this as the central depth-estimation outcome.
  • Conclusion: The inferred depths support time-coherent, visually pleasing stereo sequences from conventional 2D videos.The method is presented as a good starting point for converting legacy 2D feature films into 3D.
Loading 2001.00987v1…