Source-linked AI summary
Depth Extraction from Video Using Non-parametric Sampling
Kevin Karsch, Ce Liu, Sing Bing Kang
TL;DR
The paper addresses depth estimation for videos with dynamic scenes and non-translating cameras, where conventional methods often fail. It uses non-parametric depth transfer with motion cues and optical flow to produce temporally coherent depth, and reports state-of-the-art-or-better benchmark performance and broad video applicability.
Problem
Conventional video depth methods assume moving cameras and static scenes, leaving dynamic, stationary, rotating, or zooming sequences insufficiently handled.
Method
The method transfers depth from matched RGBD candidates, improves moving-object estimates with motion cues, and enforces temporal coherence using optical flow.
Results
The method is as good as or better than the state-of-the-art for each reported metric and works across varied indoor and outdoor video sequences.
Takeaways & Limitations
The technique supports automatic conversion of ordinary 2D videos into compelling stereo views, including videos with moving objects and rotating or zooming views.
Takeaways & Limitations
Motion-segmentation errors can cause inaccurate depth, and the method assumes moving objects contact the ground, so airborne objects may fail.
Abstract
from arXiv · showhide
We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large dataset containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.
1 Introduction
The paper targets depth recovery from videos where conventional methods fail, including dynamic scenes and cameras that do not translate. It introduces non-parametric depth transfer for videos and single images, with applications to automatic 2D-to-3D conversion.
- Conventional video depth methods typically assume moving cameras and static scenes, limiting them on dynamic, stationary, rotating, or variable-focal-length sequences.
- The paper presents non-parametric depth transfer for generating temporally consistent depth maps from arbitrary videos and for estimating depth in single images.
- The method includes a procedure for improving depth estimates of moving objects and introduces a ground-truth stereo RGBD video dataset.
- The approach extends non-parametric depth sampling from monocular-image stereo conversion to videos, reducing reliance on user interaction for 2D-to-3D conversion.
2 Non-parametric depth estimation
The method transfers depth from matched RGBD candidates, aligns them densely with SIFT flow, and globally optimizes their warped depths. Its continuous objective combines candidate agreement, spatial smoothness, and a database prior.
- Non-parametric transfer avoids explicitly defining a parametric model, requires fewer assumptions, and can use static RGBD images for video input.
- The pipeline matches database RGBD images to the input, warps candidate images and depths with SIFT Flow, then optimizes the warped candidates into an inferred depth map.
- Candidate selection uses high-level image features and restricts each database video to one matching frame, encouraging viewpoint diversity among candidates.
- SIFT Flow estimates dense, potentially one-to-many warps that establish pixel correspondences between candidate images and the input.
- The objective combines data agreement, spatial smoothness, and a database prior, with candidate depths compared using robust absolute and relative-depth differences.
3 Improving depth estimation for videos
The video method extends single-image depth transfer with temporal coherence and motion cues, targeting rotating, zooming, and dynamic scenes where conventional methods fail. A global sequence optimization combines optical flow with moving-object constraints to improve depth consistency and recovery.
- Scope: The framework targets arbitrary sequences, including static or rotating cameras, variable focal lengths, and moving scene objects, assuming zooming induces little or no parallax.
- Video optimization: The method processes entire video sequences jointly, enforcing temporal consistency and assigning moving objects depth similar to their ground contact points.
- Temporal information: Without temporal information, the car is practically ignored; adding temporal information recovers the moving car’s depth.
- Video optimization: Optical-flow temporal terms smooth inferred depth where flow estimates are reliable, while motion terms improve depth for moving objects.The video objective adds temporal-coherence and motion terms to the single-image objective, weighted by ν = 100 and η = 5.
- Motion segmentation: Motion segmentation aligns neighboring frames using homography, extracts a median-filtered background, and thresholds relative background differences weighted by stabilized-frame optical flow.The resulting mask is transformed back to each video frame and used to improve moving-object depth.
4 Dataset
The authors collected a large stereoscopic RGB-D video dataset with known depth across varied indoor scenes and used additional outdoor footage only for evaluation. The mobile capture rig combines two side-by-side Microsoft Kinects.
- Dataset composition: The dataset contains over 200 stereoscopic video sequences with corresponding depth values from four buildings in two cities.Scenes include hallways, rooms, and foyers, with substantial variation.
- Dataset composition: Sequences primarily depict people walking, sitting, or socializing, while camera viewpoints are static or slowly rotated.
- Capture system: The capture system uses two side-by-side, vertically mounted Microsoft Kinects about 5 cm apart, collecting color from both and depth from the left.
- Evaluation scope: Outdoor sequences were collected for testing and evaluation but excluded from training because sunlight interfered with Kinect depth sensing.Stereo ground truth was also not extracted outdoors because of reliability issues.
5 Application: Automatic stereoscopic view synthesis
The paper applies inferred depth maps to automatic 2D-to-3D conversion by warping each frame into stereoscopic views. It presents single-image depth and anaglyph comparisons while motivating the application as a cheaper alternative to manual film conversion.
- Motivation: The application targets the growing availability of 3D films, viewing technology, and user-generated stereoscopic content.
- Motivation: Automatic 2D-to-3D conversion is motivated as an efficient, inexpensive alternative to labor-intensive industry workflows.The paper cites a reported $10 million cost for converting at most 20 minutes of Superman Returns footage.
- Stereoscopic synthesis: The technique generates depth maps that can be used to produce stereoscopic video by warping each input frame with its corresponding depth map.The view-synthesis procedure is adapted to avoid holes at disocclusions and preserve salient regions.
6 Results
The method achieves state-of-the-art single-image depth results, generalizes across indoor and outdoor videos, and produces plausible stereo conversions, including feature-film footage.
- The method achieves state-of-the-art results on every reported Make3D depth metric.The comparison includes relative error, log10 error, and RMS error.
- Unlike Make3D, the method extracts the runner’s depth throughout the video sequence.
- Across four buildings, Building 1 performs best while the other buildings still achieve reasonable video-depth errors.Building 1 supplies training data, with hold-one-out evaluation for its test examples.
- Static RGBD images can train the system for video input, and the method usually infers accurate depth and stereo views even from another location.
- The method produces plausible depth maps and convincing automatic 3D sequences for feature-film clips, although more diverse training data is needed for commercial-quality conversion.
- The system runs at roughly one minute per 640×480 frame on a parallel quad-core implementation.
7 Discussion
The method handles varied indoor and outdoor sequences with practical training data, but depends on related training examples and has identifiable motion, object, and error-propagation limitations.
- The method works across many indoor and outdoor sequences using a practical amount of training data.
- Training on data unlike the query domain can fail, whereas combining large amounts of depth data degrades output quality little but increases candidate-search comparisons.
- Candidate matching transfers depth from semantically and geometrically related regions, such as facades, ground planes, and shrubbery.
- Motion-segmentation errors can produce inaccurate depth and 3D estimates, while the method’s ground-contact assumption may fail for airborne objects.
- Because depth estimation precedes view synthesis, errors can propagate through the pipeline and yield visually implausible results.
8 Concluding Remarks
The paper presents a fully automatic, non-parametric technique for estimating depth in videos and single images, including cases where motion-parallax and structure-from-motion methods fail. The inferred depths support time-coherent, visually pleasing stereoscopic videos and provide a starting point for converting legacy 2D feature films into 3D.
- The method estimates depth automatically for videos and single images, including dynamic scenes and cases where motion-parallax and structure-from-motion methods fail.Its non-parametric approach produces qualitatively good results and applies beyond conventional camera-motion settings.
- The inferred depth maps enable time-coherent, visually pleasing stereo sequences from conventional 2D videos.
- The technique is suitable as a starting point for converting legacy 2D feature films into 3D.