PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks

Junyuan Xie, Ross Girshick, Ali Farhadi

arXiv:1604.03650v1cs.CV

TL;DR

Automated 2D-to-3D conversion is challenging because single-view depth is under-constrained and unseen geometry must be hallucinated. Deep3D trains a deep neural network end-to-end on stereo pairs to synthesize the other eye’s view, outperforming baselines quantitatively and in human evaluations.

  • Problem

    Automated 2D-to-3D conversion remains challenging because single-view depth is under-constrained and unseen novel-view geometry requires hallucination.

  • Method

    Deep3D uses a deep neural network that estimates a soft disparity representation and renders the other eye’s view, trained end-to-end directly on stereo-frame pairs.

  • Results

    Deep3D outperforms baseline algorithms quantitatively, and human subjects consistently prefer its stereo images while finding them more confusable with ground truth.

  • Takeaways & Limitations

    Using still RGB frames as input allows the trained model to apply to both videos and photos.

  • Takeaways & Limitations

    The evaluation uses still images and ignores temporal information, while initial experiments found little quantitative gain from incorporating it.

Abstract

from arXiv · show

As 3D movie viewing becomes mainstream and Virtual Reality (VR) market emerges, the demand for 3D contents is growing rapidly. Producing 3D videos, however, remains challenging. In this paper we propose to use deep neural networks for automatically converting 2D videos and images to stereoscopic 3D format. In contrast to previous automatic 2D-to-3D conversion algorithms, which have separate stages and need ground truth depth map as supervision, our approach is trained end-to-end directly on stereo pairs extracted from 3D movies. This novel training scheme makes it possible to exploit orders of magnitude more data and significantly increases performance. Indeed, Deep3D outperforms baselines in both quantitative and human subject evaluations.

1 Introduction

Growing 3D and VR demand, together with the high cost and limitations of native stereo filming, motivates fully automated 2D-to-3D conversion. Deep3D predicts and renders a novel eye view end-to-end from stereo-frame pairs, with quantitative and human-subject evaluation against ground truth and depth-estimation baselines.

  • Motivation: 3D movies represented 14%–21% of U.S. and Canadian box-office sales from 2010 to 2014, while emerging VR head-mounted displays may increase demand for 3D content.The passage identifies both current 3D movie popularity and potential VR-driven growth.
  • Problem: Native 3D filming requires costly stereo camera rigs and can be incompatible with optical effects such as forced perspective.Forced perspective makes objects appear larger or smaller and breaks down when viewed from another angle.
  • Problem: The paper proposes fully automated, data-driven 2D-to-3D conversion to address high production costs, requiring single-image depth inference and novel-view synthesis.The approach targets a key obstacle to scaling the 3D movie industry.
  • Method: Deep3D takes the left-eye view, estimates a soft probabilistic disparity map, renders the right-eye view, and performs in-painting implicitly without post-processing.The model is designed as an end-to-end deep neural network trained on ground-truth stereo-frame pairs.
  • Evaluation: Evaluation compares reconstructed and ground-truth right views using pixel-wise metrics and human-subject experiments, including comparisons with state-of-the-art single-view depth-estimation baselines.The paper notes that evaluating generated 3D scenes from the left view is nontrivial.

2 Related Work

Automatic 2D-to-3D conversion has traditionally estimated depth before synthesizing a missing stereo view, while newer learning-based methods use image-depth supervision but face costly data collection. The proposed direction instead learns direct view prediction from stereo pairs through an end-to-end differentiable system, building on DeepStereo’s probabilistic rendering layer.

  • Traditional conversion pipelines: Most automatic 2D-to-3D pipelines estimate a depth map from the input image and then use DIBR to generate the missing stereo view.Early methods estimate depth using hand-engineered cues such as defocus, scattering, or texture gradients, often relying on only one cue.
  • Learning-based methods: Learning-based methods map single-view 2D images to depth maps using image-depth pairs as supervision and combine multiple cues for better generalization.Recent work uses deep convolutional neural networks to advance the state of the art, but collecting high-quality image-depth pairs is difficult and expensive.
  • Direct view prediction: The proposed method avoids requiring accurate monocular depth prediction by training on stereo pairs to directly predict the right view from the left view.DIBR is implemented through an internal probabilistic disparity representation within an end-to-end differentiable system.
  • Relation to DeepStereo: The approach is inspired by DeepStereo, which uses a probabilistic selection layer to model rendering differentiably and train it jointly with a DCNN.It retains the same probabilistic selection layer while improving on DeepStereo’s approach in two significant ways.

3 Method

Deep3D directly predicts the right view from the left view while using a differentiable DIBR-inspired selection layer to preserve task structure. Its multilevel architecture predicts probabilistic disparities, and high-resolution rendering couples upsampled disparity with the original left image.

  • 3 Method: Deep3D replaces separate depth estimation and DIBR rendering with direct right-view regression, while retaining a DIBR process to capture the task structure.Naive direct regression performs poorly because it does not capture the structure of 2D-to-3D conversion.
  • 3 Method: The network combines lower-level features through branches after each pooling layer that use learned upsampling filters before producing a probabilistic disparity map.The upsampled disparity map and left view are passed to a selection layer that outputs the right view.
  • 3 Method: Deconvolutional layers are initialized to perform bilinear interpolation to facilitate training.For upsampling factor S, the layer uses a 2S by 2S kernel, S by S stride, and S/2 padding, with weights initialized by the specified bilinear formula.
  • 3 Method: The selection layer makes DIBR differentiable by predicting a disparity probability distribution at each pixel and reconstructing the right view as a weighted combination of shifted left-view stacks.This enables joint neural-network training with an L1 loss against the ground-truth right view.

4 Dataset

The dataset uses stereo-video training without ground-truth depth supervision, enabling substantially more data than traditional scene-depth datasets. It comprises 27 recent non-animation 3D movies split into 18 training and 9 testing movies, totaling around 5 million frames.

  • Dataset construction: Deep3D’s direct stereo-pair training avoids requiring ground-truth depth maps and enables use of existing stereo videos instead of KITTI and NYU Depth.The dataset contains around 5 million frames, whereas KITTI and NYU Depth provide only several hundred frames.
  • Dataset construction: The collection includes 27 recent non-animation 3D movies, randomly partitioned into 18 movies for training and 9 for testing.The split is performed at the movie level.

5 Experiments · 5.1 Implementation Details

The experiments primarily use single frames for fair comparison with single-frame baselines and photo applicability, while implementation details specify the network architecture, training procedure, evaluation resolution, and runtime.

  • 5 Experiments: Deep3D uses a single RGB frame per input, enabling fair comparison with single-frame baselines and application to static photos.Additional experiments use consecutive RGB frames and computed optical flow to investigate motion cues for depth.
  • 5.1 Implementation Details: Quantitative evaluation uses a non-upsampled output of 384 by 160 pixels, whereas qualitative and human evaluations use 4× upsampling.The upsampling method is described in Section 3.3.
  • 5.1 Implementation Details: The model is based on VGG16 pretrained on ImageNet, with main-branch convolutional layers initialized from VGG16 weights.All other weights are initialized from a normal distribution with standard deviation 0.01.
  • 5.1 Implementation Details: Side branches after each pooling layer combine batch normalization, 3×3 convolution, and bilinear-initialized deconvolution to integrate lower-level features.Their deconvolution outputs match the final prediction dimensions, while batch normalization connects pretrained and randomly initialized layers to address numerical instability.
  • 5.1 Implementation Details: The top VGG16 feature connects to two fully connected layers with 4096 hidden units, followed by a linear layer reshaped into 33 channels of 12 by 5 maps.These maps are upsampled, summed, convolved into the final representation, and fed to the selection layer.
  • 5.1 Implementation Details: Training uses mini-batches of 64 for 100, 000 iterations, beginning at learning rate 0.002 and reducing it by 10× every 20, 000 iterations.No weight decay is used, and dropout rate 0.5 is applied only after the fully connected layers.
  • 5.1 Implementation Details: Training takes two days on one NVidia GTX Titan X GPU, while trained Deep3D reconstructs novel right views at more than 100 frames per second.The implementation is based on MXNet.

5.2 Comparison Algorithms

The comparison uses Global Disparity, Eigen et al.’s depth-estimation method with DIBR, and ground-truth stereo pairs. Because prior methods require ground-truth depth for training, Eigen et al.’s released NYU Depth model is evaluated directly, with lower- and upper-bound variants defined by plane-of-focus access.

  • Comparison Algorithms: Three baselines are compared: Global Disparity, Eigen et al.’s depth estimation plus DIBR, and ground-truth stereo pairs.Ground-truth stereo pairs are shown only in human-subject studies because they yield zero error in quantitative evaluations.
  • Comparison Algorithms: Global Disparity shifts the left view by a global disparity δ chosen by minimizing validation-set Mean Absolution Error (MAE).
  • Comparison Algorithms: Deep3D is trained directly on stereo pairs, whereas previous methods require ground-truth depth maps for training.Comparison algorithms cannot be retrained on the 3D movie dataset, so the model released by Eigen et al. is evaluated on the test set.
  • Comparison Algorithms: Eigen et al.’s model, trained on NYU Depth, may not generalize well to 3D movies, highlighting Deep3D’s access to vastly more training data.
  • Comparison Algorithms: Eigen et al.’s depth predictions are converted to disparity for DIBR, while unknown B and f are handled with fixed-parameter and per-frame-oracle variants.The fixed-parameter setup is the lower bound, while per-frame optimization of f is the upper bound; analogous Deep3D and Deep3D + Oracle results are also reported.

5.3 Results · Quantitative Evaluation

The quantitative evaluation uses Mean Absolute Error (MAE) to compare pixel-wise reconstruction error and finds that Deep3D outperforms baselines with and without oracle distance of focus plane. A human-subject study further reports that Deep3D is preferred to [10] in 66% of comparisons and to ground truth in 24%.

  • 5.3 Results: Mean Absolute Error (MAE) is used to measure pixel-wise reconstruction error for each method.The comparison is reported in Table 1.
  • 5.3 Results: The quantitative evaluation computes Mean Absolute Error (MAE) as its evaluation metric.The results are presented in Table 1.
  • Quantitative Evaluation: Deep3D outperforms baselines with and without oracle distance of focus plane.This observation is based on the results shown in Table 1.
  • Quantitative Evaluation: Human subjects preferred Deep3D to [10] 66% of the time.Table 2 reports pairwise preference frequencies between methods.
  • Quantitative Evaluation: Human subjects preferred Deep3D over the ground truth 24% of the time.This preference frequency is also reported in Table 2.

Qualitative Evaluation · Human Subject Evaluation

Deep3D qualitatively infers depth from multiple visual cues and focuses disparity prediction on structurally important regions. In human evaluation, it outperforms Global Disparity and [10] + Oracle, while receiving stronger preference than competing predictions against ground truth.

  • Qualitative Evaluation: Deep3D infers depth using multiple cues, including size, occlusion, and geometric structure.
  • Qualitative Evaluation: Its 12 internal disparity channels separate scene elements by relative distance, from near (-3) to far (+8).Examples place closer subjects in earlier channels and backgrounds in later channels.
  • Qualitative Evaluation: Compared with, Deep3D better delineates people and estimates their distance from the camera.
  • Qualitative Evaluation: Disparity maps can be noisy in low-horizontal-gradient regions, but this does not affect reconstruction when rows have identical pixel values.Accuracy is needed mainly around vertical edges, where Deep3D focuses its predictions.
  • Human Subject Evaluation: The human study compared Global Disparity, [10] + Oracle, Deep3D without Oracle, and ground-truth2.The evaluation randomly selected 500 test-set frames and presented pairwise anaglyph comparisons to annotators.
  • Human Subject Evaluation: Deep3D outperforms Global Disparity by a 49% margin and [10] + Oracle by a 32% margin.
  • Human Subject Evaluation: Against ground truth, Deep3D is preferred 24.48% of the time, compared with 10.27% for [10] + Oracle and 7.88% for Global Disparity.

5.4 Algorithm Analysis · Ablation Study

The ablation study shows that removing lower-level features, selection, direct stereo-pair training, or disparity-based view synthesis reduces performance. Modeling DIBR and incorporating temporal information improve Deep3D’s results.

  • Ablation Study: Deep3D + Oracle and method 2 without Oracle are omitted because of annotator-budget constraints.The passage also notes that average scene-depth changes generally alter scene distance without affecting perceived depth variation.
  • Ablation Study: Removing lower-level features and the selection layer both cause performance drops.These component removals are evaluated in the ablation study to assess their contributions.
  • Ablation Study: Removing lower-level features significantly decreases performance compared with the full Deep3D method.The ablated network uses one feed-forward path with five convolution-and-pooling modules and two fully connected layers.
  • Ablation Study: Replacing direct stereo-pair training with block-matching disparity supervision decreases performance.The predicted disparity maps are subsequently passed to DIBR to render the right view.
  • Ablation Study: Directly regressing the novel view without internal disparity representation or a selection layer also decreases performance.This result supports modeling the DIBR process within the method.
  • Ablation Study: Additional temporal information from multiple consecutive RGB or optical-flow frames leads to performance gains.Temporal information is incorporated by extending the input with consecutive frames or optical-flow frames.

Temporal Information

The main experiment used a single still RGB frame for fair comparisons and broad applicability, while preliminary temporal extensions yielded moderate pixel-wise improvements. Larger gains may require architectural changes, tuning, and explicit temporal modeling, potentially restricting use to videos.

  • Temporal Information: The main experiment used one still RGB frame as input to support fair comparisons and broader application domains.Temporal information could instead be incorporated through multiple consecutive RGB frames or optical-flow frames.
  • Temporal Information: Moderate improvements in pixel-wise metrics were observed when briefly exploring both temporal-input directions.The explored directions were consecutive RGB frames and optical-flow frames.
  • Temporal Information: Larger gains may require model-structure adjustment, hyper-parameter tuning, and explicit time modeling, at the cost of limiting applications to videos.This trade-off would restrict the method’s application domain to videos only.

6 Conclusions

The paper presents a fully automatic 2D-to-3D conversion method using end-to-end deep convolutional networks trained directly on stereo image pairs. The method outperforms baseline algorithms quantitatively and is consistently preferred in human studies, while the evaluations reveal limitations from ignoring video temporal information.

  • Contributions: The proposed method performs fully automatic 2D-to-3D conversion using deep convolutional neural networks.It is trained end-to-end directly on stereo image pairs.
  • Contributions: End-to-end training on stereo image pairs enables the method to exploit orders of magnitude more data than traditional learning-based conversion methods.
  • Evaluation: The method outperforms baseline algorithms quantitatively and is consistently preferred over baseline results in a human subject study.
  • Limitations: The evaluations used still images while ignoring temporal information, allowing the trained model to apply to both videos and photos but potentially limiting video-conversion performance.A brief experiment found little quantitative performance gain from using temporal information.
Loading 1604.03650v1…