PaperlayerView exact revision on arXiv

Source-linked AI summary

Refine and Distill: Exploiting Cycle-Inconsistency and Knowledge Distillation for Unsupervised Monocular Depth Estimation

Andrea Pilzer, Stéphane Lathuilière, Nicu Sebe, Elisa Ricci

arXiv:1903.04202v2cs.CV

TL;DR

Costly depth annotations motivate self-supervised estimation from stereo images. The paper combines cycle-inconsistency refinement with distillation from a refinement network to a student. It reports better performance than prior unsupervised approaches and results comparable to state-of-the-art supervised methods on KITTI.

  • Problem

    Supervised monocular depth methods depend on costly labeled depth data, motivating self-supervised estimation from stereo image pairs.

    Source: §1 · p. 1

  • Method

    A student predicts disparity from a stereo-training view, a backward network closes the image cycle, and a third network refines depth from cycle inconsistency before distilling its information to the student.

    Source: §1 · p. 2, §3.1 · pp. 3–4

  • Results

    The framework outperforms previous unsupervised approaches and obtains results comparable to state-of-the-art supervised methods on KITTI.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The student network can serve applications with low computation power or real-time constraints.

    Source: §1 · p. 2

    The distillation process does not yet account for teacher and student confidence, which the authors identify as future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Nowadays, the majority of state of the art monocular depth estimation techniques are based on supervised deep learning models. However, collecting RGB images with associated depth maps is a very time consuming procedure. Therefore, recent works have proposed deep architectures for addressing the monocular depth prediction task as a reconstruction problem, thus avoiding the need of collecting ground-truth depth. Following these works, we propose a novel self-supervised deep model for estimating depth maps. Our framework exploits two main strategies: refinement via cycle-inconsistency and distillation. Specifically, first a \emph{student} network is trained to predict a disparity map such as to recover from a frame in a camera view the associated image in the opposite view. Then, a backward cycle network is applied to the generated image to re-synthesize back the input image, estimating the opposite disparity. A third network exploits the inconsistency between the original and the reconstructed input frame in order to output a refined depth map. Finally, knowledge distillation is exploited, such as to transfer information from the refinement network to the student. Our extensive experimental evaluation demonstrate the effectiveness of the proposed framework which outperforms state of the art unsupervised methods on the KITTI benchmark.

1. Introduction

Supervised depth estimation depends on costly labeled data, motivating self-supervised learning from stereo reconstruction. The paper adds cycle-inconsistency refinement and distillation, reporting stronger KITTI results than prior unsupervised methods and performance comparable to supervised methods.

1. Introduction

  • Depth prediction matters in applications including robot navigation, autonomous driving, virtual reality, and 3D reconstruction.Source: §1 · p. 1
  • Self-supervised depth estimation addresses the cost of collecting large labeled datasets by learning disparity from stereo pairs and image reconstruction.Depth can then be recovered from a single image at test time.Source: §1 · pp. 1–2
  • The model cycles from a predicted opposite view back to the input, uses reconstruction inconsistency to refine depth, and distills the refined estimates into the student network.The student can be used under low-computation or real-time constraints.Source: §1 · p. 2
  • On KITTI, the framework outperforms previous unsupervised approaches and obtains results comparable to state-of-the-art supervised methods.The evaluation also covers Cityscapes.Source: §1 · p. 2

2. Related Work

Prior work spans supervised depth prediction, self-supervised stereo reconstruction, temporal methods, cycle consistency, and knowledge distillation. This paper positions its approach as frame-level monocular depth estimation using cycle inconsistency and a nested student-teacher distillation setup.

2. Related Work

  • Supervised depth methods rely on expensive ground-truth annotations, which can limit flexibility in novel environments.Source: §2 · p. 2
  • Self-supervised work reconstructs stereo views from predicted disparities, uses bidirectional disparity consistency, or exploits temporal information and camera motion.This paper focuses on frame-level learning without temporal consistency or auxiliary ego-motion supervision, making it complementary to temporal approaches [12].Source: §2 · p. 2
  • Prior cycle-consistency work addressed stereo depth, while another cycle-reconstruction refinement approach used supervision derived from Stereo Direct Sparse Odometry.The proposed architecture instead uses cycle inconsistency at training and test time, with a backward network and a distillation loss.Source: §2 · p. 2
  • Knowledge distillation trains a smaller student to match a teacher's outputs; here, the student is an unusual sub-network of the teacher.The paper presents this as its first use for depth estimation and notes its relevance when real-time constraints limit network size.Source: §2 · p. 3

3. Proposed Method

The method trains a monocular depth estimator from stereo pairs by reconstructing opposite views, then uses cycle inconsistency to refine disparity and distills that refinement back into the student network.

3.1. Overview

  • Given rectified stereo images, the model predicts right-to-left disparity from the right view, which can recover left-image depth using camera baseline and focal length.At test time, depth is estimated from a single image.Source: §3.1 · pp. 3–4

3.2. Unsupervised Monocular Cycled Network

  • The cycled network synthesizes the left view, then predicts the opposite disparity from that synthesized image to reconstruct the right view.Sharing weights exposes the networks to both training images and synthesized images; the backward network also constrains disparity globally, while cycle inconsistency can reveal errors and object edges.Source: §3.1 · pp. 3–4

3.3. Inconsistency-Aware Network

  • The inconsistency-aware network uses the input image, cycle difference, and student disparity to predict a refined right-to-left disparity.The cycle difference is between the input right image and its backward reconstruction.Source: §3.3 · p. 4, §3.1 · p. 4

3.4. Network Training and Knowledge Self-Distillation

  • Training combines reconstruction losses across the networks and scales with self-distillation from the inconsistency-aware teacher to the student.The student is part of the teacher, and the stop-gradient operation prevents the distillation target from receiving backward gradients; distillation improves the student without adding test-time computation.Source: §3.4 · p. 5

4. Experiments

Experiments on KITTI and Cityscapes test the ablation stages and compare the framework with existing depth-estimation methods. The teacher and distilled student show improvements in the reported comparisons, with the teacher reaching state-of-the-art frame-level unsupervised KITTI performance.

4.1. Experimental Setup

  • Experiments use driving-scene stereo datasets, with 22,600 training and 697 test KITTI pairs, and 22,973 training and 1,525 test Cityscapes pairs.KITTI uses the Eigen split; Cityscapes training combines densely and coarsely annotated images, while testing uses the densely annotated split.Source: §4.1 · p. 5

4.2. Baselines for Ablation.

  • Ablations progress from a forward half-cycle to a training-time backward cycle, an inconsistency-aware teacher, and a distilled student.At test time, the student has the half-cycle's output and computation complexity.Source: §4.2 · p. 6

4.3. Results

  • The teacher improves over the cycle by 14% in Abs Rel and 18% in Sq Rel on KITTI.The cycle also improves several metrics over the half-cycle without additional losses; its extra forward-propagation time is incurred during training, not testing.Source: Table 1 · p. 6, §4.3 · p. 6
  • Replacing the two-network cycle with the single-network cycle from Yang et al. yields inconsistency estimates associated with worse performance.The comparison is conducted without supervision, using the L1 loss from.Source: Table 2 · p. 6, §4.3 · pp. 7–8
  • On Cityscapes, the cycle improves five of seven metrics, the teacher improves all error metrics by 7%–20%, and distillation adds about 1.5%–5%.With the second reconstruction-loss variant, the teacher gains over 20% over the initial cycle setting, while the student improves by 2%–5%.Source: Table 3 · p. 6, §4.3 · p. 8

4.4. Comparison with State-of-the-Art

  • The teacher reaches state-of-the-art performance in frame-level unsupervised KITTI depth estimation, surpasses a depth-supervised method, and is competitive with methods using depth and video clues.The distilled student improves on unsupervised methods of similar network capacity and is outperformed only by methods exploiting additional training information.Source: Table 4 · p. 7, §4.4 · p. 8

4.3. Results

  • Student-teacher distillation improves the student consistently across metrics while keeping teacher performance constant.Disparity distillation converges faster than feature distillation, which the authors relate to disparity's more compact representation.Source: §4.3 · p. 7

5. Conclusions

The proposed monocular depth network uses cycle-reconstruction inconsistencies for depth refinement and distills information from the refinement network to improve the student. Future work will incorporate teacher and student confidence into distillation.

5. Conclusions

  • The refinement network uses inconsistencies between input and cycle-reconstructed images to generate state-of-the-art depth predictions.Source: §5 · p. 8
  • Distillation transfers information from the refinement network to improve the student network’s performance.Source: §5 · p. 8
  • Future work will account for teacher and student confidence in distillation to better guide learning and correct prediction inconsistencies.Source: §5 · p. 9

Appendix

The appendix reports implementation details and further experimental results, with additional qualitative results available in an attached video.

Appendix

  • The appendix reports implementation details and additional experimental results.Source: Appendix · p. 11
  • Additional qualitative results are available in the video attached to the document.Source: Appendix · p. 11
  • The appendix directs readers to the attached video for further qualitative results.Source: Appendix · p. 11

A. Training Details

Training uses Adam with a learning rate of 1e-5 and batches of eight stereo pairs; the student runs faster at test time than the full teacher network.

A. Training Details

  • Training uses a 1e-5 learning rate, batches of 8 stereo pairs, Adam, and weight decay of 2e-5.The reported momentum parameter is 0.9.Source: A. Training Details · p. 11
  • The reported training stages take approximately 2.7 hours for the half-cycle, 1 hour for the backward-cycle decoder, 3.5 hours for joint cycle training, 2 hours for the inconsistency network, and 6.5 hours for joint fine-tuning.Experiments used an NVIDIA Titan Xp with 12 GB of memory.Source: A. Training Details · p. 11
  • The student takes 25 ms at test time, compared with 48.5 ms for the teacher.Source: A. Training Details · p. 11

B. Experimental Results

On KITTI, qualitative comparisons show better teacher depth estimates and fewer large teacher errors; cycle inconsistency maps resemble student error maps and indicate error locations, especially at object edges.

B. Experimental Results

Related papers

Loading 1903.04202v2…