PaperlayerView exact revision on arXiv

Source-linked AI summary

Detect, Replace, Refine: Deep Structured Prediction For Pixel Wise Labeling

Spyros Gidaris, Nikos Komodakis

arXiv:1612.04770v1cs.CVcs.LG

TL;DR

Dense image labeling requires modeling dependencies between inputs and output labels, yet common feed-forward transformation and residual strategies are sub-optimal. The paper introduces an end-to-end Detect, Replace, Refine architecture, evaluates its variants for stereo disparity estimation, and reports state-of-the-art KITTI 2015 results.

  • Problem

    Common deep joint input-output models are sub-optimal because transform-based methods must learn identity behavior, while residual methods struggle with large deviations from initial labels.

  • Method

    Detect, Replace, Refine detects incorrect labels, replaces erroneous labels, and refines all outputs with residual corrections in a unified end-to-end trainable architecture.

  • Results

    The disparity estimation model achieves state-of-the-art results on the KITTI 2015 test set, outperforming all prior published work by a significant margin.

  • Takeaways & Limitations

    Separating coarse correction of hard mistakes from fine-detail residual refinement provides the paper’s structured strategy for improving dense label estimates.

  • Takeaways & Limitations

    The stereo models use only the left image and initial disparity labels, excluding the right stereo image; the framework is presented on a 2D image domain.

Abstract

from arXiv · show

Pixel wise image labeling is an interesting and challenging problem with great significance in the computer vision community. In order for a dense labeling algorithm to be able to achieve accurate and precise results, it has to consider the dependencies that exist in the joint space of both the input and the output variables. An implicit approach for modeling those dependencies is by training a deep neural network that, given as input an initial estimate of the output labels and the input image, it will be able to predict a new refined estimate for the labels. In this context, our work is concerned with what is the optimal architecture for performing the label improvement task. We argue that the prior approaches of either directly predicting new label estimates or predicting residual corrections w.r.t. the initial labels with feed-forward deep network architectures are sub-optimal. Instead, we propose a generic architecture that decomposes the label improvement task to three steps: 1) detecting the initial label estimates that are incorrect, 2) replacing the incorrect labels with new ones, and finally 3) refining the renewed labels by predicting residual corrections w.r.t. them. Furthermore, we explore and compare various other alternative architectures that consist of the aforementioned Detection, Replace, and Refine components. We extensively evaluate the examined architectures in the challenging task of dense disparity estimation (stereo matching) and we report both quantitative and qualitative results on three different datasets. Finally, our dense disparity estimation network that implements the proposed generic architecture, achieves state-of-the-art results in the KITTI 2015 test surpassing prior approaches by a significant margin.

1. Introduction

Dense image labeling must capture dependencies between input images and output labels, but existing approaches use sub-optimal refinement strategies. The paper proposes Detect, Replace, Refine and evaluates it for stereo disparity estimation, achieving state-of-the-art KITTI 2015 results.

  • The paper addresses dense image labeling, where each pixel receives a discrete or continuous value across tasks such as stereo matching, optical flow, surface-normal estimation, and semantic segmentation.
  • Independent patch-based predictors capture image representations but do not model correlations between nearby output labels.
  • Deep joint input-output models improve initial labels by predicting either entirely new estimates or residual corrections, while learning dependencies more flexibly than CRFs.
  • Detect, Replace, Refine separates label improvement into detecting incorrect labels, replacing hard errors, and refining all outputs with residual corrections in one end-to-end architecture.
  • The framework explores alternative combinations of its components and evaluates them with qualitative and quantitative disparity-prediction experiments.
  • The proposed disparity model achieves state-of-the-art KITTI 2015 test results, outperforming all prior published work by a significant margin.

2. Methodology

The proposed deep joint input-output model replaces a monolithic label update with sequential detection, replacement, and residual refinement. This decomposition targets hard mistakes first, then soft errors and fine structures, while remaining end-to-end differentiable.

  • 2. Methodology: The model maps an image X and initial labels Y to more accurate labels Y′, supporting continuous or discrete labels but focusing here on continuous labels.Initial estimates may come from another image-only model F0.
  • 2.1. Detect, Replace, Refine architecture: Detect, Replace, Refine decomposes label improvement into error detection, label replacement, and residual refinement components.The components Fe, Fu, and Fr operate sequentially on the image and initial labels.
  • Detect: Detection estimates an error-probability map E for pixels whose labels are inconsistent with the image and neighboring label structure.The detector is trained through final-label reconstruction and can implicitly model local joint input-output patches.
  • Replace: Replacement gates between old and new labels using E, producing U = E ⊙ Fu(X, Y, E) + (1 − E) ⊙ Y and training Fu mainly on erroneous regions.The error map also identifies trusted regions that can guide filling incorrect areas.
  • Refine: Refinement applies residual corrections to the renewed map, Y′ = U + Fr(X, Y, E, U), correcting soft mistakes and aligning labels with fine image structures.The three steps may be iterated, allowing recovery from severely corrupted initial labels.
  • 2.2. Discussion: The two-stage design separates coarse correction of hard mistakes from fine-detail residual refinement, enabling lower-resolution processing in the first stage and end-to-end learning.First-stage errors can be corrected by the second stage during joint training.

3. Detect, Replace, Refine for disparity estimation

The disparity experiments use initial estimates from a deep patch-matching model and train image-plus-initial-disparity refiners without the right image. Architectures are implemented with convolutional components operating at reduced resolutions where appropriate.

  • 3. Detect, Replace, Refine for disparity estimation: Disparity estimation assigns each left-image pixel a continuous horizontal displacement corresponding to its match in the right image.This task tests dense labeling under challenges requiring accurate disparity preservation.
  • 3.1. Initial disparities: Initial disparity labels come from deep patch matching, while the refinement models receive only the left image X and initial disparities Y, not the right image.Consequently, the models must learn joint input-output structure rather than rely solely on image evidence.
  • Implementation: The error detector uses five convolutional layers with sigmoid output, while Replace and Refine use encoder-decoder convolutional architectures with residual blocks and skip connections.Detection downsamples by a factor of four; Replace compresses to 1/64 resolution and Refine to 1/16 before restoring resolution.
  • Implementation: Replace stops decompression early because it performs crude fixes, then uses parameter-free bilinear upsampling by four to restore label resolution.This reflects the component’s role in correcting hard mistakes rather than fine details.
  • Alternative architectures: Alternative architectures preserve total learnable capacity by increasing layers or feature planes when components are removed.The explored variants include architectures built from Replace, Refine, Detection, and their combinations.

4. Experimental results

The paper conducts an extensive experimental evaluation of the proposed and alternative architectures for dense disparity estimation, reporting quantitative and qualitative comparisons.

  • 4. Experimental results: The experiments evaluate the proposed architecture and alternative architectures through detailed quantitative results and qualitative results for Detect, Replace, Refine and its components.The evaluation is organized around settings, quantitative architectural comparisons, and qualitative analyses.

4.1. Experimental settings

Training combines a large synthetic disparity dataset with 160 KITTI 2015 images, while evaluation spans Synthetic, KITTI 2015, and Middlebury data using EPE and thresholded disparity errors.

  • Training set: Training uses around 34k synthetic stereo images plus 160 KITTI 2015 training images.The synthetic data contains three types of image sequences.
  • Evaluation sets: Evaluation covers 2,000 Synthetic test images, 40 KITTI 2015 validation images, and 15 Middlebury training images.Models are fine-tuned on the 160 KITTI training images before KITTI validation evaluation.
  • Evaluation metrics: Performance is measured with EPE and the percentage of estimates exceeding t pixels, separately for non-occluded, all, and occluded pixels.EPE is the average Euclidean distance from ground-truth disparity.
  • Evaluation results: Table 1 reports stereo matching results on the Synthetic dataset.The supplied caption identifies the table’s dataset scope but does not provide cell values or comparisons.

4.2. Quantitative results

Across three stereo datasets, Detect + Replace + Refine performed best among the examined architectures, with two iterations further improving performance and achieving a substantial KITTI 2015 test advantage.

  • 4.2.1 Disparity estimation performance: Detect + Replace + Refine achieved the best single-iteration performance overall, while Parallel performed considerably worse, supporting the importance of sequential processing.Detect + Replace and Detect + Refine also outperformed Replace and Refine on the Synthetic and Middlebury datasets.
  • 4.2.1 Disparity estimation performance: Two iterations of Detect + Replace + Refine further improved performance over one iteration, but additional iterations produced no further improvement.The two-iteration model was pretrained from the single-iteration version and then fine-tuned with first-iteration disparity labels added to training.
  • 4.2.2 Label prediction accuracy Vs initial labels quality: When initial labels contained fewer than 10% errors, Refine was more accurate than Replace, whereas Replace was better when errors exceeded 20%.The result supports using residual corrections for soft mistakes and entirely new labels for hard mistakes.
  • 4.2.2 Label prediction accuracy Vs initial labels quality: Detect + Replace + Refine showed the strongest accuracy across initial-label qualities, with larger gains as initial labels became less accurate.Adding error detection also substantially improved both Replace and Refine.
  • 4.2.3 KITTI 2015 test set results: More than 10% relative error-rate reduction on KITTI 2015 test versus the best prior approach was achieved by Detect + Replace + Refine x2.The submission attained state-of-the-art D1-all results and surpassed prior work by a significant margin.

4.3. Qualitative results

Qualitative analyses illustrate how detection, replacement, and refinement contribute to disparity estimation. Two iterations improve fine details and visual plausibility, including when initial labels are severely corrupted.

  • 4.3.1 Detection qualitative results: Error-probability maps show that the detection component recognizes areas appearing abnormal in the joint input-output space.The maps are compared with ground-truth error maps obtained by thresholding initial-label deviations at 3 pixels.
  • 4.3.2 Replace qualitative results: Replace-step visualizations compare the input image, initial disparities, detection probabilities, and renewed label predictions.These examples isolate the operation performed by the replacement component.
  • 4.3.3 Refine qualitative results: Refine-step examples show renewed labels, residual corrections, and the resulting final label estimates.The full workflow is also compared with ground-truth disparity labels.
  • 4.3.6 Multi-iteration qualitative results: The second iteration produced finer details and a more natural-looking disparity field than the first iteration.The multi-iteration architecture achieved the most accurate results in the experiments.
  • 4.3.5 KITTI qualitative results: On KITTI 2015 validation examples, Detect + Replace + Refine x2 recovered good disparity maps even from severely corrupted initial labels.These qualitative examples used the architecture that also gave the best quantitative results.

5. Conclusions

The paper proposes an end-to-end Detect + Replace + Refine architecture that iteratively improves initial dense-label estimates through three structured subtasks. Evaluated on stereo disparity estimation, it reports state-of-the-art KITTI 2015 test results.

  • The approach learns a deep joint input-output model that iteratively improves initial estimates for dense image labeling.
  • The unified Detect + Replace + Refine architecture detects incorrect labels, replaces them, and refines the output with residual corrections.The three stages are embedded in one end-to-end trainable architecture for iteratively improving initial output-label estimates.
  • The architecture was evaluated on disparity estimation and achieved state-of-the-art results on the KITTI 2015 test set.
Loading 1612.04770v1…