PaperlayerView exact revision on arXiv

Source-linked AI summary

HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching

Vladimir Tankovich, Christian Häne, Yinda Zhang, Adarsh Kowdle, Sean Fanello, Sofien Bouaziz

arXiv:2007.12140v5cs.CV

TL;DR

Stereo matching needs accurate disparity estimation without the latency and computation of full cost-volume networks. HITNet avoids explicitly building a 3D volume by combining high-resolution multi-resolution initialization, geometric propagation, warping, and slanted-plane hypotheses. It reports state-of-the-art accuracy across multiple benchmarks at a fraction of the computation, while requiring ground-truth depth for training.

  • Problem

    Accurate stereo matching methods often rely on computationally expensive cost volumes and 3D convolutions, whereas active agents require millisecond-scale processing.

  • Method

    HITNet avoids explicit 3D cost volumes by using fast multi-resolution high-resolution initialization, learned tile features, 2D propagation, image warping, and slanted support windows.

  • Results

    HITNet achieves state-of-the-art results on multiple popular stereo benchmarks using a fraction of the computation required by other methods.

  • Takeaways & Limitations

    The architecture provides accurate real-time stereo matching through learned geometric refinement without explicitly operating on a full 3D cost volume.

  • Takeaways & Limitations

    HITNet requires training on a dataset with ground-truth depth.

Abstract

from arXiv · show

This paper presents HITNet, a novel neural network architecture for real-time stereo matching. Contrary to many recent neural network approaches that operate on a full cost volume and rely on 3D convolutions, our approach does not explicitly build a volume and instead relies on a fast multi-resolution initialization step, differentiable 2D geometric propagation and warping mechanisms to infer disparity hypotheses. To achieve a high level of accuracy, our network not only geometrically reasons about disparities but also infers slanted plane hypotheses allowing to more accurately perform geometric warping and upsampling operations. Our architecture is inherently multi-resolution allowing the propagation of information across different levels. Multiple experiments prove the effectiveness of the proposed approach at a fraction of the computation required by state-of-the-art methods. At the time of writing, HITNet ranks 1st-3rd on all the metrics published on the ETH3D website for two view stereo, ranks 1st on most of the metrics among all the end-to-end learning approaches on Middlebury-v3, ranks 1st on the popular KITTI 2012 and 2015 benchmarks among the published methods faster than 100ms.

1. Introduction

HITNet targets the latency–accuracy trade-off in stereo matching by avoiding full 3D cost volumes and combining fast initialization with learned geometric refinement. The proposed architecture uses multi-resolution matching, 2D propagation, and slanted support windows to achieve accurate results with substantially less computation.

  • Motivation: Stereo methods often achieve accuracy at high computational cost, while active agents require image-pair processing within milliseconds.Large CNNs can take a second or more, and only 4 of the top 100 KITTI 2012 methods were published approaches under 100ms.
  • Motivation: Explicit cost volumes and 3D convolutions improve accuracy but significantly increase computation.This motivates approaches that avoid operating on a full 3D cost volume.
  • Approach: HITNet integrates image warping, spatial propagation, and fast high-resolution initialization without explicitly building a 3D volume.The architecture preserves learned representations while reducing the computational disadvantages of volume-based processing.
  • Approach: The method computes high-resolution matches using learned features through a fast multi-resolution initialization step.This initialization is one of the architecture’s main design components.
  • Approach: HITNet uses efficient 2D disparity propagation with slanted support windows and learned descriptors.These components allow disparity hypotheses and attached features to be refined spatially.
  • Results: The paper reports state-of-the-art benchmark results using a fraction of the computation required by other methods.The introduction presents this as the central performance outcome of the proposed framework.

2. Related Work

Prior stereo methods trade accuracy, efficiency, and geometric reasoning in different ways. HITNet builds on classical propagation and pyramidal matching ideas while using learned, high-resolution tile initialization to recover fine details efficiently.

  • Classical methods: Traditional stereo methods use hand-crafted local correspondences and global optimization, but runtime is linked to disparity-space size.Patchmatch and super-pixel methods reduce search but avoid searching the full disparity space.
  • Learned methods: Modern learned methods use cost volumes and 3D convolutions, whereas faster downsampled approaches can sacrifice thin structures and edge accuracy.Low-resolution initialization may omit fine details that later upsampling cannot fully recover.
  • HITNet positioning: HITNet is inspired by classical algorithms that propagate sparse matches using slanted support windows while replacing hand-crafted components with end-to-end learning.The approach seeks to retain computational efficiency with learned representations.
  • HITNet positioning: Unlike pyramidal methods that refine low-resolution cost-volume results, HITNet uses fast multi-scale high-resolution initialization to recover fine details.The distinction is between initializing from a low-resolution solution and retrieving high-resolution matches directly.

3. Method

HITNet represents disparity with multi-resolution planar tile hypotheses, initializes them without storing a full cost volume, and iteratively refines them through propagation and feature warping.

  • Tile Hypothesis: HITNet represents each tile hypothesis with disparity, x- and y-direction disparity gradients, and a learned feature descriptor.The geometric component encodes a slanted plane, while the descriptor can capture matching quality or local surface properties learned end-to-end.
  • Feature Extractor: A small U-Net extracts multi-scale feature maps used for initial matching and propagation-stage warping.The decoder outputs features at all resolutions, including high-resolution features with spatial context from the encoder.
  • Initialization: Initialization evaluates disparities for 4 × 4 tiles at multiple resolutions and selects the minimum-cost match, producing fronto-parallel hypotheses.The initial hypothesis has the form h_init = [d_init, 0, 0, p_init].
  • Initialization: Although initialization exhaustively evaluates disparities, HITNet stores only the best match rather than a full 3D cost volume.The best-match location can be extracted efficiently using fast memory and a fused GPU operation.
  • Propagation: Propagation updates tile hypotheses using spatial neighborhoods, confidence-weighted fusion, and warped right-image features compared with left-image features.Each slanted hypothesis becomes a 4 × 4 local disparity map, which warps right-image features and yields a local matching-cost vector.

4. Loss Functions

HITNet is trained end-to-end across scales and pixels with initialization and propagation losses. These losses supervise matching costs, tile geometry, surface slant, and confidence using ground-truth disparities and gradients.

  • The final training loss sums losses over all scales and pixels using ground-truth disparities.
  • Initialization Loss: Initialization uses interpolated matching costs and an ℓ1 contrastive loss to make ground-truth disparities cheaper than nonmatches.Ground-truth disparity maps are maxpooled for multiple resolutions, and subpixel matching costs use linear interpolation.
  • Initialization Loss: The initialization margin is β = 1, pushing the ground-truth cost toward 0 and the lowest-cost nonmatch toward the margin.The negative sample is the lowest-cost nonmatch rather than a random or exhaustive set of nonmatching locations.
  • Propagation Loss: Propagation supervises tile geometry, surface slant, and confidence using ground-truth disparity and gradients derived from robustly fitted planes.Ground-truth planes are fitted in 9×9 windows, then used to expand tiles to full-resolution disparities before applying geometry and slant losses.
  • Propagation Loss: Confidence increases for hypotheses within threshold C1 of ground truth and decreases for hypotheses farther than threshold C2 away.The experiments use A = B = C1 = 1 and C2 = 1.5; at the final single-hypothesis levels, loss applies to all pixels with A = ∞.

5. Experiments

HITNet achieves competitive or state-of-the-art stereo results across SceneFlow, ETH3D, KITTI, and Middlebury while using a fraction of the computational time of other methods.

  • Evaluation setup: The evaluation covers SceneFlow, KITTI 2012, KITTI 2015, ETH3D, and Middlebury V3 using EPE and x-pixels error under stated evaluation settings.
  • SceneFlow: 0.36 EPE on SceneFlow finalpass is reported as 2X better than the state of the art at the time.
  • SceneFlow: HITNet achieves the lowest EPE on SceneFlow while maintaining real-time performance and consistently outperforming StereoNet.
  • Middlebury V3: On Middlebury V3, HITNet outperforms other end-to-end learning approaches on most metrics and ranks first for bad 0.5 and A50.
  • ETH3D: HITNet ranks 1st–4th on all published ETH3D two-view stereo metrics and ranks first on several listed metrics.
  • KITTI 2012 and 2015: Among published methods faster than 100ms, HITNet ranks #1 on both KITTI 2012 and KITTI 2015.

6. Conclusion

HITNet combines fast high-resolution initialization with propagation, fusion, and slanted support windows to deliver accurate real-time stereo matching. The authors identify supervised ground-truth depth training and separately trained dataset-specific experiments as limitations.

  • HITNet computes high-resolution matches efficiently, then fuses tile initializations through propagation and fusion steps.
  • Slanted support windows with learned descriptors provide additional accuracy.
  • The algorithm requires training on a dataset with ground-truth depth.
  • The experiments train different datasets separately and use slightly different model architectures.

A.1. Training Setup

Training uses dataset-specific splits and schedules across synthetic and real-world stereo benchmarks. The reported training procedure includes extended optimization and measures intended to limit overfitting.

  • SceneFlow experiments use the predefined train/test split, with additional experiments using only FlyingThings because the other components lack corresponding test sets.
  • KITTI benchmark submissions train on all 394 available images, while ablations use a 75% training and 25% validation split.
  • A learning rate of 1e−4 with longer training performed best across multiple datasets without showing signs of overfitting.
  • ETH3D training combines KITTI, Middlebury, and ETH3D images because its training set contains only a few stereo pairs.
  • Middlebury training pre-trains on SceneFlow FlyingThings before fine-tuning on 23 Middlebury14perfectH training images.

A.2. Data Augmentation

The training pipeline augments image appearance and geometry to address dataset shift, brightness changes, reflections, and calibration differences.

  • Augmentations perturb left and right image brightness and contrast symmetrically and asymmetrically to improve robustness to real-world appearance changes.
  • Middlebury-specific processing addresses color-distribution differences and miscalibrated stereo pairs through color normalization and random vertical offsets.

B. Additional Evaluations

Additional evaluations compare HITNet with accurate and fast stereo methods on real-world datasets and examine intermediate predictions and cross-domain transfer. The reported results emphasize fine structures, crisp edges, and generalization.

  • Qualitative Results: HITNet is compared with GC-Net, RTS-Net, and GA-Net, representing full-cost-volume, similarly fast, and highly accurate approaches respectively.
  • Qualitative Results: HITNet compares favorably with GC-Net and RTS-Net and is on par with GA-Net while recovering fine structures and crisp edges.
  • Middlebury Results: On Middlebury, HITNet produces crisp edges, correct occlusions, and thin structures in the considered qualitative cases.
  • Intermediate Results: Increasing resolution produces finer-grained disparity while merging high-resolution initialization details with global context from lower resolutions.
  • Cross-Domain Adaptation: Training on SceneFlow and testing on KITTI 2012 and 2015 yields superior generalization results compared with the other evaluated state-of-the-art approaches.

B.3. Ablation Study

The ablations evaluate multi-scale prediction, initialization resolution, slant prediction, tile features, warping, and model size. Results show that several geometric and multi-resolution components improve accuracy, while larger models can overfit small datasets.

  • Multi Scale Prediction: Multi-scale prediction improves real-world KITTI performance more clearly than synthetic SceneFlow performance, where additional texture reduces the need for propagation context.
  • 4x4x4 Downsampled: Full-resolution initialization is important: downsampling disparity resolution by 4x substantially drops accuracy.
  • 16x16x8 Downsampled: Downsampling the cost volume by 16x in height and width and 8x in disparity degrades accuracy.
  • 16x16x1 Downsampled: Using larger tiles while retaining disparity resolution degrades accuracy further because initialization loses precise low-resolution spatial reasoning.
  • Slant Prediction: Removing slant prediction substantially reduces precision on SceneFlow and KITTI 2012 and removes the network’s surface-normal prediction capability.
  • Tile Features: Removing tile features decreases accuracy on both SceneFlow and KITTI 2012.
  • Warping: Removing image warps hurts subpixel precision because warps compute matching costs during propagation.
  • Model Size: Increasing runtime to 54ms improves SceneFlow but not small KITTI datasets, where larger models overfit; a 114ms model increases overfitting further.The XL version reports 0.31 EPE, 15.6 bad 0.1, 3.67 bad 1.0, and 1.99 bad 3.0 on cleanpass.

C. Model Architecture Details

HITNet uses configurable multi-scale feature extractors and propagation modules, with benchmark-specific variants and a reported 19 ms runtime on KITTI-resolution inputs.

  • Architecture configuration: The default HITNet architecture uses five feature-extractor scales with 16, 16, 24, 24, and 32 channels.Initialization uses 4 × 4 tiles, while tile descriptors have 13 channels and propagation steps typically use two residual blocks.
  • Architecture configuration: HITNet variants use different feature widths and residual-block configurations for SceneFlow, Middlebury, KITTI, and ETH3D experiments.SceneFlow includes HITNet, HITNetL, and HITNetXL variants, while Middlebury uses a wider feature extractor than the default configuration.
  • Reproducibility: Benchmark submission models and execution scripts are publicly available in the authors’ GitHub repository.The cited repository contains the models used for benchmark submissions and scripts to run them.
  • Architecture configuration: Figures 9–12 detail initialization, residual-block integration, and propagation differences between single- and multiple-hypothesis settings.The figures cover initial tile-feature computation, propagation logic, and multi-hypothesis updates.
  • Runtime: 19 ms per frame is reported for the ETH3D and KITTI HITNet architecture on 0.5-megapixel inputs using a Titan V GPU.The final three high-resolution propagation steps take 7.5 ms, multi-scale propagation contributes less than 5 ms, and fused initialization takes 0.25 ms.

E. Number of Parameters

The paper emphasizes parameter and compute efficiency, reporting that HITNet achieves better results than other approaches with significantly fewer parameters and lower compute.

  • Efficiency rationale: Fewer parameters reduce required compute and memory, and may make the network less susceptible to over-fitting.The paper frames parameter count as an important efficiency consideration for neural network architectures.
  • Ablation study: The qualitative ablation study indicates that all proposed design choices contribute to performance on fine details, edges, and occluded regions.The comparison focuses on the visual quality of these structures and regions.
  • Efficiency comparison: Table 7 compares parameter counts and GMacs across methods on the Scene Flow “finalpass” dataset, where lower values are better.The table uses 960 × 540 inputs and also reports GMacs for 1280 × 384 inputs for multi-scale HITNet submissions.
  • Efficiency comparison: HITNet achieves better results than other approaches with significantly fewer parameters and compute.The comparison is presented as evidence that the network combines performance with reduced resource requirements.
  • Generalization: Training on synthetic data and testing in real-world scenarios, HITNet outperforms multiple state-of-the-art baselines.The paper connects this result to reduced susceptibility to overfitting from using fewer learnable weights.
Loading 2007.12140v5…