Source-linked AI summary
Hierarchical Neural Architecture Search for Deep Stereo Matching
Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Yuchao Dai, Xiaojun Chang, Tom Drummond, Hongdong Li, Zongyuan Ge
TL;DR
NAS has not translated readily to deep stereo matching because volumetric networks make end-to-end architecture search computationally prohibitive. LEAStereo addresses this with task-specific hierarchical search across the stereo pipeline and reports top-ranked benchmark accuracy alongside smaller, faster networks.
Problem
End-to-end NAS for volumetric stereo matching is computationally prohibitive because the search space and memory demands are large.
Method
LEAStereo uses task-specific geometric knowledge and hierarchical cell-level and network-level search to jointly optimize the feature and matching networks.
Results
LEAStereo outperforms state-of-the-art handcrafted and NAS-searched stereo architectures and ranks first on KITTI 2012, KITTI 2015, and Middlebury while improving network size and inference speed.
Takeaways & Limitations
The framework reduces search time by more than 2/3 compared with a previous method and improves performance, reducing energy consumption.
Takeaways & Limitations
The task-tailored search design may not be domain-agnostic, so other domains may require different task-dependent NAS mechanisms.
Abstract
from arXiv · showhide
To reduce the human efforts in neural network design, Neural Architecture Search (NAS) has been applied with remarkable success to various high-level vision tasks such as classification and semantic segmentation. The underlying idea for the NAS algorithm is straightforward, namely, to enable the network the ability to choose among a set of operations (e.g., convolution with different filter sizes), one is able to find an optimal architecture that is better adapted to the problem at hand. However, so far the success of NAS has not been enjoyed by low-level geometric vision tasks such as stereo matching. This is partly due to the fact that state-of-the-art deep stereo matching networks, designed by humans, are already sheer in size. Directly applying the NAS to such massive structures is computationally prohibitive based on the currently available mainstream computing resources. In this paper, we propose the first end-to-end hierarchical NAS framework for deep stereo matching by incorporating task-specific human knowledge into the neural architecture search framework. Specifically, following the gold standard pipeline for deep stereo matching (i.e., feature extraction -- feature volume construction and dense matching), we optimize the architectures of the entire pipeline jointly. Extensive experiments show that our searched network outperforms all state-of-the-art deep stereo matching architectures and is ranked at the top 1 accuracy on KITTI stereo 2012, 2015 and Middlebury benchmarks, as well as the top 1 on SceneFlow dataset with a substantial improvement on the size of the network and the speed of inference. The code is available at https://github.com/XuelianCheng/LEAStereo.
1 Introduction
Stereo matching estimates dense disparity from rectified image pairs, while NAS has succeeded mainly in high-level vision. Applying NAS end-to-end to volumetric stereo networks is computationally prohibitive, motivating LEAStereo’s task-specific hierarchical search.
- Stereo matching estimates dense disparity by finding correspondences between rectified stereo images.
- Direct regression methods infer per-pixel disparity with large 2D encoder-decoder networks but raise concerns about generalization.DispNet fails random dot stereo tests.
- Volumetric methods build 4D feature volumes and use feature and matching networks, with 3D convolutions computing matching costs.
- NAS has succeeded in high-level vision, but searching volumetric stereo architectures is prohibitive because NAS explores huge spaces and volumetric networks require 6–8 GB GPU memory per training batch.Early NAS methods also required thousands of GPU hours on CIFAR.
- LEAStereo jointly searches feature and matching networks using cell-level and network-level structures, while incorporating stereo geometry into the search.The framework also searches feature-map, feature-volume, and output-disparity sizes.
- LEAStereo outperforms handcrafted and NAS-based methods in accuracy, inference time, and resulting network size across multiple stereo benchmarks.The introduction reports comparisons with man-designed networks and AutoDispNet on these dimensions.
2 Our Method
The method uses task-specific knowledge to perform end-to-end hierarchical NAS over volumetric stereo matching, jointly searching feature and matching networks at cell and network levels. Its search space controls operations, resolutions, feature dimensions, and information flow while end-to-end disparity supervision enables joint optimization.
- Task-specific Architecture Search Space: Hierarchical NAS searches both cell-level operations and network-level structures across the Feature Net and Matching Net.The framework searches the full architecture rather than only encoder cells, including feature-map, feature-volume, and output-disparity sizes.
- Task-specific Architecture Search Space: The volumetric pipeline comprises a 2D feature net, 4D feature volume, 3D matching net, and soft-argmin disparity projection.Only the Feature Net and Matching Net contain trainable parameters and are optimized by NAS.
- Cell Level Search Space: The cell search represents intermediate nodes in a directed acyclic graph and selects candidate operations using architecture parameters before decoding a discrete architecture.The searched cell connects two preceding-layer outputs to intermediate nodes, and the final operation between nodes is the most likely candidate.
- Search-Space Design: The search uses a residual cell option, task-specific operation sets, and a four-level trellis reaching 1/24 input resolution.The chosen depths are LF = 6 for the Feature Net and LM = 12 for the Matching Net, balancing computational load and performance.
- Network Level Search Space: The network-level search finds a path through a trellis that controls cell arrangement, feature dimensionality, and information flow across resolutions.The design uses resolution adjustment between neighboring cells and commonly doubles filters when halving spatial dimensions.
- Loss Function and Optimization: End-to-end disparity supervision jointly searches the Feature Net and Matching Net with smooth ℓ1 loss and bi-level optimization of weights and architecture parameters.Separate training sets are used for network weights and architecture parameters, followed by discrete decoding of cell and network structures.
3 Experiments
LEAStereo is evaluated across SceneFlow, KITTI, and Middlebury, with ablations examining its search space, search strategy, subnet functionality, and comparison to AutoDispNet. The experiments report strong accuracy alongside a compact architecture and analyze design choices affecting performance.
- Benchmark Evaluation: LEAStereo achieves state-of-the-art SceneFlow performance with nearly one third the parameters of handcrafted methods and 20× fewer parameters than AutoDispNet.Evaluation uses EPE and bad 1.0 on the SceneFlow test set with 192 disparity levels.
- Benchmark Evaluation: LEAStereo ranks first among human-designed architectures on both KITTI 2012 and KITTI 2015 benchmarks.The reported ranking is based on the benchmark leaderboards.
- Benchmark Evaluation: LEAStereo achieves state-of-the-art ranking across multiple Middlebury 2014 metrics while processing 1500 × 1000 images with 432 disparity levels.Its compact architecture permits higher-resolution evaluation than methods operating at quarter resolution; the paper notes stronger performance at large error thresholds.
- Ablation Study: Using residual cells increases performance by 14.29% while slightly increasing parameters and FLOPs.This comparison is reported in the final two rows of the ablation table.
- Ablation Study: A larger operation set produces fewer parameters but poorer EPE because the search favors skip connections and yields a low-capacity architecture.The larger set includes convolution, pooling, and connection mechanisms, whereas the paper’s set contains 3 × 3 convolution, skip, and zero connections.
- Ablation Study: Joint-search improves EPE by 9.30% and reduces parameters by 10.50% relative to separate-search on SceneFlow.The comparison fixes the candidate operations and connection type while changing the search strategy.
4 Related Work
Deep stereo matching progressed from learned features and direct disparity regression toward volumetric methods, while NAS research mainly targeted other dense-prediction tasks.
- MC-CNN replaced handcrafted stereo features with learned features and achieved better performance.
- DispNet introduced end-to-end direct disparity regression using a large U-shape encoder-decoder with skip connections.
- NAS studies for dense predictions developed super-nets containing many architectures for tasks such as scene parsing and semantic segmentation.
5 Conclusion
The paper presents a task-informed hierarchical NAS framework that jointly searches the deep stereo pipeline. Its searched network achieves top-ranked benchmark accuracy while substantially improving network size and inference speed.
- The framework jointly optimizes the feature net, feature volume, and matching net within an end-to-end stereo pipeline.
- Task-specific human knowledge tailors the search space and enables hierarchical architecture search for deep stereo matching.
- The searched network outperforms handcrafted and NAS-searched deep stereo architectures and ranks first in accuracy on KITTI 2012, KITTI 2015, and Middlebury.
- The framework also substantially improves network size and inference speed.
- The authors plan to extend the search framework to optical flow estimation and multi-view stereo.
Broader Impact
The paper frames improved stereo architecture design as an ongoing need and presents a search framework intended to extend beyond stereo matching. Optical flow and multi-view stereo are identified as target dense-matching applications.
- Finding better architectures for stereo matching networks remains an active research topic beyond the progress of traditional and deep methods.
- The proposed search framework is presented as applicable beyond stereo matching to other dense matching tasks.
- The authors specifically identify optical flow estimation and multi-view stereo as extension tasks.