Source-linked AI summary
Cross-Scale Cost Aggregation for Stereo Matching
Kang Zhang, Yuqiang Fang, Dongbo Min, Lifeng Sun, Shiqiang Yang. Shuicheng Yan, Qi Tian
TL;DR
State-of-the-art stereo cost aggregation generally operates only at the finest scale, despite multi-scale interaction in human correspondence processing. This paper unifies aggregation methods through WLS optimization and adds inter-scale regularization, enabling cross-scale integration. Experiments on Middlebury, KITTI, and New Tsukuba show that the framework improves almost all evaluated methods, with even box filtering achieving very good performance.
Problem
Existing state-of-the-art cost aggregation methods aggregate costs at the finest scale, leaving multi-scale interaction underused despite its relevance to stereo correspondence.
Method
The paper formulates cost aggregation as WLS optimization and adds an inter-scale regularizer that integrates multiple aggregation kernels while enforcing neighboring-scale cost-volume consistency.
Results
Experiments on Middlebury, KITTI, and New Tsukuba show better disparity maps, with almost all methods improving under cross-scale aggregation.
Takeaways & Limitations
Cross-scale regularization effectively expands existing cost aggregation methods, and even simple box filtering combined with the framework achieves very good performance.
Takeaways & Limitations
The paper leaves investigation of scale-space behavior for continuous plane-parameter stereo methods as future work.
Abstract
from arXiv · showhide
Human beings process stereoscopic correspondence across multiple scales. However, this bio-inspiration is ignored by state-of-the-art cost aggregation methods for dense stereo correspondence. In this paper, a generic cross-scale cost aggregation framework is proposed to allow multi-scale interaction in cost aggregation. We firstly reformulate cost aggregation from a unified optimization perspective and show that different cost aggregation methods essentially differ in the choices of similarity kernels. Then, an inter-scale regularizer is introduced into optimization and solving this new optimization problem leads to the proposed framework. Since the regularization term is independent of the similarity kernel, various cost aggregation methods can be integrated into the proposed general framework. We show that the cross-scale framework is important as it effectively and efficiently expands state-of-the-art cost aggregation methods and leads to significant improvements, when evaluated on Middlebury, KITTI and New Tsukuba datasets.
1. Introduction
Stereo matching depends strongly on cost aggregation, yet conventional methods aggregate costs only at the finest image scale. The paper proposes a general cross-scale framework that unifies aggregation methods through WLS optimization and introduces inter-scale consistency.
- Motivation: Cost aggregation enforces piecewise-constant disparity over each pixel’s support region and strongly affects stereo-matching quality.It is central to state-of-the-art local algorithms and a building block for some global methods.
- Motivation: Conventional aggregation methods operate at the finest scale, despite evidence that human correspondence processing interacts across coarse and fine scales.The paper uses this multi-scale interaction as motivation for aggregating costs across scales.
- Framework: The proposed framework formulates diverse cost aggregation methods uniformly as weighted least-square optimization problems.Its contributions explicitly include a unified WLS formulation from an optimization perspective.
- Framework: Adding a generalized Tikhonov regularizer enforces consistency among neighboring-scale cost volumes while retaining conventional intra-scale consistency.The resulting convex objective can be solved analytically, allowing multiple aggregation methods to be integrated.
- Evaluation: The paper presents cross-scale cost aggregation as a novel framework and evaluates representative aggregation methods on Middlebury, KITTI, and New Tsukuba datasets.The introduction reports that cross-scale integration produces better disparity maps and includes quantitative evaluation across three datasets.
2. Related Work
Prior stereo methods use multi-scale information mainly through coarse-to-fine disparity evolution, whereas this paper regularizes cost-volume consistency across scales without narrowing the solution space. Its framework therefore targets a different stage and mechanism of multi-scale processing.
- Coarse-to-Fine Methods: Stereo methods have used coarse-to-fine strategies to assign coarse disparities first, reduce finer-scale search spaces, accelerate convergence, and avoid unexpected local minima.This strategy appears in global methods including dynamic programming, semi-global matching, and belief propagation.
- Cost-Volume Consistency: Most coarse-to-fine approaches model disparity evolution across scales, while the proposed method models cost-volume evolution across scales.The distinction concerns the quantity whose cross-scale consistency is represented.
- Cost-Volume Consistency: Coarse-to-fine approaches narrow the solution space, whereas this method preserves the solution space and adds inter-scale regularization to the optimization objective.The paper identifies regularization of the multiscale prior as the originality of its approach.
- Relation to Cost Aggregation: The framework is designed to improve existing cost aggregation methods rather than replace them with a wholly new aggregation method.Its regularization can be combined with conventional aggregation while maintaining intra-scale consistency.
3. Cost Aggregation as Optimization
Cost aggregation is formulated as weighted least squares, unifying existing methods through their similarity kernels. This perspective exposes why finest-scale aggregation struggles in low-texture regions and motivates scale-space reformulation.
- Unified optimization: Weighted least-squares optimization unifies cost aggregation methods through different choices of similarity kernels.The kernel determines how neighboring costs contribute to the denoised cost volume.
- Similarity kernels: Filter-based methods use local neighborhoods, whereas tree-based methods use whole-image neighborhoods and therefore produce different cost-volume structures.Filter-based aggregation preserves local similarity, while tree-based aggregation tends to create harder region boundaries.
- Scale-space motivation: State-of-the-art aggregation methods perform well in high-texture regions but commonly fail in low-texture or near-textureless regions.These regions require correspondence search to exploit coarse-scale information.
- Scale-space motivation: Finest-scale aggregation cannot adaptively use information from multiple scales, motivating a reformulation of the WLS objective from a scale-space perspective.The limitation follows because the conventional formulation aggregates costs at the finest scale.
4. Cross-Scale Cost Aggregation Framework
The proposed framework adds inter-scale consistency to unified cost aggregation by regularizing neighboring-scale cost volumes. Solving the resulting convex optimization adaptively combines scale-specific aggregation, allowing coarse-scale information to reshape ambiguous cost vectors.
- Multi-scale formulation: Direct multi-scale application of the original objective aggregates each scale independently rather than enabling interaction between scales.The scale-indexed cost volumes use corresponding pixels and disparities across resolutions, but remain separately aggregated.
- Inter-scale regularization: A Generalized Tikhonov regularizer directly enforces consistency between neighboring-scale cost volumes.The regularization strength is controlled by λ.
- Optimization solution: The regularized optimization is convex and yields S + 1 linear equations represented by an invertible tridiagonal matrix.The matrix structure permits efficient solution of the coupled scale equations.
- Framework output: The final cost volume adaptively combines aggregation results from different scales, enabling multiscale interaction within optimization.The finest-scale result is used to produce the final disparity map.
- Effect of regularization: Inter-scale regularization reshapes an ambiguous finest-scale cost vector using coarse-scale information, moving its selected disparity closer to ground truth.In the illustrated pixel, finest-scale information alone produces similar local minima and an erroneous disparity.
5. Implementation and Complexity
The implementation builds Gaussian-pyramid cost volumes, applies existing aggregation at each scale, and then combines them across scales. The added computation is small because the scale-coupling matrix is spatially invariant and can be precomputed.
- Implementation: Gaussian pyramids are built by successive smoothing and subsampling with η = 2, reducing image size exponentially at coarser scales.The smaller coarse-scale images reduce the computational cost of aggregation.
- Implementation: The workflow generates initial cost volumes at each scale, aggregates them separately, and then aggregates across scales to produce the final robust finest-scale cost volume.The algorithm accepts a stereo color-image pair as input.
- Generality: Any existing cost aggregation method can be used in the per-scale aggregation step.This preserves the framework’s compatibility with conventional aggregation methods.
- Complexity: The additional cross-scale computation requires only a small amount of extra computational load because the coupling matrix is spatially invariant and tridiagonal.Its inverse can be pre-computed, avoiding repeated matrix inversion.
6. Experimental Result and Analysis
Experiments on Middlebury, KITTI, and New Tsukuba evaluate cross-scale integration with several cost aggregation methods, showing broad improvements and a small computational overhead. The regularization study finds that moderate inter-scale coupling helps, whereas excessive coupling removes fine disparity details.
- Experimental Setup: The framework is evaluated with BF, GF, NL, ST, and BOX, with S+ denoting integration into the cross-scale framework.Table 1 also reports Middlebury ranks and errors after applying the same disparity-refinement technique for the rank comparison.
- Middlebury Dataset: Middlebury evaluation uses 31 stereo pairs in M31 and excludes four textureless-region pairs to form M27, reducing bias toward cases where local methods are fragile.The excluded pairs are Midd1, Midd2, Monopoly, and Plastic from Middlebury 2006.
- Middlebury Dataset: Cross-scale integration improves all evaluated cost aggregation methods on Middlebury, including the simple BOX baseline, which becomes comparable to state-of-the-art on M27.Avg Non-occ measures the percentage of non-occluded pixels whose absolute disparity error exceeds 1.
- Efficiency: The added computational overhead is relatively small and consists of aggregating the cost volumes at multiple scales plus computing Equation (13).Running time is reported for the Tsukuba stereo pair on a 2.83 GHz CPU with 8 GB of memory.
- KITTI Dataset: On KITTI, cross-scale aggregation significantly improves BOX and GF, while S+NL and S+ST remain nearly unchanged and can underperform S+BOX.The authors attribute the remaining textureless slant-plane errors to tree-based methods’ piecewise-constancy assumption.
- New Tsukuba Dataset: All evaluated cost aggregation methods improve with cross-scale aggregation on New Tsukuba.The experiment uses the Daylight scene and the KITTI benchmark metric with an error threshold of 3.
- Regularization Parameter Study: The regularization parameter controls inter-scale coupling: introducing it improves every method, but very large values make scales identical and increase error by removing fine details.At λ = 0, the procedure is equivalent to finest-scale aggregation; the error study uses M31.
7. Conclusions and Future Work
The paper proposes cross-scale cost aggregation for stereo matching and validates its effect across three datasets. It concludes that the framework improves nearly all evaluated methods, while future work should examine continuous plane-parameter stereo methods.
- Cross-scale cost aggregation is proposed as a framework for stereo matching, rather than as an entirely new high-accuracy aggregation method.
- Extensive experiments on three datasets validated the effect of cross-scale cost aggregation, with almost all methods showing improvements.
- Future work should investigate the scale-space behavior of stereo methods operating in continuous plane parameter space instead of discrete disparity labels.