Source-linked AI summary
LocalBins: Improving Depth Estimation by Learning Local Distributions
Shariq Farooq Bhat, Ibraheem Alhashim, Peter Wonka
TL;DR
Single-image depth estimation needs an architecture that can model local depth distributions and use distribution supervision throughout decoding rather than relying on a global, late prediction. LocalBins extends AdaBins with per-pixel local distributions and multi-scale decoder integration, and improves state-of-the-art performance across all NYU-Depth V2 metrics.
Problem
Existing AdaBins-based estimation predicts a global depth distribution and introduces distribution supervision late in the decoder, motivating local and earlier multi-scale alternatives.
Method
LocalBins predicts local depth distributions at every pixel through progressively split bins integrated across decoder layers, with query-and-response regularization.
Results
LocalBins improves upon the state of the art on all metrics of NYU-Depth V2.
Takeaways & Limitations
The architecture’s local, multi-scale design supports a lightweight module that can be used with encoder-decoder networks and may extend to other dense regression tasks.
Abstract
from arXiv · showhide
We propose a novel architecture for depth estimation from a single image. The architecture itself is based on the popular encoder-decoder architecture that is frequently used as a starting point for all dense regression tasks. We build on AdaBins which estimates a global distribution of depth values for the input image and evolve the architecture in two ways. First, instead of predicting global depth distributions, we predict depth distributions of local neighborhoods at every pixel. Second, instead of predicting depth distributions only towards the end of the decoder, we involve all layers of the decoder. We call this new architecture LocalBins. Our results demonstrate a clear improvement over the state-of-the-art in all metrics on the NYU-Depth V2 dataset. Code and pretrained models will be made publicly available.
1 Introduction
LocalBins extends AdaBins by predicting local depth distributions at each pixel and injecting distribution supervision throughout the decoder. The resulting architecture is designed to address late supervision while improving single-image depth estimation on NYU-Depth V2.
- Motivation: AdaBins predicts one global depth distribution per image, whereas LocalBins predicts a distribution for each pixel’s neighborhood.The distinction motivates replacing global adaptive bins with local adaptive bins.
- Motivation: The paper identifies a late injection problem because AdaBins estimates bins near the output, potentially limiting earlier layers’ use of distribution supervision.Attempts to estimate global adaptive bins earlier or without a transformer can produce unstable training or suboptimal convergence.
- Method: LocalBins estimates local depth distributions at every pixel and regularizes them through query-and-response training on randomly selected image bounding boxes.The training scheme targets distributions within sampled regions rather than relying on a fixed local window.
- Method: At decoder layer i, LocalBins predicts 2^iN_seed bins per pixel by splitting bins progressively from a small bottleneck seed.This coarse-to-fine, multi-scale construction addresses the late injection problem and avoids unstable training.
- Contribution: The lightweight LocalBins module adds approximately 1M parameters and can be combined with any encoder-decoder network.The paper presents the module as broadly compatible with encoder-decoder architectures.
- Results: LocalBins improves upon the state of the art on all metrics of the NYU-Depth V2 dataset.The paper summarizes its contributions as a new single-image depth architecture based on local and multi-scale histogram estimation.
2 Related Work
Related depth-estimation methods differ in supervision and scale handling, including unsupervised methods, multi-dataset methods, and single-dataset methods with consistent scale.
- Depth-estimation categories: Unsupervised methods learn depth without ground-truth depth data, generally using 3D-reconstruction self-supervision from videos or stereo videos.These methods form one category of depth-estimation approaches discussed by the paper.
- Depth-estimation categories: Methods trained jointly on multiple datasets must account for differing depth scales across scenes.The passage notes that this scale issue affects comparability with other methods.
3 Methodology
LocalBins extends AdaBins by predicting adaptive depth distributions locally at every pixel and refining them throughout the decoder. Its architecture combines pixel-wise bin estimation, coarse-to-fine splitting, and Query-Response regularization to capture local depth distributions across scales.
- Background: AdaBins predicts one adaptive global depth-bin division per image, whereas LocalBins predicts a bin division at every pixel to represent local depth distributions.LocalBins retains adaptive binning while adding pixel-level locality.
- Architecture: The architecture uses a standard encoder-decoder block together with a lightweight LocalBins module and retains AdaBins’ EfficientNet-B5 encoder-decoder for fair comparison.The LocalBins module adds approximately 1M parameters and is designed to work with encoder-decoder networks.
- Architecture: LocalBins embeds bottleneck and multi-scale decoder features into a shared 128-dimensional bin-embedding space used to determine each pixel’s bin division.The module includes bin embedding layers, seed bin-width estimators, and bin splitters implemented with pointwise MLPs.
- Architecture: Starting from Nseed bins at the bottleneck, each decoder layer splits every bin into two, yielding 2^nNseed bins after an n-layer decoder.This coarse-to-fine process progressively refines local distributions while incorporating depth-distribution prediction throughout the decoder.
- Architecture: The final depth at each pixel is computed as the expectation of pixel-specific bin centers under the output-layer softmax probabilities.Unlike AdaBins, the bin widths and centers vary from pixel to pixel.
- Training: Query-Response training regularizes local distributions by requiring features averaged within bounding boxes to predict the depth-value density inside those boxes.Bounding boxes of different sizes provide multi-scale supervision, while ROIAlign pools bottleneck and decoder features for the regularization.
4 Implementation Details
The models use a PyTorch implementation with specified MLP dimensions, batch size, optimizer, weight decay, learning-rate schedule, and a 10-epoch training budget.
- The Seed bin width MLP uses hidden dimension h = 256, while Bin embedding and Bin splitter MLPs use h = 128.
- Training uses batch size 16, AdamW with weight decay 10^-1, and an initial learning rate of 3.57 × 10^-4.
- The learning rate is decayed by a factor of 10^4 during the last 30% of iterations using cosine decay.
- All experiments train the models for 10 epochs.
5 Experiments and Results
Experiments evaluate LocalBins on NYU-Depth V2 and zero-shot transfer benchmarks, alongside ablations of its components and training choices. The model reaches state-of-the-art performance on NYU-Depth V2, while analyses report improved accuracy, efficiency, robustness, and generalization in supported comparisons.
- 5.1 Comparison to state-of-the-art: LocalBins achieves state-of-the-art performance across all NYU-Depth V2 metrics, including an approximately 4% reduction in absolute relative error.It also performs better on thin objects and planar surfaces, and uses fewer parameters than AdaBins with the same encoder-decoder backbone.
- 5.2 Zero-shot performance: On iBims-1, LocalBins outperforms prior state-of-the-art methods without fine-tuning and predicts scale-accurate depths more consistently across varying depth ranges.AdaBins noticeably underestimates scene depth ranges, with relative error growing with distance.
- 5.2 Zero-shot performance: SUN-RGBD evaluation tests models trained on NYU-Depth V2 without fine-tuning on the official 5050-image test set.
- 5.3 Analysis and Ablation Studies: Query-Response training improves absolute relative error by approximately 7% compared with the naive scheme, while foveated weighting further improves squared-error metrics.
- 5.3 Analysis and Ablation Studies: Query-Response training performs approximately 7.6% better with 8× fewer point-set comparisons than the naive scheme.PSCI measures point-set comparisons per image, while coverage measures the total pixel area involved in loss computation.
- 5.3 Analysis and Ablation Studies: LocalBins is generally more robust to the total number of bins and generally performs better than AdaBins as Nseed varies.
6 Conclusions
LocalBins extends AdaBins with local depth distributions and multi-scale decoder integration, while introducing query-response training to make local prediction practical.
- LocalBins estimates local depth distributions around each pixel through bin embedding layers, a seed bin width estimator, and bin splitters integrated with the decoder.
- Query-response acceleration makes training feasible because naive local-distribution training would consume substantial time and memory.
- Qualitative results on NYU-Depth V2 visualize LocalBins predictions alongside input images and local depth distributions.
A.1 Visualizing bin predictions
The visualizations compare LocalBins bin predictions with ground-truth depth distributions across neighborhood sizes and indicate that the effective local neighborhood varies with context.
- LocalBins visualizations compare predicted local depth distributions with ground-truth distributions across neighborhoods of various sizes.Density plots show input RGB, predicted bin-center densities, and ground-truth depth densities at selected locations.
- The effective local neighborhood is indeterminate because Query-Response training regularizes predictions over windows rather than individual pixel locations.
- Figure 9 indicates that the model can cover different-sized neighborhoods depending on image context rather than using one fixed neighborhood size.
A.2 Different Backbones
The paper evaluates LocalBins with various backbone encoders, as reported in Table 7.
- Table 7 reports LocalBins performance with various backbone encoders.