Source-linked AI summary
BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation
Zhenyu Li, Xuyang Wang, Xianming Liu, Junjun Jiang
TL;DR
Monocular depth estimation must balance continuous prediction with the visual artifacts and design limitations of regression or discretized classification. BinsFormer addresses this through Transformer-based adaptive-bin set prediction, multi-scale refinement, and auxiliary scene classification, achieving new state-of-the-art results across evaluated datasets.
Problem
Per-pixel regression can converge slowly and yield unsatisfactory results, while discretized classification causes sharp depth discontinuities.
Method
BinsFormer uses a Transformer decoder for adaptive-bin set prediction, a per-pixel representation module, multi-scale refinement, and an auxiliary scene-classification query.
Results
BinsFormer achieves new state-of-the-art monocular depth estimation performance on NYU, KITTI, and SUN-RGBD, with large margins reported using a Swin-Transformer backbone.
Takeaways & Limitations
Adaptive bins, multi-scale refinement, and auxiliary scene understanding together form an effective classification-regression depth-estimation framework with demonstrated cross-dataset generalization.
Abstract
from arXiv · showhide
Monocular depth estimation is a fundamental task in computer vision and has drawn increasing attention. Recently, some methods reformulate it as a classification-regression task to boost the model performance, where continuous depth is estimated via a linear combination of predicted probability distributions and discrete bins. In this paper, we present a novel framework called BinsFormer, tailored for the classification-regression-based depth estimation. It mainly focuses on two crucial components in the specific task: 1) proper generation of adaptive bins and 2) sufficient interaction between probability distribution and bins predictions. To specify, we employ the Transformer decoder to generate bins, novelly viewing it as a direct set-to-set prediction problem. We further integrate a multi-scale decoder structure to achieve a comprehensive understanding of spatial geometry information and estimate depth maps in a coarse-to-fine manner. Moreover, an extra scene understanding query is proposed to improve the estimation accuracy, which turns out that models can implicitly learn useful information from an auxiliary environment classification task. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that BinsFormer surpasses state-of-the-art monocular depth estimation methods with prominent margins. Code and pretrained models will be made publicly available at \url{https://github.com/zhyever/Monocular-Depth-Estimation-Toolbox}.
1 Introduction
Classification-regression reframes monocular depth estimation to combine probabilistic representations with continuous predictions, while BinsFormer improves adaptive-bin generation and prediction interaction using Transformer-based components.
- Per-pixel regression predicts continuous depth but can suffer from slow convergence and unsatisfactory results.
- Discretizing depth as classification improves performance but introduces poor visual quality through sharp depth discontinuities.
- Classification-regression predicts per-pixel probability representations and combines them linearly with bin centers to produce continuous depth.
- BinsFormer treats adaptive-bin generation as direct set prediction, using a separate Transformer decoder to produce bin lengths and embedding vectors.
- A multi-scale decoder estimates depth coarse-to-fine, while an auxiliary scene-understanding query provides environment-classification supervision.
- Experiments on NYU, KITTI, and SUN-RGBD report new state-of-the-art performance with a Swin-Transformer backbone.
2 Related Work
Prior monocular depth methods explored multi-scale convolutional prediction, classification, and classification-regression, while BinsFormer extends adaptive bins with Transformer set prediction and multi-scale refinement.
- Earlier methods used global and local networks or multi-scale convolutional fusion to predict coarse and refined depth.
- Regression-based approaches improved accuracy but were associated with relatively slow convergence and sub-optimal solutions.
- Classification-regression methods were introduced to alleviate sharp discontinuities caused by discretizing continuous depth values.
- BinsFormer treats adaptive-bin generation as set prediction and uses Transformer layers in the decoder rather than only strengthening encoder capability.
- The method also integrates multi-scale refinement and an auxiliary scene-understanding task to improve model performance.
- Its basic framework is designed to function as a strong baseline that can be plugged into other methods.
3 Methods
BinsFormer combines a pixel-level encoder-decoder with a Transformer decoder that predicts adaptive bins, bin embeddings, and auxiliary scene classification. It integrates multi-scale coarse-to-fine refinement and computes depth by combining probability maps with bin centers.
- Framework Overview: BinsFormer contains pixel-level, Transformer, and depth-estimation modules, augmented by scene classification and multi-scale refinement.The pixel-level module produces multi-scale features and per-pixel representations, while the Transformer predicts bin-related outputs.
- Adaptive Bins: A Transformer decoder treats adaptive bin generation as set-to-set prediction, producing bin lengths and bin embedding vectors from image features.Independent projections predict bin lengths b and embeddings fb from Transformer query outputs.
- Depth Estimation: Pixel representations and bin embeddings form a similarity map whose Softmax output is combined linearly with post-processed bin centers to estimate depth.Predicted bin lengths are converted into bin centers before the final linear combination.
- Design Difference: Unlike Adabins, BinsFormer disentangles bin generation from fine-grained per-pixel representations to avoid their interference and reduce large-area failures.The design leaves the per-pixel backbone untouched while using the Transformer for bin generation.
- Multi-Scale Refinement: The multi-scale decoder progressively interacts with image features at different resolutions and applies increasingly weighted losses for coarse-to-fine prediction refinement.Each Transformer decoder scale receives one resolution of the multi-scale feature maps.
- Auxiliary Scene Understanding: An auxiliary scene query predicts the input environment and provides implicit supervision that helps bin queries absorb global semantic information through self-attention.The auxiliary classification task uses a CrossEntropy loss and adds negligible training overhead compared with pixel-wise chamfer loss.
4 Experiments
BinsFormer is evaluated across KITTI, NYU-Depth-v2, and SUN RGB-D using standard protocols, comparisons, and ablations. Results report strong benchmark performance, cross-dataset generalization, adaptive-bin behavior, and benefits from multi-scale refinement and auxiliary scene classification.
- Datasets and evaluation: Experiments cover KITTI, NYU-Depth-v2, and SUN RGB-D using standard depth-estimation evaluation metrics and established dataset protocols.KITTI includes Eigen-split and online benchmark evaluations; SUN RGB-D is used for cross-dataset generalization.
- State-of-the-art comparison: BinsFormer significantly outperforms leading methods on KITTI Eigen split and improves the online KITTI SILog metric by 6.1%.The online benchmark comparison reports persistent saturation in SILog while still measuring a 6.1% improvement.
- State-of-the-art comparison: BinsFormer outperforms all competitors with prominent margins across all reported NYU-Depth-v2 metrics.Table 3 includes BinsFormer variants using different encoders.
- State-of-the-art comparison: Training on NYU-Depth-v2 and testing on SUN RGB-D without fine-tuning yields significant improvements across all metrics.This evaluates cross-dataset generalization under a direct transfer setting.
- Ablation studies: BinsFormer adaptively estimates bins for dynamic scenes, with predicted bins approaching the maximum depth when images contain large distant regions.The ablation comparison includes regression, fixed Uniform/Log-uniform bins, and Adabins strategies.
- Ablation studies: Increasing the number of queries improves performance until gains diminish above N = 64, which is used in the final model.Queries correspond one-to-one with predicted bins.
- Ablation studies: Multi-scale refinement improves predictions as feature scale increases up to three scales, producing sharper and more accurate depth maps.The study reports that performance increases significantly until the scale count reaches three.
- Ablation studies: The auxiliary scene classification task improves model performance through implicit supervision, while its query aggregates spatial information across feature scales.The classification query provides hints for generating bins embeddings.
5 Conclusion
BinsFormer combines set-based adaptive-bin prediction, high-resolution pixel representations, depth aggregation, auxiliary scene understanding, and multi-scale refinement. The method achieves new state-of-the-art performance on two benchmark datasets and demonstrates cross-dataset generalization.
- BinsFormer predicts adaptive bins in a set-to-set manner and combines them with per-pixel probability distributions to estimate depth maps.Its architecture includes Transformer-based bin prediction, per-pixel representations, and depth aggregation.
- An auxiliary scene understanding task and multi-scale prediction refinement further boost performance with negligible overhead.
- BinsFormer achieves new state-of-the-art performance on two popular benchmark datasets.
- Cross-dataset experiments further demonstrate the method’s generalization ability.
A.1 Network Structure
The network feeds multi-scale FPN outputs into a Transformer that predicts bins, auxiliary scene classifications, and depth maps through aggregation and refinement.
- FPN outputs are fed into the Transformer module, where they interact with learned queries.
- MLPs project queries into bin embeddings, bin centers, and auxiliary scene classification results.
- The depth estimation module aggregates the predicted information to produce final depth maps at each scale.Applying the module across scales enables multi-scale refinement.
A.2 More Qualitative Results
Qualitative comparisons with previous state-of-the-art methods show that BinsFormer estimates better depth, recovers more details, and maintains an accurate overall scale.
- BinsFormer is qualitatively compared with BTS, Adabins, and DORN on KITTI, KITTI online benchmark, and SUN RGB-D.
- The results show better depth estimation and more detailed recovery for BinsFormer.
- Reasonable global bin generation supports an accurate overall depth scale.
A.3 Point Cloud Visualization
The paper visualizes qualitative depth results across KITTI and SUN RGB-D and point clouds on NYUv2. The NYUv2 visualizations show satisfactory recovery of 3D structures.
- Point clouds are visualized on the NYUv2 test set by back-projecting image pixels using estimated depth maps.
- The NYUv2 point-cloud visualizations show that 3D structures can be recovered satisfyingly.
- Qualitative results are presented for KITTI, KITTI online benchmark, and the unseen SUN RGB-D dataset.