Source-linked AI summary
Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging
S. Mahdi H. Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, Yağız Aksoy
TL;DR
High-resolution monocular depth estimation is limited by low-resolution outputs and a trade-off between scene structure and fine detail. The paper merges estimates from different resolutions and adaptive patches using a pre-trained model, producing multi-megapixel depth maps with improved resolution and boundary accuracy without retraining.
Problem
Existing monocular depth estimators commonly produce sub-megapixel maps that lack fine-grained details, while high-resolution estimation must retain boundary accuracy and consistent scene structure.
Method
The method combines low- and high-resolution estimates, then selects and merges content-adaptive patches onto a structurally consistent base estimate.
Results
The pipeline generates multi-megapixel depth maps and exceeds the state of the art in resolution and boundary accuracy without retraining the original networks.
Takeaways & Limitations
Pre-trained monocular depth models can be boosted to produce highly detailed high-resolution depth maps, including for complex scenes.
Takeaways & Limitations
The method produces relative rather than absolute depth, does not meaningfully improve noisy images, and can leave low-magnitude white noise on flat surfaces.
Abstract
from arXiv · showhide
Neural networks have shown great abilities in estimating depth from a single image. However, the inferred depth maps are well below one-megapixel resolution and often lack fine-grained details, which limits their practicality. Our method builds on our analysis on how the input resolution and the scene structure affects depth estimation performance. We demonstrate that there is a trade-off between a consistent scene structure and the high-frequency details, and merge low- and high-resolution estimations to take advantage of this duality using a simple depth merging network. We present a double estimation method that improves the whole-image depth estimation and a patch selection method that adds local details to the final result. We demonstrate that by merging estimations at different resolutions with changing context, we can generate multi-megapixel depth maps with a high level of detail using a pre-trained model.
1 Simon Fraser University 2 Adobe Research
The figure presents a high-resolution depth-estimation method that merges predictions across resolutions and patches, with results shown using MiDaS.
- The method merges estimations at different resolutions and patches to generate a high-resolution depth estimate from a single image.
- The demonstrated pipeline uses MiDaS as the pre-trained network.
- An asterisk denotes equal contribution.
1. Introduction
The paper targets high-resolution monocular depth estimation by combining low-resolution structural consistency with high-resolution detail through content-adaptive estimation and merging.
- High-resolution monocular depth estimation remains difficult while maintaining good boundary accuracy and consistent scene structure.
- The method merges low- and high-resolution estimates from a pre-trained model to preserve structure while capturing high-frequency details.
- Local patches are selected using depth-cue density and merged onto a structurally consistent base estimate.
- The framework improves resolution and boundary accuracy without retraining the original networks.
2. Related Work
Existing depth-estimation methods generally operate at relatively small resolutions, while this work changes network inputs and merges estimates to recover high-frequency detail.
- Deep neural networks have become standard solutions for monocular depth estimation after advances in architectures, constraints, and supervision.
- Most existing methods use relatively small input resolutions, limiting their suitability for applications requiring high-resolution depth maps.
- Prior refinement methods use guided upsampling or residual training, whereas this approach changes network inputs and merges multiple estimations.
- Figure 3 illustrates that low resolutions preserve scene structure but miss details, while higher resolutions improve boundaries yet introduce structural artifacts.
- Figure 4 attributes degradation at higher resolutions to depth cues becoming farther apart than the fixed network receptive field.
3. Observations on Model Behavior
The model’s input resolution changes which depth cues fit within its receptive field, producing a trade-off between global structure and local detail.
- Monocular depth estimation relies on high-level contextual cues such as occlusions and perspective-related signals.
- At smaller resolutions, estimates retain consistent scene structure but often omit high-frequency details.
- Upsampling a 192 × 192 image to 500 × 500 before inference adds details to the depth estimate.
- At higher resolutions, additional details appear while gradual depth shifts create inconsistencies in scene structure.
- A receptive field is the region around a pixel that contributes to the output at that pixel.
- Limited network capacity restricts the amount of information that can be generated within one receptive field during a forward pass.
4. Method Preliminaries
The method generates multiple depth estimates under different contextual conditions, then merges them so high-frequency details are transferred while structural consistency is retained.
- Method Overview: The pipeline must estimate the distribution of contextual cues to determine network inputs and perform a structure-preserving merge.These are the two stated requirements for combining multiple depth estimates.
- Estimating Contextual Cues: Contextual-cue estimation uses image edges as a proxy because an existing cue-identification method did not produce the high-resolution estimates required for MiDaS.The cited prior method analyzes inference cues but was insufficient for this use case.
- Merging Monocular Depth Estimates: The framework targets a merged result combining a low-resolution estimate’s consistent structure with a higher-resolution estimate’s boundary details.The two inputs may represent the same image or a patch, and the high-resolution estimate can contain low-frequency artifacts.
- Merging Monocular Depth Estimates: The merging network adopts a Pix2Pix architecture with a 10-layer U-net rather than the default 6-layer generator.The deeper generator is selected to support training and inference at 1024 × 1024.
5. Double Estimation
Double estimation chooses a content-adaptive resolution and merges low- and high-resolution predictions, preserving scene structure while adding high-frequency detail beyond the receptive-field limit.
- Resolution Selection: R0 is the maximum resolution at which every pixel remains within contextual range, determined by dilating an edge map with a receptive-field-sized kernel.Resolutions above R0 are denoted Rx according to the percentage of pixels lacking nearby contextual information.
- Resolution Selection: Above R0, estimates gain richer high-frequency content but lose structural consistency as contextual cues become too far apart.This trade-off is illustrated by sharper but structurally unstable estimates at higher resolutions.
- Double Estimation: Double estimation feeds the image at two resolutions and merges the outputs, using a receptive-field-sized low-resolution estimate to determine overall structure.The merging network successfully transfers detail up to R20; artifacts beyond R20 damage merged results.
- Double Estimation: Merging preserves the low-resolution structure while integrating high-resolution details, enabling consistent results beyond R0 at the cost of a second base-network forward pass.Figure 6 shows this fusion between the low-resolution estimate and the detailed high-resolution estimate.
6. Patch Estimates for Local Boosting
Local boosting selects patches according to contextual-cue density, estimates them at higher resolution, and merges them into a whole-image base estimate for added detail.
- Patch Selection: The whole-image resolution is limited by low-context regions, while denser regions can benefit from higher-resolution estimates merged into a consistent full result.This motivates selecting different resolutions for different image regions.
- Patch Selection: The patch-selection method uses cautious design choices because available datasets cannot train a data-driven local-resolution selector or high-resolution cue estimator.The method therefore avoids requiring an additional dataset or training.
- Patch Selection: The pipeline first creates an R20 base estimate, then tiles it with receptive-field-sized patches using one-third overlap.Tiles with lower edge density than the whole image are discarded; denser tiles are enlarged until their edge density matches.
- Patch Estimates: Patch estimates use a fixed high-resolution size twice the receptive-field size and are merged sequentially onto the base estimate.The merging network handles differing depth-value ranges while transferring patch high-frequency details.
- Base Resolution Adjustment: When edge density varies strongly, the base estimate is upsampled before selection to prevent R20 from producing patches that are too small.The adjustment is especially relevant when a large image portion lacks edges.
7. Results and Discussion
The method improves monocular depth estimation across datasets and base networks by combining adaptive resolutions and multiple estimations, especially for boundaries and high-frequency details. Its gains over refinement baselines are strongest qualitatively, while resolution choice and image noise remain important constraints.
- Adaptive computation: 74.82 patches per image at average R20 = 2145 × 1501 were selected for Middleburry, versus 12.17 patches at 1443 × 1082 for IBMS-1.The different workloads reflect differing scene structures, and estimation resolution depends mainly on image content rather than original input resolution.
- Quantitative evaluation: The full content-adaptive pipeline consistently improves ORD and D3R over baselines on both datasets, indicating better depth ordering and boundary preservation.It performs comparably in RMSE and δ1.25.
- Qualitative comparison: Qualitative comparisons show drastically more high-frequency details and better boundary localization than the original networks.Figure 8 is identified as showing the strongest visible performance improvement.
- Comparison with refinement: Refinement methods show little improvement because they cannot generate details absent from the base estimate, unlike the method’s repeated network use and patch merging.The cited examples specifically mention small objects and sharp depth discontinuities.
- Resolution selection: R20 is chosen over R30 because higher-resolution estimates introduce heavy artifacts and reduce performance as more pixels lack contextual information.Table 2 also shows that double estimation outperforms fixed input resolutions and R0 estimation.
- Limitations: The method produces relative ordinal depth rather than absolute depth, and low-magnitude white noise in high-frequency estimates can make flat surfaces appear noisy.It also does not provide meaningful improvement for noisy images.
8. Conclusion
The paper demonstrates a pre-trained-model algorithm for multi-megapixel monocular depth maps, supporting detailed boundaries and applications such as image segmentation.
- The algorithm infers high-resolution depth maps from single images using pre-trained models, extending beyond previous sub-megapixel limitations.The method processes multi-megapixel images captured by modern cameras.