Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

1. Introduction

• High-resolution monocular depth estimation with accurate boundaries and consistent scene structure remains challenging despite fully convolutional models that can handle arbitrary input sizes.Practical limits include GPU memory, a shortage of diverse high-resolution datasets, and CNN receptive field size.Source: §1 · p. 1
• Low-resolution inputs yield structurally consistent estimates but lack fine details, whereas high-resolution inputs capture details better while structural consistency degrades.The paper attributes this duality to limits in model capacity and receptive field size, and merges estimates at two resolutions adaptively to image content.Source: §1 · p. 2
• Because depth cues farther apart than the receptive field can produce structural inconsistency, the method selects local patches and adapts their input resolution to cue density.Their estimates are merged onto a structurally consistent base estimate to produce a highly detailed result.Source: §1 · p. 2, Figure 2 · p. 2
• Without retraining, the method exceeds state of the art in resolution and boundary accuracy using two state-of-the-art models [34]; double estimation alone improves performance with limited computational overhead.The full pipeline produces highly detailed results, including for complex scenes.Source: §1 · p. 2

Related Work

• Rather than introducing a new depth estimation method, the work adapts existing models by merging estimates from different resolutions and patches.Recent progress has also relied on large-scale datasets and advances in architectures, geometric and semantic constraints, loss functions, and supervision strategies.Source: §2 · p. 2
• Most existing depth methods are trained at relatively small input resolutions, limiting their use where high-resolution depth maps are desirable.Prior refinement approaches use guided upsampling alone [30] or combine it with residual training; this work instead changes network inputs and merges estimates to generate high-frequency detail.Source: §2 · p. 2
• Although the patch framework resembles patch-based editing, matting, and synthesis [49] [51] [53], those techniques do not directly address depth-specific patch challenges.Depth estimates vary in range, depend strongly on patch context, and can contain characteristic low-frequency artifacts at high resolution.Source: §2 · p. 2
• Figure 3 shows a resolution trade-off: small inputs preserve overall scene structure but miss details, while higher inputs improve boundaries yet can produce low-frequency artifacts.The resolution where artifacts appear depends on the distribution of contextual cues in the image.Source: Figure 3 · p. 3
• As input resolution increases, depth cues become farther apart within the fixed receptive field, progressively degrading estimation accuracy.Figure 4 attributes this degradation to the model seeing less of the scene as resolution changes.Source: Figure 4 · p. 3

3. Observations on Model Behavior

• Monocular depth models use contextual cues such as occlusions and perspective-related cues that are also used by human vision [17].The authors examine how the density of these cues in the input affects network performance.Source: §3 · p. 3
• Lower-resolution inputs produce coherent overall scene structure but fewer high-frequency details, whereas higher-resolution inputs add details while introducing gradual depth shifts between regions.The authors explain this trade-off through limited network capacity and receptive-field size.Source: §3 · p. 3
• When contextual cues are farther apart than the receptive field, the network cannot estimate depth coherently; MiDaS’s 384 × 384 receptive field is associated with inconsistencies at larger inputs [25].The authors point to inconsistent estimates on a flat wall as further support for this observation.Source: §3 · p. 3
• Upsampling a 192 × 192 image to 500 × 500 increases depth-map detail despite unchanged high-frequency input information, which the authors attribute to limited network capacity [3].They argue that dense contextual cues help produce larger-scale structure but limit the details generated in one forward pass.Source: §3 · pp. 3–4, Figure 5 · p. 3

Method Preliminaries

• The method seeks multiple estimates of one image, using contextual cues to determine network inputs and a merge operation to transfer detail while retaining structural consistency.These preliminaries motivate the pipeline described in Sections 5 and 6.Source: §4 · p. 4

Estimating Contextual Cues

• The method uses thresholded RGB gradients as an approximate edge-map proxy for contextual cues, because Hu et al. [17]’s method could not generate the high-resolution MiDaS estimates needed.The choice follows their observation that image edges correlate reasonably with contextual cues.Source: §4 · p. 4

Merging Monocular Depth Estimates

• The low-resolution estimate provides consistent structure and a fixed depth range, while the higher-resolution estimate adds detail near discontinuities but contains low-frequency artifacts.The merge embeds the higher-resolution details into the low-resolution map, whether the estimate concerns a full image or a patch.Source: §4 · p. 4
• A Pix2Pix [19] network with a 10-layer U-net [35] learns to transfer fine detail from high-resolution estimates into low-resolution estimates.The 10-layer generator supports training and inference at 1024 × 1024. Training pairs use patches from Middlebury2014 and Ibims-1 [24]; since consistent high-resolution ground truth is unavailable, 672*672 inputs maximize artifact-free estimates across both datasets, and guided-filtered patches serve as proxy ground truth.Source: §4 · p. 4

5. Double Estimation

• R0 is the image-content-dependent maximum resolution for consistent structure, estimated by dilating an edge map with a receptive-field-sized kernel until it no longer covers every pixel.Resolutions above R0 are denoted Rx, where x is the percentage of pixels lacking contextual information.Source: §5 · pp. 4–5
• Above R0, predictions gain high-frequency content but lose structural consistency as contextual cues become farther apart than the network’s receptive field.Figure 6 shows sharper estimates at higher resolution but unstable depth structure, including an incorrect bench depth range and unrealistic gradients around tires.Source: §5 · p. 5, Figure 6 · p. 5
• Double estimation feeds the image at two resolutions: the receptive-field size sets overall structure, while merging can incorporate high-frequency details up to R20.Below the receptive-field size, estimates do not improve structure and can perform worse; artifacts beyond R20 begin to damage merged results, and R20 may exceed the original resolution.Source: §5 · p. 5
• The merging network preserves low-resolution structure while adding high-resolution detail, producing consistent estimates beyond R0 with a second base-network forward pass.Figure 6 illustrates the fusion of a consistent estimate with finer details from a structurally unstable higher-resolution estimate.Source: §5 · p. 5, Figure 6 · p. 5

6. Patch Estimates for Local Boosting

• Low contextual-cue density sets the maximum whole-image estimation resolution, while denser regions can gain high-frequency detail from higher-resolution patch estimates merged into a consistent result.The method selects different estimation resolutions for different image regions.Source: §6 · p. 5
• Because available datasets lack sufficient resolution to train a data-driven patch selector, the method uses cautious design choices and requires no additional dataset or training.The ideal selector would assess high-resolution network performance and contextual cues, but the available datasets cannot support that training.Source: §6 · p. 5
• Patch selection tiles the base-resolution image with receptive-field-sized patches and one-third overlap, discards tiles below the image-wide edge density, and enlarges denser tiles until densities match.The base estimate is generally fixed at R20; matching edge density is intended to give each selected patch a stable structure.Source: §6 · pp. 5–6
• Patch estimates use a second double-estimation scheme with high-resolution size fixed at twice the receptive field, then merge onto the base estimate despite differing ordinal depth ranges.The merging network is designed to handle non-metric depth estimates and transfer high-frequency patch detail.Source: §6 · p. 6
• For images with highly variable edge density, the base estimate is upsampled before selection using max(1, Rmax/(4KR20)), where Rmax = 3000×3000 and K is the percentage of pixels near edges.This adjustment addresses cases where large edge-poor areas would otherwise lead to patches that are too small; K is computed from a dilated edge map.Source: §6 · p. 6

7. Results and Discussion

• Runtime depends mainly on base-network calls; adaptive selection averaged 74.82 patches and R20 = 2145 × 1501 on Middleburry, versus 12.17 patches and R20 = 1443 × 1082 on IBMS-1.Although IBMS-1 images are 640 × 480, estimation resolution depends mainly on scene content rather than original input resolution; upscaling can help generate high-frequency detail.Source: §7 · p. 6

7.1. Boosting Monocular Depth Estimation Models

• The full pipeline considerably improves most metrics over the baselines, consistently improving ORD and D3R while performing comparably on RMSE and δ1.25.The double-estimation method already provides a good improvement at small computational overhead.Source: §7.1 · pp. 6–7, Table 1 · p. 7
• Adaptive adjustment to R0 meaningfully increases performance, while qualitative comparisons show more high-frequency detail and better boundary localization than the original networks.The qualitative comparison uses MiDaS and SGR outputs at their default sizes.Source: §7.1 · p. 7, Figure 8 · p. 7
• Refinement methods add little because they cannot recover details absent from the base estimate, such as small objects or sharp depth discontinuities.The paper contrasts these methods with its repeated network estimates and patch-based merging, which can add such details.Source: §7.1 · p. 7

7.2. Double Estimation and Rx

• Double estimation outperforms fixed input resolutions and R0; using R30 instead of R20 decreases performance as high-resolution estimates develop artifacts without context.R20 is selected as the high-resolution estimate in the double-estimation framework.Source: §7.2 · pp. 7–8, Table 2 · p. 8

7.3. Limitations

• The method produces relative ordinal rather than absolute depth, struggles with noisy images, and can leave white noise on flat surfaces.It uses RGB edges as a proxy for depth cues and makes ad-hoc patch-selection choices; the authors identify contextual cues and patch selection as areas for future work.Source: §7.3 · p. 8

• The method infers multi-megapixel depth maps from a single image using pre-trained models, beyond the sub-megapixel resolutions of previous work.Source: §8 · p. 8
• The approach builds on a careful characterization of existing depth-estimation networks and the factors that influence their abilities.Source: §8 · p. 8
• Thresholding estimated depth values provides a simple image-segmentation example and demonstrates boundary localization.The paper identifies image segmentation as one potential application of high-quality, high-resolution monocular depth estimation.Source: §8 · p. 8