Source-linked AI summary
Self-Supervised Perceptually Interpretable Monocular Depth Estimation
Zain Ul Abidin, George Dimas, Dimitris K. Iakovidis
TL;DR
Self-supervised monocular depth estimation avoids ground-truth depth labels, but RGB-based approaches can obscure which image cues influence predictions. PIMDE decomposes images into perceptual feature maps and processes them in separate depth branches; on KITTI, it achieves competitive performance while offering clearer insight into cue contributions.
Problem
RGB-based depth predictions can obscure which visual cues influence the result, leaving the internal decision process difficult to interpret.
Source: §1 · p. 1Method
PIMDE decomposes RGB images into perceptual feature maps and processes each cue in a separate depth-estimation branch before combining the predictions.
Source: §1 · p. 1Results
On KITTI, PIMDE achieves competitive performance compared with various baselines while clarifying how visual components influence the final prediction.
Source: §4 · p. 5Takeaways & Limitations
PIMDE allows depth predictions to be examined at individual perceptual components rather than only as black-box outputs.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Self-supervised monocular depth estimation (MDE) enables depth prediction from monocular images without requiring ground-truth supervision, making it attractive for large-scale and real-world applications. Despite steady improvements in accuracy, most existing methods remain difficult to interpret, as depth is inferred from RGB representations that obscure the impact of individual perceptual image components. This lack of transparency limits systematic analysis of failure cases and reduces confidence in safety-critical settings. This paper presents a self-supervised framework for perceptually interpretable monocular depth estimation (PIMDE), designed to associate depth predictions with distinct perceptual components of the input image. Rather than operating directly on RGB inputs, the proposed method decomposes each image into a set of perceptual feature maps (PFMs), each encoding a specific visual cue. Distinct depth estimation branches process these PFMs independently to produce depth estimates (PIDEs), which are subsequently combined through an explicit fusion strategy. This formulation allows us to examine directly the contribution of each perceptual cue to the final depth prediction. Experiments conducted on the KITTI benchmark dataset demonstrate that PIMDE achieves performance comparable to established self-supervised MDE methods while providing additional insight into how different perceptual cues influence depth estimation. These results indicate that perceptual decomposition can support interpretability without sacrificing depth estimation accuracy.
1. INTRODUCTION
Self-supervised monocular depth methods can predict depth without ground-truth training data, but their RGB-based predictions often obscure which visual cues shape the result. PIMDE separates perceptual cues into independently processed branches, making their influence examinable.
1. INTRODUCTION
- Self-supervised MDE methods remain difficult to interpret, leaving it unclear whether errors arise from missing texture, misleading color, or illumination changes.The paper notes that this lack of transparency limits user trust and can matter in safety-sensitive applications.Source: §1 · p. 1
- Post-hoc tools can highlight influential image regions but do not explain how specific visual cues contribute to depth, while RGB networks entangle those cues.The paper proposes separating perceptual components before depth estimation as a more transparent alternative.Source: §1 · p. 1
- PIMDE decomposes RGB images into perceptual feature maps representing chromatic, structural, and luminance cues, then estimates depth from each cue in parallel branches.The framework is described as self-supervised and inspired by human visual perception.Source: §1 · p. 1
- A subtraction-based analysis examines how each perceptual branch influences the final depth prediction, and the framework is validated on KITTI.The framework’s known combination of branch predictions permits assessment of each PFM’s contribution.Source: §1 · p. 1
2. METHODOLOGY
PIMDE decomposes images into perceptual feature maps and processes them with parallel depth-estimation branches whose outputs are explicitly fused. Its self-supervised training and branch-removal analysis make each branch’s contribution to the final prediction assessable.
2.1. Framework Overview
- Each branch processes a distinct perceptual representation and produces a complete inverse-depth map; branch outputs are explicitly fused into the final prediction.The framework uses recurrent multi-scale refinement, and converts the final inverse depth to metric depth for evaluation.Source: §2.1 · p. 2, §2.3 · p. 2
2.2. Perceptual Feature Map Decomposition
- The four PFMs encode red–green and blue–yellow opponency, coarse–fine texture, and light–dark luminance, rather than complete geometric depth cues.CIE-Lab a and b channels provide the chromatic maps; wavelet luminance components provide the texture and lightness maps.Source: §2.2 · p. 2
2.3. Monocular Depth Estimation Model
- Identical CNN branches estimate inverse depth from separate PFMs, with recurrent refinement incorporating multi-scale information at each iteration.The branch-wise outputs are depth approximations, not intermediate feature maps.Source: §2.3 · p. 2
- Fusion weights are estimated by measuring each branch’s effect on the fused prediction, softmax-normalized, and held fixed during final training.The process begins with uniform weights and uses subtraction-based branch analysis to obtain relative-difference values.Source: §2.3 · pp. 2–3
- Training applies self-supervised photometric reprojection and edge-aware smoothness losses to the fused prediction, with the same loss also supervising each branch.The reprojection loss uses masking and auto-masking; the smoothness term helps smooth homogeneous regions while preserving changes at image boundaries.Source: §2.3 · p. 3
2.4. Interpretability
- Branch influence is mapped by subtracting the fused prediction without a branch from the full fused prediction, with fusion held identical between the two.This isolates the prediction change associated with removing that perceptual branch.Source: §2.4 · p. 3
3. EXPERIMENTS AND RESULTS
PIMDE is evaluated on KITTI using quantitative, qualitative, and branch-contribution analyses. The reported results show competitive performance, improved predictions with more refinement iterations, and distinct effects from perceptual branches.
3.1. Experimental Setup and Dataset
- Experiments use the KITTI Eigen split, remove static training sequences for valid photometric supervision, and resize inputs and outputs to 640 × 192.The filtered training set contains 39,810 sequences, with 4,424 validation sequences.Source: §3.1 · p. 3
3.2. Quantitative Results
- On KITTI, both PIMDE variants achieve competitive performance across metrics, and the six-iteration variant consistently outperforms the three-iteration variant.The evaluation compares PIMDE-3 and PIMDE-6 with established self-supervised MDE baselines.Source: §3.2 · p. 4, Table 1 · p. 4
- The full PIMDE configuration outperforms every single-branch setting, while the CF branch performs strongly on its own.The comparison evaluates PIMDE-6 models trained on individual PFMs.Source: §3.2 · p. 4, Table 2 · p. 4
3.3. Qualitative Results
- Six refinement iterations produce smoother, more spatially consistent maps in planar regions, while the branches show distinct depth-map characteristics.CF shows clearer depth changes at structures, RG and BY mainly affect the road, and LD is smoothest but shows fewer details.Source: §3.3 · p. 4, Figure 3 · p. 5, Figure 4 · p. 5
3.4. Interpretability Analysis
- Removing CF produces the largest prediction changes, especially around object boundaries; LD affects broader smooth regions, while RG and BY changes are more localized.The authors report that CF also has the highest relative-difference values in the branch-contribution statistics.Source: §3.4 · p. 5, Table 3 · p. 5
4. CONCLUSION
PIMDE separates perceptual cues into branches and fuses their depth predictions, enabling component-wise examination. On KITTI it achieves competitive performance, while broader datasets, domains, and branch-selection strategies remain future work.
4. CONCLUSION
- PIMDE decomposes input into chromatic, luminance, and structural cues, processes them in parallel depth branches, and combines them with fixed fusion.The design supports examining predictions by individual perceptual component rather than only as black-box outputs.Source: §4 · p. 5
- On KITTI, PIMDE achieves competitive performance against various baselines while clarifying how visual components influence the final prediction.Source: §4 · p. 5
- Future work will examine additional and automatic PFM selection, branch-combination analysis, stronger domain-specific backbones, and evaluation on more datasets and domains.Source: §4 · p. 5