Source-linked AI summary
How do neural networks see depth in single images?
Tom van Dijk, Guido C. H. E. de Croon
TL;DR
The paper addresses the lack of analysis explaining how neural networks estimate depth from single images. It probes MonoDepth with controlled image modifications and finds that the network primarily uses vertical object position rather than apparent size, with camera-pose and obstacle-recognition limitations. These findings clarify learned behavior while motivating tests on other networks and datasets.
Problem
Prior monocular-depth work showed accurate depth estimation but did not explain what visual cues neural networks had learned or how they behaved beyond tested cases.
Method
The authors treat MonoDepth as a black box, perturbing images with controlled or conflicting cues and measuring changes in its depth maps.
Results
MonoDepth primarily uses vertical object position rather than apparent size, partially accounts for camera-pose changes, and can detect some unseen obstacles under suitable visual conditions.
Takeaways & Limitations
Analyzing learned cues reveals limitations affecting behavior under camera-pose changes and conditions such as weak ground-contact evidence, informing transfer and training considerations.
Takeaways & Limitations
The findings come from one MonoDepth network trained on one KITTI dataset, so their applicability to other networks and datasets remains unknown.
Abstract
from arXiv · showhide
Deep neural networks have lead to a breakthrough in depth estimation from single images. Recent work often focuses on the accuracy of the depth map, where an evaluation on a publicly available test set such as the KITTI vision benchmark is often the main result of the article. While such an evaluation shows how well neural networks can estimate depth, it does not show how they do this. To the best of our knowledge, no work currently exists that analyzes what these networks have learned. In this work we take the MonoDepth network by Godard et al. and investigate what visual cues it exploits for depth estimation. We find that the network ignores the apparent size of known obstacles in favor of their vertical position in the image. Using the vertical position requires the camera pose to be known; however we find that MonoDepth only partially corrects for changes in camera pitch and roll and that these influence the estimated depth towards obstacles. We further show that MonoDepth's use of the vertical image position allows it to estimate the distance towards arbitrary obstacles, even those not appearing in the training set, but that it requires a strong edge at the ground contact point of the object to do so. In future work we will investigate whether these observations also apply to other neural networks for monocular depth estimation.
I. INTRODUCTION
Monocular depth estimation has advanced through learned pictorial cues, but prior work emphasized accuracy rather than analyzing what networks learned. This paper studies MonoDepth’s high-level behavior and why that understanding matters for robustness, training, and transfer.
- Motivation: Geometric methods estimate scene depth from stereo, optical flow, SLAM, or Structure-from-Motion while largely ignoring image content.Single-image methods instead require pictorial cues within the image.
- Motivation: CNNs made it possible to learn pictorial depth cues from data rather than program them manually.Monocular depth estimation progressed from early machine-learning methods to CNNs and stereo-supervised approaches.
- Research gap: Recent studies showed that networks estimate monocular depth on datasets such as KITTI and NYUv2, but did not investigate how they do so.The paper identifies this missing analysis as its central research gap.
- Why analysis matters: Understanding learned behavior can help assess unexpected scenarios, improve training and augmentation, and anticipate transfer to changed camera setups.The authors specifically discuss camera-height changes as a transfer concern.
- Study scope: The study analyzes MonoDepth’s high-level behavior, focusing on its depth cues, camera-pose assumptions, and obstacle recognition.The work examines vertical position versus apparent size, pose changes, and detection of unseen obstacles.
II. RELATED WORK
Prior monocular-depth research identified many pictorial cues and analyzed human perception, but had not established whether neural depth networks use those cues similarly. The paper therefore adopts black-box image perturbations to test learned behavior.
- Analysis approach: The authors treat the network as a black box, modifying images with conflicting cues and measuring correlations in the resulting depth maps.This avoids optimizing inputs with respect to a loss function.
- Pictorial cues: Single images provide pictorial cues including position, occlusion, texture density, linear perspective, apparent size, and shading.These cues can indicate depth or depth order, although their informational roles differ.
- Cue selection: For KITTI, position and apparent size are considered the most applicable cues because other cues are limited by resolution, depth range, or relevance.The paper excludes or downplays texture density, focus blur, aerial perspective, occlusion, perspective, and shading for this setting.
- Human perception: Human studies have observed both vertical position and apparent size as influences on distance perception.Vertical position depends on context such as background and horizon or eye level.
- Research gap: No prior work had investigated whether human depth-perception observations also apply to neural monocular-depth networks.The gap concerns learned cue use rather than whether analysis methods exist for CNNs generally.
III. POSITION VS. APPARENT SIZE
The study contrasts apparent size with vertical ground-contact position as cues for estimating obstacle distance. MonoDepth primarily uses vertical position, relying on assumptions about ground geometry and camera pose.
- Cue models: Apparent-size estimation uses known focal length, true object size H, and image size h to infer distance.This cue requires the obstacle’s true size to be known, which may be plausible for recurring KITTI object classes.
- Results: Figure 3 reports successful distance estimation with both cues, slight overestimation with position alone, and no distance estimation with scale alone.The position-only condition also has increased measurement variability.
- Cue models: Vertical-position estimation uses the ground-contact coordinate y and camera height Y to infer distance.It does not require knowledge of the object’s true size.
- Cue assumptions: The vertical-position method assumes flat ground and fixed or known camera pose, assumptions considered valid for KITTI.These assumptions distinguish it from apparent-size estimation.
A. Evaluation method
The evaluation isolates apparent size and vertical position by inserting cropped KITTI objects under controlled transformations. Relative depth responses are measured on localized car regions rather than entire objects.
- Evaluation design: Three test sets vary both cues together, vary vertical position alone, or vary apparent size alone.The combined condition serves as a control resembling real-world image changes.
- Image generation: Objects are cropped from KITTI scene-flow images and inserted into labeled image positions where obstacles could appear.The construction preserves object and scene-placement metadata from the source images.
- Distance manipulation: Relative distance is defined against the crop’s original scale and position as Z′/Z = 1.0, then increased in 0.1 steps up to 3.0.The transformation controls object scaling and ground-contact position.
- Coordinate conventions: Image coordinates and horizon height are measured from the image center, with horizon height assumed constant throughout the dataset.The transformed ground-contact coordinates determine the inserted object’s placement.
- Measurement: Depth is averaged over a flat front- or rear-car region near the lights to avoid vehicle length affecting the estimate.The region is indicated in the disparity maps and excludes the entire object footprint.
B. Results
MonoDepth estimates obstacle distance primarily from vertical image position rather than apparent size. Varying position preserves depth estimation, while changing apparent size alone does not change the estimated distance.
- When both position and scale vary, MonoDepth estimates obstacle depth close to the true distance.
- Varying only vertical position still permits distance estimation, but slightly overestimates distance and increases estimate variability.
- These findings indicate that MonoDepth relies primarily on vertical position rather than apparent size, implying assumptions about flat ground and camera pose.
IV. CAMERA POSE: CONSTANT OR ESTIMATED?
Using vertical image position for depth requires knowledge of camera height and pitch. MonoDepth could infer camera pose from image cues or assume a fixed pose, but fixed-pose behavior would limit transfer across vehicles.
- Vertical image position requires knowledge of the camera’s height and pitch for estimating obstacle depth.
- MonoDepth could obtain camera-pitch information from image cues such as the horizon or vanishing points, or assume a constant camera pose.
- The horizon level is estimated by fitting the road surface and evaluating the fitted line at disparity 0, corresponding to infinite distance.
- A fixed camera-pose assumption is plausible for KITTI but would prevent direct transfer to a vehicle with a different camera setup.
A. Camera pitch
MonoDepth detects changes in camera pitch but only partially incorporates them into estimated depth maps. Consequently, pitch changes disturb estimated distances toward obstacles.
- Unmodified KITTI images test whether estimated horizon levels track true horizon variation caused by camera motion and terrain.
- A regression coefficient of 0.60 shows that MonoDepth’s estimated horizon changes only partially reflect true horizon shifts.
- Cropping images at offsets between -30 and 30 pixels approximates camera-pitch changes of ±2-3 degrees and evaluates reflected horizon shifts.
- A regression coefficient of 0.71 shows that pitch changes are detected but not fully accounted for in the depth estimate.
- Changes in camera pitch affect the estimated distances toward obstacles.
B. Camera roll
Camera roll influences MonoDepth’s obstacle-distance estimates, introducing dependence on horizontal image position. The network detects roll but underestimates its effect in the resulting depth maps.
- A nonzero camera roll makes obstacle distance depend on horizontal as well as vertical image position.
- Roll angles are tested by cropping smaller tilted regions from the original images and extracting the road-surface angle from estimated depth maps.
- MonoDepth detects camera roll, but the resulting depth maps represent an angle smaller than the true roll angle.
V. OBSTACLE RECOGNITION
MonoDepth can estimate depth for arbitrary obstacles when it identifies their ground contact point, but obstacle recognition is unreliable for unfamiliar objects.
- V. OBSTACLE RECOGNITION: Objects absent from training are not reliably detected when pasted into images.The car appears as an obstacle, while other inserted objects can appear as flat road surface.
- V. OBSTACLE RECOGNITION: MonoDepth requires the obstacle’s ground contact point to estimate its distance and its outline to fill the corresponding depth-map region.The car is recognized, whereas inserted fridge and dog objects are not consistently recognized.
A. Color and Texture
MonoDepth is largely insensitive to exact obstacle color when value information is preserved, but performance drops when objects lose value structure through flat-color replacement.
- A. Color and Texture: Color changes that preserve value information leave MonoDepth’s performance roughly unchanged, with only slight error increases for grayscale and false-color images.These results suggest exact obstacle color is not a strong determinant of the depth estimate.
- A. Color and Texture: Flat-color replacements cause a noticeable performance drop, even when colors are more realistic than semantic false colors.This suggests contrast between adjacent regions or bright and dark regions may matter more than exact color.
- A. Color and Texture: Objects do not need familiar texture or shape to be detected in the tested examples.The distances of unfamiliar triangles appear to follow their lowest image point.
B. Shape and contrast
MonoDepth relies strongly on object edges and contrast, especially at the bottom, while camera-pose changes and missing ground-contact cues limit depth accuracy.
- B. Shape and contrast: MonoDepth appears to fill in objects mainly from their bottom and side edges, with both edges sometimes sufficient for near-complete detection.Removing the bottom edge instead leads to detection of the car sides as separate structures.
- B. Shape and contrast: Unfamiliar objects can be detected when a strong shadow or contrasting edge is added at their ground contact point.Adding a car-like shadow enables detection of previously undetected inserted objects, although the fridge remains vertically undersized in depth.
- B. Shape and contrast: The network primarily uses vertical object position rather than apparent size, but incomplete camera-pose correction causes pitch-related distance errors.These observations are reported as important limitations of the learned behavior.
- B. Shape and contrast: The reported findings come from one MonoDepth network trained on KITTI, so their applicability to other networks and datasets remains unknown.The authors plan to repeat the experiments across other networks and datasets.