PaperlayerView exact revision on arXiv

Source-linked AI summary

Illusion of Depth: Revealing Hidden Stereo Vision Vulnerabilities in Depth Estimation

Sri Hrushikesh Varma Bhupathiraju, Tetsu Ishizue, Nicholas U. Costagliola, Ozora Sako, Kentaro Yoshioka, Takeshi Sugawara, Sara Rampazzi

arXiv:2609.16336v1cs.CR

TL;DR

The study asks what causes repeated-element depth errors in classical and deep-learning stereo estimation and whether an adversary can induce consistent, controlled manipulation. The paper analyzes sampling artifacts and calibration errors, uses simple repeated patterns to manipulate estimated depth, and proposes detecting and suppressing shifts through local matching scores. BM depth shifts reach 19m closer or 5m farther, and real-world errors lasting at least 0.5 seconds can trigger emergency braking.

  • Problem

    The study asks what causes repeated-element depth errors in classical and deep-learning stereo estimation and whether an adversary can induce consistent, controlled manipulation.

    Source: §1 · p. 2

  • Method

    The paper analyzes sampling artifacts and calibration errors, uses simple repeated patterns to manipulate estimated depth, and proposes detecting and suppressing shifts through local matching scores.

    Source: §1 · p. 2

  • Results

    BM depth shifts reach 19m closer or 5m farther, and real-world errors lasting at least 0.5 seconds can trigger emergency braking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Simple repeating patterns let adversaries systematically bias local matching scores and controllably manipulate depth estimates in both classical and deep-learning SDE models.

    Source: §1 · p. 2

    The threat model assumes oracle access to the victim’s depth-estimation algorithm and focuses on simple repeated shapes projected over flat or semi-flat surfaces.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Stereo cameras are integrated into autonomous systems such as self-driving cars, drones, and robots to offer precise depth estimation in a cost-effective manner compared to LiDAR technology. In this work, we reveal an intrinsic vulnerability in stereo cameras that stems from their pixel sampling and calibration processes, which can influence the outputs of stereo matching algorithms. Attackers can achieve fine-grained control over the estimated depth of real obstacles using simple repeating patterns, without relying on sophisticated adversarial machine learning techniques. Furthermore, deep learning-based depth estimation models exhibit a similar vulnerability. We evaluate the impact of this attack on two widely used stereo matching algorithms (BM and SGBM), three deep learning models (PSMNet, MoCha-Stereo, and UniMatch), a stereo-LiDAR fusion model (SGM-DDC), and two popular commercial stereo cameras, the ZED2 and Intel RealSense D435. For example, in the ZED2 camera, an attacker can displace obstacles up to 20~meters farther or 12~meters closer. In our real-world evaluation in a driving setting, a brief 0.5~second attack can trigger emergency braking in a popular autonomous driving framework. We further demonstrate the feasibility at driving speeds up to 40~km/h using CARLA. Finally, we confirm the ineffectiveness of state-of-the-art defenses, and we propose a novel strategy that leverages similarity scores to dynamically detect and suppress the depth discrepancies. Our work highlights vulnerabilities hidden in stereo matching and deep learning depth estimation models, addressing critical limitations in autonomous system deployments.

1 Introduction

The introduction identifies sampling artifacts and calibration errors as hidden sources of stereo-depth vulnerability, exploitable with simple repeated patterns. It evaluates attacks across stereo algorithms and cameras, and presents a defense based on local matching scores.

1 Introduction

  • Prior repeated-pattern attacks induced depth errors but did not explain their cause or control the errors’ magnitude and direction, motivating the study’s research questions.The questions ask what causes these errors in block-matching and deep-learning models and whether an adversary can induce consistent, controlled manipulation.Source: §1 · p. 2
  • Sampling artifacts and calibration errors let simple repeated patterns bias local matching scores and control depth estimates in classical and deep-learning models.The deep-learning models learn and replicate the same vulnerability; the projection attack does not require sophisticated adversarial optimization or tracking.Source: §1 · p. 2
  • BM can be shifted up to 19m closer or 5m farther, while attacks also affect deep-learning models, stereo-LiDAR fusion, and commercial cameras in driving scenarios.The introduction also reports feasibility up to 40 km/h in CARLA and errors lasting at least 0.5 seconds, sufficient to trigger emergency braking or unsafe maneuvers.Source: §1 · p. 2
  • The proposed defense detects repeated patterns by analyzing local-region matching scores and suppresses resulting depth shifts, with reported success rates of 96.5% for classical and 100% for deep-learning SDE algorithms.The introduction contrasts this approach with prior suggestions of adversarial training and impractical manual parameter tuning.Source: §1 · p. 2

2 Background and Previous Work

Stereo depth estimation derives depth from disparity between two horizontally aligned camera views. Prior attacks used optimized adversarial examples or sensor glare, while this paper’s background describes limits in their control and real-world applicability.

2.1 Depth Estimation in Stereo Cameras

  • Stereo depth estimation converts horizontal disparity between corresponding regions in two camera images into depth; block matching uses local pixel similarity, while learned methods use semantic features and representations.The paper distinguishes block matching for resource-constrained devices from deep-learning methods that offer higher accuracy at greater computational cost.Source: §2.1 · p. 2
  • In classical block matching, the algorithm searches horizontally along an epipolar line for the most similar right-image block, and its displacement determines disparity.BM and SGBM are used in commercial cameras; SGBM aggregates matching costs along multiple 1D paths, while deep models infer disparity from stereo pairs using learned features and cost volumes.Source: §2.1 · p. 3

2.2 Attacks against Stereo Depth Estimation

  • Earlier adversarial-pattern attacks increased depth errors but were model-specific and did not control their magnitude, direction, or consistency over time and movement.The cited work describes optimized printed patches and notes that prior light-injection attacks produced glare without control over its shape or the resulting obstacle.Source: §2.2 · p. 3

3 Adversary Model and Vulnerability

The section describes how sampling artifacts and calibration errors in stereo cameras let repeated patterns manipulate depth estimates. Experiments show the effects in block matching and PSMNet, including under real-world conditions.

3.1 Threat Model

  • The threat model assumes a remote attacker who knows the camera, SDE algorithm, parameters, and calibration, and has oracle access to depth estimates.The attacker can briefly project simple repeated shapes onto flat or semi-flat surfaces to manipulate perceived distance without hardware access or scene-specific optimization.Source: §3.1 · pp. 3–4

3.2 Vulnerability in Block Matching Algorithms

  • The vulnerability arises from pixel-level sampling artifacts and calibration errors, which alter local image features and matching scores in stereo cameras.Sampling averages light across pixel regions, creating intermediate intensities at high-contrast boundaries; calibration distortions warp corresponding image features.Source: §3.2.1 · p. 4, §3.2 · p. 4, §3.2.2 · p. 5

3.2.1 Sampling Artifacts.

  • Repeated high-contrast patterns create periodic SAD minima, so BM can select a repeated block with a matching artifact rather than the true geometric match.Changing pattern spacing can shift the minima and systematically bias estimated depth; a ZED2 real-world test confirmed that sampling artifacts can control disparity.Source: §3.2.1 · p. 5

3.2.2 Calibration Error.

  • At 3 pixels of calibration error, BM’s simulated mean disparity rises from 9.9 to 22.8, then plateaus beyond error magnitude 12; real-world values reach 63.The simulation plateau follows the algorithm’s disparity-range cap of 64; the real-world increase is attributed to pixel imperfections further amplifying the effect.Source: §3.2.2 · p. 6

3.3 Vulnerability in Deep Learning Models

  • The section evaluates PSMNet on 100 KITTI stereo pairs with synthesized patterns placed 10 m in front of the camera, testing whether deep models share the vulnerability.The patterns are luminance-matched to each image, and the evaluation applies the sampling-artifact and calibration-error scenarios.Source: §3.3 · p. 6

3.3.1 Sampling Artifacts.

  • In PSMNet, shifting sampling artifacts farther from the ground-truth position increases average maximum predicted disparity across 100 KITTI images.The learned feature correspondences are influenced by artifact location and magnitude, and similar-magnitude artifacts can produce incorrect depth.Source: §3.3.1 · p. 6

3.3.2 Calibration Error.

  • PSMNet’s estimated disparity increases with calibration-error magnitude, but calibration’s effect on the chessboard pattern and disparity is reduced above 40 mm block width.This extends the calibration-error vulnerability observed in BM to PSMNet.Source: §3.3.2 · p. 7

4 Attack Characterization

The section characterizes how simple repeated patterns exploit sampling artifacts and calibration errors to control stereo disparity and depth. Pattern granularity, physical size, camera distance, and contrast determine the attack’s effects and limits.

4 Attack Characterization

  • The attack uses a repeated pattern on a flat or semi-flat surface, with pattern design, matching parameters, camera distance, size, and contrast as adjustable controls.The characterization uses Gazebo simulations and controlled indoor projections, with BM and the SAD score as its reference.Source: §4 · p. 7

4.1 Pattern Design

  • At pattern granularities up to 6 pixels, calibration error can dominate sampling artifacts and drive BM disparity to its 64-pixel range limit; at higher granularities, selected ranges support controlled disparity increases.The reported controllable ranges are 14–23 pixels and 23–50 pixels in PG; BM’s result is unaffected by PRS because it matches blocks horizontally.Source: §4.1 · p. 8
  • Larger BM blocks require greater pattern granularity to induce disparity errors, while a wider disparity range raises the maximum error without changing granularity’s influence.The block size shifts the threshold for manipulation; the disparity range caps the cyclic error amplitude.Source: §4.1 · p. 8, Figure 11 · p. 8

4.2.1 Distance from the Stereo Camera.

  • As pattern distance increases, its projected pixel size and BM disparity error decrease because fewer pixels provide potential matching candidates.The trend was observed in simulation and real-world tests, and the passage reports that it also persists for RealSense D435.Source: §4.2.1 · p. 8

4.2.2 Physical Size of the Pattern.

  • Pattern width increases disparity error approximately linearly from 44 pixels at W=0.5m until the disparity range is reached, after which the error saturates.The pattern can be scaled with attack distance to preserve its projected region and induce the same depth error.Source: §4.2.2 · p. 9

4.2.3 Pattern Contrast.

  • Disparity error persists through an 80% contrast reduction in simulation and real-world tests, but falls away when the sampling artifact becomes too small.The authors state that an attacker can analytically derive pattern design for a target depth, distance, size, and contrast without optimizing for a specific scene.Source: §4.2.3 · p. 9

5 Evaluation on Driving Dataset

The authors evaluate repeated-pattern attacks on classical stereo matching, learned depth models, and stereo-LiDAR fusion using KITTI driving data. The attacks control estimated depth across these model types, although fusion constrains the shift toward the LiDAR prior.

5 Evaluation on Driving Dataset

  • The evaluation covers BM, SGBM, PSMNet, MoCha-Stereo, UniMatch, and SGM-DDC using synthesized patterns over KITTI stereo pairs.The setup overlays patterns on all 7,518 pairs and tests 627 patterns across seven true disparities, for 4,389 combinations per pair; attack success requires a 10-pixel region with the desired disparity.Source: §5 · p. 9, §5.1 · p. 9

5.2 Stereo Matching Algorithms

  • BM permits single-pixel disparity control over a 1–13m fake-depth range, while SGBM’s disparity rises with both PG and PRS and permits control from 11.5m to 2.5m.The SGBM multi-direction cost aggregation suppresses outliers and reduces sensitivity to lower PG, but makes it susceptible to higher PG.Source: §5.2 · p. 9
  • Under simulated motion blur corresponding to 40 km/h, controlled depth manipulation persists, reaching up to 12m for BM and 9m for SGBM.The reported BM control range is PG 23–63 pixels; the authors attribute SGBM’s smaller maximum error to multi-path aggregation.Source: §5.2 · p. 10

5.3 Deep Learning SDE algorithms

  • PSMNet shifts estimated depth by up to 59m toward the camera, MoCha-Stereo by up to 52m, and UniMatch by up to 60.5m from a pattern at 62m.The learned models also permit fine-grained control; reported disparity steps are about 1 pixel for PSMNet, 0.9–1.55 pixels for MoCha-Stereo, and 0.5–1.2 pixels for UniMatch, with an outlier at true disparity 10.Source: §5.3 · p. 10

5.4 SDE Stereo - LiDAR Fusion Model

  • SGM-DDC shifts a 62m target to 16m, but its LiDAR prior constrains the attack and makes the fusion model more resilient than non-fusion models.At distances within 31m, the reported possible shift is 4.4m with a more restricted control range.Source: §5.4 · p. 10

6 Real World Evaluation

Real-world tests with ZED2 and RealSense D435 cameras examine projected repeated patterns indoors, outdoors, and on a van obstacle. The cameras produce false depth across varied distances and lighting, including shifts lasting long enough to persist during a driving scenario.

6 Real World Evaluation

  • The real-world evaluation tests ZED2 and RealSense with projected patterns on indoor surfaces and outdoor obstacles, assessing whether false depth is detected as an obstacle.The setup uses Autoware Euclidean clustering for obstacle detection.Source: §6 · pp. 10–11

6.1 Static Indoor Scenario

  • Indoors at 3m, the attack shifted depth by up to 2.4m on RealSense and 2.2m on ZED2, while pattern granularity enabled camera-specific control.RealSense depth was adjustable in 0.1m increments from 0.4–2.6m; ZED2 showed coarser changes and shifts farther by up to 4m at lower PG values.Source: §6.1 · p. 11

6.1.1 Controllability at Increasing Distances.

  • At pattern distances up to 15m for ZED2 and 16m for RealSense, induced false depth could place a surface as close as 2m when the pattern was 10m away.The depth trend initially increases exponentially, then approaches linear saturation beyond 8m for ZED2 and 12m for RealSense.Source: §6.1.1 · p. 11, Figure 13 · p. 11

6.1.2 Scene Illumination.

  • Indoor, both cameras produced false depth across illumination levels from 0 to 1,000 lux, remaining at 0.8m for ZED2 and 0.6m for RealSense.The illumination tests were repeated 10 times at each level.Source: §6.1.2 · pp. 11–12

6.2 Outdoor Evaluation

  • At night, patterns induced false depth out to 16m for ZED2 and 18m for RealSense; in daytime, they produced obstacle detections at 2.5m depth from an 8m pattern distance for ZED2 and 1.8m from 10m for RealSense.The passage attributes the reduced daytime range to higher illumination diminishing pattern brightness, contrast, and sampling artifacts.Source: §6.2 · p. 12

6.2.1 Driving Scenario.

  • A van outside the vehicle’s trajectory produced false obstacles lasting 1.06 seconds by day and 1.23 seconds by night for RealSense, and 1 second by day and 0.7 seconds by night for ZED2.The false depth shifted the van into the victim’s path along the camera frustum.Source: §6.2.1 · p. 12, Figure 14 · p. 12

7 High Speed Evaluation

A CARLA evaluation tests pattern-based depth manipulation as a vehicle passes a proxy billboard at driving speeds. The attack remains consistent for more than the stated 0.5-second response threshold, including at 40 km/h.

7 High Speed Evaluation

  • At 10, 20, 30, and 40 km/h, consistent depth manipulation lasts 5.3, 2.6, 2.3, and 2.1 seconds, respectively.Each duration exceeds the 0.5-second chattering threshold described as the minimum time required for an autonomous driving framework such as Autoware to react.Source: §7 · pp. 12–13

8 Defenses

Manual parameter tuning is impractical in dynamic settings and addresses only one pattern granularity. The proposed defense detects repeated structures from matching-score behavior and reduces depth errors.

8 Defenses

  • Continuous parameter adaptation is impractical in dynamic environments, and vendor tuning addresses only a single pattern granularity.An attacker can vary patterns across multiple granularities.Source: §8 · p. 13
  • 13% of evaluated stereo image regions were successfully mitigated by the FFT-based detector, which struggled with non-uniform real-world patterns.The method’s assumptions require near-perfectly repeated structures.Source: §8 · p. 13

8.1 Proposed Defense

  • The defense detects repeated patterns through periodic matching-score peaks, then uses regular, consistent oscillations to mark attacked blocks and estimate disparity across a bounding box.It normalizes block-matching SAD scores and identifies minima induced by repeated structures.Source: §8.1 · p. 13
  • 96.5% success rate in mitigating the attack, with post-mitigation depth error below 0.5m across 200 real-world stereo image pairs.Mitigation success was 98% at night and 94.5% during the day.Source: §8.1 · p. 13
  • 100% success rate in suppressing depth estimation error to below 0.1m on 200 KITTI image pairs using PSMNet cost-volume features.The defense also mitigates natural repeated patterns that can unintentionally trigger the same effects.Source: §8.1 · p. 13

9 Discussion

The discussion describes how training data, illumination, and surface properties affect attack patterns and their projection. It also bounds the real-world experiments to particular patterns, surfaces, cameras, and models.

9 Discussion

  • With SceneFlow rather than KITTI pretraining, UniMatch pattern-region maximum disparity falls by over 50 pixels at 60 m but rises by over 50 pixels at 10 m.The effective patterns also differ, showing that dataset, architecture, and training methodology are among the factors involved.Source: §9 · p. 13
  • Projected brightness can be adjusted for ambient illumination and surface reflectance; daytime outdoor evaluation used 1000–1500 lux and a conventional projector at 4 meters.The paper gives Bp = B + k · I for this adjustment.Source: §9 · p. 13
  • Projection patterns can be contorted for curved surfaces, while stealth approaches include naturalistic textures and infrared projection.The paper verifies infrared projection against the RealSense D435.Source: §9 · p. 14
  • Experiments use a single attack pattern on ZED2 and RealSense cameras and flat or semi-flat surfaces; adaptive patterns and arbitrary obstacle shapes are not evaluated.The analysis covers four state-of-the-art stereo matching algorithms and deep learning models, with potential extension to related architectures and cameras.Source: §9 · p. 14

10 Related Works

Related work covers physical sensor spoofing, attacks in the ADAS domain, and adversarial perturbations of deep neural networks. This work focuses on depth ambiguity from repeated elements in stereo-camera scenes.

10 Related Works

  • Prior physical spoofing work includes camera laser injection [37] [44] [50] and electromagnetic interference [37], with similar concerns for LiDARs and radars [13] [14] [31] [34].Source: §10 · p. 14
  • ADAS examples include thermal-source attacks causing downstream object-detection failures, repeated-pattern attacks on monocular SLAM, and detectors that ignore obstacle depth.This work instead considers stereo-camera depth estimation and ambiguity caused by repeated scene elements.Source: §10 · p. 14
  • Prior work shows deep neural networks are susceptible to adversarial perturbations, including patch attacks [12]; this work reports controlled depth manipulation without adversarial optimization.Source: §10 · p. 14

11 Conclusion

The conclusion identifies structured repeated patterns as a means of inducing depth errors in stereo cameras and deep learning SDE algorithms. It reports evaluation in simulated and controlled real-world driving scenarios and a defense that detects repeated elements.

11 Conclusion

  • Structured repeated patterns can induce depth-estimation errors in stereo cameras and affect deep learning SDE algorithms without adversarial examples.Source: §11 · p. 14
  • The evaluation shows adversarial control of classical block-matching algorithms and machine-learning models, including two widely used stereo cameras.Source: §11 · p. 14
  • High-speed simulated and controlled real-world scenarios are used to assess practicality in driving conditions, and the proposed defense detects repeated elements to mitigate depth estimation errors.Source: §11 · p. 14

12 Ethical Considerations

The study used public datasets and software and conducted indoor and outdoor attack evaluations in controlled settings. It reports operating within institutional safety policies and speed limits, disclosing findings to vendors, and involving no human studies.

12 Ethical Considerations

  • Experiments used publicly available datasets and software to support transparency and reproducibility.Source: §12 · p. 14
  • Indoor and outdoor evaluations used controlled environments, with vehicles operating at up to 15 km/h under institutional safety policies and speed limits.Source: §12 · p. 14
  • The authors disclosed findings to ZED and RealSense and were awaiting their responses.Source: §12 · p. 14
  • The research involved no human studies.Source: §12 · p. 14

A Influence of Sampling Artifact on SAD Score

The section explains how sampling artifacts can bias stereo matching toward false disparities, and reports related behavior in deep-learning models. It also describes edge effects, a motion-blur simulation, and the release of research artifacts.

A Influence of Sampling Artifact on SAD Score

  • Sampling artifacts can dominate BM’s SAD minima, favoring a false match with lower cost and shifting the estimated disparity.Because repeated elements create multiple candidate matches, periodic artifact magnitudes can bias which candidate minimizes SAD and enable pattern-based depth manipulation.Source: A Influence of Sampling Artifact on… · pp. 16–17

B Sampling Artifact Consistency in PSMNet

  • PSMNet consistently matches artifacts with higher normalized pixel similarity, extending the sampling-artifact vulnerability observed in classical block matching.The study measures maximum disparity in 0.2 increments across normalized similarity values.Source: B Sampling Artifact Consistency in… · p. 17

C Global Properties of Deep-learning Models.

  • Convolutional aggregation can blend attack-pattern and foreground features at pattern boundaries, causing incorrect depth estimates for edge objects.These edge-induced errors are excluded from the evaluation, although adversaries could exploit them to extend the affected scene region.Source: C Global Properties of Deep-learning… · p. 17

D Motion Blur Simulation

  • The motion-blur simulation models forward camera motion by integrating changing images over exposure, with radial displacement varying by image position and scene depth.The section also reports that average maximum disparity increases with higher normalized pixel similarity.Source: D Motion Blur Simulation · pp. 17–18, Figure 15 · p. 17

E Open Science Policy

  • The authors provide scripts for attack synthesis, model evaluation, and defense testing, along with point clouds from real-world experiments.Research artifacts and demo recordings are made available through the project website and Zenodo.Source: E Open Science Policy · p. 18
Loading 2609.16336v1…