PaperlayerView exact revision on arXiv

Source-linked AI summary

A Degradation-Tolerance Benchmark for Camera-Only End-to-End Driving

Haohua Que, Handong Yao

arXiv:2608.29005v1cs.RO

TL;DR

Camera-only E2E driving lacks planning-level evidence about tolerance to degraded camera streams, despite deployment-relevant blur, noise, weather, frame loss, and memory faults. DriveDegrade addresses this gap with a controlled benchmark spanning sixteen corruption families, five severities, fifteen policies, and open- and closed-loop evaluation. It finds corruption-dependent planning fragility and uses separate quality-degradation and information-removal axes to distinguish camera reliance from apparent robustness.

  • Problem

    Existing corruption benchmarks measure detection or bird’s-eye-view perception rather than the planning output of camera-only E2E driving under degraded images.

  • Method

    DriveDegrade injects sixteen corruption families at five severities inside image loaders and evaluates fifteen policies on nuScenes, NAVSIM, and a CARLA closed-loop anchor.

  • Results

    Mild degradation often has little effect, while blur, JPEG, noise, raindrop, and camera loss reveal corruption-dependent breakpoints; weather and bit-error are often tolerated.

  • Takeaways & Limitations

    Separating image-quality degradation from information removal makes flat curves interpretable as possible quality insensitivity rather than blindness or robustness.

  • Takeaways & Limitations

    Open-loop nuScenes measures replayed planning error, while the closed-loop anchor covers only two agents, three towns, and few routes; corruptions are synthetic stressors.

Abstract

from arXiv · show

Camera-only end-to-end (E2E) driving models are nearing deployment, where the camera stream is degraded by blur, noise, low light, weather, frame loss, and memory faults. How much a policy tolerates before its driving breaks is unclear. Corruption-robustness benchmarks target detection or bird's-eye-view perception, not the planning output that drives the car. We present DriveDegrade, a benchmark for image-degradation tolerance in camera-only E2E driving. Sixteen corruption families at five severities are injected on the fly inside the image loader, one operator reaching fifteen policies, and we evaluate open-loop planning on nuScenes and NAVSIM plus a CARLA closed-loop anchor. First, mild degradation barely affects planning, and the families that break it have a clear threshold at mid severity. Second, fragility is corruption-dependent: blur, JPEG, and raindrop damage planning most, while weather and bit error are tolerated far into the range. Third, a flat curve is ambiguous, so we separate corruptions that degrade the image from those that remove it. A planner that reads its camera must lose accuracy when information is deleted, whatever it does under quality loss. On these two axes the planners separate sharply, quantifying the ego-status shortcut without mistaking indifference for robustness. A released vision-language-action planner is flat on both axes, and blinding all six of its cameras costs it only 11.5 percent.

1. Introduction

DriveDegrade addresses the missing planning-level evidence on how camera-only E2E policies tolerate real image degradation. It applies controlled corruptions across architectures and adds a diagnostic that distinguishes image-quality insensitivity from reliance on visual information.

  • Motivation: Camera-only E2E driving is approaching deployment, but existing corruption benchmarks measure perception rather than the trajectory that drives the vehicle.Deployment degradations include blur, noise, weather, compression, frame loss, and memory faults.
  • Approach: DriveDegrade injects sixteen corruption families at five severities into each model’s image loader, using one operator across fifteen policies.The evaluation covers open-loop planning on nuScenes and NAVSIM, plus a CARLA closed-loop anchor.
  • Contributions: The benchmark locates per-corruption planning thresholds, including four driving-specific contamination families with spatially concentrated damage.The same severity-controlled injector reaches every architecture without writing corrupted copies to disk.
  • Contributions: Separating quality degradation from information removal prevents a flat degradation curve from being mistaken for robustness when a planner may simply ignore image detail.The diagnostic is intended to quantify camera reliance rather than reward input indifference.

2. Related Work

Prior work benchmarks perception robustness or planning under selected perturbations, but does not provide DriveDegrade’s combined image-degradation, per-family, and vision-reliance analysis across modern planners. DriveDegrade therefore complements existing benchmarks by evaluating the planning output under controlled camera corruption.

  • Benchmark gap: Common corruption benchmarks report degradation in perception, not in the planned trajectory that moves the vehicle.This boundary spans image robustness and driving perception benchmarks.
  • Prior planning robustness: RobustE2E evaluates adversarial and natural corruptions on three planners, while Bench2Drive-Robust evaluates four closed-loop agents under deployment perturbations.Their perturbations include attacks, frame drops, partial observation, GPS and speed noise, and inference latency.
  • Diagnostic gap: Neither prior work separates image-quality degradation from information removal, so neither distinguishes a robust curve from an unread camera.This diagnostic gap is central to interpreting flat performance curves.
  • Contribution: DriveDegrade adds per-family knee locations, an approximate-memory fault family, and a memory-interface energy proxy while covering architectures absent from RobustE2E.It is complementary to prior planning-robustness benchmarks rather than a replacement for them.
  • Benchmark scope: Existing planning protocols measure open- or closed-loop behavior but do not systematically degrade camera images across modern nuScenes planners at per-family resolution.Table 1 contrasts evaluation level, image corruption, policy coverage, and vision-reliance diagnostics.

3. The DriveDegrade Benchmark

DriveDegrade applies reproducible, on-the-fly camera corruptions and evaluates normalized planning degradation across multiple E2E architectures. Its five-level, physically specified severity ladder and retention-based metrics support comparable tolerance curves while preserving absolute planning measures.

  • On-the-fly injection: The injector corrupts decoded images after loading and before normalization, so every model receives the same operator and severity without stored corrupted copies.The implementation uses numpy/cv2 and can serve multiple model environments and simulators.
  • Corruption taxonomy: The taxonomy combines four inherited corruption groups with vehicle-specific occlusion, mud splash, raindrop, and lens dirt.The driving-specific families remove information spatially, leaving some pixels pristine while opaque deposits cannot be recovered by global denoising.
  • Severity specification: Each corruption family uses a physical parameter across S1–S5, with endpoints fixed from routine deployed conditions to a human-unusable view.Snow and blackout instead use ordinal parameters, and blackout ramps over camera identity.
  • Evaluated planners: Six camera-input planners spanning full-stack, vectorized, sparse, generative, diffusion, and latent-world-model architectures are evaluated on 6019 nuScenes frames from six cameras.LAW consumes surround images despite being trained without perception supervision.
  • Metrics: Retention normalizes degraded trajectory L2 by each planner’s clean run, while absolute L2, collisions, comfort, and energy are also reported where interpretable.For reward-valued metrics, the reciprocal is used so higher retention is always better.
  • Metrics: Tolerance thresholds s∗_10 and s∗_25 mark the first severity where retention falls below 0.90 and 0.75, respectively.Open-loop metrics are deterministic population means over fixed validation splits, with bootstrap confidence intervals below 0.01 in retention.

4. Results

DriveDegrade shows that degradation tolerance varies sharply by corruption and architecture, while flat quality curves can reflect insensitivity rather than camera blindness. Separating image-quality loss from information removal reveals how planners actually rely on visual input.

  • Corruption-dependent tolerance: Mild degradation barely affects UniAD, but blur and noise reach clear knees around s*₁₀ ≈ 2.2–2.5, whereas weather and bit-error remain tolerated.Camera blackout is reported separately because it removes whole cameras rather than applying a gradual severity.
  • Cross-model reliance: At S5, full-stack and diffusion planners are most camera-reliant, sparse and generative planners are intermediate, and vectorized and latent-world-model planners move least.Under blur, DiffusionDrive and UniAD retain only 0.42 and 0.52 of clean L2, respectively.
  • Quality versus removal: Separating quality degradation from information removal distinguishes tolerance to poor pixels from reliance on visual evidence.A camera-reading planner should lose accuracy when information is removed, regardless of its response to quality loss.
  • Quality versus removal: VAD and LAW retain 0.97 and 1.00 under quality degradation but lose 7.7% and 9.7% when the front camera is removed.Their flat quality curves therefore indicate insensitivity to image detail, not blindness to the camera.

4.3. Absolute performance and rank reversal

Absolute performance and retention tell different stories: degradation can reverse clean-data rankings, while high absolute accuracy alone does not establish meaningful camera use. The visual and two-axis diagnostics expose these distinctions.

  • Rank reversal: GenAD beats VAD clean at 0.908 against 0.969 m but loses under gaussian noise S5 at 1.133 against 0.871 m.This rank reversal shows why clean-data performance does not determine the best planner under degradation.
  • Retention curves: Mean retention curves average twelve core corruptions, with steeper declines for full-stack and diffusion planners and flat curves for vectorized and latent-world-model planners.OpenDriveVLA has full ladders for four families, while the ego-only MLP is 1.000 everywhere by construction.
  • Two-axis diagnosis: Table 3 separates quality degradation from removal, preventing quality-immune VAD and LAW from being mistaken for camera-ignoring planners.None of the six core planners is flat on both axes, although the released VLA nearly is.
  • Qualitative trajectories: Trajectory plots show UniAD and SparseDrive swinging under blur, noise, and JPEG while VAD remains almost straight; LAW is omitted because only aggregate outputs were stored.The comparison uses clean and corrupted trajectories across six disruptive families at S5.
  • Absolute performance: LAW achieves 0.524 m clean and never exceeds 0.581 m, despite being trained without perception supervision.An ego-status-only MLP reaches 0.731 m with no image, showing that absolute open-loop L2 alone cannot certify camera use.

4.4. Secondary metrics and checks

Secondary metrics reinforce the camera-reliance ordering for path stability, comfort, and vehicle collision, while exposing limits of rare-event and energy-proxy measurements.

  • Planning and safety metrics: Under gaussian blur S5, UniAD’s 3 s endpoint shifts 11.4 m, DiffusionDrive’s 6.7 m, and VAD’s 2.1 m.Comfort and vehicle collision follow the same ordering: UniAD jerk rises from 1.92 to 2.69 m/s3 and 3 s collision rate from 0.65% to 1.14%.
  • Metric checks: VRU collision remains 0.02–0.25% and flat across severity because replayed-scene drift does not preferentially strike VRUs.This makes VRU collision a rare-event metric in the reported setting.
  • Energy proxy: The memory-interface energy proxy estimates image-byte movement and storage rather than total system power, whose dominant network-computation term degradation leaves unchanged.Within that scope, low light and color quantization reduce write energy while retaining 0.88 and 0.72.
  • Reproducibility check: Three fresh seeds shift retention by 0.004 and 0.002 on the two most stochastic cells, versus per-family effects of 0.1–0.5.The reported stochastic-family trends are therefore much larger than the observed seed variation.

4.5. Driving-specific contamination

Driving-specific contaminations expose failures that global image corruptions miss because they damage only parts of the frame. Raindrop is especially damaging, while occlusion shows that planners integrate evidence across the full image.

  • Driving-specific contamination: Spatially concentrated contamination families test failure modes that generic global corruptions cannot represent.Their damage leaves part of the frame pristine while removing information elsewhere.
  • Driving-specific contamination: GenAD crosses only four generic-family thresholds but breaks barely past the mildest raindrop setting.Local refractive distortion therefore affects GenAD more than global degradation.
  • Driving-specific contamination: Occlusion reaches the same band as gaussian blur at S5 despite leaving half of each frame pristine.This indicates that planners integrate evidence across the full frame rather than relying only on an occlusion-free region.
  • Driving-specific contamination: NAVSIM’s PDMS conclusions hold beyond open-loop imitation L2, with blur and front-camera blackout lowering TransFuser’s safety-related scores.TransFuser’s milder retention reflects untouched LiDAR, not immunity of its camera stream.

4.7. Four NAVSIM planners on the same slice

Across four NAVSIM planners, quality degradation is mild, whereas deleting the front camera causes large losses. Clean-score reproduction and rear-camera blackout provide null controls for the comparison.

  • Four NAVSIM planners on the same slice: Blackout S3 leaves all four planners’ scores bit-for-bit unchanged because it darkens rear cameras that none reads.The match extends to fourteen decimal places.
  • Four NAVSIM planners on the same slice: Published clean results are reproduced before corrupted cells are evaluated, including ZTRS 45.45 versus 45.5 and RAP 93.80 versus 93.8.This validates the separately patched evaluation setups.
  • Four NAVSIM planners on the same slice: Quality-axis retention is mild for all four planners and almost absent for RAP, ranging from 0.992–1.001 on a DINOv3 backbone.Deleting the front camera instead costs RAP half its score and GTRS two thirds.
  • Four NAVSIM planners on the same slice: OpenDriveVLA-0.5B remains nearly flat across a forty-eight-cell sweep, with retention in [0.909, 0.997].Its clean L2 reproduces the published 0.35 m, while front-camera blackout costs 3.6%.
  • Four NAVSIM planners on the same slice: Blackout S5 costs every NAVSIM planner between a quarter and two thirds of its score.The planners use different protocols, so absolute scores are not pooled or ranked.
  • Four NAVSIM planners on the same slice: Blinding all six OpenDriveVLA-0.5B views leaves retention 0.885, an 11.5% loss, whereas UniAD loses 32% from the front camera alone.The result supports the paper’s ego-status-shortcut interpretation.

4.9. Closed-loop anchor in CARLA

The CARLA anchor confirms that severe camera removal can break closed-loop driving even when image-quality conditions remain strong. Closed-loop contamination also changes the relative ordering and distribution of losses across families.

  • Closed-loop anchor in CARLA: Front-camera blackout at S5 reduces SimLingo’s mean Driving Score on Town10 from 100 to 19.4.The corresponding image-quality conditions remain above 81.
  • Closed-loop anchor in CARLA: Closed-loop evaluation uses completed scored runs, with the CARLA table covering three towns and a second agent.The table also reports Town10 corruptions and Town03 contamination families.
  • Closed-loop anchor in CARLA: On Town03, lens dirt remains mildest, while occlusion, mud splash, and raindrop converge at S5 within 1.3 Driving Score points.Open-loop separates these families by more than a severity step, so closed-loop loss distributions differ even when endpoints converge.

5. Evaluation Scope and Limitations

DriveDegrade supports controlled comparisons of camera degradation and vision reliance, but its results do not directly establish deployment safety. Open-loop replay, limited simulator coverage, synthetic stressors, and proxy energy measurements constrain interpretation.

  • Evaluation scope: DriveDegrade measures tolerance thresholds and relative vision reliance rather than deployment safety.Its controlled comparisons fix the model, split, evaluator, and operator while degrading only the camera stream.
  • Evaluation scope: A flat quality-degradation curve is ambiguous because open-loop nuScenes can reward ego-status shortcuts.The quality-versus-removal split and an ego-status-only blind-floor regressor distinguish robustness from camera indifference.
  • Limitations: Open-loop nuScenes measures replayed planning error, so collision and comfort do not capture feedback compounding.This limits what the open-loop metrics can say about closed-loop behavior.
  • Limitations: The CARLA anchor covers two agents and three towns in one simulator with few routes.Identification bounds protect blackout separation from unscored outcomes but do not support fine-grained ranking of image-quality conditions.
  • Limitations: The corruptions are controlled synthetic stressors rather than a complete model of real sensor physics.The energy figure is an analytical proxy, not measured watts.

6. Conclusion

DriveDegrade measures camera degradation tolerance at the planning level, locating corruption-specific breakpoints and exposing vision reliance that clean-set scores can hide.

  • DriveDegrade evaluates camera degradation across sixteen corruption families, identifying clear breakpoints for blur, JPEG, noise, and camera loss while weather and bit-error are often tolerated.
  • The same degradation curves reveal vision reliance that clean-set performance can hide, while a CARLA anchor confirms camera loss remains load-bearing under feedback.
  • The released artifact makes results traceable to checkpoints, splits, corruptions, severities, and seeds, with tables and figures regenerable on CPU.
Loading 2608.29005v1…