PaperlayerView exact revision on arXiv

Source-linked AI summary

Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation

Weizhi Tao, Zengwang Jin, Xiao Wang, Hailong Huang

arXiv:2608.24366v1cs.CVcs.RO

TL;DR

Asymmetric camera and LiDAR degradation can leave useful information in unaffected cells while corrupting other regions, but existing fusion and uncertainty methods do not provide calibrated spatial reliability for planning. VG-SAF supplies that reliability through physical mask supervision, cross-branch variance distillation, and hybrid spatial-plus-cross-modal attention. On CARLA Longest6, it improves robustness across camera-only, LiDAR-only, and joint corruption regimes, while its waypoint scale signals severe degradation; limitations include complete-outage localization and simulation-to-real transfer.

  • Problem

    Existing fusion methods and downstream uncertainty diagnostics do not provide calibrated dense reliability for localized asymmetric sensor faults, where unaffected cells remain useful.

  • Method

    VG-SAF uses physical corruption masks, cross-branch dense distillation, and hybrid local-gating plus cross-modal-trust attention to make dense variance actionable before waypoint decoding.

  • Results

    VG-SAF exceeds TransFuser by 10.5, 8.5 and 13.7 driving-score points under camera, LiDAR and both-modality corruption, respectively, on Longest6.

  • Takeaways & Limitations

    Reliability-aware spatial fusion improves robustness while preserving useful evidence from surviving regions and providing interpretable attention, trust, and waypoint-uncertainty signals.

  • Takeaways & Limitations

    Complete signal loss prevents the per-cell gate from localizing reliable evidence, and all quantitative results are obtained in CARLA, limiting real-world fault-transfer evidence.

Abstract

from arXiv · show

End-to-end multimodal driving has progressed rapidly by fusing camera and LiDAR streams. Existing pipelines remain fragile under asymmetric sensor degradation, where either an entire modality or only a localized region is corrupted while other regions remain useful. The key difficulty is not simply to add an uncertainty head, but to obtain dense reliability supervision, calibrate this reliability against physical fault severity, and use it before unreliable features bias the planner. We propose Variance-Guided Spatial Attention Fusion (VG-SAF), in which dense heteroscedastic reliability estimates act as interpretable spatial gates. The framework couples three components. First, a physically grounded augmentor simulates representative camera and LiDAR failures and emits a continuous spatial mask, providing dense supervision without additional annotation. Second, modality-specific experts predict per-pixel reliability scales through cross-branch dense distillation in log space, enforcing a monotone severity-to-scale response. Third, calibrated reliability maps drive a hybrid attention mechanism that suppresses unreliable cells with a local spatial gate and arbitrates between modalities through a cross-modal trust softmax. A Laplace uncertainty head emits a systemic waypoint uncertainty scale that signals severe or combined sensor degradation, including severities outside the training ranges. On the CARLA Longest6 benchmark, VG-SAF consistently improves closed-loop robustness over the baselines across camera-only, LiDAR-only, and joint degradation regimes, as measured by driving score, route completion, and infraction score.

I. INTRODUCTION

VG-SAF addresses localized, asymmetric sensor faults by calibrating dense reliability from physical corruption masks and using it to gate fusion before planning. It improves closed-loop robustness while exposing interpretable spatial and systemic uncertainty signals.

  • Motivation: Localized asymmetric faults require suppressing corrupted cells while retaining useful evidence from unaffected cells in the same modality.Modality-level dropout either passes damaged cells or discards surviving lane, obstacle, and traffic-light evidence.
  • Contributions: Hybrid attention combines within-modality spatial suppression with cross-modal trust arbitration, allowing unreliable regions to be discounted while surviving modality content remains usable.The local gate and trust softmax are presented as alternatives to trust-everywhere cross-attention.
  • Motivation: Existing uncertainty estimates are primarily downstream diagnostics rather than calibrated dense variance fields used structurally in cross-modal fusion.This leaves the planner without a calibrated spatial reliability field.
  • Contributions: VG-SAF uses physically grounded camera and LiDAR fault injection with continuous corruption masks to provide dense reliability supervision without manual annotations.The masks calibrate predicted log-variance against physical fault severity through cross-branch dense distillation.
  • Contributions: A Laplace uncertainty head emits a systemic waypoint scale that increases under severe or combined degradation and can support downstream safety control.The signal also responds to severity extrapolation beyond the training ranges.
  • Positioning: VG-SAF is positioned as an integrated reliability-aware fusion pipeline spanning physical mask supervision, dense variance calibration, and local-plus-trust attention.The paper evaluates this design on CARLA Longest6 under camera, LiDAR, and joint degradation regimes.

III. METHODOLOGY

VG-SAF uses a two-stage pipeline in which augmentor masks supervise modality-specific uncertainty-aware experts before frozen experts feed a variance-guided fusion stage. Cross-branch distillation calibrates dense variance to corruption severity, while hybrid attention gates modalities before waypoint decoding.

  • Overview: VG-SAF trains modality-specific perception experts first, then freezes them while learning fusion components driven by their dense reliability maps.The augmentor’s continuous corruption mask is the shared physical supervision signal across both stages.
  • Augmentor and Corruption Mask: Each augmentor outputs a corrupted input, dense mask M, sample-level severity K = 100 E[M]%, and failure-mode identifier.Mask values represent untouched, partially damaged, or fully destroyed cells.
  • Augmentor and Corruption Mask: Global masks model frame-wide failures, whereas geometrically bounded masks model localized faults such as lens contamination or LiDAR angular blockage.The continuous mask supplies dense supervision for reliability prediction without additional annotation.
  • Augmentor and Corruption Mask: Camera augmentation covers multiple optical failures, and held-out routes, seeds, severities, and generic masks test robustness beyond exact training perturbations.The evaluation protocol includes perturbations not tied to a named fault mode.
  • Uncertainty-Aware Perception Experts: Mask-weighted supervision preserves full clean-branch training while reducing noisy-branch supervision as corruption severity increases.A confidence-aware consistency term also aligns noisy predictions with clean predictions only on reliable cells.
  • Uncertainty-Aware Perception Experts: Cross-branch dense distillation uses the clean branch as self-reference and the corruption mask as a dense physical severity label, avoiding manual uncertainty annotation.The noisy log-variance is targeted to increase linearly with local corruption severity; fully destroyed cells target e^α times clean variance.
  • Training Curriculum: A staggered four-phase curriculum introduces dual-branch consistency and calibration gradually to avoid conflict between prediction recovery and variance inflation.The initial phase stabilizes the backbone before calibration weights are activated.

E. Variance-Guided Spatial Attention Fusion

The fusion stage turns calibrated per-cell variance into hybrid spatial attention, combining within-modality suppression with cross-modal trust arbitration. Independent gains preserve sharp local attenuation and smooth modality-level splitting without extra sensor inputs.

  • Calibrated variance maps are normalized, pooled to fusion resolution, and combined into a local spatial gate plus cross-modal trust softmax.Normalization makes task-specific scales comparable before the two reliability mechanisms operate.
  • Worst-case max-pooling down-weights a pooled feature stack whenever one cell in its pooling window has high uncertainty.The design prefers safety-oriented worst-case aggregation over averaging for gating decisions.
  • Positive learned gains enforce monotonic behavior: higher variance can only reduce local attention or modality trust.The gain parameterization prevents unreliable regions from being rewarded by the attention mechanism.
  • The cross-modal trust scalar summarizes each modality with its variance-map lower quartile, asking whether enough reliable cells remain for trust.The 25th-percentile summary is insensitive to isolated noisy cells already handled by the local gate.
  • A trust floor of τ = 0.15 prevents complete suppression of either modality, preserving potentially useful local cues under global degradation.The floor is described as a safety guard rather than a performance trick.
  • Independent local and trust gains resolve the tension between sharp per-cell attenuation and smooth cross-modal splitting without additional parameter cost or sensor inputs.The decoupling adds negligible architectural overhead.

3) Fusion, Waypoints and Systemic Uncertainty:

The fusion output is pooled into a waypoint decoder while a separate Laplace head estimates systemic trajectory uncertainty from fused features and worst-case modality summaries. Training and evaluation preserve clean waypoint targets and test held-out routes, persistent corruptions, unseen severities, and reproducible multi-run metrics.

  • Fusion, Waypoints and Systemic Uncertainty: Global-average-pooled gated features are fused and decoded autoregressively into waypoint sequences, while a systemic head predicts a positive Laplace scale.The uncertainty head uses the fused state together with two worst-case modality summaries.
  • Fusion, Waypoints and Systemic Uncertainty: The two-dimensional Laplace negative log-likelihood lets large residuals under severe corruption increase the predicted scale instead of forcing the decoder to fit unreliable evidence.Vwp is a trajectory-level Laplace scale shared across waypoint steps and coordinates, not a variance.
  • Fusion, Waypoints and Systemic Uncertainty: Large Vwp provides a safety signal for severe, combined, or severity-extrapolation faults, while a clean-input trust regularizer prevents drift toward one modality.The trust-balance regularizer is applied only to no-fault inputs.
  • Experimental Setup: Longest6 evaluation uses 36 held-out routes across six towns, persistent route-level corruptions, and selected severities beyond training ranges.Generic uniform and elliptical masks are also used in ablations.
  • Experimental Setup: Robustness is measured by closed-loop Driving Score, Route Completion, and Infraction Score, with higher values better and inference running at 34 ms per frame.Reported results are averaged over three independent runs of the full route set per regime.

B. Closed-Loop Longest6 Comparison under Sensor Corruption

VG-SAF improves closed-loop driving under camera-only, LiDAR-only, and joint corruption by using reliability-guided attention to suppress damaged features and reweight modalities. Its trust and local attention components remain effective across corruption severity, with dense variance responses tracking modality impairment.

  • Closed-loop comparison: VG-SAF achieves the highest driving score in every corruption regime, exceeding TransFuser by 10.5, 8.5, and 13.7 points under camera, LiDAR, and both-modality corruption.The three regimes are camera, LiDAR, and both modalities corrupted.
  • Attention ablation: The attention-free baseline peaks near 1.3 m waypoint L1 error, whereas trust and full attention remain near 0.4 m and 0.3 m in the both-corrupted scenario at K ≈50%.The four attention configurations are evaluated across corruption severity and three corruption scenarios.
  • Attention ablation: The trust softmax provides most of the robustness gain under heavy corruption, while the local gate corrects residual within-modality damage.The trust-over-local gap is roughly 40% of the local loss, and the full-over-trust gap is roughly 30% of the trust loss.
  • Attention redistribution: When LiDAR is corrupted, its mean attention decreases from 0.25 at K = 0% to 0.01 at K = 90%, while RGB attention rises from 0.50 to 0.69.Camera corruption produces a mirrored but smaller redistribution: RGB decreases from 0.50 to 0.38 while LiDAR increases from 0.25 to 0.34.
  • Reliability response: Twelve of thirteen trained corruption modes produce variance ratios of at least 3.2× the clean baseline, with the largest responses for severe camera and LiDAR signal drops and LiDAR frustum occlusion.The reported peak ratios are 37.5×, 57.9×, and 37.7×, respectively.
  • Qualitative analysis: Qualitative cases show local reliability suppression preserving useful complementary information: camera faults leave LiDAR trusted, while LiDAR occlusion preserves the camera-supported trajectory.The visualizations cover a camera exposure pulse at K = 48% and LiDAR frustum occlusion at K = 59%.

V. LIMITATIONS AND FUTURE DIRECTIONS

The paper identifies calibration, total-signal-loss saturation, and simulation-to-real transfer as key boundaries, while proposing future validation and complementary safeguards.

  • Reliability-scale calibration and encoder blind spots: The LiDAR reliability head has a larger dynamic range than the camera head under matched corruption, leaving residual miscalibration between modality reliability distributions.The paper suggests post-hoc calibration, distribution-level regularization, or head-specific temperature calibration.
  • Reliability-scale calibration and encoder blind spots: Encoder-invariant faults such as color shift may produce weak reliability responses, motivating a lightweight input-space shift detector.The proposed detector is intended to complement, rather than replace, reliability gating.
  • Closed-loop evidence: In a night lowlight example, VG-SAF confines the road class to the carriageway and completes the route, whereas TransFuser exits at the curve and fails.The comparison illustrates the intended spatially selective robustness behavior.
  • Saturation under total signal loss: At K = 100%, the perception encoder leaves its variance-informative range, so the per-cell gate cannot localize reliable evidence.The systemic waypoint uncertainty scale Vwp still reflects planning error; an external sensor-health signal or fallback policy could improve complete-outage behavior.
  • Simulation-to-real transfer: All quantitative results are obtained in CARLA, and real sensor faults may differ in texture, temporal persistence, and spatial statistics.Real-vehicle fault validation remains a future step toward deployment.

S2. NETWORK ARCHITECTURE DETAILS

The architecture uses modality-specific experts, aligned reliability heads, gated feature pooling, autoregressive waypoint prediction, and a systemic Laplace uncertainty head.

  • Backbones: Both modality experts use RegNetY backbones, with 512-channel projected features and modality-specific spatial resolutions.The camera backbone uses ImageNet initialization; the LiDAR backbone is trained from scratch with a two-channel input stem.
  • Reliability heads: Each task head shares a decoder body with its reliability head, keeping task predictions and reliability outputs spatially aligned.This alignment supports cell-level reliability-aware fusion.
  • Modality experts: The camera expert produces seven-class segmentation and normalized depth outputs, while the LiDAR expert produces BEV segmentation and CenterNet detection reliability scales.The corresponding scales are V seg, V dep, V bev, and V det.
  • Fusion: The two gated feature maps are globally averaged, concatenated, and projected back to width C =512 by a fusion projection.This forms the fused representation used by the waypoint pipeline.
  • Planning and uncertainty: The waypoint head is an autoregressive GRU, while the systemic uncertainty head consumes the fused representation and scalar variance summaries.Vwp is a positive trajectory-level Laplace scale shared across all waypoint steps and coordinates.

S3. AUGMENTATION CATALOG

The augmentation catalog defines physically motivated camera and LiDAR faults, trains on online phase-dependent corruptions, and evaluates held-out routes with persistent severities and extrapolation.

  • Catalog scope: The catalog records each fault’s real-world analogue, description, spatial scope, and unified severity range for camera and LiDAR modes.Figures S1 and S2 show one realization of every mode on a validation frame.
  • Training and evaluation: Training samples corruptions online with phase-dependent probabilities, while evaluation uses independent routes, random seeds, and route-level persistent severities.Evaluation severities are not reused from training mini-batches.
  • LiDAR noise modeling: LiDAR Bernoulli noise injectors are modulated by 1−r, where r is normalized distance from the sensor origin, with peak densities capped between 1% and 3%.This approximates inverse-range ray-density falloff and prevents near-field speckles from dominating severity.
  • Uncertainty objectives: The uncertainty derivation uses Gaussian NLL for depth and uncertainty-weighted classification surrogates for segmentation and detection, with a Laplace counterpart for sparse regression.These forms connect predicted scales to task-specific residual or loss magnitudes.
  • Mask-weighted supervision: The noisy branch receives mask-weighted supervision, retaining full gradients on clean cells while reducing supervision as corruption removes reliable evidence.The ground-truth label remains valid, and CenterNet uses sum(·)/af normalization for positive object-center cells.

C. Confidence-Aware Consistency

Confidence-aware consistency preserves agreement on uncorrupted cells while calibrating noisy-branch reliability to corruption severity across camera and LiDAR augmentations.

  • Consistency masking: A mask-weighted consistency term ties noisy predictions to clean predictions only on uncorrupted pixels.Without the mask, corrupted pixels would be penalized for differing from clean counterparts.
  • Camera faults: The camera catalog contains eight trained corruption modes, including global and local failures such as night lowlight, local noise, and local occlusion.Global-mask modes include signal drop, night lowlight, motion blur, ghosting, and color shift.
  • Severity calibration: The calibration term sets the noisy branch’s log-reliability target as linear in local corruption severity.This directly links the reliability scale to the augmentor’s per-cell mask.
  • Head calibration: Head-specific scaling compensates for different task-loss baselines, using αseg = 3 for a 20× ratio at M=1 and αdet =4.6 for a matching 100× ratio.The camera and LiDAR segmentation heads use cross-entropy baselines, while CenterNet uses a focal-loss baseline.
  • LiDAR faults: The LiDAR catalog contains five trained geometric corruption modes, with signal drop, range dropout, and feature noise global and other modes local.Frustum occlusion and local speckle carry local masks.

B. Softplus Reparameterization of the Gains

The method constrains learned gains and trust weights to remain safe during optimization and fusion. A trust floor prevents excessive suppression of a modality that remains spatially recoverable.

  • Softplus Reparameterization of the Gains: Softplus reparameterization keeps learned quantities positive during transient optimizer steps.The reparameterization is applied at the start of every forward pass.
  • Softplus Reparameterization of the Gains: The gain parameterization permits linear growth for large positive γr.This preserves an expressive response while maintaining positivity.
  • Softplus Reparameterization of the Gains: The trust floor prevents over-suppression of a modality whose unreliable regions can still be spatially recovered.Setting τ=0 recovers the unclamped softmax.
  • Softplus Reparameterization of the Gains: A regularizer pulls the trust split toward its prior on samples without augmentation.The regularizer addresses underdetermined trust on no-fault frames and is zero when no clean samples occur in the minibatch.

E. Quantile-versus-Maxpool Aggregation

VG-SAF uses different variance aggregators for modality arbitration and systemic safety signaling, then trains perception and fusion components through staged curricula. The inference procedure applies local and cross-modal reliability to frozen expert features before waypoint rollout.

  • E. Quantile-versus-Maxpool Aggregation: The trust softmax uses the variance-map 25th percentile, while the systemic head uses the spatial maximum.The lower-quantile statistic supports arbitration; the maximum preserves worst-case local evidence for the safety alarm.
  • E. Quantile-versus-Maxpool Aggregation: Late perception training emphasizes camera motion blur, lowlight, and ghosting together with LiDAR frustum occlusion and range-dependent dropout.The late budget targets corner cases where calibrated reliability heads provide safety value.
  • E. Quantile-versus-Maxpool Aggregation: Mixed clean–noisy batches can collapse the variance head under BatchNorm drift, especially during severe corruptions.The reported failure includes negative pre-activation drift and variance saturation at the floor across the frame.
  • E. Quantile-versus-Maxpool Aggregation: Inference obtains frozen expert features and variance maps, computes local gains and trust weights, gates features, fuses them, and rolls out the GRU.The procedure returns waypoints, systemic waypoint uncertainty, modality trust values, variance maps, and attention maps.

C. Fusion-Stage Curriculum

Fusion training uses a four-phase curriculum that gradually introduces single-modality faults, joint faults, and hard degradation modes. The staged procedure trains navigation and systemic uncertainty behavior while calibrating trust on clean samples.

  • C. Fusion-Stage Curriculum: Phase 1 trains the GRU and systemic uncertainty head on clean inputs only.Both modality fault flags remain zero throughout this phase.
  • C. Fusion-Stage Curriculum: Phase 2 samples clean data and single-modality faults to exercise attention gains without stressing the systemic head.The sampled flag pairs are (0,0), (1,0), and (0,1).
  • C. Fusion-Stage Curriculum: Phase 3 independently samples modality faults, producing approximately one quarter double-fault samples for systemic-head training.Each fault flag has probability 0.5.
  • C. Fusion-Stage Curriculum: Phase 4 retains Bernoulli fault sampling but switches augmentation toward hard failures and calibrates Vwp for severe combined faults and out-of-range severities.This phase targets degradation beyond the training ranges.
  • C. Fusion-Stage Curriculum: The fusion stage runs after two-stage training with frozen perception experts, and its loss combines waypoint uncertainty with trust-balance regularization.Algorithm S3 defines Lfusion=Lwp+λtbLtb.
  • C. Fusion-Stage Curriculum: Training takes approximately 30 hours per perception modality and 18 hours for fusion, while inference runs at 34 ms per frame on one RTX 6000 Ada.The reported optimizer is AdamW with cosine decay.
Loading 2608.24366v1…