PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating Adversarial Robustness of Heterogeneous Cooperative Perception

Chenyi Wang, Yutong Liu, Qingzhao Zhang, Ming F. Li

arXiv:2609.17856v1cs.CVcs.CRcs.MAcs.RO

TL;DR

Heterogeneous cooperative perception is thought to gain protection from translation modules, but the evidence here shows tuned attacks can bypass that apparent robustness. The paper introduces a label-free, single-pass attack generator and a lightweight trust-layer defense; HetShield recovers 83–95% of clean accuracy against HetPoison.

  • Problem

    Heterogeneous translation modules are widely assumed to blunt feature attacks, while prior cooperative-perception attacks largely target homogeneous systems with victim-detector access.

    Source: §1 · p. 1

  • Method

    The paper evaluates tuned attacks with a matched-objective harness, introduces the label-free single-pass HetPoison generator, and proposes HetShield, a spatiotemporal trust layer.

    Source: §1 · p. 1, §3.4 · p. 4

  • Results

    HetShield recovers 83–95% of clean accuracy against HetPoison, which matches or exceeds expensive per-frame optimization-based attacks.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The paper identifies discrete codebooks as the only evaluated architectural choice that actively resists adaptive attacks, while describing diffusion robustness as an illusion.

    Source: Does heterogeneity defend, and why was… · p. 7

    A fully adaptive white-box attacker with hundreds of iterations can optimize through HetShield, which does not resolve the underlying theoretical vulnerability.

    Source: Does heterogeneity defend, and why was… · p. 7

Abstract

from arXiv · show

Heterogeneous cooperative perception (CP) enables connected vehicles with diverse sensor setups to share spatial awareness via compact feature maps, where receivers reconcile these maps using learned translation modules for fusion and inference. Prior attacks against CP in a homogeneous setting reveal that the data exchange introduces a critical attack surface: a single malicious agent can transmit crafted features that erase real objects from a neighbor's fused scene. Yet, it is widely hypothesized that heterogeneity naturally defends against these attacks, as the attacker lacks knowledge of the victim's detector and the translation module scrambles adversarial gradients. We demonstrate that this protection is largely an illusion. Using a matched-objective harness to standardize the perturbation budget, objective, and forward path, we show that properly tuned iterative attacks close or reverse the apparent robustness gap. However, these optimization-based attacks require ground-truth labels and iterative backpropagation, meaning they do not represent a practical field threat running in real-time. To bridge this gap, we introduce HetPoison, a learned generator that crafts a removal perturbation in a single, label-free forward pass. HetPoison transfers across major heterogeneous designs without requiring access to the victim's detector, matching or exceeding the effectiveness of expensive optimizer-based attacks. Since heterogeneity itself is not a defense, we propose HetShield, a lightweight trust layer that validates the spatiotemporal consistency across features, recovering 83--95% of the accuracy degraded by attacks, outperforming prior art.

1 Introduction

The paper questions whether heterogeneous translation modules genuinely protect cooperative perception from feature attacks, then tests adaptive attacks and introduces a deployable generator and trust-layer defense. Its findings challenge the assumed robustness while presenting HetShield as a lightweight mitigation.

1 Introduction

  • A single malicious agent can transmit crafted features that make neighbors ignore a real object even when other agents report it correctly.Cooperative perception shares compact feature maps for receiver-side inference.Source: §1 · p. 1
  • Under standard sign-PGD, heterogeneous models appear less vulnerable than homogeneous V2VAM, supporting the perceived robustness gap that adaptive attacks later challenge.The reported AP values are V2VAM 0.608, HEAL 0.740, and CodeFilling 0.795.Source: §1 · p. 1
  • A matched-objective harness shows tuned iterative attacks close or reverse the apparent robustness gap across most designs; diffusion falls to near-zero accuracy, while only a discrete codebook survives strong adaptive attacks.The harness fixes the perturbation budget, objective, and forward path to isolate the optimizer.Source: §1 · pp. 1–2
  • HetPoison generates label-free removal perturbations in one forward pass, transfers across heterogeneous designs without victim-detector access, and matches or exceeds per-frame optimizers.The paper motivates it as a practical alternative to attacks requiring labels and iterative backpropagation.Source: §1 · pp. 1–2
  • HetShield recovers 83–95% of clean accuracy against HetPoison, outperforming LUCIA and ROBOSAC with minimal computational overhead.The paper presents it as a lightweight trust layer.Source: §1 · p. 2

2 Background and Related Work

Heterogeneous cooperative perception uses learned translation modules to reconcile incompatible feature maps before fusion, creating an attack surface at the shared representation. The background reviews four bottleneck designs, prior attack settings, and adaptive evaluation and defense approaches.

Heterogeneous Cooperative Perception

  • Heterogeneous CP uses a learned translation module, or bottleneck, to reconcile incompatible feature maps before fusion; an attacker can manipulate the shared map while detection heads remain private.The paper evaluates four translation designs with mechanistically distinct approaches to feature reconciliation.Source: §2 · p. 2
  • The reviewed bottlenecks include common-space reverters, paired adapters/reverters, nearest-entry discrete codebooks, and diffusion-seed reconstruction.These are identified respectively with HEAL, STAMP, CodeFilling, and GenComm.Source: §2 · p. 2

Adversarial Attacks on CP

  • Prior test-time evasion assumes victim-detector access and per-frame labels, while BadMDA requires poisoning training data; HetPoison instead uses one test-time forward pass without either access requirement.The paper distinguishes these threat models by their deployment prerequisites.Source: §2 · p. 2

Evaluating and Defending Robustness

  • The related evaluation checklist includes BPDA, EOT, APGD/AutoAttack, and gradient-free checks; the paper compares adaptable CP defenses LUCIA and ROBOSAC and considers quantization and diffusion purification.The passage notes that existing CP countermeasures were designed for homogeneous architectures.Source: §2 · p. 2

3 Threat Model and Method

The section defines an object-removal threat in heterogeneous cooperative perception, controls attack comparisons with a matched harness, and introduces HetPoison and HetShield for attack and mitigation.

3.1 Threat Model

  • The attacker controls one collaborating agent and targets object removal with bounded feature perturbations, while lacking knowledge of the ego vehicle’s private backbones.The threat model gives the attacker access to the public translation bottleneck and fusion stack; the per-frame optimizer is a theoretical upper bound, unlike deployable HetPoison.Source: Capabilities · pp. 2–3

3.2 Matched-Objective Attack Harness

  • The matched harness holds perturbation budget, removal objective, and forward path fixed, isolating optimization strategy when comparing iterative attacks with HetPoison.It includes sign-PGD, Adam-PGD with restarts, EOT-PGD, and BPDA/SPSA for the discrete codebook.Source: §3.2 · p. 3

3.3 HetPoison: A Deployable Attack Generator

  • HetPoison amortizes per-frame search in a context-conditioned generator, producing in-budget perturbations in one forward pass that transfer across heterogeneous designs.Offline training uses a surrogate removal loss and an adversarial term to make perturbations plausible; labels and the surrogate architecture are not needed at deployment.Source: §3.3 · p. 4

3.4 HetShield: A Lightweight Trust Layer

  • HetShield combines temporal forecast deviation and spatial agreement into a per-agent trust score for an ego-anchored blend before fusion.Temporal inconsistency compares features with a self-supervised forecast; spatial consistency checks overlap with the ego's perception.Source: §3.4 · p. 4
  • ∼1ms per frame and 0.90M parameters; HetShield lets trusted agents pass nearly unchanged and blends suspicious features toward ego before fusion.It costs almost no clean accuracy.Source: §3.4 · pp. 4–5

4 Evaluation

Matched attacks show that weak sign-PGD overstates heterogeneous CP’s resistance: HetPoison damages multiple designs, while resistance varies by bottleneck and HetShield recovers attacked accuracy. The evaluation compares these attacks and defenses across heterogeneous architectures and transfer settings.

4.1 Experimental Setup

  • The evaluation reports AP@0.5 across the OPV2V test set, comparing heterogeneous HEAL, STAMP, CodeFilling, and GenComm victim pairs with homogeneous V2VAM as a control.OPV2V-H pairs LiDAR PointPillar-64 with Camera LSS-EfficientNet.Source: §4.1 · p. 5

4.2 Attack Effectiveness

  • Adam-PGD drops V2VAM from 0.608 to 0.109 AP and HEAL from 0.740 to 0.405, reversing sign-PGD’s apparent heterogeneous protection.Increasing sign-PGD steps from 50 to 400 does not help, which the authors attribute to stalled gradients.Source: §4.2 · p. 5
  • HetPoison drives HEAL to 0.209 AP and STAMP to 0.072 AP, outperforming Adam-PGD on continuous architectures; CodeFilling remains at 0.601 AP, a 16.6-point drop from clean.The authors characterize the residual CodeFilling damage as meaningful despite its relative resilience.Source: §4.2 · p. 5

4.3 A Defense Taxonomy

  • EOT-PGD drives GenComm to 0.054 AP, while CodeFilling withstands tested per-frame attacks at 0.730 AP; HetPoison reaches 0.710 AP through the BPDA straight-through surrogate.The authors attribute GenComm’s apparent robustness to stochastic averaging, which EOT-PGD removes.Source: §4.3 · pp. 5–6

4.4 Cross-Method Transferability

  • HetPoison drops continuous HEAL and STAMP targets to 0.03–0.04 AP in both directions, but HEAL→CodeFilling remains 0.79 while CodeFilling→STAMP reaches 0.04.Adam-PGD transfers strongly into STAMP (HEAL→STAMP 0.30) but not into CodeFilling, which remains at 0.82, approximately its noise floor.Source: §4.4 · p. 6

4.5 Defense Effectiveness

  • Against HetPoison, HetShield recovers 78.6%–84.5% of clean AP across continuous families, with approximately 1 ms overhead and almost zero clean-accuracy cost.It outperforms LUCIA and ROBOSAC; ROBOSAC requires multiple full CP forward passes.Source: §4.5 · p. 6

5 Discussion

The discussion attributes over-credited robustness to weak sign-PGD baselines and identifies shared representations as an attack surface. It recommends adaptive evaluation and practical threat testing, while noting that HetShield is not a worst-case guarantee.

  • GenComm is excluded from cross-design comparisons because its 2-channel diffusion message is dimensionally incompatible with the other designs’ 128-channel BEV grid.Source: Does heterogeneity defend, and why was… · p. 7
  • Weak sign-PGD stalls on flat loss surfaces or shattered gradients; the authors identify shared representations in interoperability machinery as the attack surface.Their additional experiments find content-conditioned transfer rather than one persistently effective fixed perturbation.Source: Does heterogeneity defend, and why was… · p. 7
  • HetShield is a practical mitigation, not a worst-case guarantee: a fully adaptive white-box attacker with defense knowledge and hundreds of iterations can optimize through it.The authors leave the fundamental vulnerability introduced by data exchange as an open question.Source: Does heterogeneity defend, and why was… · p. 7
  • The authors recommend discrete bottlenecks over diffusion, adaptive-optimizer ceilings before robustness claims, and HetPoison as the deployable threat baseline for defense testing.Source: Does heterogeneity defend, and why was… · p. 7

6 Conclusion

Heterogeneity does not prevent tuned attacks: HetPoison performs label-free removal attacks in a single forward pass across four heterogeneous architectures without victim access. HetShield recovers 83–95% of attack-degraded accuracy.

6 Conclusion

  • Tuned iterative attacks bypass translation modules, while HetPoison attacks all four major heterogeneous architectures without victim access in a single label-free forward pass.HetPoison matches the damage of computationally heavy optimizers.Source: §6 · p. 7
  • HetShield is a lightweight spatiotemporal trust layer that recovers 83–95% of accuracy degraded by attacks and outperforms prior defenses.Source: §6 · p. 7
  • The paper concludes that translation modules’ presumed protection against feature perturbation attacks is an illusion.Source: §6 · p. 7

Ethics and Broader Impact

The experiments used simulated OPV2V only; no physical vehicle or person was involved. The work studies attacks on safety-critical perception to inform defenses and releases HetShield alongside the attack.

Ethics and Broader Impact

A.1 Generator and Discriminator Architecture

HetPoison uses a three-scale U-Net generator and PatchGAN discriminator for feature-map perturbations. Its tanh parameterization keeps perturbations within a relative-ε budget based on ego-feature activation scale.

A.1 Generator and Discriminator Architecture

  • The three-scale U-Net accepts different BEV grid sizes by resizing decoder activations before skip concatenation, including LiDAR 64 × 128 and camera 64 × 64 grids.Its encoder uses three Conv–InstanceNorm–ReLU blocks, followed by three residual blocks; the decoder returns a C-channel output through tanh.Source: §A.1 · p. 10
  • The PatchGAN discriminator emits a patch-wise real/fake map; C=128 for HEAL, STAMP, and CodeFilling, and C=2 for GenComm.Only the attacker’s feature is discriminated.Source: §A.1 · p. 10

A.2 Perturbation Parameterization

  • The perturbation budget is a relative-ε ball scaled by the floored mean absolute ego activation, and tanh keeps δ feasible without deployment-time clipping.Source: §A.2 · p. 10

A.3 Training Objective and Loss Weights

HetPoison trains its generator with removal, adversarial, and hinge terms, using fixed settings per bottleneck family. Training is offline, and only the generator is needed at deployment.

A.3 Training Objective and Loss Weights

  • The removal loss suppresses mean objectness confidence on ground-truth-positive anchors, with no regression-confusion term to displace boxes.The objective also includes a least-squares GAN term and a hinge guard that measured zero at every logged step because tanh already enforces the perturbation ball.Source: §A.3 · p. 10
  • Only Gθ is needed at deployment; discriminator D is discarded after offline training.Table 5 gives module sizes, but the supplied passage does not state their values.Source: Table 5 · p. 10
  • Loss weights were set once per bottleneck family rather than tuned per victim; GenComm uses 10× larger GAN and hinge weights than the three 128-channel designs.Source: §A.3 · p. 10

A.4 Optimization Schedule and Cost

  • Each generator is trained for 1,000 steps at batch size one frame, costing 6–12 GPU-minutes per generator on one RTX 5090.One generator is trained per design and ε pair; main results use ε=1.0 except GenComm at ε=1.5.Source: §A.4 · p. 10

A.5 Surrogate Training: What the Attacker Needs

HetPoison is trained offline against a surrogate cooperative-perception stack, while deployment uses only observable features and requires no victim access. A quantization-aware variant trains perturbations to survive feature rounding.

A.5 Surrogate Training: What the Attacker Needs

  • HetPoison’s offline surrogate includes the public translation bottleneck, fusion module, detection head, and ground-truth labels for offline frames.The surrogate itself is never updated.Source: §A.5 · pp. 10–11
  • When the surrogate differs from the victim, the generator transfers across heterogeneous families without training access to the victim’s bottleneck, fusion stack, or detection heads.A HEAL-trained generator drives STAMP to 0.03; CodeFilling resists as a victim.Source: §A.5 · p. 11

A.6 Quantization-Aware Variant

  • A per-channel scalar quantizer with a straight-through backward pass trains the generator to produce perturbations that survive rounding.The variant uses production-practice INT4/INT8 bit widths, with other hyperparameters unchanged from Table 6.Source: §A.6 · p. 11

B HetShield: Hyperparameters and Training

The section describes HetShield’s trust gate and temporal predictor, then details a random-perturbation floor used to interpret transfer results. Against that floor, HetPoison transfers strongly to STAMP, while transfer into CodeFilling is not detected.

B.1 Trust Gate

  • HetShield scores collaborators on warped, post-bottleneck, pre-fusion features, then blends in the original feature space using a combined anomaly score.The reported configuration uses α=0.6, τgate=0.3, and T=0.1; fewer than 10% overlapping energetic cells makes the spatial term uninformative and zero-valued.Source: §B.1 · p. 11

B.2 ConvGRU Temporal Predictor

  • A single ConvGRU cell predicts the next feature from three buffered frames, with a 0.90M-parameter defense budget and clean-sequence self-supervised training.One predictor is trained per victim design; no attacked frames or attack labels are used.Source: §B.2 · p. 11

C.2 Construction

  • The random-δ floor uses a fresh uniform draw from the same per-element ℓ∞ budget, inserted in the collaborator slot and evaluated through the victim’s true stack.It is also the initializer for Adam-PGD restarts after the first, so it measures the attack’s starting point before optimization.Source: §C.2 · p. 12

C.3 Measured Floors

  • CodeFilling’s random-δ floor coincides with clean AP because vector quantization snaps noise back to essentially the selected codeword, while STAMP loses 0.217 AP to noise.Table 7 reports the random-δ floor at ε=1.0 using AP@0.5; STAMP’s noise permissiveness also makes it the most transfer-susceptible victim.Source: §C.3 · p. 12, Table 7 · p. 12

C.4 How Cells Are Read Against It

  • 0.03 AP@0.5 for HetPoison HEAL→STAMP leaves the floor far behind; sign-PGD HEAL→STAMP is within noise, and Adam-PGD into CodeFilling does not transfer.Adam-PGD HEAL→STAMP scores 0.30, 0.38 below the floor.Source: §C.4 · p. 12

C.5 Choice of Distribution, and Limitations

The transfer floor uses uniform noise, whose energy differs from sign-PGD’s, and the authors qualify near-floor interpretations. Cross-method comparisons pair loaders by scene and verify ego-feature agreement.

C.5 Choice of Distribution, and Limitations

  • The uniform floor has about 0.58 of the budget’s per-element RMS, making it lenient for transfer; cells within roughly 0.05 of the floor indicate no transfer detected, not a measured null.The authors state that their transfer conclusions are unaffected because the relevant gaps are 0.3–0.65 AP.Source: §C.5 · p. 12
  • Floors are reused across fixed-δ and generator evaluation passes because loaders traverse the same scenes in the same order; STAMP clean AP is 0.899 versus 0.898..Source: §C.5 · p. 12

C.6 Scene-Matched Pairing for STAMP

  • HEAL and STAMP loaders are paired index-by-index on identical scenes and aligned pre-fusion grids, with ego-feature cosine agreement of 0.9945 and no frames dropped.GenComm is excluded from Table 3 because its two-channel message is dimensionally incompatible with the 128-channel grid.Source: §C.6 · pp. 12–13

Related papers

Loading 2609.17856v1…