PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent-Wise Test-Time Normalization for Camera-Based Cooperative Perception with Corrupted Collaborators

Yu Wang, Shaohua Wang, Yicheng Li, Kecheng Sun

arXiv:2610.39505v1cs.CVcs.AI

TL;DR

When only some collaborators’ cameras are corrupted, their shared features can shift away from training data, and pooled test-time normalization mixes clean and corrupted agents’ statistics. AgentNorm normalizes each collaborator using its own current observation while anchoring the ego to source statistics. It outperforms pooled normalization across the reported OPV2V fog and motion-blur checkpoint combinations, while occlusion remains a limitation.

  • Problem

    Pooled test-time batch normalization mixes statistics from clean and corrupted agents, and that mixing can lower detection from the untouched ego’s own features.

    Source: §I · pp. 1–2

  • Method

    AgentNorm computes normalization statistics per collaborator from its current observation and keeps the ego on source statistics.

    Source: §I · p. 2, §V · p. 4, C. Ego Anchoring · p. 5

  • Results

    5.53 to 12.94 AP@0.5 points over Pooled BN, AgentNorm is best on all eight checkpoint-and-corruption combinations under fog and motion blur.

    Source: B. Main Results · p. 7

  • Takeaways & Limitations

    AgentNorm preserves the ego’s pre-trained path while adapting collaborators locally, without extra communication.

    Source: §I · p. 2, C. Ego Anchoring · p. 5

    Under severity-5 occlusion, AgentNorm is below no adaptation on three of four checkpoints because renormalization cannot restore missing image content.

    Source: §VII · p. 10

Abstract

from arXiv · show

Intermediate-fusion cooperative perception lets a vehicle detect objects beyond its own field of view by fusing features shared by nearby agents. When only a collaborator's cameras are degraded by fog, motion blur or exposure errors, the features it shares drift away from the training distribution while the ego input stays clean, and cooperation can become worse than not cooperating at all. Test-time batch-normalization adaptation, the standard training-free remedy, estimates statistics over the pooled batch of all agents and therefore normalizes both the clean ego and the corrupted collaborator with a mixture of two distributions. We show that this mixing harms even the clean ego: when the ego shares BN statistics with a corrupted collaborator, its detection from its own features drops by 3.58 to 6.68 AP@0.5 points although its cameras are untouched. We propose AgentNorm, an agent-wise test-time normalization that i) estimates the normalization statistics of every collaborator from its own current observation and ii) anchors the ego to its source statistics, so that the ego path remains exactly the original model. AgentNorm has no learnable parameters, needs no gradients or labels, is independent of the order of test frames, and runs locally on each agent without changing the transmitted features. On camera-based OPV2V with four fusion checkpoints and collaborator-only corruptions, AgentNorm raises AP@0.5 under severity-5 fog from 29.42-37.25 to 43.32-58.86 and outperforms pooled normalization by 5.53-12.94 points under fog and motion blur, at an extra latency of 2.3 ms per frame. On the real-world DAIR-V2X-C dataset, it outperforms pooled normalization in all ten infrastructure-corruption settings. We also report the cases it does not fix: occlusion, which removes image content rather than shifting feature statistics, and the case in which all agents share the same shift.

I. INTRODUCTION

Agent-specific camera corruption can make cooperation perform worse than using the ego alone, while pooled test-time normalization can also damage the clean ego’s detection. AgentNorm adapts each collaborator using its own statistics while preserving the ego’s source-model path.

I. INTRODUCTION

  • Under severity-5 fog, an attention-fusion detector reaches 37.25 AP@0.5 with a corrupted collaborator, below the 38.31 AP@0.5 obtained by discarding it.Camera degradation may affect one agent while another remains clean, shifting the collaborator’s features away from what the fusion module expects.Source: §I · p. 1
  • Pooled batch-normalization statistics mix clean and corrupted agents, lowering ego-only detection from 38.31 to 31.63–34.73 AP@0.5.The paper identifies agent-specific corruption as a setting where this standard training-free remedy contaminates the clean ego.Source: §I · pp. 1–2
  • As corruption severity rises, collaborator BEV features move farther from source statistics and their local variance shrinks, while ego features remain unaffected.These trends were observed for all four tested corruption types.Source: §I · p. 1
  • AgentNorm normalizes each collaborator from its own current observation and anchors the ego to source statistics, leaving the ego path as the unmodified pre-trained model.It requires no parameters, gradients, or labels, is independent of test-frame order, and changes neither message format nor bandwidth.Source: §I · p. 2
  • 5.53 to 12.94 AP@0.5 points over pooled normalization under collaborator fog and motion blur, AgentNorm improves every tested fusion checkpoint.The evaluation uses four public fusion checkpoints and also reports cases where AgentNorm does not help, including occlusion and agent-homogeneous shift.Source: §I · p. 2

A. Cooperative Perception

This section situates intermediate-fusion cooperative perception among robustness and test-time adaptation methods, then describes the camera-based fusion pipeline. The paper’s setting differs from prior adaptation work because only a subset of agents may experience a distribution shift.

A. Cooperative Perception

  • Intermediate fusion is a cooperative-perception strategy in which agents share sensor-derived features, while the fusion architecture remains unchanged in this work.The related work describes max pooling, attention, transformers, graph distillation, and spatially selective communication as existing fusion approaches.Source: A. Cooperative Perception · p. 2

B. Robustness of Cooperative Perception to Corruptions

  • Prior corruption remedies include training-time robustness methods, while AgentNorm targets shifts that differ between agents without retraining.The cited approaches include domain generalization, corruption training, feature restoration, and sim-to-real adaptation.Source: B. Robustness of Cooperative… · p. 2

C. Test-Time Adaptation and Test-Time Normalization

  • Prior test-time normalization methods treat the test batch as one target distribution; AgentNorm computes statistics per agent and keeps the ego at source statistics.The adaptation setting retains test-time BN constraints but allows the shift to affect only a subset of agents processed together.Source: C. Test-Time Adaptation and Test-Time… · p. 2

A. Camera-Based Intermediate Fusion

  • A shared per-agent encoder lifts each agent’s images into a BEV feature map, and collaborators transmit those maps to the ego for warping, fusion, and detection.The camera models use an image backbone, Lift-Splat-Shoot view transformer, and BEV backbone.Source: A. Camera-Based Intermediate Fusion · pp. 2–3

B. Batch Normalization in the Per-Agent Encoder

The per-agent encoder contains the batch-normalization layers that process multiple agents together. Training accumulates source statistics; inference uses those source estimates.

B. Batch Normalization in the Per-Agent Encoder

  • During training, batch normalization uses current mini-batch statistics and accumulates running source estimates; inference uses the source statistics.The normalization operates on channel-wise statistics.Source: B. Batch Normalization in the… · p. 3
  • In the cooperative encoder, the BN batch contains slices from multiple agents: each agent’s camera images in the image backbone and one BEV map per agent in the BEV backbone.In the cited AttFuse checkpoint, all 75 BN layers are in the encoder, not the fusion module or detection head.Source: B. Batch Normalization in the… · p. 3
  • The AttFuse checkpoint’s fusion module and detection head contain no normalization layers; its 75 BN layers lie in the encoder.Source: B. Batch Normalization in the… · p. 3

C. Agent-Specific Corruption at Test Time

The empirical analysis examines collaborator-only corruption in camera-based cooperative perception, showing both how it affects detection and why local feature gating is not an effective fix. Corruption also produces measurable shifts in collaborator feature statistics.

C. Agent-Specific Corruption at Test Time

  • The test-time setting has clean ego images and collaborators that may follow distinct corrupted distributions, with no labels, source data, retraining, or repeated frame processing.The system is constrained to retain the transmitted message format and size.Source: C. Agent-Specific Corruption at Test… · p. 3

IV. EMPIRICAL ANALYSIS

  • The experiments use the public camera AttFuse checkpoint of HEAL on the OPV2V test split and corrupt only collaborator cameras.The section examines what goes wrong and why obvious remedies do not fix it.Source: §IV · p. 3

A. A Corrupted Collaborator Can Be Worse Than None

  • 37.25 AP@0.5 under severity-5 fog and 36.48 under severity-5 motion blur are below the 38.31 AP@0.5 from ego-only detection.With clean collaborators the model reaches 56.81 AP@0.5; in the other six settings, cooperation still helps, but by less than with clean inputs.Source: A. A Corrupted Collaborator Can Be… · p. 3

B. Corruption Shifts the Collaborator’s Feature Statistics

  • For all four corruptions, normalized distance from source statistics rises from 0.36 on clean data to 0.54–0.63 at severity 5, while local variance falls from 0.51 to 0.34–0.39.The means are computed over 3,445 collaborator instances; the figure describes these statistics against corruption severity.Source: B. Corruption Shifts the… · p. 3, Figure 1 · p. 3

C. Why Not Gate Corrupted Regions?

  • Feature-distance gating localizes shifted cells with AUROC 0.65–0.84, yet performs worse than random-cell gating in all 24 tested setting-and-k combinations.Even oracle gating improves over the better of no gating and ego-only by at most 2.31 points; the table reports AP@0.5 and AUROC for region-level gating.Source: C. Why Not Gate Corrupted Regions? · pp. 3–4, Table II · p. 4

D. A Tent-Style Entropy Baseline

A Tent-style entropy objective collapses under collaborator corruption, while pooled BN mixes clean-ego and corrupted-collaborator statistics. Agent-wise normalization separates those statistics and keeps the ego on its source statistics.

D. A Tent-Style Entropy Baseline

  • 37.25 to 1.59 or 3.94 AP@0.5 under severity-5 fog: Tent-style adaptation collapses, and the tested entropy objective is unreliable on the dense binary map.At threshold 0.2, entropy minimization drives low-confidence anchors below the threshold; the authors tested two thresholds, two learning rates, and one seed, but not sample selection or periodic resets.Source: D. A Tent-Style Entropy Baseline · p. 4

V. AGENT-WISE TEST-TIME NORMALIZATION

  • AgentNorm uses each collaborator’s current-observation statistics while keeping the ego on source statistics, rather than normalizing both with pooled mixture statistics.The figure contrasts pooled BN, which shifts the clean ego and under-corrects the collaborator, with AgentNorm’s agent-wise statistics and ego anchoring.Source: §V · p. 4, Figure 2 · p. 4

A. Pooled Test-Time Normalization Is Mis-Specified

  • Pooled BN leaves each agent with residual offset and scale error unless all agents share the same statistics, and its between-agent variance term shrinks collaborator features.With one collaborator and equal shares, the clean ego’s offset is half the collaborator’s deviation; when all agents share a shift, the residuals vanish.Source: A. Pooled Test-Time Normalization Is… · p. 4
  • 3.58 to 6.68 AP@0.5 points: pooled statistics with a corrupted collaborator reduce the clean ego’s own-feature detection despite untouched cameras.Pooling with a clean collaborator costs 0.89 points; corrupted collaborators also degrade the fused output through their shifted features.Source: A. Pooled Test-Time Normalization Is… · p. 5

B. Agent-Wise Statistics

Agent-wise statistics are computed separately from each collaborator’s current feature slice and used for its batch-normalization layers. The method uses frame-wise statistics rather than carrying a running average across test frames.

B. Agent-Wise Statistics

  • AgentNorm estimates each collaborator’s statistics from its own slice and normalizes that slice with its own mean and variance, retaining the original BN affine parameters.The statistics are computed separately by channel across the slice’s batch and spatial positions.Source: B. Agent-Wise Statistics · p. 5
  • In the image backbone, statistics pool over one agent’s camera views; in the BEV backbone, they use that agent’s map positions, while unsplittable layers retain source statistics.Source: B. Agent-Wise Statistics · p. 5
  • Current-frame statistics avoid carrying one collaborator’s conditions into another frame and make each output independent of previously processed frames.The method therefore has no state that can drift over the test stream.Source: B. Agent-Wise Statistics · p. 5

C. Ego Anchoring

AgentNorm anchors the ego to source batch-normalization statistics, preserving its pretrained feature path when only collaborators are corrupted. Its inference-time changes can be computed locally without changing transmitted feature size or adding an ego-side message.

C. Ego Anchoring

  • Keeping source statistics makes the ego’s features exactly those of the pretrained model, so adapting collaborators cannot alter the ego’s own evidence.If the ego itself may be corrupted, the paper also evaluates a symmetric variant that uses its own statistics.Source: C. Ego Anchoring · p. 5

D. Properties and Deployment

  • AgentNorm has no learnable parameters or backward pass, needs no labels or source data, and lets collaborators normalize locally without changing transmitted feature size or requiring an extra ego message.Its per-layer statistics add a few reductions and no additional memory.Source: D. Properties and Deployment · pp. 5–6
  • 7.3 × 10^-5 versus 0.016: AgentNorm’s ego-feature change is at floating-point level, compared with the change under Pooled BN.The comparison is from an OPV2V AttFuse frame with a clean ego and a severity-5 fog-corrupted collaborator.Source: Figure 4 · p. 6

VI. EXPERIMENTS

The experiments use camera-based OPV2V checkpoints and a camera-only DAIR-V2X-C model, with collaborator-side image corruptions and clean ego cameras unless stated otherwise. The setup compares inference-time normalization against unadapted, ego-only, alignment, entropy, gating, and oracle methods.

A. Experimental Setup

  • The OPV2V test split has 2,170 frames, including 2,051 with a collaborator, and the study evaluates four public camera checkpoints; DAIR-V2X-C evaluation uses 1,789 validation frames.The four OPV2V checkpoints are AttFuse, V2X-ViT, DiscoNet, and AttFuse-B; the DAIR-V2X-C camera-only CoAlign configuration was trained for 30 epochs.Source: A. Experimental Setup · p. 6
  • Collaborator cameras receive fog, motion blur, brightness, or occlusion on OPV2V and fog, snow, frost, brightness, or motion blur on DAIR-V2X-C; ego cameras stay clean unless stated otherwise.Corruptions are evaluated at severities 3 and 5 with fixed per-image, agent, camera, type, and severity seeds.Source: A. Experimental Setup · p. 6, Figure 5 · p. 7
  • The evaluation reports AP@0.5 and AP@0.7, compares methods including Pooled BN and AgentNorm, and uses the study’s own pipeline numbers rather than mixing them with distributed checkpoint results.The compared methods also include no adaptation, ego only, feature-statistics alignment, Tent-style entropy minimization, region gating, and a label-supervised oracle reference.Source: A. Experimental Setup · p. 6
  • AgentNorm and Pooled BN use released weights unchanged, and deterministic methods are evaluated once because their outputs do not depend on frame order.OPV2V uses separate identical-weight encoder copies for ego and collaborators; DAIR-V2X-C uses a module that splits BN inputs by agent.Source: A. Experimental Setup · pp. 6–7

B. Main Results

AgentNorm outperforms pooled normalization under collaborator fog and motion blur, while its gains vary with corruption type. It is less reliable for occlusion, and its advantage over pooling shrinks as corruption severity decreases or agents share a shift.

B. Main Results

  • 5.53 to 12.94 AP@0.5 points: AgentNorm beats Pooled BN on all eight checkpoint–corruption combinations of severity-5 fog and motion blur.On V2X-ViT under fog, AgentNorm reaches 58.86 AP@0.5 versus 45.92 for Pooled BN; on AttFuse-B under fog, it reaches 43.32 versus 30.79 for Pooled BN.Source: B. Main Results · p. 7
  • AgentNorm improves over no adaptation by 9.09 to 13.22 AP@0.5 points under brightness, while AgentNorm and Pooled BN are within 2.47 points and each wins on two checkpoints.Brightness shifts collaborator feature statistics less than fog, making pooled statistics closer to the collaborator’s own.Source: B. Main Results · p. 7
  • Neither method is reliable under occlusion: AgentNorm falls below no adaptation on three of four checkpoints, and Pooled BN does so on two.Occlusion removes image content rather than shifting feature statistics, which normalization cannot restore.Source: B. Main Results · p. 7
  • Averaged over fog, motion blur, and brightness, AgentNorm reaches 52.47, 59.08, 51.34, and 44.71 AP@0.5 on AttFuse, V2X-ViT, DiscoNet, and AttFuse-B, respectively.The corresponding Pooled BN scores are 45.81, 52.01, 47.68, and 36.23; without adaptation they are 38.70, 38.98, 36.33, and 33.22.Source: B. Main Results · p. 7
  • On clean collaborators, AgentNorm does not reduce AP@0.5 relative to the source model on AttFuse or V2X-ViT, though Pooled BN scores higher on the reported clean checkpoints.For AttFuse, AP@0.5 changes from 56.81 to 57.85 with AgentNorm; for V2X-ViT, it changes from 66.46 to 68.50.Source: B. Main Results · p. 7

C. Robustness to Corruption Severity

  • On AttFuse, severity-5 fog reduces AP@0.5 by 19.56 points without adaptation, 18.52 with Pooled BN, and 5.87 with AgentNorm.Under motion blur, the respective drops are 20.33, 16.27, and 8.02 points; AgentNorm remains more than 11 points above ego-only detection at severity 5.Source: C. Robustness to Corruption Severity · p. 7

D. Real-World Data

On DAIR-V2X-C, AgentNorm beats Pooled BN in all ten roadside-corruption settings, though the authors caution that absolute camera-only accuracy is low. Comparisons on OPV2V also favor AgentNorm over alternative remedies in the hardest setting, and a selected example illustrates its detections alongside results across candidate frames.

D. Real-World Data

  • On AttFuse under fog and motion blur, AgentNorm degrades more slowly than Pooled BN as collaborator-corruption severity increases and stays above ego-only detection; no method helps under occlusion.Source: Figure 6 · p. 8
  • In all ten roadside-corruption settings on DAIR-V2X-C, AgentNorm exceeds Pooled BN; it exceeds no adaptation in nine, with motion blur at severity 3 the exception (8.93 versus 8.97 AP@0.5).Under severity-5 fog, snow, and frost, AgentNorm restores AP@0.5 to 5.17–5.88 from cooperative scores of 1.16–1.50, below the vehicle-only score of 1.65.Source: D. Real-World Data · p. 8, Table IV · p. 8
  • DAIR-V2X-C camera-only detection is low for this model: 10.83 AP@0.5 with clean inputs and 1.65 for the vehicle alone, so the authors frame the results as directional rather than magnitude evidence.With clean inputs, AgentNorm is 0.85 points below the source model and 1.12 points above Pooled BN.Source: D. Real-World Data · p. 8

E. Comparison with Alternative Remedies

  • On the hardest AttFuse setting, AgentNorm gains 14.73 points over no adaptation, compared with 3.40 for Pooled BN; only the label-supervised oracle scores higher.Discarding the collaborator, region gating, and feature-statistics alignment remain within 1.2 points of no adaptation.Source: E. Comparison with Alternative Remedies · p. 8

F. Qualitative Results

  • Across 115 candidate frames, AgentNorm detects 747 of 1,390 objects at IoU 0.5 (53.7%), versus 621 (44.7%) with Pooled BN and 555 (39.9%) without adaptation.The displayed frame was selected as the most favorable of those candidates; it shows 13 true positives for AgentNorm, 8 for Pooled BN, and 6 without adaptation.Source: F. Qualitative Results · p. 8

G. Ablation and Analysis

The ablations identify the image encoder as the key location for agent-wise statistics and show a trade-off in ego anchoring between AP@0.5 and AP@0.7. The advantage over pooled normalization is tied to heterogeneous agent shifts; running statistics and shift-based gating perform less favorably in the reported tests.

G. Ablation and Analysis

  • The symmetric variant scores 0.86 to 1.43 AP@0.5 points above AgentNorm on all five columns, while AgentNorm scores 0.47 to 0.80 AP@0.7 points higher on all five.Ego anchoring remains the default because it leaves ego features identical to the source model.Source: G. Ablation and Analysis · p. 8, Table V · p. 9
  • In the illustrated severity-5 fog frame, AgentNorm detects 13 of 21 objects at IoU 0.5, versus 8 with Pooled BN and 6 without adaptation.The frame is the most favorable of 115 candidates, rather than a typical example.Source: Figure 8 · p. 9
  • Under fog and motion blur, applying agent-wise statistics only in the image encoder gives 51.92 and 49.39 AP@0.5, within 0.44 points of applying them everywhere.Applying them only in the BEV backbone gives 31.37 and 31.43, 5.88 and 5.05 points below no adaptation, respectively.Source: G. Ablation and Analysis · p. 9
  • Replacing frame-wise collaborator statistics with an exponential moving average yields 51.12 AP@0.5 under fog, 0.86 points below frame-wise statistics, and makes results depend on frame order.Source: G. Ablation and Analysis · p. 9
  • When ego and collaborators share the same corruption, agent-wise statistics and Pooled BN perform within 1.01 AP@0.5 points on all four corruption types.The reported comparison attributes AgentNorm’s advantage to heterogeneous shifts across agents.Source: G. Ablation and Analysis · p. 9, Table VI · p. 9
  • The shift-detection rule triggers on only 5% of AttFuse and 4.2% of V2X-ViT collaborator frames under motion blur, yielding 36.99 and 34.13 AP@0.5 instead of 49.83 and 55.45 with AgentNorm.Because AgentNorm does not reduce accuracy on clean collaborators, the authors report unconditional application as simpler and more accurate than gating.Source: G. Ablation and Analysis · p. 9

H. Efficiency

AgentNorm improves over pooled normalization under fog and motion blur on OPV2V and DAIR-V2X-C, with a small measured runtime cost. Its benefits are bounded by the type and scope of shift: it does not recover lost image content or help when all agents share the same shift.

H. Efficiency

  • AgentNorm adds 2.33 ms to mean latency of 254.32 ms (0.9%), with unchanged peak memory.The added latency is comparable to the 0.87 ms difference between repeated unmodified-model runs; AgentNorm adds no parameters, gradient computation, or transmitted data.Source: H. Efficiency · p. 10

VII. DISCUSSION AND LIMITATIONS

  • Under severity-5 occlusion, AgentNorm falls below no adaptation on three of four checkpoints because renormalization cannot restore missing image content.For deployments expecting lens blockage or heavy dirt, the authors recommend combining AgentNorm with detection of missing content and reduced fusion weight for the affected collaborator.Source: §VII · p. 10
  • When all agents undergo the same corruption, per-agent and pooled statistics coincide in expectation, and AgentNorm brings no gain over Pooled BN.Its benefit is specific to shifts that differ between agents.Source: §VII · p. 10
  • On clean OPV2V data, Pooled BN is 0.73 to 3.51 AP@0.5 points above AgentNorm across four checkpoints; the proposed explanation of shared train-test shift is unverified.The authors say the loss on clean data is small relative to AgentNorm’s gains under fog and motion blur.Source: §VII · p. 10
  • Evaluation covers camera-based models, not LiDAR encoders, and AgentNorm applies to encoders with BN rather than only layer or group normalization.The real-world evaluation uses a camera-only DAIR-V2X-C model with low absolute accuracy; per-agent statistics also use one frame of a few images, which may be noisier for very small inputs.Source: §VII · p. 10
  • Gradient-based baseline results use a single seed, so their exact values should not be overinterpreted.The authors state that the observed collapse is far larger than seed variation.Source: §VII · p. 10

VIII. CONCLUSION

  • AgentNorm improves all four OPV2V checkpoints under fog and motion blur and gains consistently over pooled normalization on real-world DAIR-V2X-C.The conclusion also notes no advantage when all agents share the same shift and no remedy for occlusion-type content loss.Source: §VIII · p. 10

Related papers

Loading 2610.39505v1…