PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Residual Bottleneck for Robust Asynchronous Collaborative Perception

Melih Yazgan, Ahmed Abouelazm, J. Marius Zöllner

arXiv:2610.10090v1cs.CV

TL;DR

Delayed or incomplete collaborator features make asynchronous fusion difficult, and existing compensation can be unreliable when correspondence estimates are inaccurate. The paper retains a pose-warped feature as an anchor and applies temporal evidence as gated residual corrections. Experiments report improved robustness across DAIR-V2X and OPV2V under communication degradation.

  • Problem

    Delayed or incomplete collaborator features make alignment and fusion difficult, while compensation can be unreliable when motion or correspondence estimates are inaccurate.

    Source: §1 · pp. 1–2

  • Method

    The Temporal Residual Bottleneck preserves a deterministic pose-warped collaborator feature and adds gated residual corrections from a ∆t-conditioned temporal branch.

    Source: §1 · p. 2

  • Results

    Experiments on DAIR-V2X and OPV2V show improved robustness under severe delay, packet loss, and pose noise.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pose-warped anchor and gated corrections avoid replacing reliable geometric structure with uncertain transported estimates.

    Source: §1 · p. 2

    The method's safety is architectural and empirical, not a formal hard bound on residual corrections.

    Source: §3.4 · p. 7

Abstract

from arXiv · show

Collaborative perception extends the sensing range of autonomous vehicles, but its performance degrades when shared features arrive stale or incomplete. Most latency-robust methods compensate delayed collaborator features through flow-guided alignment or direct feature transport. In this work, we formulate asynchronous collaborative perception as temporal residual prediction. Our Temporal Residual Bottleneck keeps a deterministic pose-warped collaborator feature as a conservative anchor and uses a $\Delta t$-conditioned xLSTM to extract residual temporal evidence from the available history. A detector-facing residual bottleneck then applies only gated, regularized corrections before ego-side fusion, reducing the risk of overwriting reliable static structure when temporal correspondence is uncertain. Experiments on DAIR-V2X and OPV2V show that our method is especially effective under severe fixed/irregular delays and packet drops. On DAIR-V2X, the reported checkpoint trades a small amount of synchronized peak accuracy for better robustness under stronger communication degradation. Controlled diagnostics further indicate that direct feature transport has oracle headroom but can become unreliable when deployed without accurate correspondence. These results support temporal residual fusion as a practical alternative for asynchronous and incomplete collaborative perception. Code will be publicly released at https://url.fzi.de/8dk38.

1 Introduction

Asynchronous and incomplete collaborator features challenge reliable fusion, while existing compensation can falter when motion or correspondence estimates are inaccurate. The paper instead preserves a pose-warped anchor and applies temporal evidence through gated residual corrections, distinguishing its approach from feature reconstruction or replacement.

1 Introduction

  • Stale or incomplete collaborator information degrades feature alignment and fusion, and geometric alignment alone leaves moving objects temporally inconsistent.Source: §1 · p. 2

2 Related Work

  • Existing latency-aware methods use synchronization, flow prediction, or direct feature transport, which can become unreliable when motion or correspondence estimates are inaccurate.Source: §1 · p. 2, §2 · p. 3

1 Introduction

  • The Temporal Residual Bottleneck retains a deterministic pose-warped collaborator feature and uses a ∆t-conditioned xLSTM to add only gated residual corrections.Source: §1 · p. 2
  • Experiments on DAIR-V2X and OPV2V show improved robustness under severe delay, packet loss, and pose noise.Source: §1 · p. 2

2 Related Work

  • Unlike methods that reconstruct or replace the full current collaborator representation, this approach keeps the pose-warped feature as its detector-facing anchor.Source: §2 · p. 3
  • The temporal branch estimates residual evidence relative to the warped baseline, operates on dense BEV features, and does not maintain object identities or trajectories.Source: §2 · p. 4

3 Method

The method treats asynchronous collaborative perception as temporal residual prediction: it preserves a deterministic pose-warped collaborator feature and applies temporal evidence as a gated correction before ego-frame fusion. A Δt-conditioned recurrent branch summarizes collaborator history, and the resulting representation is optimized for downstream detection.

3.1 Problem Setup

  • The available collaborator history can be irregular or partially missing, and the realized feature age conditions the temporal branch without requiring delay-regime prediction or model switching.The history records temporal offsets and whether each feature is available.Source: §3.1 · p. 5

3.2 Pose-Warped Geometric Baseline

  • The pose-warped baseline compensates known ego/collaborator pose changes and is reliable for static structure, but cannot fully correct independently moving objects.The warp uses localization metadata rather than an object-motion oracle.Source: §3.2 · p. 6

3.3 Time-Conditioned Temporal Evidence

  • A recurrent predictor summarizes each BEV cell’s short history; multi-frequency delay features condition it through FiLM, while a pose/time token adds relative translation and periodic yaw information.The resulting temporal evidence is treated as a residual, not a standalone transported feature.Source: §3.3 · p. 6

3.4 Detector-Facing Residual Bottleneck

  • The attention-free bottleneck converts recurrent evidence into a gated correction while preserving the baseline; its object-focused gate uses collaborator objectness maps rather than labels or oracle boxes.The gate is obtained from cached collaborator features, so it requires no additional dense objectness transmission.Source: §3.4 · pp. 6–7

3.5 Fusion and Detection

  • After temporal compensation, updated collaborator features are aligned to the ego frame and fused with the ego feature using multi-scale intermediate fusion before detection.The detector architecture remains fixed; the temporal residual interface prepares collaborator features before fusion.Source: §3.5 · p. 8

3.6 Training Objective

  • Training combines the standard detection objective with an applied-residual objective, supervising the correction exposed to the detector rather than reconstructing full BEV features.The synchronized current feature supplies a training target for the correction but is not an inference input; remaining objective terms discourage harmful corrections.Source: §3.6 · p. 8

4 Experiments

The experiments test asynchronous collaborative perception across two benchmarks, delay and packet-drop conditions, and pose noise. Results show a robustness trade-off on DAIR-V2X and stronger performance across reported OPV2V scenarios; ablations examine the temporal residual interface and its components.

4.1 Experimental Setup

  • Evaluation uses LiDAR-only intermediate fusion for single-class vehicle detection on DAIR-V2X and OPV2V, with AP@0.7 as the primary metric and shared comparison protocols.The seven communication settings include synchronized input, fixed and irregular average delays of 100/300/500 ms, and joint delay-plus-packet-drop stress.Source: §4.1 · p. 9

4.2 Implementation Details

  • The method uses four cached collaborator frames from an ego-side history cache, leaving current communication payload unchanged while increasing receiver-side memory and computation.Packet drop is activated only at inference as a severe missing-feature stress test, not as a calibrated wireless-channel model.Source: §4.1 · p. 9, §4.2 · p. 10

4.3 Comparison to DAIR-V2X Baselines

  • −0.058 AP@0.7 for LRCP versus −0.029 for our method from synchronized input to irregular 500 ms delay; the DAIR-V2X comparison reflects a robustness trade-off, not broad synchronized SOTA.Our method is near the best AP@0.7 at 300 ms and achieves the best AP@0.5 at 500 ms, while published CoDynTrust and W2C+SyncNet are stronger at synchronized AP@0.7.Source: Table 1 · p. 10, §4.3 · p. 10

4.4 OPV2V Culver City Results

  • Our residual temporal model is strongest in all reported OPV2V scenarios, improving AP@0.5 and AP@0.7 under synchronized and irregular 300/500 ms delays over the strongest listed CoDynTrust baseline.The comparison is not driven by weakened reproduced baselines; the authors relate the pattern to OPV2V’s cleaner pose/time consistency and richer temporal redundancy versus DAIR-V2X.Source: Table 3 · p. 11, §4.4 · p. 11

4.5 Robustness to Delay and Packet Drop

  • Our method degrades less than LRCP as delay grows and is comparable around 300 ms, then stronger at 500 ms in fixed and irregular settings; packet drop is a severe burst-interruption stress test, not a calibrated V2X channel.From synchronized input to irregular 500 ms delay, AP@0.7 degradation is reduced by 50%.Source: Table 4 · p. 12, §4.5 · p. 12

4.6 Robustness to Pose and Localization Error

  • Under inference-time fusion-pose noise, LRCP’s AP@0.7 drops sharply as error increases, while our model better preserves strict localization at medium and high noise.The evaluation perturbs fusion poses while keeping detector checkpoints fixed; the zero-noise case corresponds to irregular 300 ms delay.Source: §4.6 · pp. 12–13, Table 5 · p. 13

4.7 Ablation and Diagnostic Findings

  • At irregular 300 ms, the complete residual bottleneck reaches 0.640 AP@0.7 versus 0.622 for the pose-warped baseline; neither generic temporal prediction nor the residual interface alone explains the gain.Oracle full residual reaches 0.660 AP@0.7, but predicted transport reaches 0.631; removing the object-focused gate also reduces AP@0.7 to 0.630.Source: §4.7 · p. 13, Table 6 · p. 14
  • The final m-s-m xLSTM with four history frames performs best across the three evaluated settings, while a three-layer GRU remains effective at 0.630 AP@0.7 under irregular 300 ms delay.Shorter histories and alternative xLSTM block patterns underperform; the residual formulation is not specific to xLSTM.Source: Table 7 · p. 14, §4.7 · p. 14

5 Conclusion

The Temporal Residual Bottleneck anchors collaborator features with deterministic pose warping and applies gated residual corrections from a Δt-conditioned xLSTM. Results show stronger robustness under severe DAIR-V2X degradation and improved synchronized and delayed performance on OPV2V Culver City, with limits on the packet-drop test and safety guarantees.

5 Conclusion

  • The Temporal Residual Bottleneck preserves a deterministic pose-warped collaborator feature and injects only gated residual corrections from a Δt-conditioned xLSTM.Its detector-facing interface avoids direct feature overwrite when temporal correspondence is uncertain.Source: §5 · p. 14
  • Under severe fixed delay, irregular delay, and joint packet-drop stress, the DAIR-V2X model degrades more gracefully than flow-guided LRCP.The reported model does not maximize peak synchronous AP.Source: §5 · p. 14
  • On OPV2V Culver City, the same residual formulation improves both synchronized and delayed performance.Source: §5 · p. 14
  • The packet-drop benchmark is an aggressive stress test, and the residual gate is not a formal safety guarantee.Source: §5 · p. 14
  • Future work will study residual-norm diagnostics and communication-aware residual selection under more realistic packet-loss processes.Source: §5 · p. 14

A Additional Implementation Details

Supplementary Table 8 lists implementation settings for the DAIR-V2X and OPV2V temporal residual model, covering training and model-specific configurations.

A Additional Implementation Details

B Contextualizing Packet Drop as Severe V2X Degradation

The joint delay-and-packet-drop stress test evaluates whether fusion preserves detection performance when delayed collaborator evidence is spatially incomplete. It is a severe proxy for impaired V2X communication, not a calibrated wireless-channel simulator.

B Contextualizing Packet Drop as Severe V2X Degradation

C Qualitative Examples of Detections under Delay and Pose Noise

Representative DAIR-V2X examples compare the method with LRCP under irregular 500 ms delay, with and without pose noise. The examples show differing error structure, not failure-free detection.

C Qualitative Examples of Detections under Delay and Pose Noise

  • Under severe delay and perturbation, our method tends to keep more detections near visible object support, while LRCP often produces duplicated or spatially shifted boxes in dense vehicle regions.This qualitative behavior aligns with the AP@0.7 trend of better preserved localization, though both methods can produce false positives.Source: C Qualitative Examples of Detections… · p. 17
  • The examples are representative cases and are not intended to show failure-free detection; both methods can produce false positives under severe temporal misalignment.Source: C Qualitative Examples of Detections… · p. 17
  • Figure 4 compares our method with LRCP for irregular 500 ms delay and for the same delay with 0.3 m / 0.3° pose noise.Source: Figure 4 · p. 17, C Qualitative Examples of Detections… · p. 17

D Runtime Analysis

Runtime was measured per sample on an RTX 3090 under matched inference conditions. The lightweight xLSTM-m variant is substantially faster than flow-guided baselines, while the final configuration has runtime comparable to LRCP and much lower than CoDynTrust.

D Runtime Analysis

  • The final xLSTM-m-s-m configuration has runtime comparable to LRCP and much lower than CoDynTrust.The lightweight xLSTM-m variant is substantially faster than flow-guided baselines.Source: D Runtime Analysis · p. 17
  • Runtime was measured per sample on an RTX 3090 with batch size 1, identical BEV resolution and collaborator count, 50 warm-up iterations, and 200 measured iterations.CUDA synchronization was performed before and after each measurement.Source: D Runtime Analysis · p. 17, Figure 5 · p. 18
  • The comparison indicates moderate recurrent overhead for the residual temporal interface without the high cost of trust-filtered dense compensation.Source: D Runtime Analysis · p. 17

E CoBEVFlow reproduction diagnostic.

The CoBEVFlow reproduction uses a hard multiplicative ROI mask after BEV-flow warping, suppressing non-ROI background features. At DAIR-V2X p = 0.3, a background-preserving blend improves AP@0.7, while the flow warp itself does not improve over identity.

E CoBEVFlow reproduction diagnostic.

  • The released CoBEVFlow path applies a hard multiplicative ROI mask after BEV-flow warping, suppressing all non-ROI background features.The diagnostic compares the hard-mask path, no-mask, no-warp/no-mask, and background-preserving blend at the standard DAIR-V2X p = 0.3 setting.Source: E CoBEVFlow reproduction diagnostic · p. 18
  • At DAIR-V2X p = 0.3, the background-preserving blend improves AP@0.7 from 0.597 to 0.611, matching the no-mask and identity-warp controls.The Table 9 caption identifies this as a CoBEVFlow ROI-mask diagnostic reporting DAIR-V2X AP for the reproduced method.Source: Table 9 · p. 18, E CoBEVFlow reproduction diagnostic · p. 19
  • The flow warp itself does not improve over identity in this setting, so the released hard-mask path remains the reproduced baseline for the main-paper comparison.The remaining Table 9 rows are used as diagnostics for why the reproduced baseline is lower than the published reference.Source: E CoBEVFlow reproduction diagnostic · p. 19

F Synchronized/asynchronous training trade-off

A checkpoint fine-tuned with mostly causal-asynchronous samples reaches higher reported AP@0.7 across the synchronized and delayed settings than the reported model. The authors interpret the synchronized gap as a training-distribution trade-off rather than an intrinsic limit of the residual architecture.

F Synchronized/asynchronous training trade-off

Related papers

Loading 2610.10090v1…