Source-linked AI summary
EgoRefine: Ego-Referenced Predictive Alignment and Trajectory-Conditioned Reliability-Aware Fusion for Asynchronous Collaborative Perception
Lingzhao Kong, Yongsheng Zang, Yu Kang, Kailun Yang, Jie Fu, Yukun Zuo, Zhiyong Li
TL;DR
Asynchronous communication delays cooperative features, and prediction from the transmitting agent’s history can leave residual misalignment while fusion overlooks spatial differences in alignment quality. EgoRefine guides delayed-feature prediction with the current ego feature and uses alignment cues to adaptively fuse features. It outperforms TraF-Align by 1.6 points in AP@0.5 and 2.9 points in AP@0.7 on average.
Problem
Existing asynchronous methods predict delayed cooperative features mainly from the transmitting agent’s history and do not directly use alignment-stage cues to assess local reliability, leaving residual misalignment.
Source: §I · p. 2Method
EgoRefine uses the current ego feature to guide delayed-feature prediction and uses trajectory discrepancy and directional refinement magnitude to condition reliability-aware fusion.
Source: §I · p. 2Results
EgoRefine outperforms TraF-Align by 1.6 points in AP@0.5 and 2.9 points in AP@0.7 on average across V2V4Real and DAIR-V2X-Seq.
Source: §I · p. 2Takeaways & Limitations
The reported gains hold under every tested latency from 0 to 400 ms without extra supervision, annotation, or communication overhead.
Source: §I · p. 2
Abstract
from arXiv · showhide
Collaborative perception enables connected agents to share complementary observations for 3D object detection, extending sensing range and mitigating occlusion. Under asynchronous communication, however, cooperative features arrive with temporal delay. Existing prediction-based methods compensate for these features mainly from the transmitting agent's own history, leaving residual misalignment with the ego agent's current observation; subsequent fusion also often overlooks spatial variations in alignment quality. We propose EgoRefine, an ego-referenced predictive alignment and reliability-aware fusion framework for asynchronous collaborative perception. Its Ego-referenced Predictive Alignment module uses the current ego feature to guide cooperative trajectory-field prediction and refines the sampling offsets along an ego-referenced trajectory direction. Its Trajectory-conditioned Reliability-aware Fusion module treats the trajectory discrepancy between the ego and cooperative streams and the directional refinement magnitude as alignment cues, using them to condition the relation between aligned features and adaptively reweight the two streams before convolutional fusion. Experiments on V2V4Real and DAIR-V2X-Seq show that EgoRefine outperforms TraF-Align by 1.6 and 2.9 points on average in AP@0.5 and AP@0.7, respectively. The source code will be made publicly available at https://github.com/godk0509/EgoRefine.
I. INTRODUCTION
Asynchronous communication leaves cooperative features misaligned with the ego agent’s current observation, while existing prediction and fusion approaches do not fully address residual, spatially varying alignment quality. EgoRefine uses the current ego feature to guide delayed-feature alignment and reliability-aware fusion, outperforming TraF-Align across the tested latencies.
I. INTRODUCTION
- Communication latency makes cooperative features correspond to past timestamps; for moving objects, direct fusion can produce mislocalized boxes and duplicate detections.Existing compensation predicts mainly from the cooperative agent’s own history, leaving residual misalignment with the ego’s current observation, and common fusion methods do not directly use alignment-stage cues to assess local reliability.Source: §I · pp. 1–2
- EgoRefine treats the current ego feature as a temporal and spatial reference for both prediction and fusion, with trajectory discrepancy and directional refinement magnitude serving as alignment cues.The directional correction follows the predicted motion direction rather than using an independent unconstrained offset.Source: §I · p. 2
- EPA anchors delayed-feature prediction to the ego timestamp and refines sampling offsets along the predicted motion direction, while TRF uses trajectory discrepancy and refinement magnitude to adaptively reweight aligned features before fusion.TRF conditions the relation between aligned features to produce a spatial gate for ego and CAV representations.Source: §I · p. 2
- EgoRefine outperforms TraF-Align under every latency from 0 to 400 ms, with average gains of 1.6/2.9 points in AP@0.5/AP@0.7.The largest margins occur under the stricter AP@0.7 criterion, with no extra supervision, annotation, or communication overhead.Source: §I · p. 2
II. RELATED WORK
The related work situates EgoRefine among collaborative 3D detection and asynchronous perception methods. It distinguishes its focus on aligning delayed cooperative features to the ego timestamp and fusing them with reliability-aware weighting.
II. RELATED WORK
- The paper frames its work across two directions: collaborative perception for 3D detection and asynchronous collaborative perception under communication latency.Its focus is asynchronous feature-level detection, aligning delayed CAV features to the ego’s current timestamp before reliability-aware fusion.Source: §II · p. 2, A. Collaborative Perception for 3D… · p. 2
A. Collaborative Perception for 3D Object Detection
- Collaborative 3D detection methods share vehicle and infrastructure observations, with feature-level collaboration prominent for balancing accuracy and communication cost.Prior approaches include graph-based aggregation, transformer interaction, pose-error handling, spatial communication selection, and heterogeneous collaboration, while temporal inconsistency is not their primary focus.Source: A. Collaborative Perception for 3D… · p. 2
B. Asynchronous Collaborative Perception
Asynchronous collaboration relies on delayed cooperative features, but existing compensation is driven mainly by the transmitting agent’s history and fusion often omits alignment cues. EgoRefine uses current ego information to guide alignment and carries alignment cues into fusion.
B. Asynchronous Collaborative Perception
- Existing latency methods forecast or warp delayed features, but compensation relies mainly on transmitting-agent history and fusion does not directly reuse alignment cues.EgoRefine couples alignment and fusion using the current ego representation and cues produced during alignment.Source: B. Asynchronous Collaborative Perception · pp. 2–3
A. Problem Definition and Preliminaries
- At ego timestamp t, the system must compensate for a cooperative feature captured at t −τ before fusion and detection.Trajectory fields encode likely object locations and motion directions; offset generation converts that prior into sampling displacements for reconstruction.Source: A. Problem Definition and Preliminaries · p. 3
- Latency leaves two gaps: trajectory and offsets rely mainly on the transmitting agent’s history, and alignment cues are not explicitly reused despite spatially varying alignment quality.The resulting need is both predictive alignment and careful feature selection.Source: A. Problem Definition and Preliminaries · p. 3
B. Ego-referenced Predictive Alignment
- EPA keeps the ego stream independent while selectively injecting ego cues into the cooperative feature using local cross-agent discrepancy before trajectory-field prediction.Its residual update preserves the original cooperative representation, including regions observed only by the CAV.Source: B. Ego-referenced Predictive Alignment · p. 4
- The reconstructed ego stream supplies a current-time motion reference, while EPA passes trajectory fields and directional refinement steps onward as alignment cues.The ego representation is reconstructed along predicted motion trajectories, and the ego trajectory field guides the cooperative stream.Source: B. Ego-referenced Predictive Alignment · p. 4
- EPA predicts a signed scalar for each sampling point and applies it along the predicted trajectory direction, avoiding unconstrained lateral corrections.This reduces each residual correction from a two-dimensional vector to a scalar step that adjusts distance along the predicted motion direction.Source: B. Ego-referenced Predictive Alignment · p. 4
C. Trajectory-conditioned Reliability-aware Fusion
TRF uses motion-specific alignment cues from EPA to condition the relationship between aligned ego and cooperative features, then learns spatially varying fusion preferences before convolutional aggregation.
C. Trajectory-conditioned Reliability-aware Fusion
- TRF conditions feature relations on alignment cues because raw feature differences can reflect either residual misalignment or legitimate cross-view complementarity.The latter can arise from viewpoint, point density, or occlusion.Source: C. Trajectory-conditioned… · pp. 4–5
- The ego-CAV trajectory discrepancy directly compares trajectory structures inferred by the two streams, providing a more motion-specific alignment indicator than raw feature differences.It is one of two intermediate quantities reused from EPA without additional supervision.Source: C. Trajectory-conditioned… · p. 5
- The magnitude of directional refinement is averaged over sampling points; larger values indicate that base offsets require stronger ego-referenced correction and signal greater local alignment difficulty.This serves as a complementary cue to trajectory discrepancy.Source: C. Trajectory-conditioned… · p. 5
- A lightweight cue encoder uses the alignment cues to modulate the feature-relation representation, rather than concatenating cues with semantic features.The modulation lets similar feature discrepancies yield different fusion preferences according to inferred alignment status.Source: C. Trajectory-conditioned… · p. 5
- A spatial fusion gate assigns complementary ego and cooperative weights before a two-layer convolutional fusion operator aggregates the streams.The learned gate is a fusion preference, not an explicitly calibrated reliability probability.Source: C. Trajectory-conditioned… · p. 5
D. Joint Optimization of Alignment and Fusion
EPA and TRF form a jointly optimized differentiable path from temporally misaligned inputs to collaborative predictions, without module-specific objectives.
D. Joint Optimization of Alignment and Fusion
- EPA determines where delayed cooperative information is reconstructed, while TRF determines how that information contributes to the fused representation.The two stages form a unified differentiable path from misaligned inputs to collaborative predictions.Source: D. Joint Optimization of Alignment and… · p. 5
- Alignment and fusion are optimized jointly with the collaborative detector, allowing both stages to co-adapt to the detection task.No module-specific objectives are introduced.Source: D. Joint Optimization of Alignment and… · p. 5
A. Experimental Setup
Experiments use two real-world collaborative perception datasets and evaluate detection under fixed communication latencies with AP at two IoU thresholds.
A. Experimental Setup
- V2V4Real contains vehicle-to-vehicle sequences, while DAIR-V2X-Seq provides sequential vehicle-infrastructure observations at intersections.Both datasets support evaluation under temporal asynchrony, and the experiments follow their official splits and protocols.Source: A. Experimental Setup · pp. 5–6
- Table II studies EPA and TRF ablations on DAIR-V2X-Seq across communication latencies, using TraF-Align as the first-row baseline.The ablation reports AP@0.5/0.7.Source: Table II · p. 6
- AP@0.5 and AP@0.7 measure 3D detection at IoU thresholds 0.5 and 0.7, with AP@0.7 more sensitive to localization accuracy.The non-ego agent is evaluated at fixed latencies of 0, 100, 200, 300, and 400 ms.Source: A. Experimental Setup · p. 6
B. Comparison with State of the Art
EgoRefine outperforms TraF-Align on both datasets across evaluated latencies, with larger average gains at the stricter localization metric. The reported severe-latency and zero-latency comparisons also show gains in those settings.
B. Comparison with State of the Art
- 1.60/3.25 points on V2V4Real and 1.58/2.49 on DAIR-V2X-Seq: EgoRefine improves over TraF-Align in average AP@0.5/AP@0.7 across five latency settings.EgoRefine achieves the best AP@0.5 and AP@0.7 on both datasets at every evaluated latency.Source: B. Comparison with State of the Art · p. 6
- EPA anchors delayed-feature prediction to the current ego representation and constrains offset refinement along inferred trajectory direction; TRF uses trajectory discrepancy and refinement magnitude to modulate feature relations.The authors attribute the improvements to these complementary designs.Source: B. Comparison with State of the Art · p. 6
- At 400 ms, gains are 1.81/3.32 points on V2V4Real and 1.51/2.41 on DAIR-V2X-Seq in AP@0.5/AP@0.7.The advantage remains substantial under severe latency.Source: B. Comparison with State of the Art · p. 6
- At 0 ms, AP@0.7 gains are 3.39 points on V2V4Real and 3.00 on DAIR-V2X-Seq, where ego-referenced alignment and reliability-aware fusion also correct cross-view inconsistency.The paper concludes that ego-CAV feature consistency improves generally, with latency making inconsistency most severe.Source: B. Comparison with State of the Art · p. 6
C. Ablation Studies
EgoRefine improves over TraF-Align, with gains from both ego-guided predictive alignment and trajectory-conditioned fusion. Its EPA components also provide complementary benefits, while qualitative comparisons show stable detection as latency increases.
C. Ablation Studies
- 1.58/2.49 points are EgoRefine’s average AP@0.5/AP@0.7 gains over TraF-Align, with the larger gain at AP@0.7.The reported component averages are 1.38/1.80 points for EPA and a further 0.20/0.69 points from adding TRF, in AP@0.5/AP@0.7 order.Source: C. Ablation Studies · p. 6
- Feature-level Guidance improves AP@0.7 by 1.58 points on average, Trajectory-Directional Refinement by 1.44, and their combination by 1.80.The paper attributes the combined gain to complementary feature-level ego guidance and trajectory-level offset refinement.Source: C. Ablation Studies · pp. 6–7
- EgoRefine preserves more stable detection quality from 0 to 400 ms, while competing methods show more localization errors, missed detections, or inconsistent predictions.Fig. 4 compares CoBEVT, ERMVP, TraF-Align, and EgoRefine; green boxes mark ground truth and red boxes mark predictions.Source: Figure 4 · p. 7
- Fig. 5 examines αd and αt sensitivity using DAIR-V2X-Seq AP@0.7 averaged across 100–400 ms, with all EgoRefine components enabled.The caption specifies the hyperparameters and evaluation scope, but does not provide plotted values.Source: Figure 5 · p. 7
D. Hyperparameter Analysis
The analysis varies EPA’s feature-guidance and trajectory-refinement strengths one at a time while evaluating latency-averaged AP@0.7. Performance peaks at moderate settings and declines as either strength increases further.
D. Hyperparameter Analysis
- The sensitivity test varies one parameter at a time with all EgoRefine components enabled, fixes smax = 0.01, and averages AP@0.7 over 100–400 ms on DAIR-V2X-Seq.The other parameter is held fixed during each variation.Source: D. Hyperparameter Analysis · p. 7
- AP@0.7 first improves and then decreases as either guidance strength increases; excessive αt may weaken CAV complementary information, while excessive αd may overadjust base offsets.The reported pattern motivates moderate rather than stronger guidance and directional correction.Source: D. Hyperparameter Analysis · p. 7
- The best average AP@0.7 occurs at αt = 0.2, while αd = 0.3 gives the best trajectory-directional refinement performance; both values are used in all experiments.Source: D. Hyperparameter Analysis · p. 7
E. Qualitative Results
Qualitative comparisons show more accurate localization and stable detections across communication latencies, with differences from competing methods becoming clearer at higher latency. The conclusion reports consistent improvements on two real-world datasets and identifies broader multi-agent settings as future work.
E. Qualitative Results
- At 0 ms, EgoRefine gives more precise localization in highlighted regions; at 200 and 400 ms, it maintains better ground-truth alignment as competitors show more errors, missing or duplicated boxes, and unstable predictions.The qualitative examples compare CoBEVT, ERMVP, TraF-Align, and EgoRefine.Source: E. Qualitative Results · pp. 7–8
V. CONCLUSIONS
- The conclusion reports consistent improvements across V2V4Real and DAIR-V2X-Seq and proposes extending EgoRefine to multiple collaborators and dynamically varying communication conditions.The proposed future direction includes jointly modeling agent-wise alignment confidence and adaptive multi-agent fusion.Source: §V · p. 8