PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Distributed Eyes: Leveraging Collaborative Perception for Automated Model Adaptation

Yanan Ma, Yihang Tao, Zhengru Fang, Zihan Fang, Yiqin Deng, Xianhao Chen, Yuguang Fang

arXiv:2609.18511v1cs.CVcs.LG

TL;DR

Autonomous-driving perception models struggle to generalize to new environments, while ego-only adaptation can yield unreliable pseudo-labels. LDE uses collaborative perception for model-adaptation supervision, with selective feature sharing, FoV filtering, and confidence-based curriculum learning; it outperforms pre-trained models and existing unsupervised adaptation methods in the reported 3D detection evaluations.

  • Problem

    Ego-only test-time adaptation can produce unreliable pseudo-labels and is limited by the vehicle’s view, while manual labeling for new environments is costly.

    Source: §I · p. 1

  • Method

    LDE uses collaborative perception as teacher supervision, combining adaptation-oriented feature sharing, FoV-based label filtering, and confidence-based curriculum learning.

    Source: §I · p. 2

  • Results

    LDE consistently improves performance over pre-trained models and existing unsupervised adaptation methods on 3D object detection tasks.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    The evaluations support CP-generated pseudo-labels as a source for automated perception-model adaptation without manual labeling.

    Source: §VI · pp. 7–8

Abstract

from arXiv · show

In autonomous driving, perception models often struggle to generalize to new environments due to domain shifts. While unsupervised model adaptation offers a feasible solution without labor-intensive manual labeling, existing methods that rely solely on the ego-vehicle's data often lead to inferior pseudo-labeling performance. To address this critical issue, we propose LDE, Learning from Distributed ``Eyes", a novel framework that transforms collaborative perception (CP) into a source of high-quality supervision for model adaptation. This pseudo-labeling approach is hyperparameter-insensitive and relatively reliable, assuming CP often outperforms single-agent's perception. However, naively implementing this approach encounters (1) the communication bottleneck of sharing rich features under time and bandwidth constraints, (2) the view discrepancy between the CP view and the learner's Field of View (FoV), and (3) the unreliability even in CP-generated labels. To address these issues, we design an adaptation-oriented feature sharing mechanism that selectively transmits the most critical information for adaptation, an FoV filtering method that meticulously eliminates mismatched labels, and a curriculum learning strategy to progressively exploit pseudo labels. Extensive experiments on 3D object detection tasks demonstrate that LDE consistently outperforms both the pre-trained models and state-of-the-art unsupervised adaptation methods.

I. INTRODUCTION

LDE uses collaborative perception (CP) as a source of pseudo-labels for adapting an ego vehicle’s perception model without manual annotation. It addresses communication, field-of-view mismatch, and label-reliability challenges with selective feature sharing, filtering, and curriculum learning.

I. INTRODUCTION

  • Existing test-time adaptation can produce unreliable pseudo-labels and is limited by the ego vehicle’s view, which hampers labeling occluded objects and resolving perceptual ambiguities.These limitations constrain unsupervised adaptation for autonomous driving.Source: §I · pp. 1–2
  • CP aggregates multiple vehicles’ data and can provide teacher predictions for ego-model adaptation, assuming it often outperforms single-vehicle perception.The paper characterizes this pseudo-labeling approach as hyperparameter-insensitive and relatively reliable.Source: §I · p. 2
  • LDE selectively shares adaptation-relevant features under communication constraints, filters labels outside the learner’s FoV, and progressively uses pseudo-labels through confidence-based curriculum learning.The design targets communication bottlenecks, view discrepancies, and noise in CP-generated labels.Source: §I · p. 2
  • LDE outperforms non-adaptive baselines and existing unsupervised benchmarks on 3D object detection tasks.Evaluations use the V2X-Sim and DAIR-V2X datasets.Source: §I · p. 2

II. RELATED WORK

Prior unsupervised adaptation methods have limited applicability to autonomous driving, while collaborative-perception research has not used CP outputs to adapt an individual vehicle’s model. Directly applying CP to adaptation also raises feature-selection, view-mismatch, and label-reliability issues.

A. Test-time Adaptation for Object Detection

  • Prior test-time adaptation approaches have limited scopes, including dependence on better predictions, repeated route traversals, or shared CAV pose and shape information.The review notes that these constraints affect their applicability to individual vehicles and object classes beyond CAVs.Source: A. Test-time Adaptation for Object… · p. 2

B. Collaborative Perception

  • Existing collaborative-perception work improves communication efficiency or model-agnostic fusion, but does not use CP to adapt an individual vehicle’s model.Examples include feature sparsification, graph-based aggregation, and cross-model feature transformation.Source: B. Collaborative Perception · pp. 2–3
  • Directly applying CP to adaptation risks transmitting features of low adaptation value, using out-of-view detections, and overfitting to unreliable pseudo-labels.These are the three issues identified for adapting an individual learner with existing CP methods.Source: B. Collaborative Perception · p. 3

III. PROBLEM DEFINITION

The problem is to adapt an ego vehicle’s source-pretrained predictor to an unlabeled target dataset, where unavailable ground truth prevents direct optimization of target risk. LDE considers collaboration with surrounding CAVs and uses CP results as pseudo-labels for adaptation.

III. PROBLEM DEFINITION

  • The objective is to find an ego-vehicle predictor that minimizes empirical risk on samples from the target dataset.The predictor belongs to a hypothesis class, and the objective uses a detection loss.Source: §III · p. 3
  • Target-domain ground-truth labels are unavailable, making direct optimization of the stated objective infeasible.The unlabeled target dataset is the setting for adaptation.Source: §III · p. 3
  • The learner collaborates with a movement-dependent set of surrounding CAVs that transmit sparse intermediate features, and the resulting CP predictions serve as adaptation pseudo-labels.The passage motivates their use by stating that CP results are generally more accurate than the learner’s predictions alone.Source: §III · p. 3

IV. PROPOSED LDE FRAMEWORK

LDE adapts a learner through a five-step pipeline: CAVs create spatial utility maps, the learner requests selected features, and fusion produces collaborative representations. FoV filtering and confidence-based curriculum learning then support progressive adaptation.

IV. PROPOSED LDE FRAMEWORK

  • Each CAV extracts features and generates a spatial utility map, which informs the learner’s selective requests for neighboring features under communication constraints.The framework then fuses the received features.Source: §IV · p. 3
  • A FoV filtering stage removes out-of-range pseudo-labels that are not visible to the learner.This follows feature fusion in the pipeline.Source: §IV · p. 3
  • Confidence-based curriculum learning progressively adapts the learner’s model using pseudo-labels.This is the final step in the described pipeline.Source: §IV · p. 3

A. Feature and Utility Map Generation

LDE represents vehicle observations as globally aligned BEV features and utility maps, then selects and fuses sparse neighboring-vehicle features to produce collaborative predictions for adaptation.

A. Feature and Utility Map Generation

  • Each CAV encodes sensor observations as BEV feature maps in a unified global coordinate system and derives a spatial utility map with a detection decoder.The learner also computes its own feature and utility maps.Source: A. Feature and Utility Map Generation · pp. 3–4

B. Adaptation-oriented Feature Sharing and Fusion

  • Neighboring CAV feature grids are selected by utility gain under a latency budget, prioritizing cells with the highest utility-to-latency ratios.The learner ranks candidate cells and selects them until the budget is exhausted or all candidates have been evaluated.Source: B. Adaptation-oriented Feature Sharing… · p. 4
  • A transformer-based fusion module aggregates the spatially aligned sparse features, and a detection decoder turns the fused map into 3D boxes with confidence scores.Detections below the predefined confidence threshold are discarded.Source: B. Adaptation-oriented Feature Sharing… · p. 4

C. FoV-based Pseudo Label Filtering

LDE filters collaborative pseudo-labels to align them with the learner’s view, using LiDAR line-of-sight checks and conservative box processing to assess visibility and occlusion.

C. FoV-based Pseudo Label Filtering

  • Collaborative predictions can include objects outside the learner’s effective FoV, so the method retains only detections judged visible to the learner.Out-of-view objects, including those in blind spots, can mislead adaptation.Source: C. FoV-based Pseudo Label Filtering · pp. 4–5
  • Boxes are processed from nearest to farthest and conservatively shrunk, with stronger shrinkage at longer range to reduce false positives from noisy detections.The shrinkage also mitigates potential occlusion errors from bounding-box inaccuracies and shape mismatches.Source: C. FoV-based Pseudo Label Filtering · p. 5
  • LiDAR ray-casting checks visibility toward each shrunken box, then BEV overlap with closer accepted boxes is used to reject potential misdetections.A box is considered partially visible if at least one ray reaches it unobstructed.Source: C. FoV-based Pseudo Label Filtering · p. 5

D. Confidence-based Curriculum Learning

LDE progressively expands the pseudo-label set during adaptation by lowering a confidence threshold from an initially high value.

D. Confidence-based Curriculum Learning

  • Adaptation begins with a high confidence threshold to select a high-quality subset containing the most certain detections.The method uses a dynamic threshold rather than a single fixed confidence cutoff.Source: D. Confidence-based Curriculum Learning · p. 5
  • The confidence threshold is gradually decayed to introduce lower-confidence samples and extend learning to broader, more challenging data distributions.The intended progression transfers knowledge from the initial reliable subset.Source: D. Confidence-based Curriculum Learning · p. 5
  • The curriculum is intended to prevent early-stage model corruption and make adaptation more stable and robust.Source: D. Confidence-based Curriculum Learning · p. 5

A. Experimental Setup

Experiments use simulated V2X-Sim and real-world DAIR-V2X collaborative-perception datasets, with PointPillars as the default detector. The evaluation adapts a pretrained model using collaborative predictions as pseudo-labels and reports detection AP at two IoU thresholds.

A. Experimental Setup

  • The experiments use V2X-Sim and DAIR-V2X, with PointPillars as the default LiDAR-based 3D detector.Source: A. Experimental Setup · p. 5
  • V2X-Sim contains 10,000 frames from 5 CAVs and 501,000 annotated 3D boxes; its BEV representation is 256 × 256 × 13 at 0.4 m/pixel.The dataset is simulated using SUMO and CARLA.Source: A. Experimental Setup · p. 5
  • DAIR-V2X pairs synchronized vehicle and infrastructure data over a 201.6 m × 80 m perception range, represented as a 200 × 504 × 64 BEV map at 0.4 m/pixel.The dataset is real-world.Source: A. Experimental Setup · p. 5
  • Detection accuracy is evaluated using AP@0.3 and AP@0.5; the label-quality table reports recall and precision at IoU@0.3 and IoU@0.5.Source: A. Experimental Setup · p. 6, Table II · p. 6

B. Quantitative Evaluation

LDE outperforms non-adaptive and unsupervised adaptation baselines across both datasets and initial-model setups, while ablations support its feature selection, FoV filtering, and curriculum components. More communication time improves adaptation, and label-filtering confidence involves a trade-off between noisy and excluded labels.

B. Quantitative Evaluation

  • 55.60% AP@0.3 and 49.11% AP@0.5 on DAIR-V2X Setup 1; on V2X-Sim Setup 1, LDE gains 18.95% AP@0.3 and 17.51% AP@0.5 over CP4Adaptation.Across both datasets and setups, LDE substantially outperforms existing TTA baselines; LDE-Full has the highest unsupervised accuracy and narrows the gap to the fully supervised upper bound.Source: B. Quantitative Evaluation · p. 6
  • At IoU@0.3 on V2X-Sim, LDE's pseudo-labels achieve 79.20% recall and 64.72% precision, exceeding existing unsupervised methods.The authors attribute the improvement to multi-vehicle spatially diverse features that mitigate occlusions and false negatives.Source: B. Quantitative Evaluation · p. 6

C. Ablation Study

  • Random feature selection wastes bandwidth and lowers pseudo-label quality; removing FoV filtering or confidence-based curriculum learning harms adaptation.FoV filtering avoids misleading labels outside the learner's view, while curriculum learning starts with reliable labels to limit early error accumulation.Source: C. Ablation Study · pp. 6–7, Table III · p. 7

B. Quantitative Evaluation

  • LDE's visualized detections suppress erroneous predictions and improve localization relative to the original model and baselines in complex driving scenes.Source: Figure 3 · p. 7

C. Ablation Study

  • Increasing the communication time allocation improves pseudo-label quality and adaptation performance as the learner aggregates richer collaborative information.Source: Table IV · p. 7, C. Ablation Study · p. 7
  • Bounding-box shrinkage has little effect on detection accuracy, whereas confidence thresholds that are too low or too high degrade adaptation.Low thresholds admit noisy pseudo-labels; high thresholds exclude informative labels.Source: C. Ablation Study · p. 7, Table V · p. 7

D. Qualitative Analysis

LDE suppresses false detections and improves box localization in qualitative comparisons; the conclusion reports consistent gains over pre-trained models and existing unsupervised adaptation methods. The authors also identify potential applications beyond 3D object detection.

D. Qualitative Analysis

  • In Fig. 3, LDE suppresses erroneous detections and produces more accurate bounding-box localization than the compared baselines.The baselines frequently show false positives and poor localization in complex driving scenes.Source: D. Qualitative Analysis · p. 7

VI. CONCLUSION

  • LDE delivers robust, consistent performance improvements over pre-trained models and existing unsupervised adaptation methods.The conclusion reports this result from extensive simulations.Source: §VI · p. 8
  • LDE uses collaborative perception for pseudo-labels, with adaptation-oriented feature sharing, field-of-view filtering, and confidence-based curriculum learning.These components address communication constraints, view discrepancies, and label noise, respectively.Source: §VI · p. 8
  • The authors suggest LDE may extend beyond autonomous-driving 3D object detection to depth estimation, segmentation, drones, and robotic systems.Source: §VI · p. 8

Related papers

Loading 2609.18511v1…