PaperlayerView exact revision on arXiv

Source-linked AI summary

FedCKA: Representation-Guided Layer Personalization for Federated 3D Perception Across Driving Domains

Jolle Verhoog, Ali Burak Ünal, Holger Caesar

arXiv:2610.01510v1cs.CVcs.RO

TL;DR

Federated 3D detection must address domain shifts and sparse labeled data, while existing personalization strategies rely on fixed partitions or ratios. FedCKA uses layer-wise CKA to select what to share, improving average NDS by 7 percentage points over the strongest competitor on the unified benchmark.

  • Problem

    Federated 3D object detection faces scarce labeled data in niche domains and heterogeneous location, weather, and illumination shifts that existing studies do not systematically isolate.

    Source: §I · p. 1

  • Method

    FedCKA uses layer-wise representation similarity to create client-specific masks that select which layers are globally aggregated and which remain personalized.

    Source: §I · p. 1, B. FedCKA System Architecture · p. 2

  • Results

    FedCKA improves average NDS by 7 percentage points over the strongest competitor.

    Source: §I · p. 1

  • Takeaways & Limitations

    On this benchmark, adaptive partial personalization can outperform a single consensus model, and representation similarity is a useful criterion for balancing global sharing and local adaptation.

    Source: §VIII · p. 8

    Evaluation is limited to one detector, dataset, and domain split, which is highly imbalanced.

    Source: §VII · p. 7

Abstract

from arXiv · show

Robust perception in intelligent vehicles demands 3D object detectors that remain dependable under domain shifts, such as changes in time of day, location, or weather. However, due to costly annotation and rare shifts, some environments lack sufficient data to train a standalone detector. Federated learning offers a privacy-preserving framework for collaborative model training, enabling clients to benefit from shared learning across diverse environments. Yet, this framework traditionally relies on a single global consensus model, which struggles to perform across heterogeneous local data distributions. Local conditions are better captured by adapting a subset of the model, but many personalization approaches rely on predefined layer partitions or fixed personalization ratios, thereby limiting adaptation to client-specific divergence. To reduce this rigidity, we propose FedCKA, a Centered Kernel Alignment (CKA)-based strategy that dynamically handles the personalization-globalization trade-off. Specifically, FedCKA computes layer-wise feature similarities between local client models and the global consensus model during training. By converting layer-wise similarity scores into client-specific aggregation masks, FedCKA selectively shares representation-consistent layers. Evaluation on a unified multi-domain benchmark based on nuScenes shows that FedCKA outperforms established federated baselines, including FedBN, FedRep, and FedSelect, improving average NDS by 7 percentage points over the strongest baseline. The findings offer both a comparative benchmark and a promising direction for robust federated 3D perception across shifts in location, weather, and illumination. Code is available at https://github.com/j-verhoog/FedCKA.

I. INTRODUCTION

The introduction motivates federated 3D object detection for scarce, privacy-constrained driving domains and identifies a gap in evaluating structured domain shifts. It presents a unified benchmark and FedCKA as a representation-guided approach to layer personalization.

I. INTRODUCTION

  • Scarce labeled data in niche domains, privacy barriers, and large multimodal data volumes motivate collaborative training rather than standalone or centralized approaches.Naive localized retraining is also computationally expensive.Source: §I · p. 1
  • Federated 3D-OD studies do not systematically isolate location-, weather-, and illumination-induced heterogeneity or compare personalization strategies in a unified benchmark.The study focuses on labeled client-level domain adaptation and leaves client reliability and sensor-configuration variability to future work.Source: §I · p. 1
  • FedCKA improves average NDS by 7 percentage points over the strongest competitor by identifying domain-sensitive layers.The work also establishes a unified multi-domain federated 3D object detection benchmark.Source: §I · p. 1

II. RELATED WORK

  • Prior personalization methods use fixed layer partitions or parameter- and gradient-based importance, while FedCKA selects personalized layers using representational consistency.FedBN and FedRep require predefined partitions; FedSelect and FedMC infer personalization from parameter updates or gradient magnitudes.Source: §II · p. 2

III. PROPOSED METHOD

FedCKA uses layer-wise representation similarity to decide which model layers to share and which to personalize. Clients compare locally updated models with a common previous global model, and the server aggregates only shared layers.

A. Motivation

  • FedCKA compares layer representations and masks the most dissimilar layers from global aggregation, allowing those layers to adapt locally while similar layers transfer shared knowledge.The approach compares activations rather than parameter-space differences and does not require fixed layer partitions or personalization ratios.Source: A. Motivation · p. 2

B. FedCKA System Architecture

  • At each communication round, clients locally train their models and compare the updates with the previous global model using layer-wise CKA on a small unlabeled subset of local training data.Using the previous global model gives clients a common reference for measuring layer divergence; clients send unmasked layers and a binary mask.Source: B. FedCKA System Architecture · p. 2
  • The server aggregates shared parameters layer by layer, then clients overwrite their unmasked layers with the updated global parameters while retaining personalized layers locally.This combines cross-client knowledge with client-specific adaptation.Source: B. FedCKA System Architecture · p. 2

C. Centered Kernel Alignment

FedCKA uses Centered Kernel Alignment to compare layer representations rather than parameter values. CKA normalizes empirical HSIC between the layers’ Gram matrices to measure representational similarity.

C. Centered Kernel Alignment

  • CKA compares activations rather than parameters, addressing cases where different layer parameterizations produce similar representations.In this work, it measures whether a locally trained client layer remains aligned with the previous global model.Source: C. Centered Kernel Alignment · p. 3
  • CKA computes representational similarity by normalizing empirical HSIC between Gram matrices formed from aligned layer outputs.The outputs have n aligned activation samples and feature dimensions p1 and p2.Source: C. Centered Kernel Alignment · p. 3

HSIC(K, K) HSIC(L, L)

CKA captures whether layers organize the same samples similarly, independent of neuron ordering. FedCKA converts these scores into personalization masks, with percentage-based and threshold-based strategies that differ in mask persistence and personalization footprint.

HSIC(K, K) HSIC(L, L)

  • CKA measures whether the same samples have similar relational structure across layers, regardless of neuron ordering.Mean-centering and normalization make it invariant to isotropic scaling and orthogonal transformations.Source: HSIC(K, K) HSIC(L, L) · p. 3

D. Personalization Strategies

  • For each layer, the server aggregates weights only from clients that keep it shared; if all clients personalize it, the previous global layer is retained.The mask value 1 denotes personalization.Source: D. Personalization Strategies · p. 3
  • Clients update their local models layer by layer using the selected personalization mask.The persistence of that mask across rounds depends on the chosen strategy.Source: D. Personalization Strategies · p. 3
  • Percentage-based masking selects the least-similar layers until their parameter count reaches a target ratio, and selected layers remain personalized across rounds.Because selection is layer-granular, the realized personalized ratio can exceed the target when the final selected layer crosses it.Source: D. Personalization Strategies · pp. 3–4
  • Threshold-based masking personalizes layers whose CKA falls below a chosen threshold, allowing the personalized parameter volume to vary by client and round.Unlike percentage-based masking, this mask is not cumulative, so a layer can become globally shared again when its similarity rises above the threshold.Source: D. Personalization Strategies · p. 4

A. Dataset

The benchmark assigns clients to five environmental domains spanning Boston and Singapore under varied weather and lighting conditions.

A. Dataset

  • Five client domains (A–E) span Boston and Singapore across varying weather and lighting conditions.The domains are defined in Table I.Source: A. Dataset · p. 4, Table I · p. 4
  • Each client holds data for one distinct domain, with scenes of roughly 40 annotated samples in a 20-second window.The domains also differ in scene count and class composition.Source: A. Dataset · p. 4
  • Singapore training data average three times as many annotated motorcycles, while Boston has more than twice as many cars.Domain E’s three validation scenes may yield noisy standalone metrics, but its compounded shifts and small training set provide a stress test for knowledge transfer.Source: A. Dataset · p. 4

B. Detector

The experiments use a fixed CMT detector and compare federated strategies across domain-specific clients. FedCKA leads the federated baselines on validation-scene-weighted NDS, with qualitative recovery shown under severe shift.

B. Detector

  • CMT is held constant across experiments to isolate the impact of federated strategies.The detector uses attention-based multimodal fusion and is described as robust to synthetic corruption.Source: B. Detector · p. 4

C. Federated Strategies

  • The benchmark includes globalization methods FedAvg, FedDyn, and PCGrad, plus layer-wise FedBN and FedRep and parameter-selection FedSelect and FedMC.All methods run for 40 rounds with one local epoch.Source: C. Federated Strategies · p. 4

D. Evaluation

  • mAP and NDS use only classes or attributes available in each validation domain; CAP provides a single-class comparison, and averages are weighted by validation-scene counts.Domains C, D, and E lack 1, 3, and 6 of the 10 target classes, respectively.Source: D. Evaluation · p. 4

V. EXPERIMENTS

  • 0.63 validation-scene-weighted NDS for FedCKA, versus 0.55 for FedSelect and 0.56 for FedMC, with FedCKA closest to centralized training.Table II compares federated methods with non-federated references.Source: §V · p. 4, Table II · p. 5
  • Under Singapore night rain, the FedCKA predictions recover detections missed by FedAvg in the BEV comparison.The figure places ground-truth boxes on the left, FedAvg predictions in the middle, and FedCKA predictions on the right.Source: Figure 3 · p. 5, §V · p. 5

A. Globalization versus Personalization

FedCKA selectively personalizes layers based on representation similarity, and the results favor adaptive personalization over a single global model or fixed personalization schemes. In cross-domain evaluation, it improves both own-domain and transfer performance.

A. Globalization versus Personalization

  • FedCKA achieves the highest performance across mAP, NDS, and CAP, while fixed BN- or head-only personalization performs comparably to globalization methods.BN and head layers have fewer personal parameters and do not influence the intermediate cross-modal merging stages; they may therefore lack enough capacity to move client models toward local optima.Source: A. Globalization versus Personalization · p. 5

B. Cross-Domain Generalization

  • Isolated models can transfer poorly: Model A reaches NDS 0.15 on domain E despite strong own-domain performance.This supports the benchmark’s focus on domain heterogeneity rather than local dataset size alone.Source: B. Cross-Domain Generalization · pp. 5–6
  • FedCKA improves both own-domain and cross-domain NDS over FedSelect and surpasses the previously evaluated federated methods.It is the only federated method for which all five clients improve over local-only training, and it has the highest average off-diagonal NDS.Source: Table III · p. 6, B. Cross-Domain Generalization · p. 6
  • FedAvg, FedDyn, and PCGrad sacrifice local performance on richer domains A–C while maintaining stable but limited cross-domain performance.Their cross-domain matrices have identical rows because all clients share one global model.Source: B. Cross-Domain Generalization · p. 6, Table III · p. 6
  • FedSelect improves average cross-domain transfer over isolated training, showing that personalized models can benefit from shared knowledge.Source: B. Cross-Domain Generalization · p. 6

C. Masking Strategy Ablation

The ablation frames masking as a tunable sharing-specialization trade-off, while the footprint analysis shows that FedCKA prioritizes dynamic personalization over communication compression and adds computation.

C. Masking Strategy Ablation

  • The ρ = 0.40 final personalization target serves as the ablation reference for comparing FedSelect’s and FedMC’s fixed personalization ratio; masks remain stable across repeated runs, isolating the same structural components.Source: C. Masking Strategy Ablation · p. 6

D. Communication and Computation Footprint

  • FedCKA's upload payload is comparable to full-model aggregation, underscoring that its objective is dynamic personalization rather than communication compression.Table V reports cumulative client-to-server upload over 40 rounds with five clients and a 125 MB base model.Source: D. Communication and Computation… · p. 6, Table V · p. 7
  • At matched personalization ratios, FedCKA incurs lower masking metadata because its masks operate at the layer rather than parameter level.Parameter-level masking methods reduce transmitted model payload more substantially in the reported experiments because they use higher personalized parameter fractions.Source: D. Communication and Computation… · p. 7
  • 14,000 seconds (2.7% of total training time) and 6 GB peak VRAM are FedCKA's added CKA-computation costs.The additional cost comes from client-side forward passes to compute layer-wise CKA similarities.Source: D. Communication and Computation… · p. 7

E. Robustness Analysis

Repeated-run and sample-size tests support stable FedCKA behavior in the evaluated setting. The discussion links its performance to similarity-guided personalization and finds that learned masks leave task heads largely shared.

E. Robustness Analysis

  • 0.635±0.008 average NDS and 0.810 ± 0.014 CAP leave FedCKA the strongest personalized federated method across three repeated runs.The reported performance margin over competing personalized methods exceeds the corresponding run-to-run standard deviations.Source: E. Robustness Analysis · p. 7
  • Across three runs, 10 randomly sampled inputs produced the same layer-selection decisions as 100, but this sample choice is specific to the evaluated dataset and detector.Masking decisions are reevaluated every round, allowing layers personalized by noisy CKA estimates to revert to global aggregation.Source: E. Robustness Analysis · p. 7

VI. DISCUSSION

  • On this benchmark, global aggregation underperforms certain personalization methods, and representation similarity further boosts performance over magnitude- or gradient-based personalization.The reported gains include out-of-domain and smaller-domain performance.Source: §VI · p. 7
  • FedCKA replaces hand-picked layer types or fixed ratios with measured representation similarity, comparing activations rather than assuming same-index parameters serve the same function.Source: §VI · p. 7
  • Learned masks frequently personalize normalization and deeper representation layers while leaving the task head largely shared, preserving a common detection-task structure.The discussion suggests shared task heads may preserve knowledge of classes absent from particular local domains.Source: §VI · p. 7

VII. LIMITATIONS

The study’s scope is limited by its evaluation setting, supervised-label assumption, and added client computation. Its conclusion reports benchmark-specific benefits while identifying broader evaluation and privacy questions for future work.

VII. LIMITATIONS

  • Evaluation is restricted to one detector, dataset, and domain split, with an imbalanced split that may make smaller-domain gains depend on scene distribution and validation composition.Source: §VII · p. 7
  • The framework assumes supervised local training with reliable client-side labels or a labeling pipeline, and does not evaluate pseudo-label noise from offline autolabelling.Source: §VII · p. 7
  • CKA adds client computation and flattening representations may simplify positional information, although flattened features remain effective for layer-wise similarity in the evaluated setting.Source: §VII · p. 8

VIII. CONCLUSION AND FUTURE WORK

  • Under weather, time, and location shifts, adaptive partial personalization outperforms a single consensus model on this benchmark; representation similarity is a useful criterion for balancing shared and local adaptation.Source: §VIII · p. 8
  • Benefits across other detectors, datasets, domain definitions, and heterogeneity types remain to be evaluated; proposed extensions include adaptive thresholds, smoothed masks, and CKA-based domain clustering.Source: §VIII · p. 8
  • FedCKA alone provides no formal privacy guarantees, making combination with differential privacy an important future direction.Source: §VIII · p. 8
Loading 2610.01510v1…