PaperlayerView exact revision on arXiv

Source-linked AI summary

From Classifier Heads to Encoder Prototypes: Personalized Federated Learning with Head-Derived Clustering

Amirardalan Dehghanpour, Arian Noori, Ehsan Tanghatari, Naimeh Omidvar, Hamid R. Rabiee

arXiv:2610.30075v1cs.LG

TL;DR

Personalized federated learning must determine which clients should share representations, since a shared encoder can limit clients with different data conditions. HDC-FL uses cached private-head geometry to group clients and return soft mixtures of encoder prototypes while preserving a single shared training trajectory. Across 11 configurations, it is competitive in mean accuracy and achieves the highest macro-averaged worst-client and bottom-20% accuracy among evaluated methods.

  • Problem

    Personalized FL must determine which clients should share representations, because a shared encoder can limit clients with different label support or acquisition conditions.

    Source: §1 · p. 2

  • Method

    HDC-FL groups clients using cached private-head geometry and returns each client a soft mixture of encoder prototypes paired with its unchanged head.

    Source: §1 · p. 2

  • Results

    Across 11 configurations, HDC-FL is competitive with the strongest baselines in macro-averaged mean accuracy and attains the highest macro-averaged worst-client and bottom-20% accuracy.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Client-level tail metrics distinguish methods that can appear similar when compared only by mean accuracy.

    Source: §1 · p. 2

    The evaluated setting assumes a common encoder and label space, short local updates with frequent aggregation, and server access to client-specific heads.

    Source: §5.1 · pp. 10–11

Abstract

from arXiv · show

Personalized federated learning must decide which clients should share representations. We introduce HDC-FL (Head-Derived Clustering for Personalized Federated Learning), which reads that decision off a signal shared-encoder training already produces: the client-specific classifi- er heads. At periodic refresh rounds the server projects the cached heads into a normalized low-rank descriptor space, groups the descriptors, builds one encoder prototype per group from the cached client encoders, and returns to each client a soft mixture of those prototypes paired with its own unchanged head. Client-specific heads are never averaged or distributed to oth- er clients; the shared-encoder trajectory is left intact; and no second optimization trajectory, additional local objective, or post-training fine-tuning stage is required. Across 11 CIFAR-10 and CIFAR-100 configurations covering label-subset, Dirichlet and rotation heterogeneity with ResNet-18 and MobileNetV2 encoders, HDC-FL is competitive with the strongest personalized and clustered baselines in macro-averaged mean accuracy and attains the numerically highest macro-averaged worst-client and bottom-20% accuracy among the evaluated methods. The tail results distinguish methods that look similar when only their means are compared. A controlled donor-swap diagnostic further finds positive donor-ranking associations between head similarity and encoder compatibility in all three settings tested, most clearly under covariate shift.

1 Introduction

Personalized federated learning must decide which clients should share representations because a shared encoder can limit clients with different label support or acquisition conditions, while mean accuracy can hide poor service to some clients. HDC-FL uses private-head geometry to form soft encoder-prototype mixtures, retaining one shared training trajectory; across 11 configurations it is competitive in mean accuracy and consistently strong on client-level tails.

1 Introduction

  • Personalized FL must choose which clients share representations because a shared encoder can limit clients with different label support or acquisition conditions.Mean-only evaluation can mask severe degradation for the worst-performing clients.Source: §1 · pp. 1–2
  • Existing affinity signals may vary with local updates, require extra computation or communication, or impose hard group assignments.The challenge is to obtain stable relatedness from information already produced during training and support graded sharing without separate group trajectories.Source: §1 · p. 2
  • HDC-FL reads client relatedness from private-head geometry, which can reflect label support and covariate shifts in clients’ class directions.The signal comes from the client-specific heads produced along the shared-encoder trajectory, without an additional data summary or local objective.Source: §1 · p. 2
  • Across 11 configurations, HDC-FL is competitive with the strongest evaluated baselines in macro-averaged mean accuracy while providing consistently strong worst-client and bottom-20% accuracy.Tail results distinguish methods that appear similar when only their means are compared.Source: §1 · p. 2
  • HDC-FL forms soft mixtures of encoder prototypes from low-rank head descriptors while keeping heads private and avoiding multiple training trajectories.The approach does not change the local objective or require a second local training stage.Source: §1 · p. 2

2 Related Work

HDC-FL combines model decomposition with similarity-based collaboration: it keeps client-specific heads and uses their geometry to decide which clients share encoder information. Unlike approaches that personalize decision layers or rely on hard assignments, it mixes encoder prototypes from one shared optimization trajectory.

Shared representations and client-specific components

  • Shared-encoder methods retain one representation for all clients, while HDC-FL uses private-head geometry to determine which clients should share encoder information.FedPer, FedRep, and FedBABU use client-specific prediction layers; FedBN keeps batch normalization local to address feature shift.Source: §2 · p. 3

From hard clustering to soft association

  • Clustered methods infer client groups from different signals, while graded-collaboration methods allow broader sharing but may use time-varying evidence or maintain multiple cluster models.HDC-FL instead mixes encoder prototypes drawn from one shared optimization trajectory.Source: §2 · p. 3

Classifier-geometry-based collaboration

  • PFedCS and FedPAC personalize decision layers, whereas HDC-FL keeps heads private and uses their geometry to mix encoder prototypes.An HDC-FL encoder prototype averages encoder parameters; FedProto instead communicates class-level feature prototypes.Source: §2 · p. 3

3 HDC-FL

HDC-FL adds server-side personalization to shared-encoder training: cached client heads determine groups, whose cached encoders form prototypes mixed for each client. Personalization leaves the shared training trajectory and local objective unchanged.

3 HDC-FL

  • At refreshes, HDC-FL converts cached heads into client descriptors, groups them, builds encoder prototypes from cached encoders, and returns soft prototype mixtures paired with clients’ own heads.The personalization step does not change the local objective or introduce a separate optimization trajectory.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Setting and Cached States

  • The server caches each observed client’s most recent encoder–head pair, refreshed together so the head was trained on the encoder stored beside it.Under partial participation, clients not selected in the current round remain represented by their latest cached state.Source: §3.1 · p. 4

3.2 Head-Derived Descriptors

  • Short local updates keep client encoders near a common trajectory, making their associated linear-head coordinates approximately aligned for comparison.Head geometry can reflect label support or, under covariate shift, changing class directions despite similar label frequencies.Source: §3.2 · pp. 4–5
  • The low-rank projection reduces descriptors from Cd to rd values and yields higher silhouette scores and greater agreement between consecutive cluster assignments than raw heads.Mean predictive accuracy changes little, so the projection is used primarily for compact, empirically more stable grouping.Source: §3.2 · p. 5

3.3 Clustering and Soft Membership

  • The number of groups is selected from descriptor geometry at each refresh, while soft membership lets a client draw on multiple prototypes.The temperature controls whether weights concentrate on the most similar prototype or spread across groups.Source: §3.3 · p. 5

3.4 Encoder Prototypes and Personalization

  • Each group’s prototype averages its cached encoders weighted by local dataset size, and clients combine prototypes using membership weights.The personalized model is not written back into cached states, preserving one shared training trajectory; averaging is supported in the studied regime of short local updates and frequent aggregation.Source: §3.4 · p. 6

3.5 Refresh Schedule and Algorithm

  • Personalization waits until clustering is stable across refreshes, then delivers personalized encoders; the procedure adds server-side computation, storage, and communication but no additional client-side optimization.Stability uses silhouette score, consecutive-assignment agreement, and descriptor change, with a maximum activation round.Source: §3.5 · pp. 6–7

4 Experiments

Across 11 configurations, HDC-FL is comparable to the strongest methods in mean accuracy and leads the aggregate client-tail metrics. Ablations and donor swaps examine head-derived grouping, soft membership, and the relationship between head similarity and encoder compatibility.

4.1 Setup

  • The evaluation spans CIFAR-10 and CIFAR-100 with label-subset, Dirichlet, and rotation heterogeneity, using ResNet-18 and MobileNetV2 encoders.It uses 20 clients and reports mean, worst-client, and bottom-20% accuracy; the latter averages the four lowest clients.Source: §4.1 · pp. 7–8

4.2 Mean Client Accuracy

  • 88.08%, 87.86%, and 87.58% are the macro-averaged mean accuracies of HDC-FL, PFedCS, and FedBABU, respectively; the paper treats them as comparable given seed variation.HDC-FL and FedBABU also tie for best average rank on the eight ResNet-18 settings, at 2.12.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 Client-Level Tail Performance

  • FedBABU’s 87.58% aggregate mean accompanies 55.32% worst-client accuracy and a 38.37-point fairness gap; IFCA and EMD-CFL fall below FedAvg-SE in worst-client accuracy.These results illustrate how mean accuracy can conceal weak client-level outcomes.Source: §4.3 · pp. 8–9
  • 82.05% worst-client accuracy and 84.44% bottom-20% accuracy are HDC-FL’s highest aggregate results across 11 configurations.Its margins over PFedCS are 0.36 and 0.24 percentage points, respectively, and their fairness gaps are nearly identical.Source: §4.3 · p. 8, Table 2 · p. 9

4.4 Ablations

  • On CIFAR-10 rotation, four head-derived groups add 0.74 mean-accuracy points beyond population-level prototype averaging, improving both tail metrics as well.Prototype averaging alone raises mean accuracy from 77.57% to 80.30%; low-rank projection improves cluster separation and temporal stability without materially changing predictive accuracy.Source: §4.4 · p. 9
  • On CIFAR-100 with α = 0.3, hard assignment costs 0.85 mean-accuracy points and 1.08 and 0.95 points on the two tail metrics versus soft mixing.This setting has continuously varying client distributions rather than discrete groups.Source: §4.4 · p. 9

4.5 Does Head Similarity Predict Encoder Compatibility?

  • Head similarity positively ranks donor-transfer benefit in all three tested settings; the strongest reported association is under rotation, with ρ = 0.243 and AUROC = 0.783.Nearest-head donor selection gains 2.50 accuracy points under rotation versus a random donor, but label-skew associations are largely explained by label-support overlap.Source: §4.5 · p. 10

5 Discussion

The discussion supports a narrower claim than uniform superiority: head geometry is a useful model-side signal for representation sharing, with positive donor-ranking associations in tested settings. The method’s evaluated scope is bounded by architectural, information, and scale assumptions.

5 Discussion

  • HDC-FL is not uniformly more accurate than the strongest evaluated baselines, and cross-method comparisons cannot isolate head geometry from other algorithmic differences.The donor-swap diagnostic finds positive ranking associations in all three settings, but under label skew much of the association is explained by shared label support.Source: §5 · p. 10

5.1 Limitations

  • The method assumes a common encoder and label space, short local updates with frequent aggregation, and server access to client-specific classifier heads.The current implementation is not suited to settings where model parameters must remain hidden from the server.Source: §5.1 · pp. 10–11
  • Evaluation uses 20 clients; substantially larger and naturally partitioned populations remain an untested scale setting.The authors note that server-cache growth and less frequent client refreshes are relevant at larger scale.Source: §5.1 · p. 11

6 Conclusion

HDC-FL uses private-head geometry to guide representation sharing, pairing each client’s unchanged head with a soft mixture of encoder prototypes. Across 11 configurations and two backbones, it is competitive in mean accuracy and achieves the highest macro-averaged worst-client and bottom-20% accuracy, with small margins over PFedCS and no additional client-side optimization.

6 Conclusion

  • Across 11 configurations and two backbones, HDC-FL is competitive with the strongest baselines in mean accuracy and has the highest macro-averaged worst-client and bottom-20% accuracy.Its margins over PFedCS are small, and it requires no additional client-side optimization.Source: §6 · p. 11

A EXPERIMENTAL DETAILS

Experiments use client-specific CIFAR partitions and a shared training protocol, while HDC-FL selects prototype counts from descriptor geometry and delays personalization until grouping is stable. The section also specifies evaluation metrics and reports the MobileNetV2 mean-accuracy results in Table 6.

A EXPERIMENTAL DETAILS

  • The 20-client experiments cover label-subset, Dirichlet, and rotation partitions, using official CIFAR splits and client-private test sets matching each client’s training distribution.Label-subset clients own 20% or 30% of classes; rotation assigns five clients to each of four rotations.Source: A EXPERIMENTAL DETAILS · p. 14

A.1 Dynamic Cluster Selection

  • HDC-FL selects the prototype-group count by choosing the highest-silhouette k-means clustering among 2–8 candidates, allowing the count to change as descriptor geometry evolves.Hard assignments construct distinct encoder prototypes, while personalization remains soft through prototype-membership weights.Source: §A.1 · p. 15

A.2 Stability-Based Warmup and Activation

  • Prototype personalization waits for sufficient cluster separation and either persistent assignments or small descriptor drift, with stability required across consecutive refresh checks.The maximum activation round provides a fallback if the stability criterion has not held for the required checks.Source: §A.2 · p. 16
  • Baselines share partitions, initialization, participation, communication rounds, and local-training budgets, while method-specific optimization stages are adapted to the common protocol.The reported metrics include mean, worst-client, and bottom-20% accuracy, plus the fairness gap.Source: §A.2 · p. 17

B MOBILENETV2 MEAN ACCURACY

  • Table 6 reports MobileNetV2 mean client test accuracy across three seeds, with best and second-best entries marked for each setting.The accompanying text identifies these three settings as the MobileNetV2 configurations omitted from Table 1.Source: B MOBILENETV2 MEAN ACCURACY · p. 17, Table 6 · p. 18

C SETTING-LEVEL TAIL RESULTS

The setting-level tail tables report worst-client accuracy, bottom-20% accuracy, and fairness gap across configurations. The reported tail variability differs substantially by method, while HDC-FL and PFedCS have nearly tied fairness gaps.

C SETTING-LEVEL TAIL RESULTS

D ADDITIONAL TRAINING DYNAMICS AND DIAGNOSTICS

The training-dynamics figures track accuracy before and after personalization activation, and the grouping diagnostics trace how clusters evolve. Reported accuracy gains are stable after activation, while diagnostic trends indicate increasingly structured and confident groupings.

D ADDITIONAL TRAINING DYNAMICS AND DIAGNOSTICS

  • Figure 2’s HDC-FL accuracy curve switches from the shared trajectory to the personalized prototype mixture at activation, so it depicts evaluated-model accuracy rather than optimization speed.Prototypes are rebuilt from the current cache at each refresh.Source: D ADDITIONAL TRAINING DYNAMICS AND… · p. 19, Figure 2 · p. 20
  • After activation, HDC-FL improves or remains level in every remaining configuration, with larger activation steps in label-subset and rotation settings than in Dirichlet settings.For the three representative settings, the activation step is most visible on CIFAR-10 rotation and CIFAR-100 LS3, and smaller on CIFAR-100 α = 0.3.Source: D ADDITIONAL TRAINING DYNAMICS AND… · p. 19, Figure 3 · p. 21
  • On CIFAR-10 rotation without pretrained initialization, silhouette rises, consecutive-refresh ARI remains high after settling, and membership entropy falls as clients associate more confidently with prototypes.The personalization gain peaks just after activation and decreases as the shared encoder catches up.Source: D ADDITIONAL TRAINING DYNAMICS AND… · p. 19, Figure 4 · p. 22

E ADDITIONAL ABLATIONS AND DESCRIPTOR DIAGNOSTICS

The ablations separate descriptor quality from predictive performance and examine how rank, prototype membership, and mixing strength affect HDC-FL. The rank-8 projection improves clustering separation and refresh stability, while soft mixing and the tested full prototype weight perform better in their respective diagnostics.

E.1 Low-Rank Descriptor

  • 85.24% against 85.27% accuracy accompanies silhouette rising from 0.027 to 0.108 and adjusted Rand index from 0.152 to 0.401 with the rank-8 descriptor versus raw heads.The projection improves grouping separation and stability between refreshes, not predictive accuracy.Source: §E.1 · p. 23, Table 10 · p. 23

E.2 Descriptor Rank

  • Predictive performance varies by less than one point across ranks, while silhouette decreases monotonically as rank increases; r = 8 has the highest silhouette among tested values.This single-seed diagnostic uses CIFAR-100 rotation without pretrained initialization.Source: §E.2 · p. 23, Table 11 · p. 23

E.3 Prototype Membership

  • Soft mixing beats hard assignment on all three metrics on CIFAR-100 with α = 0.3, by 0.85 points on the mean and 1.08 and 0.95 points on the tail metrics.The authors suggest the larger tail improvements may reflect clients between groups that fare poorly with one prototype.Source: §E.3 · p. 24, Table 12 · p. 24

E.4 Mixing Coefficient

  • 76.95% mean accuracy at λ = 0 rises to 83.72% at λ = 0.5 and 85.22% at λ = 1, with worst-client and bottom-20% accuracy following the same ordering.This single-seed CIFAR-100 LS3 diagnostic finds the highest values among the three tested λ settings at λ = 1, which is also used in the main experiments.Source: §E.4 · p. 24, Table 13 · p. 24

F DONOR-COMPATIBILITY DIAGNOSTIC

The donor-swap diagnostic tests whether head similarity ranks encoder compatibility, using fixed target heads and swapped cached encoders. Head proximity tracks transfer outcomes across label-skew and rotation settings, but the projection is not universally more predictive and the diagnostic has a narrow scope.

Distance quintiles

  • In all three settings, closer head-distance quintiles show monotone or nearly monotone accuracy-gain ordering, although swaps reduce accuracy on average in every quintile for CIFAR-10 label skew and CIFAR-100 rotation.The ranking reduces the average cost of a single encoder swap in those two settings; it does not turn the swaps into gains.Source: F DONOR-COMPATIBILITY DIAGNOSTIC · p. 25, Table 14 · p. 25

Rotation structure

  • In rotation, nearest-head donors share the target’s rotation in 75.0% of cases versus a 21.1% base rate; same-rotation swaps yield +0.83 accuracy points, while different-rotation swaps yield −3.46 points.Head similarity detects same-rotation pairs with AUROC 0.887 and average precision 0.606.Source: F DONOR-COMPATIBILITY DIAGNOSTIC · p. 25

Label-support overlap

  • In label-skew settings, head similarity is strongly associated with label-support overlap, but its partial correlation with loss benefit after residualizing overlap ranks is indistinguishable from zero at this sample size.The partial correlation is exploratory, not a causal estimate.Source: F DONOR-COMPATIBILITY DIAGNOSTIC · p. 25

Low-rank descriptor versus raw head

  • The low-rank descriptor is clearly better than raw head similarity only on CIFAR-100 label skew and marginally worse in the other two settings, so it is not universally more predictive of transfer.This accords with evidence that projection improves grouping separation and temporal stability rather than predictive accuracy.Source: F DONOR-COMPATIBILITY DIAGNOSTIC · pp. 25–26, Table 15 · p. 26

Scope

  • The diagnostic measures zero-shot encoder interchangeability with a fixed target head at one training point and on one backbone; it does not test HDC-FL’s multi-encoder prototypes.It establishes association and donor-ranking results, not causal or general encoder equivalence.Source: F DONOR-COMPATIBILITY DIAGNOSTIC · p. 26

Related papers

Loading 2610.30075v1…