PaperlayerView exact revision on arXiv

Source-linked AI summary

Is High Local Accuracy Enough? Geometric Consensus with Support-Adaptive Statistical Personalization for Federated Learning

Junyu Lin, Bosong Qi, Pengcheng Shi, Jiayuan Li

arXiv:2610.31133v1cs.LG

TL;DR

High local accuracy can conceal weaknesses in recognition across classes and clients. FedGRC combines shared representation and covariance learning with support-dependent class-mean adaptation, alongside an evaluation protocol that measures rare, unseen, and weak-client performance. Its comparisons report improved global recognition and personalization floors across heterogeneous settings, with specific gains on FedRS.

  • Problem

    High local accuracy can conceal weak recognition of locally rare or absent classes, so PFL evaluation must also assess all-class recognition and weak clients.

    Source: §1 · p. 1

  • Method

    FedGRC combines shared directional and magnitude guidance for representations, a shared covariance metric from multilayer statistics, and support-dependent shrinkage of local class means.

    Source: §1 · p. 2

  • Results

    FedGRC improves global recognition and personalization floors across heterogeneous settings; on FedRS, it gains 3.30% in global accuracy and 4.61% in the worst-performing clients’ recognition floor over the respective best baselines.

    Source: §1 · p. 2, §6 · p. 16

  • Takeaways & Limitations

    Evaluating local adaptation alongside all-class, rare- and unseen-class recognition and weak-client floors exposes weaknesses that aggregate or local scores can conceal.

    Source: §1 · pp. 1–2

    Under extreme label heterogeneity, broader recognition gains accompany substantially lower local accuracy than the strongest local baselines, and shrinkage does not explicitly account for client-specific domain-shift magnitude.

    Source: §6 · pp. 16–17

Abstract

from arXiv · show

Real-world deployment of personalized federated learning (PFL) requires adapting to heterogeneous clients while preserving recognition performance beyond local distributions. However, high local accuracy can conceal failures on locally rare or absent classes. Effective personalization should therefore strengthen global recognition while raising the personalization floor. We propose an evaluation protocol that complements local accuracy with global, rare-class, and unseen-class evaluations to reveal generalization obscured by local scores, and assesses personalization floors among the worst-performing clients to expose weaknesses hidden by aggregate performance. To meet these deployment challenges, we propose FedGRC, a unified PFL framework integrating generalization and personalization through (i) Consensus-Guided Representation Learning (CGRL), which learns discriminative representations through shared directional and magnitude guidance; (ii) Multilayer Metric Synthesis (MMS), which constructs a shared covariance metric from these multilayer representations for statistical classification; and (iii) Support-Adaptive Shrinkage Personalization (SASP), which adapts class means within this metric by shrinking local estimates toward shared means according to local sample support, enabling adaptation while retaining shared knowledge. Extensive experiments, including benchmarking against 21 methods across 5 heterogeneity settings on 3 datasets and comprehensive ablation studies, validate the effectiveness of FedGRC. On CIFAR-10 and CIFAR-100, FedGRC surpasses state-of-the-art (SOTA) baselines in global recognition and personalization floors by at least 2.17% and 2.13%, respectively. On the real-world heterogeneous FedRS dataset, it outperforms the respective best baselines by 2.15%, 3.30%, and 4.61% in local adaptation, global recognition, and the personalization floor.

1 INTRODUCTION

High local accuracy can conceal weak recognition of locally rare or absent classes, motivating evaluation beyond local scores. FedGRC combines shared geometric representation learning with statistical personalization and support-dependent class-mean adaptation.

Motivation

  • As label heterogeneity increases, several methods show rising local accuracy but falling global accuracy, because local evaluation gives little weight to locally rare or absent classes.The paper asks whether local accuracy alone adequately assesses recognition across the full label space.Source: §1 · p. 1

Unified Personalization and Generalization

  • FedGRC keeps the representation and covariance metric shared during personalization, adapting class means according to local sample counts.Equal class priors avoid adding a prediction preference for locally frequent classes.Source: §1 · p. 2

Comprehensive PFL Evaluation beyond Local Accuracy

  • The proposed evaluation jointly assesses local adaptation, all-class recognition, locally rare and absent classes, and the weakest assessed abilities of poorly performing clients.The recognition floor summarizes the lowest client-level scores across local adaptation and rare- and unseen-class recognition.Source: §1 · pp. 1–2

Unified Personalization and Generalization

  • FedGRC combines shared directional and magnitude guidance for representation learning with multilayer statistics that produce a shared covariance metric.Support-dependent shrinkage adapts class means within that shared geometry.Source: §1 · p. 2

Systematic Benchmarking and Validation

  • On FedRS, FedGRC gains 3.30% in global accuracy and 4.61% in the worst-performing clients’ recognition floor over the respective best baselines, with higher local personalization accuracy.The benchmark covers 21 external methods, five heterogeneity settings, and three datasets.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses personalization, shared representations, and statistical classifiers through varied mechanisms, but these approaches leave distinct challenges in evaluating all-class recognition and adapting statistical summaries to local support. FedGRC keeps representations and covariance shared while adapting class means.

2.1 Balancing Personalization and Generalization in Federated Learning

  • Personalized predictors may balance local and shared models or use test-time selection, while the proposed protocol evaluates deployed models on local, all-class, rare, and unseen classes and measures weak-client floors.FedGRC completes adaptation before deployment, avoiding test-time predictor selection or mixture-weight estimation.Source: §2.1 · pp. 2–3

2.2 Learning and Preserving Shared Representations under Data Heterogeneity

  • Representation methods share references or constrain feature properties, but these constraints alone do not specify classifier use of within-class variation and feature correlations.FedGRC combines final-layer directional guidance and class-specific magnitude references with statistical classification based on shared multilayer features.Source: §2.2 · p. 3

2.3 Federated Statistical Classifier Construction and Local Adaptation

  • Federated statistical classifiers differ in decision rules and modeled variation; covariance-based approaches can add communication or matrix-processing costs compared with mean-only summaries.FedGRC separates shared within-class covariance estimation from local class-mean adaptation.Source: §2.3 · p. 3
  • FedGRC keeps the representation and covariance metric shared, retaining shared means for absent classes and shrinking sparse local estimates toward them while allowing stronger adaptation with more support.Equal priors avoid favoring locally frequent classes.Source: §2.3 · p. 4

3 PRELIMINARIES

The preliminaries frame personalized federated learning as adapting client predictors while retaining shared knowledge, and statistical classification as a rule built from class means, covariance, and priors. The problem formulation assesses the same predictors across local and broader class distributions, including weak client- and class-level performance.

3.1 Personalized Federated Learning

  • Personalized federated learning allows client-specific models while retaining collaboration through shared parameters, representations, or regularization.The paper assesses the same deployed predictor for local adaptation and broader recognition, with class-level statistics allowing shared and local evidence to be combined.Source: §3.1 · p. 4

3.2 Statistical Classification

  • Under a shared-covariance Gaussian model, class means set centers, inverse covariance defines the distance metric, and log priors add class-specific offsets.With equal priors, classification is equivalent to minimizing the covariance-defined distance to a class mean.Source: §3.2 · p. 4

3.3 Problem Formulation

  • The label space includes all federation-known classes, although a client may have no examples for some classes, and predictors classify across all classes.Local and global risks evaluate the same predictor under different distributions; global all-class risk weights each known class equally.Source: §3.3 · pp. 4–5
  • The formulation jointly targets local adaptation, full-label-space recognition, and stronger weakest-client abilities, assessed through local, all-class, rare-class, unseen-class, and client-floor metrics.Averaging over clients can hide particularly weak predictors, and all-class risk can conceal large errors on locally rare or unseen classes.Source: §3.3 · p. 5

4 METHOD

This section presents FedGRC’s shared geometric representation learning, covariance metric, and support-adaptive class-mean personalization. It also bounds metric sensitivity and describes classification across federation-known classes using shared geometry.

4 METHOD

  • FedGRC proceeds through CGRL, MMS, and SASP, which adapts class means according to local sample support within the shared geometry.The stages integrate geometric representation consensus with statistical personalization for local adaptation and full-label-space recognition.Source: §4 · p. 5

4.1 Consensus-Guided Representation Learning

  • CGRL combines batch-level discrimination with shared class-direction references and class- and layer-specific magnitude references to guide representations for statistical classification.Shared directional references include classes absent from a local batch; magnitude guidance regulates log-norm dispersion and deviation from shared references.Source: §4.1 · pp. 5–7

4.2 Multilayer Metric Synthesis

  • MMS builds a shared covariance metric from frozen multilayer features, capturing within-class variation, cross-client mean shifts, and cross-layer covariance for statistical classification.An isotropic ridge proportional to average variance regularizes the covariance before inversion.Source: §4.2 · pp. 7–8
  • MMS downweights deviations along high within-class-variance directions and ridge-limits low-variance sensitivity, bounding amplification by 1/λA and condition number by 1 + d/γ, even for singular covariance.The bounds apply to the metric’s operator norm and spectral condition number.Source: §4.2 · pp. 8–9

4.3 Support-Adaptive Shrinkage Personalization

  • SASP weights each local class-mean shift by α(n) = n/(n + m0), so local influence rises with support while larger m0 favors the shared mean.This is the regularized estimator balancing local evidence against a shared reference, with m0 acting as a fixed effective sample weight.Source: §4.3 · p. 9
  • SASP combines adapted means with the shared MMS metric and equal class priors to classify across federation-known classes by nearest personalized mean.The backbone, layer scales, and covariance metric remain shared; the classifier is the equal-prior Bayes rule under the stated Gaussian model.Source: §4.3 · p. 9
  • SASP’s perturbation budget is no larger than direct local-mean adaptation’s, making prediction preservation easier; absent-class scores remain unchanged.The comparison holds for fixed class means, metric, and query.Source: §4.3 · pp. 9–10

5 EXPERIMENTS

The experiments evaluate FedGRC’s local adaptation, all-class generalization, and recognition floors across heterogeneous federated settings, alongside ablations of its components. They examine how CGRL and MMS+SASP contribute to F@25 across five settings and report the effect of shrinkage on the perturbation budget when fitted statistics are held constant.

5.1.1 Baselines and Datasets; 5.1.2 Evaluation Protocol

  • The evaluation benchmarks 21 external methods across five heterogeneity settings and measures local, unseen-class, and rare-class recognition with ability-specific lower-tail metrics and an overall personalization floor.F@25 takes each client’s minimum across local, unseen-class, and rare-class accuracy before averaging the lowest 25% of client scores; when a client has no locally unseen classes, it uses local and rare-class accuracy. This captures the weakest clients even when their weakest ability differs.Source: §5.1.1 · p. 10, §5.1.2 · p. 11

5.2 Comprehensive Evaluation of Personalization, Generalization, and Recognition Floors

  • FedGRC achieves the highest mean global, unseen, rare, and F@25 scores in all five heterogeneity settings, with gains of 2.17%–4.15% in global accuracy and 2.13%–6.46% in F@25.It also leads in local accuracy on both CIFAR datasets under Dir(0.3) and on FedRS; under Dir(0.1), broader recognition gains come with a local-accuracy trade-off.Source: §5.2 · p. 11, Table 1 · p. 12, Table 2 · p. 13

5.4.1 Ablation of FedGRC: Component Contributions to Generalization and Personalized Recognition

  • The complete CGRL+MMS+SASP configuration improves all five metrics over the initial system on CIFAR-100 Dir(0.3) and FedRS.On CIFAR-100, CGRL alone reduces F@25 from 37.77% to 29.17%, while replacing the linear head with MMS raises it to 46.93%; SASP further improves local adaptation and recognition floors.Source: §5.4.1 · pp. 12–13

5.4.2 Ablation of CGRL: Directional Alignment and Magnitude Regulation for Statistical Classification

  • With MMS+SASP fixed, CGRL improves global accuracy and F@25 over cross-entropy training by 6.15% and 7.37% on CIFAR-100 Dir(0.3), and 4.68% and 5.12% on FedRS.Directional and radial guidance are complementary: adding radial guidance to directional guidance improves CIFAR-100 rare accuracy from 57.13% to 58.00% and F@25 from 46.44% to 47.56%.Source: §5.4.2 · p. 13

5.2 Comprehensive Evaluation of Personalization, Generalization, and Recognition Floors

  • On FedRS, FedGRC reaches 89.11% local and 87.08% global accuracy, beating the best baselines by 2.15% and 3.30%, and raises F@25 by 4.61%.It also improves L@25, U@25, and R@25 by 4.24%, 4.83%, and 2.09%, respectively, over the best baseline for each metric.Source: §5.2 · p. 11, Table 3 · p. 14

5.3 Recognition Before and After Personalization

  • On CIFAR-100 Dir(0.3), FedGRC’s SASP adaptation raises local accuracy from 55.53% to 55.61% and F@25 from 46.93% to 47.56%, with a 0.27% global-accuracy decrease.On FedRS, all five metrics improve, including 1.41% in local accuracy and 1.61% in F@25; reference methods’ local gains can instead accompany substantial generalization and floor losses.Source: §5.3 · p. 12, Table 4 · p. 14

5.4.3 Ablation of MMS: Contributions of Statistical Geometry and Multilayer Feature Integration

  • The MMS statistical metric raises F@25 by 2.52% on CIFAR-100 Dir(0.3) and 6.25% on FedRS, while multilayer integration adds 1.62% and 2.22%, respectively.Together on FedRS, the two components raise global accuracy by 2.79% and F@25 by 8.47%; the metric accounts for within-class variability and multilayer features contribute information from several representation depths.Source: §5.4.3 · p. 14

5.4.4 Ablation of SASP: Contributions of Support-Adaptive Mean Estimation and Local Class Priors

  • On CIFAR-100 Dir(0.3), SASP restores rare accuracy to 58.00% and F@25 to 47.56%, after direct local means reduce them to 1.17% and 0.00%.Support-based shrinkage gives shared means greater influence when local support is small; equal priors further improve unseen and rare recognition and raise F@25 by 3.23% on CIFAR-100 and 0.24% on FedRS, at some local-accuracy cost.Source: §5.4.4 · p. 15
  • With fitted statistics held constant, SASP shrinkage reduces the perturbation budget described in Corollary 4.1.Source: §5.4.4 · p. 16

5.5 Performance Gains over Strong Baseline Combinations and Backbone–Classifier Synergy

  • F@25 is higher with both FedGRC components than with either single-component replacement in all five settings; on FedRS, the joint replacement gains 4.61%.On CIFAR-100 Dir(0.3), CGRL raises F@25 by 3.97% over FedYoYo with CCVR, and MMS+SASP adds 2.49% on CGRL but decreases F@25 by 0.12% on FedYoYo.Source: §5.5 · p. 16
  • Excess gain in F@25 is positive in all five settings, ranging from 0.86% to 2.61%; global accuracy shows positive excess gain in four settings.The passage presents these results as support for jointly designing geometric representation learning and statistical personalization to strengthen client recognition floors.Source: §5.5 · p. 16

6 CONCLUSION

FedGRC combines geometric representation consensus, multilayer statistical classification, and support-adaptive class-mean shrinkage to address global recognition and personalization floors. The conclusion reports broad recognition gains while noting a remaining local-adaptation trade-off under extreme label heterogeneity and a limitation in how personalization weights handle domain shift.

6 CONCLUSION

  • FedGRC couples geometric representation consensus, multilayer statistical classification, and support-adaptive class-mean shrinkage to improve global recognition and personalization floors across heterogeneous settings.The conclusion also reports simultaneous gains in local adaptation on FedRS.Source: §6 · p. 16
  • Under extreme label heterogeneity, broader recognition gains accompany substantially lower local accuracy than the strongest local baselines.FedGRC retains a shared representation and covariance metric during personalization.Source: §6 · pp. 16–17
  • FedGRC’s shrinkage weights depend on local class sample counts and do not explicitly account for the magnitude of client-specific domain shifts.The authors propose exploring personalization guided jointly by class support and distribution shift.Source: §6 · p. 17

APPENDIX A THEORETICAL ANALYSIS AND IMPLEMENTATION DETAILS

Appendix A provides theoretical analysis and implementation details, while retaining the main article’s equation and theoretical-result numbering.

APPENDIX A THEORETICAL ANALYSIS AND IMPLEMENTATION DETAILS

  • Supplementary equations use the prefix S, while equations and theoretical results from the main article keep their original numbering.The supplement uses notation introduced in Section 3.Source: Appendix A · p. 19

A.1 Proof of Lemma 4.1

The appendix develops conditional-moment and feature-geometry arguments supporting the paper’s theoretical results. Its complementary geometry analysis connects directional and magnitude objectives to within-class covariance structure.

A.1 Proof of Lemma 4.1

  • Under positive class-indicator probability and a finite conditional second moment, conditional centering makes the cross term vanish; the identity also holds for a fixed batch’s empirical distribution.When the indicator probability is zero, its contribution is zero.Source: §A.1 · p. 19

A.2 Complementary Feature Geometry

  • The geometry analysis links class- and layer-specific scatter to pooled within-class covariance, while distinguishing CGRL’s magnitude regulation and directional between-class supervision from covariance-trace minimization.The full covariance retains directional structure and cross-layer correlations.Source: §A.2 · p. 20

A.3 Proof of Proposition 4.1

The appendix derives properties of the shared covariance metric and statistical classifier, including a construction from client moments and a support-dependent personalized class mean.

A.3 Proof of Proposition 4.1

  • Summing over nonempty groups and dividing by N yields the stated covariance decomposition.Source: §A.3 · p. 20

A.4 Regularized Metric Bounds

  • The regularized metric is positive definite, with eigenvalues bounded through the empirical covariance spectrum and ridge term; the bound covers singular empirical covariances.The bound depends only on dimension and relative ridge strength.Source: §A.4 · p. 20

A.5 Construction from Additive Moments

  • The shared covariance can be constructed from client counts and moments while retaining cross-layer covariance blocks, matching pooled computation in exact arithmetic.Source: §A.5 · p. 20

A.6 Proof of Proposition 4.2

  • The support-dependent class-mean minimizer equals the shared mean when local support is zero, and its local-center residual decreases as support increases.For n > 0, the residual weight is 1 − α(n) = m0/(n + m0).Source: §A.6 · p. 20

A.7 Statistical Head Construction

  • The statistical head follows by dropping a class-independent term from personalized squared distance; under Gaussian class-conditionals and equal priors, the Bayes rule applies.Source: §A.7 · p. 21

A.8 Proof of Corollary 4.1 and Risk Bounds

The risk analysis relates personalized errors to shared-predictor errors and margin-limited changes, while the implementation conventions specify how shared references are accumulated and updated.

A.8 Proof of Corollary 4.1 and Risk Bounds

  • The risk analysis bounds personalized errors through shared-predictor errors and margin-limited prediction changes, with corresponding class-wise and rare- or unseen-class subset bounds.Taking expectations gives bounds for labeled target distributions; setting the distribution to the class-balanced target yields the all-class bound.Source: §A.8 · p. 21

A.9 Reference Updates and Implementation Conventions

  • Directional references use ξ to moderate changes between rounds; absent classes retain their previous directions, and prototype loss begins after the first aggregation.Once enabled, prototype cross-entropy uses all class references, including zero references.Source: §A.9 · p. 21
  • For classes observed in successive rounds, reference updates blend the prior estimate with the current estimate using weight ζ, and the references and flags are broadcast for the next round.Training occurrences contribute to the reference statistics whether or not the radial penalty is applied.Source: §A.9 · p. 22

APPENDIX B SUPPLEMENTARY EXPERIMENTS

  • Appendix B covers dataset partitions and client-level distributions, then provides further personalization-floor analysis and resource-overhead results.Source: Appendix B · p. 22

B.1 Datasets and Client Distributions

The appendix partitions CIFAR-10 and CIFAR-100 across clients with Dirichlet label distributions and uses a multi-source, 15-class FedRS partition to represent heterogeneous remote-sensing data.

CIFAR-10 and CIFAR-100

  • CIFAR-10 and CIFAR-100 each use 54,000 images across 20 clients, with Dirichlet concentrations 0.3 and 0.1 and approximately 75%/25% client training/local-test splits.The shared global test sets contain 600 images per class for CIFAR-10 and 60 per class for CIFAR-100.Source: §B.1 · p. 22
  • Dir(0.1) produces more concentrated CIFAR client-class distributions and more missing classes than Dir(0.3).Source: §B.1 · p. 22

FedRS

  • FedRS uses 135 clients in the NIID-1 partition, each drawing from one source dataset, and a shared global test set of 940 images covering all 15 classes.Its client data combine label and sample-size imbalance with domain heterogeneity across eight remote-sensing sources.Source: §B.1 · p. 22

B.2 Personalization Floors: Analysis and Results

FedGRC improves lower-tail recognition on CIFAR while local adaptation can trail under the more extreme label partition. The appendix also distinguishes cross-ability client floors from ability-specific tails and reports communication and computation overhead.

B.2 Personalization Floors: Analysis and Results

  • F@25 aggregates clients by their weakest assessed ability, whereas L@25, U@25, and R@25 summarize ability-specific tails that can reflect different groups of clients.Clients without unseen classes still contribute to F@25 through local and rare-class accuracies.Source: §B.2 · p. 22
  • 2.13% and 2.82% are FedGRC’s gains over the best F@25 baseline on CIFAR-10 under Dir(0.3) and Dir(0.1); the corresponding CIFAR-100 gains are 6.46% and 3.34%.FedGRC achieves the highest U@25, R@25, and F@25 in all four CIFAR settings; under Dir(0.1), L@25 trails the best baseline by 13.57% on CIFAR-10 and 3.59% on CIFAR-100.Source: §B.2 · p. 22, Table 11 · p. 24, Table 12 · p. 25

B.3.1 Communication Cost

  • FedGRC’s total communication volume is 0.08%, 0.41%, and 0.25% higher than FedAvg on CIFAR-10, CIFAR-100, and FedRS, respectively; constructing the statistical heads requires one additional communication round.The terminal payload is independent of local sample size for fixed feature dimensions and class count.Source: §B.3.1 · p. 25

B.3.2 Computational Cost

  • CGRL adds a second augmented view and a forward-only class-direction pass during training, while MMS and SASP construct heads from a frozen-backbone pass; inference overhead is negligible relative to FedAvg.Inference uses a single backbone pass followed by an O(Cd) linear head, compared with FedAvg’s O(CdL) head.Source: §B.3.2 · p. 25

Related papers

Loading 2610.31133v1…