PaperlayerView exact revision on arXiv

Source-linked AI summary

FedSAP: Federated Learning with Structured Adaptive Partitioning for Multi-Domain Heterogeneous Edge Devices

Wentao Yue, Tianyou Lai, Hongji Li, Qingyu Mao, Qilei Li

arXiv:2610.01638v1cs.LG

TL;DR

Heterogeneous federated learning must handle unequal device budgets and domain-shifted local data, while prior approaches do not jointly decide where retained capacity should reside and how it should be shared. FedSAP allocates channels across layers and sharing roles, and its results show higher mean global accuracy on both evaluated benchmarks under pruning ratios up to 80%.

  • Problem

    Existing resource-adaptive methods do not specify where retained capacity should reside or how it should be shared, while federated domain-generalization methods usually assume a shared full architecture.

    Source: Introduction · p. 1

  • Method

    FedSAP assigns non-uniform layer budgets and partitions channels into Global, Private, or Dropped states, using pseudo-domain assignment and type-matched aggregation to set sharing scope.

    Source: Introduction · p. 1, Problem Formulation and Overview · p. 3, Adaptive Layer Budget and Selective… · p. 4

  • Results

    76.00% on Digits (1.70 points above DapperFL) and 72.67% on Office-Caltech (4.92-point gain) were FedSAP’s highest global accuracies across three seeds and 0%-80% client pruning.

    Source: Main Results · p. 6

  • Takeaways & Limitations

    The results support role-aware capacity allocation across domains, although FedSAP does not dominate every individual domain.

    Source: Conclusion · p. 7

    DGA is a heuristic proxy for latent domains, and first-layer gradients may not separate domains defined by high-level semantics or label mechanisms.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Federated learning (FL) on heterogeneous edge devices must jointly accommodate unequal resource budgets and domain-shifted local data. Existing resource-adaptive methods decide how much of a model each client trains but not where retained capacity should reside or how it should be shared, whereas federated domain-generalization methods usually assume a shared full architecture. Uniform compression can therefore discard high-utility channels, and a single aggregation path can mix transferable features with domain-sensitive updates. We propose FedSAP, a domain-aware heterogeneous FL framework that casts structured pruning as budget-constrained tri-state channel allocation. FedSAP converts each keep ratio into non-uniform layer budgets, assigns stable channels to a Global pool, useful domain-sensitive channels to pseudo-domain-specific Private pools, and low-utility channels to a Dropped state. This partition lets broadly useful features benefit from cross-client pooling while isolating domain-sensitive updates from incompatible clients. Domain-Guided Assignment infers pseudo-domains from shallow-gradient similarity, while Type-Matched Aggregation restricts each channel to its intended sharing scope. Across three random seeds, FedSAP reaches 76.00% and 72.67% mean global accuracy on Digits and Office-Caltech, exceeding the strongest baseline by 1.70 and 4.92 percentage points while supporting client pruning ratios of up to 80% across heterogeneous clients.

Introduction

Heterogeneous federated learning must accommodate unequal device budgets and domain-shifted data, but existing approaches do not jointly decide where retained capacity should reside and how it should be shared. FedSAP combines adaptive layer budgets, channel roles, and scope-matched aggregation.

Introduction

  • Resource-adaptive FL assigns smaller subnetworks to weak devices, while domain-generalization methods usually assume a shared full architecture.Neither approach alone specifies where limited retained capacity should reside or with whom each feature should be shared.Source: Introduction · p. 1
  • Uniform layer policies can waste scarce capacity, single-path aggregation can overwrite domain-sensitive channels, and shape reconstruction alone cannot determine which clients should share them.FedSAP treats layer budget, channel role, and sharing scope as a joint allocation decision.Source: Introduction · pp. 1–2
  • FedSAP combines adaptive layer-wise budgets with Global, Private, and Dropped channel states, using utility, stability, and client constraints to assign roles.Domain-Guided Assignment infers pseudo-domain groups from shallow-gradient similarity, and Type-Matched Aggregation restricts Private-channel aggregation to the corresponding group.Source: Introduction · pp. 1–2
  • Across three seeds, experiments report higher global mean accuracy under heterogeneous compression, alongside domain-level trade-offs.The supplement also reports budget–aggregation and domain-wise budget experiments, sensitivity results, conditional theory, and privacy analysis.Source: Introduction · p. 2

Related Work

Prior work addresses resource heterogeneity, domain generalization, and federated pruning, but often treats these challenges separately. FedSAP instead allocates active channels across layers and Global or Private aggregation paths, dropping low-utility channels to meet client capacity constraints.

Related Work

  • Resource-adaptive methods such as HeteroFL, FjORD, and FedRolex vary client subnetworks, but generally overlook visual domain shifts across clients.HeteroFL trains width-scaled models, FjORD uses ordered dropout, and FedRolex rotates partial subnetworks across rounds (Diao, Ding, and Tarokh 2021; Horvath et al. 2021; Alam et al. 2022).Source: Related Work · p. 2
  • Federated domain-generalization methods use regularization, prototype alignment, domain-wise heads, or gradient signals, but assume identical full-model capacity across clients.Examples include FedSR (Caldarola, Caputo, and Ciccone 2022), FPL, FedDAR, FedOMG, and FedGaLA.Source: Related Work · p. 2
  • FedDrop, PruneFL, and FedMP prune for resource constraints but do not explicitly decide whether retained capacity supports transferable or domain-specific knowledge (Bouacida et al. 2021; Jiang et al. 2022a; Jiang et al. 2022b).DapperFL supports differently compressed clients under domain shifts, but retained channels follow one aggregation path after reconstruction.Source: Related Work · p. 2
  • FedSAP allocates active channels non-uniformly across layers, assigns Global or Private aggregation paths, and drops low-utility channels to satisfy each client’s capacity constraint.This contrasts with methods that control retained capacity without specifying its layer placement or sharing scope.Source: Related Work · pp. 2–3

Methodology Problem Formulation and Overview

FedSAP treats each client’s pruning ratio as a total capacity budget, allocating it across layers and assigning retained channels to sharing roles. It uses gradient similarity to estimate pseudo-domains and matches aggregation scope to channel role.

Methodology Problem Formulation and Overview

Domain-Guided Assignment

  • Domain-Guided Assignment groups clients using first-layer gradient similarity, updating pseudo-domains periodically to limit communication and noisy early changes.This lightweight proxy suits domains distinguished by low-level appearance, but may fail when domain identity is mainly semantic.Source: Domain-Guided Assignment · p. 3

Adaptive Layer Budget and Selective Global Protection

  • Small cross-domain activation dispersion favors Global channels, while large dispersion favors Private channels; an exponential moving average smooths the scores.FedSAP also enforces a target Global-channel ratio to reduce fragmentation when early domain estimates are noisy.Source: Adaptive Layer Budget and Selective… · pp. 4–5
  • Clients train masked models with classification loss and a lightweight representation regularizer, set to γ = 0.01 in reported experiments.Source: Adaptive Layer Budget and Selective… · p. 5

Type-Matched Aggregation and Training Procedure

  • Type-matched aggregation pools Global channels across clients and Private channels only within their pseudo-domain group, with weights normalized by data size.The server combines shared Global parameters with the Private parameters for each client’s pseudo-domain group.Source: Type-Matched Aggregation and Training… · p. 5

Experiments

FedSAP is evaluated on Digits and Office-Caltech under heterogeneous client compression, with adaptive layer budgets and channel-role allocation compared against established baselines and diagnostic variants. It achieves the highest reported global accuracy on both benchmarks, while ablations and sensitivity analyses examine its components and settings.

Experimental Setup

  • Experiments evaluate FedSAP on Digits and Office-Caltech using ResNet10 and ResNet18, respectively, with heterogeneous clients assigned pruning ratios of 0%, 20%, 40%, 60%, or 80%.Digits distributes 10 clients across four domains and uses 1% of each client’s local data; Office-Caltech uses 20% of each client’s local data.Source: Experimental Setup · p. 5

Main Results

  • 76.00% on Digits is 1.70 points above DapperFL, and 72.67% on Office-Caltech is a 4.92-point gain, averaged over three seeds.Client pruning ratios range from 0% to 80%; global accuracy is the best-so-far unweighted mean of domain accuracies evaluated at the same round.Source: Table 1 · p. 6, Main Results · p. 6
  • Relative to DapperFL, FedSAP gains 6.59 points on SYN, 10.72 on Caltech, and 6.66 on DSLR, but loses 5.94 on SVHN, 2.11 on Amazon, and 3.45 on Webcam.FPL is 7.23 points higher on SVHN and NeFL is 6.90 points higher on Webcam; the reported global mean does not imply per-domain dominance.Source: Main Results · p. 7

Ablation Studies

  • On Digits, FedSAP reaches 76.0% average accuracy; removing DGA or SGP lowers it to 75.5% or 75.4%, and removing both lowers it to 74.3%.The reported domain-wise effects vary: removing SGP slightly improves SVHN but lowers the source-domain average.Source: Figure 4 · p. 6, Ablation Studies · p. 7
  • The coordinate-wise selected channel-role settings are gtarget = 0.9, τvar = 0.05, and βEMA = 0.5, with competitive accuracy across broad one-factor ranges.The sweeps use a fixed-mask, uniform-budget reference setting before transferring the selected values to the final adaptive configuration.Source: Figure 5 · p. 6, Ablation Studies · p. 7
  • Under the fixed-mask reference on Digits, Conv1, Layer1, and Layer2 gradient sources give comparable results, with Layer1 highest and Conv1 requiring the smallest payload.The authors interpret this diagnostic as supporting shallow gradients for appearance-driven domains.Source: Ablation Studies · p. 7, Table 2 · p. 7

Conclusion

FedSAP treats heterogeneous federated pruning as a joint allocation of layer capacity and channel-sharing scope. Its results improve global mean accuracy on two vision benchmarks, while the paper identifies privacy, pseudo-domain assignment, and evaluation scope as boundaries.

Conclusion

  • Across three seeds, FedSAP improves global mean accuracy on Digits and Office-Caltech under client pruning ratios up to 80%, but does not dominate every domain.The reported global mean can hide worst-domain regressions.Source: Conclusion · p. 7
  • The evidence is limited to two vision benchmarks and ResNet-style backbones, leaving semantic shifts, larger architectures, partial participation, and imbalanced domains for future evaluation.Worst-domain or fairness-aware objectives are also identified as needed evaluation directions.Source: Conclusion · p. 7
  • DGA is a heuristic proxy rather than a domain-identification guarantee, and uploaded shallow gradients and channel statistics create privacy exposure without a formal privacy guarantee.Incorrect pseudo-domain assignments can replace useful global transfer with private estimation noise or mix incompatible private updates.Source: Conclusion · p. 7

Supplementary Material

The supplementary material describes comparison methods spanning standard, domain-generalization, and resource-adaptive federated learning, then distinguishes FedSAP’s main adaptive configuration from its diagnostic reference. The main setting uses dynamic masks and selected channel-role parameters; the reference fixes masks after warm-up.

A Comparison Frameworks

  • The comparison also includes standard FL methods, domain-generalization methods, and heterogeneous or pruning-based methods, including FedAvg, FedProx, MOON, FedSR, FPL, FedDrop, FedMP, and NeFL.NeFL scales nested subnetworks by depth and width; FedDrop trains selected subsets of the global model.Source: A Comparison Frameworks · p. 10
  • DapperFL is the closest baseline, combining Model Fusion Pruning with Domain Adaptive Regularization and reconstructing heterogeneous pruned models before aggregation.It supports different client pruning ratios under domain shift but does not explicitly allocate capacity across layers or retained channels by sharing role.Source: A Comparison Frameworks · p. 10
  • FedSPU keeps the full architecture and randomly freezes neurons rather than training a smaller submodel through dropout-based pruning.It is included as a resource-constrained FL baseline, evaluated on the same multi-domain client split.Source: A Comparison Frameworks · p. 10

B FedSAP Settings

  • The main FedSAP configuration uses adaptive layer budgets, dynamic masks, and gtarget = 0.9, τvar = 0.05, and βEMA = 0.5; the diagnostic reference uses uniform budgets and fixed masks after warm-up.The reference instead uses τvar = 0.1 and βEMA = 0.3, and its 75.54% Digits accuracy is distinct from the final configuration’s 76.00%.Source: B FedSAP Settings · p. 10

C Implementation Details

FedSAP’s implementation combines periodic pseudo-domain grouping, utility-based layer budgets, and channel-role-specific aggregation. The method uses shallow gradients and channel statistics to guide these decisions while preserving each client’s target keep ratio.

C Implementation Details

  • DGA uploads each client’s first-convolution gradient after local training, groups clients by pairwise cosine similarity at τsim = 0.5, and updates groups every five rounds.Connected components define pseudo-domains; agglomerative clustering on distance 1 − similarity is an alternative.Source: C Implementation Details · p. 11
  • Adaptive layer budgets use client-level keep ratio κi and global layer utility from aggregated channel activation- and gradient-magnitude statistics, then normalize to preserve κi.The method sets (λa, λg) = (0.5, 0.5), softmax temperature 1.0, and clips layer keep ratios before final normalization.Source: C Implementation Details · p. 11
  • SGP assigns active channels to Global or Private using importance and cross-domain activation stability, with gtarget = 0.9, τvar = 0.05, and βEMA = 0.5.τvar distinguishes stable from domain-sensitive channels, and βEMA smooths channel statistics.Source: C Implementation Details · p. 11
  • Type-Matched Aggregation pools Global channels across all clients, aggregates Private channels only within corresponding pseudo-domain groups, and excludes Dropped channels.This avoids forcing domain-invariant and domain-specific channels through the same aggregation path.Source: C Implementation Details · p. 11

D Extended Ablation and Hyperparameter Sensitivity

The ablations separate the effects of adaptive layer budgeting and tri-state aggregation, and the sensitivity study selects channel-role hyperparameters for the final configuration.

D Extended Ablation and Hyperparameter Sensitivity

  • Adaptive budgeting improves both aggregation variants, while tri-state aggregation adds a gain under either budget rule.Table 5 reports best-so-far global accuracy on Digits, averaged over three seeds.Source: D Extended Ablation and Hyperparameter… · p. 11, Table 5 · p. 12
  • Domain-level gains from adaptive budgeting are small on MNIST and USPS, moderate on SVHN, and largest on SYN.Table 6 compares uniform-budget tri-state aggregation with the complete adaptive-budget configuration; domain entries may come from different rounds.Source: D Extended Ablation and Hyperparameter… · p. 11
  • The coordinate-wise best Digits sensitivity values are 0.9, 0.05, and 0.5 for gtarget, τvar, and βEMA, respectively.The study varies one channel-role hyperparameter at a time around (0.9, 0.1, 0.3), using uniform layer budgets and fixed masks; the selected combination transfers to the final dynamic-mask, adaptive-budget configuration.Source: D Extended Ablation and Hyperparameter… · p. 11
  • Table 4 lists general training hyperparameters and directs readers to Table 3 for FedSAP-specific configuration differences.Table 4 itself does not provide those FedSAP-specific differences.Source: Table 4 · p. 12

E.1 A Bias–Variance–Compression View

The analysis frames Global, Private, and Dropped channel choices as a conditional trade-off among sharing bias, estimation variance, and compression. Its role rule depends on a separable quadratic approximation and an additive channel budget, while practical assignments rely on proxies.

E.1 A Bias–Variance–Compression View

  • The analysis aims to explain when the three states are preferable, not to establish a distribution-free guarantee for non-convex neural networks.It considers one channel across K latent domains and evaluates domain risk locally.Source: §E.1 · p. 12
  • A separable quadratic model isolates sharing bias from the variance of estimating separate channel parameters with fewer samples.The local risk approximation uses a positive channel-specific curvature, hc > 0.Source: §E.1 · p. 12
  • Under the stated approximation, unbiased estimates, and an additive active-channel budget, an active channel is Global when its sharing-risk condition holds and Private otherwise; low marginal utility channels are Dropped.The Dropped set follows by selecting channels with the greatest marginal risk reduction under the cardinality budget.Source: §E.1 · pp. 12–13
  • Global channels trade a small sharing bias for lower variance, Private channels avoid cross-domain bias at higher estimation variance, and Dropped channels give up low marginal utility to meet the budget.The corresponding domain-level accuracy table reports best-so-far entries that may come from different communication rounds.Source: §E.1 · p. 13, Table 6 · p. 13
  • FedSAP uses activation dispersion to proxy cross-domain stability and utility measures to estimate the cost of dropping a channel; gtarget limits assignments to small Private groups.The mechanism depends on these proxies preserving the relevant ordering and does not prove globally optimal neural-network masks.Source: §E.1 · p. 13

E.2 Pseudo-Domain Error and the Boundary of DGA

The analysis bounds the cost of pseudo-domain assignment error under stated smoothness and diameter assumptions, but systematic proxy errors remain possible. Its implications are scoped to appearance-driven benchmarks and global mean accuracy, not per-domain guarantees.

E.2 Pseudo-Domain Error and the Boundary of DGA

  • Assigning an ε fraction of domain mass to an incompatible Private group adds at most εLcDc to weighted channel risk under the stated Lipschitz and diameter assumptions.The bound makes explicit that noisy grouping can erase the bias reduction motivating Private channels.Source: §E.2 · p. 13
  • Repeated grouping and exponential smoothing improve temporal stability but do not remove systematic proxy error.The bound describes the consequence of incompatible assignments rather than eliminating that error source.Source: §E.2 · p. 13
  • First-layer gradients are plausible domain signals for shifts in local image statistics, but can miss semantic shifts or split clients on nuisance appearance changes.The shallow-layer ablation supports robustness only on evaluated appearance-driven benchmarks, not high-level semantic domains.Source: §E.2 · p. 13

E.3 What the Analysis Does and Does Not Establish

  • The analysis links channel-wise sharing and utility-based allocation to improved average accuracy, while fixed budgets and pseudo-domain error can still reduce particular domains.FedSAP targets the evaluation protocol’s global mean and offers neither monotonic improvement for every domain nor a worst-domain guarantee.Source: §E.3 · p. 14

F Privacy Risk of Uploaded Signals

Uploaded gradients and channel statistics may expose client or domain information, and the protocol assumes a trusted server without formal privacy guarantees. The paper also estimates modest extra computation and channel-scaled communication overhead.

F Privacy Risk of Uploaded Signals

  • Client-indexed first-layer gradients and channel statistics may reveal visual domain identity, coarse sample attributes, class imbalance, or feature prevalence.Pseudo-domain membership can itself be sensitive when domains correspond to hospitals, sensors, regions, or demographic groups.Source: F Privacy Risk of Uploaded Signals · p. 14
  • The protocol assumes a trusted server and provides no differential-privacy, cryptographic, or empirical attack-resistance guarantee.The privacy claim is limited to avoiding direct raw-data transfer; communication size is not a privacy proxy.Source: F Privacy Risk of Uploaded Signals · p. 14
  • Compatible defenses include secure aggregation, clipping with calibrated noise, quantization, random projection, and infrequent updates, but can change utility.Noise may corrupt cosine similarity and destabilize DGA; a deployment study should assess inference and inversion attacks, privacy–utility curves, grouping accuracy, and worst-domain performance.Source: F Privacy Risk of Uploaded Signals · p. 14

G Computation and Communication Overhead

  • FedSAP’s estimated additional computation is about 3%–5% over FedAvg, with DGA’s gradient pass amortized because reassignment occurs every five rounds.Local training remains the dominant cost, and mask-based aggregation contributes only minor computation.Source: G Computation and Communication Overhead · p. 14
  • The extra communication consists of intermittent first-layer gradients, channel-level statistics, and tri-state masks, with masks and statistics on the order of 10^3 values for ResNet10.The passage compares this payload with model updates containing millions of parameters and cautions that size does not establish privacy.Source: G Computation and Communication Overhead · p. 14

H Additional Results

Additional results compare FedSAP configuration variants, show that accuracy gains vary across domains, and visualize distinct cross-domain structure in Global and Private representations.

Per-domain accuracy curves

Representation visualization

Training curves

Related papers

Loading 2610.01638v1…