PaperlayerView exact revision on arXiv

Source-linked AI summary

SWB-DM: A Calibrated Sliced-Wasserstein-Barycenter Aggregator with Delayed-Momentum Caching for Byzantine-Robust Federated Learning under Partial Participation

Saranraj S, Saranya M S, Alex David S, Ajay Kumar A

arXiv:2609.16099v2cs.LGcs.CRcs.DC

TL;DR

Partial participation can make a round’s attackers unrepresentative of the full population, putting robust aggregators’ finite-sample assumptions at risk. SWB-DM combines sliced-Wasserstein-inspired aggregation with full-population delayed-momentum caching. Its reported gains are disproportionately large on CIFAR-10 after equalizing round budgets, while FLTrust benefits more on CIFAR-100.

  • Problem

    Partial participation can make the sampled attackers unrepresentative of the population, while robust aggregators’ preconditions are stated for the full client population.

    Source: §I · pp. 1–2

  • Method

    SWB-DM pairs a trimmed sliced-Wasserstein-inspired aggregator with delayed-momentum caching across the full client population.

    Source: §I · p. 2

  • Results

    After extending all six baselines to the same round budget, SWB-DM’s gain is disproportionately large on CIFAR-10, while FLTrust benefits more on CIFAR-100 for a noncaching reason.

    Source: §I · p. 2

  • Takeaways & Limitations

    Per-round parameter-norm and prediction-histogram tracking is needed to distinguish failure mechanisms that can look similar in aggregate accuracy.

    Source: §VIII · p. 7

    At 500 clients, SWB-DM reaches 17.16% while DelayedMomentum reaches 20.61%; the proposed stale-entry explanation remains an open question.

    Source: §VIII · p. 7

Abstract

from arXiv · show

Robust aggregation methods for federated learning quietly rest on a fragile assumption: that whoever shows up in a given round is a fair sample of the full population. In practice, they rarely are. When only a handful of clients participate per round, even a modest fraction of adversaries can dominate that sample and silently invalidate the finite-sample guarantees that coordinate-wise median, Krum, Bulyan, and trimmed mean all depend on. We introduce SWB-DM to address this directly. SWB treats each slice of a client update as a one-dimensional distribution, computes a trimmed Wasserstein barycenter across clients, and recovers coordinate identity via a medoid-based gauge-fixing step -- a heuristic we developed and do not claim it belongs to standard optimal-transport theory. DeMoA-style delayed momentum then caches updates across the full client population each round, decoupling robustness from whoever happened to be sampled. Trim ratio calibration is not cosmetic: under-trimming causes collapse at corruption levels a properly calibrated model survives. Across 448 CIFAR-10 configurations, plus CIFAR-100, FEMNIST, and a 500-client scalability run, we find several mechanistically distinct failure modes. Even-sample coordinate-wise median degrades to a deterministic wrong answer. Krum silently violates its own n greater than 2f+2 precondition and diverges without warning. Bulyan's n greater than or equal to 4f+3 threshold produces a sharp pass/fail boundary. On attacks, IPM defeats order-statistic defenses -- including SWB -- more reliably than ALIE, confirmed through delta-space measurements against a convergence bound. SWB-DM's cache carries a real warm-up cost, but extending all baselines to the same round budget shows its CIFAR-10 gains are disproportionately large. On CIFAR-100, FLTrust benefits more -- for reasons entirely unrelated to caching.

I. INTRODUCTION

Partial participation can expose robust aggregators to round-wise corruption that violates assumptions stated for the full population. SWB-DM pairs a sliced-Wasserstein-inspired aggregator with delayed-momentum caching; the reported comparison favors it on CIFAR-10, while FLTrust benefits more on CIFAR-100.

I. INTRODUCTION

  • With partial participation, sampling or strategic skipping can make a round’s attacker fraction exceed the population corruption fraction, invalidating an aggregator’s safety precondition.Source: §I · p. 2

II. RELATED WORK

  • DeMoA caches every client’s latest update and aggregates over the full cache each round, removing selection luck from the sample-size equation.Source: §I · p. 2, §II · p. 2

I. INTRODUCTION

  • SWB treats rotated chunks as one-dimensional empirical distributions, computes trimmed Wasserstein barycenters, and restores coordinate identity with a medoid-based heuristic.The authors explicitly distinguish this gauge-fixing heuristic from established sliced optimal-transport theory.Source: §I · p. 2, §II · p. 2
  • The study spans 448 CIFAR-10 configurations and additional CIFAR-100, FEMNIST, and 500-client experiments, tracing distinct failure mechanisms and attack-error profiles.The contribution passage reports non-monotonic ALIE error and strictly rising IPM error.Source: §I · p. 2
  • On CIFAR-10, SWB-DM’s gain is disproportionately large after all six baselines are extended to the same round budget; on CIFAR-100, FLTrust benefits more for a noncaching reason.Source: §I · p. 2

III. THREAT MODEL

The threat model samples only part of a population with a fixed Byzantine fraction, so the malicious share of a round can vary. SWB-DM refreshes sampled clients’ cache entries and aggregates the full cache.

III. THREAT MODEL

  • The model samples n = pN clients from N each round while a fixed population fraction β remains Byzantine, though the attackers appearing in a sample vary.Source: §III · pp. 2–3
  • For Krum and Bulyan, the assumed attacker count is f = ⌊nβ⌋, a population-level estimate rather than an oracle count of actual round attackers.Source: §III · p. 3
  • Each round may inject label-flipping, sign-flipping, Gaussian-noise, ALIE, or IPM attacks; sampled clients refresh cache entries while unsampled clients retain their latest updates before SWB aggregates the full cache.Source: §III · p. 3, Figure 1 · p. 3

IV. METHOD

SWB chunks client model states, applies randomized rotations and rank-wise trimming to form barycenters, then uses a medoid’s coordinate permutation to restore identity before averaging views.

A. SWB: A Randomized-Slicing, Wasserstein-Inspired Aggregator

  • SWB takes full locally trained model states, not global-model deltas, and divides client vectors into chunks of size C.Source: A. SWB: A Randomized-Slicing,… · p. 3
  • For each chunk, SWB projects the client matrix through a random orthogonal rotation and sorts each projected client row so its values can be treated as one-dimensional samples.Source: A. SWB: A Randomized-Slicing,… · p. 3
  • At each sorted-rank position, SWB trims ⌊n · r⌋ values from each tail, uses the nearest-to-barycenter client’s rank-to-coordinate permutation, and averages results from P=2 rotations.The trim ratio is r = min(0.45, β + 0.05).Source: A. SWB: A Randomized-Slicing,… · p. 3
  • Medoid gauge fixing performs well empirically, but its borrowed coordinate permutation lacks a formal guarantee.Source: A. SWB: A Randomized-Slicing,… · p. 3

B. Delayed-Momentum Caching

Delayed-momentum caching refreshes sampled clients’ latest states and runs the base aggregator over all cached client states. The accompanying error bound is a general shape whose coefficient is fit empirically for SWB.

B. Delayed-Momentum Caching

  • Sampled clients’ cache entries are replaced by their newly trained states, while the global model is updated using an aggregate over the full cache.Source: B. Delayed-Momentum Caching · p. 3
  • SWB-DM uses SWB as the cached-population aggregator, while a DelayedMomentum baseline uses plain median to separate caching from the inner statistic.Source: B. Delayed-Momentum Caching · pp. 3–4

V. THEORETICAL ANALYSIS

  • The analysis uses E(β) ≤ E0 + C√β, with E0 representing benign finite-sample error and C reflecting benign-update spread.C is fitted to SWB’s measured error rather than derived tightly for SWB, and tighter analysis is left to future work.Source: §V · p. 4

VI. EXPERIMENTAL SETUP

Experiments use CIFAR-10 as the primary testbed, with CIFAR-100 and FEMNIST for generalization and a 500-client scalability study. The evaluation varies client partitions, attacks, corruption, and participation across 448 configurations.

Datasets and model

  • The study evaluates a compact two-convolutional-layer CNN on CIFAR-10, CIFAR-100, and FEMNIST, with N=500 clients in the scalability study.Most experiments use N=20 clients, two local epochs per round, and SGD with learning rate 0.01 and momentum 0.9.Source: §VI · p. 4

Partitioning

  • Clients receive a fixed Dirichlet non-IID partition with α=0.5; at N=500, client sample counts range from 13 to 288, a 22× spread.The partition seed is held fixed across methods and random seeds so each method trains on the same split.Source: §VI · p. 4

Baselines and grid

  • The 448-configuration main grid compares seven methods, while the 500-client study adds DeMoA with coordinate-wise median to isolate the caching mechanism from the inner statistic.Main-grid baselines are FedAvg, Median, Krum, Bulyan, FLTrust, SWB, and SWB-DM.Source: §VI · p. 4

Attacks

  • The attack suite includes clean runs and five attacks: label-flip, sign-flip, Gaussian noise, ALIE, and IPM.ALIE's perturbation magnitude is computed analytically from each round's (n, f) values.Source: §VI · p. 4

Grid

  • The grid uses participation rates p∈{0.5, 0.1}, two seeds, and 10 rounds; Krum and Bulyan use f=⌊nβ⌋, with Bulyan's n=11 meeting its bound at β=0.2 but not β=0.3.The Bulyan precondition violation at β=0.3 is retained as a stress test.Source: §VI · p. 4

VII. RESULTS

Across the main grid and failure-mode diagnostics, aggregation outcomes depend on the method's preconditions and attack, with distinct mechanisms behind collapse, divergence, and sharp pass/fail boundaries. Diagnostics also reveal warm-up and sampling-related behavior in the evaluated settings.

A. Main Grid

  • At p=0.1, SWB-DM falls behind all other methods within 10 rounds, which Section VII-E attributes to caching warm-up; FLTrust remains stable across corruption and participation levels.FLTrust's trust scores use a server-held clean dataset rather than statistics from the participating pool.Source: A. Main Grid · p. 4
  • Krum's CIFAR-10 accuracy is higher at p=0.1 than p=0.5 when β=0 (43.10 vs. 34.11), as it selects one update and discards the rest.With p=0.5 and n=10, nine updates are discarded each round; under non-IID partitions, the resulting information loss accumulates across rounds.Source: A. Main Grid · p. 4

B. Failure-Mode Diagnostics

  • With n=2 under IPM, the coordinate-wise median returns the smaller value per coordinate, producing 10.00% accuracy across 20 rounds and the same predicted label for every test sample.The parameter norm rises from 9.89 at round 1 to 31.56 at round 20, despite updates not reducing training loss.Source: B. Failure-Mode Diagnostics · p. 4
  • At n=2, Krum violates n>2f+2; the unguarded implementation selects the corrupted client by round 7, after which 545,066 parameters become NaN and 32 become Inf.Krum copies its selected update as the global state, so a corrupted selection can permanently corrupt the model.Source: B. Failure-Mode Diagnostics · pp. 4–5

A. Main Grid

  • Table II defines collapse as accuracy ≤10.5% (essentially chance on CIFAR-10); FLTrust had no collapses across 64 configurations, while Median collapsed in 10, predominantly under IPM.Source: A. Main Grid · p. 4, Table II · p. 5

B. Failure-Mode Diagnostics

  • On CIFAR-100 under label-flip corruption, one Krum seed falls from 13.82% at round 10 to 1.96% by round 25, while another seed with the same settings keeps improving.The passage characterizes this as inconsistency rather than collapse and ties run outcomes to which clients were sampled.Source: B. Failure-Mode Diagnostics · p. 5
  • Bulyan meets n≥4f+3 at n=11, f=2, with 51.70–51.98% accuracy under both attacks; at β=0.3 the requirement rises to n≥15 and accuracy falls to 10.00%.The β=0.3 result was rerun and did not change.Source: B. Failure-Mode Diagnostics · p. 5
  • Even within Krum's safe zone (n=10, f=2), accuracy is 51.15% under ALIE but 10.00% under IPM; IPM's low neighbor-distance score can win selection despite the precondition.IPM negates and norm-matches the honest gradient direction, targeting the neighbor-distance score Krum uses.Source: B. Failure-Mode Diagnostics · p. 5

C. Bound vs. Empirical Error, Two Attacks

ALIE and IPM produce distinct delta-space error patterns for SWB, while SWB’s aggregation error remains higher than Median’s across both attacks.

C. Bound vs. Empirical Error, Two Attacks

  • Under ALIE, SWB’s error fell from 1.478 to 1.101 and 1.087 at f=1 and f=2, then rose at f=3 and f=4.The passage attributes the initial dip to ALIE tuning poisoned updates just inside a robust statistic’s tolerance threshold.Source: C. Bound vs. Empirical Error, Two… · p. 5
  • Under IPM, SWB’s delta-space error rose at every corruption count, from 1.483 at f=0 to 3.680 at f=4.IPM pushes updates along the negative gradient direction and scales the push with each additional corrupted client.Source: C. Bound vs. Empirical Error, Two… · pp. 5–6
  • SWB’s aggregation error was higher than Median’s at every f under both ALIE and IPM, despite SWB winning downstream.The passage notes that end-to-end accuracy tends to hide this per-round delta-space cost.Source: C. Bound vs. Empirical Error, Two… · p. 6

D. Generalization: CIFAR-100 and FEMNIST

The CIFAR-100 and FEMNIST runs show distinct generalization patterns: SWB-DM has a 10-round warm-up deficit on CIFAR-100, while several robust methods perform well on FEMNIST.

D. Generalization: CIFAR-100 and FEMNIST

  • On FEMNIST at β=0.2 and p=0.5, FedAvg scored 71.24% under label-flip and 74.50% under sign-flip, while Bulyan, FLTrust, and Median scored 76–85%.The reported FEMNIST results average over 62 classes and 10 rounds.Source: D. Generalization: CIFAR-100 and FEMNIST · p. 6
  • On FEMNIST, the authors report that robust aggregation makes a stronger case on harder tasks, not a weaker one.This conclusion follows their comparison of FedAvg’s attack-related losses with the reported Bulyan, FLTrust, and Median accuracies.Source: D. Generalization: CIFAR-100 and FEMNIST · p. 6
  • On CIFAR-100, Krum scored 13–15%, FedAvg 19–32%, and SWB-DM 8.43–14.49% at 10 rounds.The passage attributes SWB-DM’s 10-round deficit to the cache warm-up effect.Source: D. Generalization: CIFAR-100 and FEMNIST · p. 6

E. SWB-DM’s Warm-Up Cost

Extending training shows a substantial SWB-DM warm-up effect, especially on CIFAR-10, but the relative gain varies by dataset and method.

E. SWB-DM’s Warm-Up Cost

  • All 64 CIFAR-10 configurations improved by round 25, averaging 13.12 percentage points; at p=0.1 gains averaged 15.25 points versus 11.00 at p=0.5.All 8 CIFAR-100 configurations also improved, by 10.39 points on average, from 11.93% to 22.32%.Source: E. SWB-DM’s Warm-Up Cost · p. 6
  • The cache refreshes only for sampled clients, so full warm-up takes longer when participation is low or the task is harder.The authors extended all six baselines to the same round budget to distinguish this cost from gains due to continued training alone.Source: E. SWB-DM’s Warm-Up Cost · p. 6
  • With all methods extended to 25 rounds, SWB-DM’s CIFAR-10 accuracy gain was 1.7–5.4× larger than any baseline’s, while CIFAR-100 FLTrust gained +14.19 versus SWB-DM’s +10.39.The comparison covers the full CIFAR-10 grid and a CIFAR-100 four-attack subset.Source: E. SWB-DM’s Warm-Up Cost · p. 6, Table IV · p. 6
  • Krum’s CIFAR-100 gain was only +0.93, concealing divergence in one of its two seeds rather than a warm-up cost resolved by more rounds.The passage links this poor performance to the structural fragility discussed earlier in the paper.Source: E. SWB-DM’s Warm-Up Cost · p. 6

F. Scalability at 500 Clients

The 500-client run shows substantial caching-method recovery by round 40, alongside a high SWB-DM aggregation cost and noisier recovery than DelayedMomentum.

F. Scalability at 500 Clients

  • For the N=500, p=0.1 clean-data run, SWB-DM’s aggregation cost was 13.35s per round versus FedAvg’s 0.01s.The run sampled 50 clients per round and lasted 10 rounds; the caching methods were also extended to 40 rounds.Source: F. Scalability at 500 Clients · p. 6
  • At round 40, SWB-DM reached 17.16% versus DelayedMomentum’s 20.61%, with SWB-DM dipping from 18.14% at round 35.This reverses the smaller-scale pattern, where SWB’s trimmed statistic consistently beat plain median under the same cache.Source: F. Scalability at 500 Clients · p. 7
  • The authors suggest chunk-rotation variance may explain SWB-DM’s noisier recovery when each cache slot refreshes on average once every 10 rounds.They leave this explanation as an open question.Source: F. Scalability at 500 Clients · p. 7
  • Krum’s clean-data accuracy rose from 10.10% to 23.09% by round 40, but non-monotonically, including a dip to 11.59% at round 20.The passage associates this behavior with single-client selection and a 22× spread in client data sizes.Source: F. Scalability at 500 Clients · p. 7

VIII. DISCUSSION AND LIMITATIONS

Partial participation can undermine robust-aggregation assumptions, and the resulting failure modes require per-round diagnostics. The discussion flags an unproven medoid step, a 500-client recovery gap, and remaining theory and evaluation limits.

VIII. DISCUSSION AND LIMITATIONS

  • The medoid gauge-fixing step lacks a proof of robustness; a corrupted medoid permutation could pass into the aggregate, though this did not occur in tested runs.Source: §VIII · p. 7
  • At 500 clients, DelayedMomentum reaches 20.61% while SWB-DM stalls at 17.16%, with a non-monotonic dip around round 35.The proposed explanation—chunked rotation leaving cache entries stale—is presented as an open question, not a confirmed cause.Source: §VIII · p. 7
  • At p=0.5 with low corruption, FedAvg, Bulyan, and Median beat SWB-DM as the warm-up cost bites harder.Source: §VIII · p. 7
  • The bound borrows robust-mean theory’s functional form, while tested adaptive attacks omit cache-state knowledge, leaving SWB-specific analysis and cache-aware attacks for future work.Source: §VIII · p. 7

IX. CONCLUSION

  • On CIFAR-10, SWB-DM’s caching warm-up cost exceeds that of all baselines extended to the same round budget.Source: §IX · p. 7
  • Across CIFAR-10, CIFAR-100, FEMNIST, and 500 clients, partial participation can violate finite-sample assumptions, producing distinct failure modes that require per-round diagnostics.Source: §IX · p. 7, §VIII · p. 7

Related papers

Loading 2609.16099v2…