PaperlayerView exact revision on arXiv

Source-linked AI summary

Private and Distributionally Robust Federated Multiclass Classification

Andrea P. Hernandez, Michael Ibrahim, Nagi Gebraeel

arXiv:2610.13631v1cs.LGmath.OC

TL;DR

Industrial fault diagnosis requires multiclass learning across sites, but sensitive data, feature and label uncertainty, and update leakage complicate federated training. This paper combines federated Wasserstein DRO with native multiclass classification and sample-level privacy for client updates. DP-Fed DRO achieves the best private performance on most benchmark datasets and is strongest among private methods at moderate privacy budgets on simulated fault data.

  • Problem

    Industrial fault diagnosis needs multiclass learning across sites, but data are sensitive, features and labels uncertain, and existing federated DRO work lacks native multiclass classification with sample-level privacy.

    Source: §1 · pp. 2–3

  • Method

    The paper combines federated Wasserstein DRO with the Crammer–Singer multiclass loss and sample-level differential privacy for released client updates.

    Source: §1 · p. 3

  • Results

    DP-Fed DRO achieves the best private performance on most benchmark datasets and is the strongest private method at moderate privacy budgets on simulated industrial fault data.

    Source: §8 · p. 16

  • Takeaways & Limitations

    The paper establishes sample-level privacy across training rounds and an O(1/T) convergence rate for its federated multiclass DRO framework.

    Source: §8 · p. 16

    Jointly optimizing client communication schedules and privacy-budget allocation across rounds remains future work under fixed per-client privacy budgets.

    Source: §8 · p. 17

Abstract

from arXiv · show

Fault diagnosis in asset-intensive industries requires multiclass models trained on noisy, sensitive data collected across geographically distributed client sites. Federated learning enables collaborative training without centralizing data, but most approaches do not provide formal privacy guarantees for individual records stored at client sites and can degrade under label noise. To address these limitations, we develop a differentially private, distributionally robust framework for federated multiclass classification that provides sample-level privacy for records stored at each client. We build on a federated Wasserstein distributionally robust multiclass objective with the Crammer-Singer loss, solved using an Alternating Direction Method of Multipliers (ADMM)-based algorithm. To protect the released client updates, our method injects calibrated Gaussian noise into the proximal linearized client updates of both the local model and the local distributionally robust variable before aggregation at the central server, and we quantify the cumulative privacy loss using the moments accountant. We provide per-round sample-level $(ε,δ)$-differential privacy guarantees and an overall composed privacy guarantee across all training rounds, together with convergence results and an explicit privacy-utility tradeoff. Experiments on simulated industrial fault data and UCI benchmark datasets show that our method outperforms differentially private federated empirical risk minimization and stochastic gradient descent baselines in most tested settings, with the largest gains under strong privacy-induced noise.

1 Introduction

Industrial fault diagnosis needs multiclass models trained across sites, but sensitive data, noisy features and labels, and update leakage complicate federated learning. The paper combines federated Wasserstein DRO, multiclass classification, and sample-level differential privacy, then evaluates the method on simulated fault data and UCI benchmarks.

1 Introduction

  • Industrial data are distributed across sites and raise privacy concerns; noisy sensor readings and inconsistent fault labels add uncertainty to classification.Federated learning keeps raw data local, but shared model updates may still leak sensitive information.Source: §1 · p. 2
  • Existing federated DRO multiclass approaches often use one-vs-one or one-vs-all decompositions, which can be costly or fail to capture class relationships.Prior federated DRO work includes binary classification, while multiclass DRO has primarily been studied centrally without privacy guarantees.Source: §1 · p. 2
  • The framework combines federated Wasserstein DRO, native multiclass loss, and sample-level differential privacy for released client updates.It builds on the mixture-of-Wasserstein-balls formulation and uses the Crammer–Singer loss for multiclass classification.Source: §1 · p. 3
  • The method privatizes both the local model and local DRO variable with calibrated Gaussian noise, under weaker assumptions than common DP methods.The stated contrast is with methods requiring strong convexity and smoothness.Source: §1 · p. 3
  • The paper derives sample-level privacy guarantees for noisy client updates and composes privacy loss across training rounds using the moments accountant.Source: §1 · p. 3
  • The authors analyze convergence and characterize the resulting privacy–utility tradeoff.Source: §1 · p. 3
  • The proposed method remains competitive under privacy constraints and includes scalability experiments varying the number of clients and classes.The evaluation covers simulated industrial fault diagnosis data and several UCI benchmark datasets.Source: §1 · p. 3

2 Problem Setup and Preliminaries

The setup considers clients training a linear multiclass classifier without sharing raw data. The preliminaries specify Wasserstein DRO for feature and label perturbations and differential privacy for protecting individual records.

2.1 Problem Setup

  • Each data point pairs a feature vector and class label; the goal is a linear multiclass classifier using the Crammer–Singer loss.Labels are represented with one-hot encoding, and y can denote either the class index or its one-hot encoding.Source: §2.1 · pp. 3–4

2.2 Distributionally Robust Learning

  • DRO minimizes worst-case expected loss over an ambiguity set centered on the empirical distribution, using a type-1 Wasserstein distance to model feature and label perturbations.The separable transportation cost penalizes label changes with κ.Source: §2.2 · p. 4

2.3 Differential Privacy

  • Clients communicate updates to a central server rather than sharing raw data, but those updates can reveal local dataset information.The privacy goal is protecting individual records within each client, not hiding client participation; the paper considers an honest-but-curious server and an external eavesdropper.Source: §2.3 · pp. 4–5
  • Differential privacy is defined for adjacent datasets differing in one element, and the paper uses Gaussian noise on each model update before server transmission.Smaller ε means stronger privacy; δ is a small failure probability.Source: §2.3 · p. 5

3 Federated Multiclass Distributionally Robust Formulation

The formulation combines multiclass loss with client-specific Wasserstein ambiguity sets in a federated objective. An ADMM consensus constraint separates client and server updates for distributed optimization.

3 Federated Multiclass Distributionally Robust Formulation

  • The federated multiclass formulation uses the Crammer–Singer loss and a global ambiguity set formed as a weighted mixture of local client Wasserstein balls.The local balls are centered at each client’s empirical distribution; the construction preserves separable structure and inherits finite-sample generalization guarantees of classical Wasserstein DRO.Source: §3 · p. 5
  • An ADMM-based consensus constraint sets each client’s local model equal to the global model, and the augmented Lagrangian introduces scaled dual variables.The penalty parameter ρ is associated with the consensus constraint.Source: §3 · pp. 5–6
  • Clients solve local convex subproblems and send updated models to the server, which updates the global model before clients update their multipliers and repeat.The server update is given in closed form.Source: §3 · pp. 6–7

4 DP Federated DRO Algorithm

The private algorithm replaces the exact client DRO update with a proximal linearized approximation, then perturbs local updates before release. This supports sensitivity analysis without requiring smoothness or strong convexity of the original objective.

4 DP Federated DRO Algorithm

  • The algorithm uses a proximal linearized approximation of the client objective because directly privatizing the exact local DRO update makes sensitivity analysis difficult.The exact objective does not satisfy the smoothness and strong-convexity assumptions commonly used in sensitivity analysis.Source: §4 · p. 7
  • The approximation moves the local DRO-variable norm constraint into the objective with a quadratic penalty before linearizing the client problem.The multiplier is updated each round, and the penalty parameter is γ.Source: §4 · p. 7
  • Clients add Gaussian noise to both local model and local DRO-variable updates before sending the perturbed updates to the server.The matrix-normal noise for the model has zero mean and entrywise variance (σM,g)^2.Source: §4 · pp. 7–8
  • The linearized surrogate enables tractable sensitivity bounds, while proximal terms keep updates near prior perturbed iterates and step sizes jointly control optimization progress and noise.The step sizes determine update sensitivities and therefore the amount of injected noise.Source: §4 · p. 9

5 Differential Privacy Guarantees

The section bounds the sensitivity of the local model update and DRO variable, then calibrates Gaussian noise to establish per-round privacy guarantees. It composes those guarantees across training rounds with the moments accountant.

Sensitivity bounds

  • The local model update and local DRO variable each have sensitivity bounds that determine their respective Gaussian perturbations.The update is obtained from the proximal linearized subproblem, and the DRO variable is obtained from that subproblem as well.Source: §5 · p. 9

Per-round guarantees

  • Theorem 3 gives a finite-round privacy guarantee for the perturbed client state using Gaussian noise calibrated to the sensitivity of each released variable.The sensitivity analysis considers adjacent local datasets differing in one sample; the model-update bound assumes bounded feature vectors.Source: §5 · p. 9

Composition across rounds

  • The moments accountant composes the two Gaussian mechanisms adaptively over T iterations into an overall (ε̄, δ̄)-differential privacy guarantee.The cumulative privacy loss grows roughly as √T rather than linearly in T.Source: §5 · p. 10

6 Convergence Analysis

The convergence analysis bounds the noisy federated algorithm over a finite training horizon using a combined objective and consensus metric. It also characterizes how privacy noise and local sample size affect the bound.

Convergence metric

  • The convergence analysis tracks both the objective gap and violation of consensus between averaged local and global models.The theorem states convergence in terms of a combined metric.Source: §6 · p. 11

Proof strategy

  • The analysis uses a one-step inequality for convex local objectives as the basis for the final convergence theorem.The stated step-size conditions require ηg > 0 and νg > 0.Source: §6 · p. 11

Convergence bound

  • The finite-horizon convergence bound has a leading term of order O(M1,T/T) and an additional lower-order term of order O(T^-1).The result applies under convex local objectives and bounded subgradients along the algorithm's trajectory.Source: §6 · pp. 11–12

Privacy–utility tradeoff

  • Stronger privacy requires larger Gaussian noise, increasing the convergence-bound constants and potentially reducing utility; privacy costs diminish as client sample size Ng grows.The privacy-dependent terms decrease with the local sample size.Source: §6 · p. 12

7 Experimental Evaluation

The evaluation tests privacy–utility behavior on simulated pump faults, benchmarks five UCI datasets, and measures runtime scalability as client and class counts grow. DP-Fed DRO is strongest among private methods on most UCI datasets and scales similarly to private ERM and SGD baselines.

7.1 Simulated Industrial Fault Diagnosis Data Experiments

  • At informative privacy budgets, DP-Fed DRO is the best private method on simulated pump data, with its clearest margin at ε ∈{0.05, 0.1}.Its performance is also more robust to tighter δ than DP-Fed SGD, whose performance degrades noticeably as δ tightens.Source: §7.1 · pp. 13–14

7.2 UCI Data Experiment

  • DP-Fed DRO achieves the best private performance on Iris, Wine, and Seeds, ranks second on Palmer Penguins, and performs strongest overall across the five datasets.DP-Fed ERM leads on Palmer Penguins and DP-Fed SGD on Dermatology; privacy reduces performance relative to non-private models on every dataset.Source: §7.2 · pp. 14–15, Table 1 · p. 14

7.3 Scalability Experiments

  • With total sample size fixed, private-model runtime remains relatively stable as clients increase; with samples per client fixed, runtime generally increases with client count.Non-Private Fed DRO is at least an order of magnitude slower in the fixed-total-sample setting, while private models remain within an order of magnitude of one another in the fixed-local-sample setting.Source: §7.3 · p. 16
  • As the number of classes grows, runtime rises for all methods, but more mildly for private models; DP-Fed DRO’s relative overhead stays roughly constant and its runtime remains orders of magnitude below Non-Private Fed DRO.Figure 3 reports log-scale runtime to peak mCCR across client- and class-scalability settings.Source: Figure 3 · p. 15, §7.3 · p. 16

8 Conclusion

The conclusion presents a private federated multiclass DRO framework with convergence and empirical results across simulated and benchmark data. The appendix proofs detail the client update formulation and the single-sample dependence underlying a sensitivity argument.

8 Conclusion

  • DP-Fed DRO combines Wasserstein robustness, multiclass classification, and sample-level differential privacy, with an O(1/T) convergence rate and strong private-method results across the reported experiments.The conclusion reports best private performance on most benchmark datasets and runtime scaling similar to private baselines.Source: §8 · p. 16

A.1 Proof of Proposition 1

  • The client update minimizes an augmented local Lagrangian, and the robust-loss constraints are rewritten using the multiclass Crammer–Singer loss.Substituting the rewritten constraints yields the clientwise subproblem stated in Proposition 1.Source: §A.1 · pp. 19–20

A.2 Proof of Lemma 1

  • For adjacent local datasets differing in one sample, the fixed-state update comparison cancels unchanged terms, leaving the changed sample’s robust-loss contribution as the source of the difference.The proof uses this reduction to establish the stated sensitivity bound.Source: §A.2 · p. 20

A.3 Proof of Lemma 2

The appendix establishes a sensitivity result for the local DRO-variable update, develops an inequality using blockwise bounds and server aggregation, and proves finite-horizon boundedness of the generated trajectory.

A.3 Proof of Lemma 2

  • For adjacent datasets differing in one sample, the λ_g update comparison leaves only that sample’s robust-loss contribution, yielding the stated sensitivity result.The proof conditions on the algorithmic state at the start of the iteration.Source: §A.3 · p. 21

A.4 Proof of Lemma 3

  • The proof combines blockwise bounds, sums them across clients with weights α_g, and uses the dual update and server aggregation identities to derive the inequality.It then adds the scaled-dual consensus term in the bound.Source: §A.4 · pp. 21, 23, 24

A.5 Auxiliary Lemmas

  • For a fixed finite training horizon, the generated trajectory contains finitely many points and admits a finite upper bound.The proof establishes finiteness of the iterates over the horizon by induction.Source: §A.5 · pp. 27–28

A.6 Proof of Theorem 2

The proof analyzes the Gaussian-perturbed model and local DRO-variable updates under adjacent datasets. It composes their privacy losses across rounds with the moments accountant to obtain an overall differential privacy guarantee.

A.6 Proof of Theorem 2

  • The proof defines privacy losses for the perturbed model update and local DRO variable, and analyzes neighboring Gaussian output laws using their log moments.The Gaussian mechanisms are calibrated using the respective update sensitivities.Source: §A.6 · pp. 28–30
  • The moments accountant adds the log moments of both mechanisms within each iteration and across all T iterations.The proof sets δ_M = δ_λ = δ for the two mechanisms.Source: §A.6 · p. 31
  • A tail bound is used to choose the composed privacy parameter, yielding an overall (ε̄, δ̄)-differential privacy guarantee for the adaptive composition.The proof chooses ε̄ so the tail bound is at most δ̄.Source: §A.6 · p. 31

A.7 Proof of Theorem 3

The proof establishes the convergence bound under stated regularity conditions and noise assumptions. The supplementary material also documents the experimental setup, software, and simulated pump dataset.

A.7 Proof of Theorem 3

  • The convergence-bound proof uses a convex local objective, independent zero-mean perturbations, and uniform bounds on penalty subgradients along the finite training trajectory.The proof concludes that these steps establish the claimed bound.Source: §A.7 · pp. 32, 35, 37

B Further Experimental Details and Supplementary Results

B.1 Software and Hardware Details

  • Experiments ran on Intel Xeon Gold 6226 processors, with two CPU cores allocated to each job-array task.Table 2 presents the software-environment details.Source: §B.1 · p. 37, Table 2 · p. 38

B.2.1 Simulated Industrial Fault Diagnosis Dataset

  • The simulated pump dataset contains 2,000 runs across healthy operation and three single-fault classes, represented by 13 extracted tabular features.Fault severities were sampled from selected upper levels, with different levels used for each fault type.Source: §B.2.1 · p. 38

B.2.2 UCI Datasets

The experiments include five public benchmark datasets from the UCI Machine Learning Repository.

B.2.2 UCI Datasets

B.3 Additional Convergence Figures

In the controlled overlap study, native multiclass Fed DRO outperforms its OvA counterpart across all three regimes. Additional convergence plots separate the objective and consensus-violation traces across privacy budgets.

B.3 Additional Convergence Figures

  • The additional plots separate the running average augmented objective from consensus violation across privacy budgets; consensus violation falls toward zero after initially rising.The consensus-violation peak is smaller and its decay faster as ϵ increases.Source: §B.3 · p. 38, Figure 4 · p. 39
  • At ϵ = 0.05, the objective rises before settling higher, with the highest level at the tightest δ values; at ϵ ≥ 0.15, it decreases after its initial peak.At ϵ = 0.1, it remains nearly flat after the first rounds, and δ effects fade as ϵ increases.Source: §B.3 · p. 39

B.4 Controlled Study on Correlated Fault Classes

  • The controlled study varies Fault A/Fault B overlap in a four-class synthetic setting with correlated features and 10% training-label noise.It uses 860 training samples, 216 balanced test samples, four clients, and 50 communication rounds.Source: §B.4 · p. 39
  • Native Fed DRO outperforms Fed OvA DRO across all three Fault A/Fault B overlap regimes, with the clearest gains in moderate- and low-overlap settings.The gains include higher mCCR, Macro-F1, and worst-class recall, driven largely by improved Fault A recovery.Source: Table 4 · p. 40, §B.4 · p. 40
Loading 2610.13631v1…