Ditto: Fair and Robust Federated Learning Through Personalization

Tian Li; Shengyuan Hu; Ahmad Beirami; Virginia Smith

Source-linked AI summary

Ditto: Fair and Robust Federated Learning Through Personalization

Tian Li, Shengyuan Hu, Ahmad Beirami, Virginia Smith

arXiv:2012.04221v3Dec 8, 2020cs.LGstat.ML

TL;DR

Federated learning must reconcile fairness and robustness, which can conflict in statistically heterogeneous networks. The paper proposes Ditto, a scalable personalized federated multi-task framework, and finds that it delivers competitive personalization performance alongside improved accuracy, robustness, and fairness across federated benchmarks.

Problem
Fairness and robustness can compete when federated learning trains a single global model over statistically heterogeneous devices.
Method
Ditto is a scalable federated multi-task learning framework that adds lightweight personalization to global federated objectives while retaining similar efficiency and privacy properties.
Results
Across federated datasets, Ditto provides more accurate, robust, and fair models than strong fair or robust baselines, including a ∼6% absolute test-accuracy gain over the strongest robust baseline.
Takeaways & Limitations
Personalized federated learning can jointly address accuracy, fairness, and robustness rather than optimizing these constraints separately.
Takeaways & Limitations
The theoretical analysis is limited to simplified attacks and problem settings, with linear-regression results requiring additional feature-covariance assumptions.

Abstract

from arXiv · show

Fairness and robustness are two important concerns for federated learning systems. In this work, we identify that robustness to data and model poisoning attacks and fairness, measured as the uniformity of performance across devices, are competing constraints in statistically heterogeneous networks. To address these constraints, we propose employing a simple, general framework for personalized federated learning, Ditto, that can inherently provide fairness and robustness benefits, and develop a scalable solver for it. Theoretically, we analyze the ability of Ditto to achieve fairness and robustness simultaneously on a class of linear problems. Empirically, across a suite of federated datasets, we show that Ditto not only achieves competitive performance relative to recent personalization methods, but also enables more accurate, robust, and fair models relative to state-of-the-art fair or robust baselines.

1. Introduction

Federated learning must balance accuracy with fairness and robustness, but these constraints can conflict under heterogeneous data. Ditto uses personalization to address them jointly and achieves strong accuracy, robustness, and fairness results.

Motivation: Fairness and robustness can directly compete when a single global model must satisfy both under heterogeneous data.Fairness methods may increase susceptibility to attacks, while robust methods may discard rare informative updates.
Approach: Ditto is a scalable federated multi-task framework that adds lightweight personalization to global federated learning.It applies to convex and non-convex objectives while retaining similar privacy and efficiency properties to traditional FL.
Results: Ditto improves robustness across diverse data and model poisoning attacks, raising test accuracy by ∼6% absolute over the strongest robust baseline.The improvement is averaged across all datasets and attacks.
Results: Ditto reduces test-accuracy variance across devices by ∼10% while maintaining similar or superior accuracy to state-of-the-art fair FL methods.This result measures fairness through performance uniformity across devices.
Implications: The framework is intended for applications that simultaneously require accuracy, fairness, and robustness.The paper motivates this use through theoretical analysis and experiments across federated datasets.

2. Background & Related Work

Federated-learning fairness concerns performance disparities across heterogeneous devices, while robustness concerns resilience to malicious updates. Prior defenses and fairness methods address these dimensions separately and can create trade-offs, motivating Ditto’s joint treatment.

Robustness: Robustness is evaluated by mean test performance on benign devices after training under a specified attack.The threat model includes Byzantine devices that may send arbitrary updates.
Threat Model: The evaluated attacks include label poisoning, random Gaussian updates, and model replacement through scaled adversarial updates.These attacks corrupt data, labels, or model updates at different points in the training process.
Robustness and Trade-offs: Common robust defenses can improve robustness but may produce unfair models by filtering informative updates in heterogeneous settings.The paper compares Ditto with median, clipping, Krum, Multi-Krum, anomaly detection, and another proposed defense.
Fairness: Fairness is defined by the uniformity of test performance across devices, measured using the standard deviation of device test losses.In adversarial settings, fairness is measured only across benign devices.
Fairness: Lowering performance variance generally must be balanced against maintaining reasonable average test accuracy.The paper identifies variance and utility as competing objectives in its fairness definition.
Personalization: Ditto achieves similar or superior performance to common personalized methods and jointly explores personalization’s fairness and robustness benefits.The paper distinguishes this joint focus from prior work treating the two connections separately.

3. Ditto: Global-Regularized Federated Multi-Task Learning

Ditto formulates personalized federated learning as a global objective paired with device-specific objectives regularized toward the global model. This scalable, modular framework supports convergence guarantees and is analyzed and evaluated for accuracy, fairness, and robustness.

Ditto objective: Ditto jointly learns a global model and personalized device models, with regularization coupling each personalized model to the global model.The framework treats global and local objectives as two related tasks in federated multi-task learning.
Ditto objective: λ interpolates between local and global learning: λ=0 yields local models, while λ→+∞ recovers the global objective.The parameter controls how strongly personalized models are regularized toward the global model.
Solver and modularity: Ditto is modular: existing global-model optimizers can update the global component while the personalization add-on preserves compatible convergence benefits, privacy, and communication properties.The framework can plug in methods beyond FedAvg and communicates the same information as typical global-objective federated solvers.
Solver and guarantees: Under strong convexity and smoothness, Ditto inherits local convergence from the global solver; with FedAvg, it achieves O(1/t) convergence under suitable conditions.The convergence result applies when the global iterates converge at rate g(t).
Fairness and robustness analysis: For the analyzed point-estimation problem, a proper λ makes Ditto Bayes optimal, while λ* decreases with more local samples, less-related devices, or stronger attacks.The theoretical analysis identifies a parameter choice that simultaneously achieves the most accurate, robust, and fair solution for this problem instance.
Fairness and robustness analysis: Empirically, λ* minimizes average test error and cross-device standard deviation simultaneously, and outperforms local and global models as device relatedness changes.The reported comparison covers clean and adversarial cases; λ* should increase as relatedness, measured by 1/τ, increases.

4. Experiments

Experiments evaluate Ditto’s accuracy, fairness, and robustness across federated benchmarks and poisoning attacks. Ditto generally outperforms strong robust and fair baselines while remaining competitive with other personalization methods.

Robustness: Ditto achieves the highest accuracy under most poisoning attacks, especially when a large fraction of devices is malicious.Across datasets and attacks, it improves absolute accuracy by approximately 6% over the strongest robust baseline.
Fairness: Ditto reduces test-accuracy variance across devices by approximately 10% while improving clean-data accuracy by 5% over TERM.These results compare Ditto with a recent fair federated-learning objective across the evaluated datasets.
Competing Constraints: Fair methods can become particularly susceptible to attacks because they may overfit to corrupted devices by assigning them greater weight.The experiments examine this behavior under data poisoning in statistically heterogeneous environments.
Competing Constraints: Robust baselines are either more robust but less fair, or fail to provide robustness under heterogeneity, whereas Ditto is more robust, accurate, and fair.Figure 5 positions Ditto in the preferred lower-right region of the robustness–fairness comparison.
Additional Properties: Ditto achieves similar or superior accuracy with slightly lower standard deviation than recent personalization methods.The comparison uses tuned baseline methods and evaluates average test accuracy and its standard deviation across devices.

5. Conclusion and Future Work

The paper proposes Ditto as a lightweight personalization framework for jointly addressing accuracy, fairness, and robustness in federated learning. Its analyses and experiments support benefits across diverse attacks, while identifying several directions for future study.

Conclusion: Ditto is a simple multi-task-learning framework that acts as a lightweight personalization add-on for global federated objectives.It is designed to preserve the privacy and communication efficiency of the global solver.
Conclusion: The paper theoretically analyzes Ditto’s ability to mitigate fairness–robustness tension on a class of linear problems.The theoretical scope is limited to the analyzed linear setting.
Conclusion: Empirical results show that Ditto can produce more robust and fairer models than strong baselines across diverse attacks.The conclusion presents this as evidence that personalization can address competing federated-learning constraints.
Future Work: Future work includes backdoor attacks, other personalized methods, and additional constraints such as privacy.The paper specifically identifies these as directions for extending the analysis.

A. Analysis of the Federated Multi-Task Learning Objective Ditto

The theoretical analysis studies Ditto in linear settings, covering its training properties and its accuracy, fairness, and robustness benefits. It also considers federated linear regression and federated point estimation.

Theoretical Analysis: The analysis first establishes general training-performance properties of Ditto for strongly convex functions.These properties are presented before the main linear-model results.
Theoretical Analysis: The main theoretical results characterize Ditto’s accuracy, fairness, and robustness benefits for federated linear regression.Federated linear regression is treated as the central linear-model setting.
Theoretical Analysis: A special case of linear analysis is presented for federated point estimation.The point-estimation case follows the general strongly convex and linear-regression analyses.

A.1. Properties of Ditto for Strongly Convex Functions

Ditto’s strongly convex analysis characterizes its solutions and shows how personalization strength trades off local fit against proximity to the global model. The theoretical guarantees require an explicit joint data distribution, examined later through a Bayesian linear framework.

Ditto’s strongly convex objective is analyzed without distributional assumptions on the tasks.
As λ increases, local empirical training loss increases while personalized models move closer to the global model.At λ = 0, separate local models minimize training loss and are most robust and fair in training performance when generalization is ignored.
Test-performance guarantees require explicitly modeling the joint distribution of data across devices.The next analysis examines generalization, fairness, and robustness on test data within a Bayesian linear setting.
The analysis uses a Bayesian framework with K device parameters and a non-informative prior over the shared parameter θ.Device-specific parameters are modeled through noisy observations of θ.
The linear-regression analysis derives optimal device estimators from observations across devices and establishes their multivariate normal characterization.

Further let

For the no-adversary linear setting, Ditto’s regularization parameter can be chosen to optimize estimation accuracy and fairness simultaneously. The resulting solution connects personalization strength to sample size and task heterogeneity.

Ditto’s local objective and estimator are analyzed using device-specific empirical losses and a global-model regularization parameter λ.
λ∗ = σ2/(nτ2) yields the MMSE estimator and is optimal among federated linear regression algorithms for device estimation.The paper states that this choice realizes the MMSE estimator for each device.
The same λ∗ minimizes expected mean square error and the variance of test performance across devices.Thus, test error and fairness can be jointly minimized with one λ.
As n →∞, λ∗→0, making local models optimal for generalization and fairness.
As τ →∞, λ∗→0, whereas as τ →0, λ∗→∞, favoring local models for unrelated tasks and a global model for identical tasks.
Without adversaries, the same λ∗ achieves the best expected mean square error and fairness under the analyzed model.

A.2.2. WITH ADVERSARIES: DITTO FOR ACCURACY, FAIRNESS, AND ROBUSTNESS

With adversaries, Ditto can jointly optimize robustness and fairness within its solution space, while the optimal personalization level increases with the number and capability of corrupted devices. The analysis reports a fundamental fairness–robustness tradeoff and a scope limited to the specified model.

The adversarial model distinguishes benign devices from malicious devices through their differing distributions relative to θ.
In the adversarial analysis, robustness is mean test performance on benign devices and fairness is performance variance across benign devices.
The same λa can minimize mean test error and performance variance across benign devices in the presence of adversaries.
As adversary task distance τa increases, λ∗a decreases toward local-model training as the robustness- and fairness-optimal choice.
λ∗a decreases as the number and capability of corrupted devices increase, so more adversarial attacks require more personalization.
Increasing adversary count Ka or power τa raises both the smallest test error and optimal fairness variance, revealing a fairness–robustness tradeoff.
The analysis concludes that an appropriate λ makes Ditto more accurate, robust, and fair than global or local models in the specified linear problem.The authors identify generalization to more general models as future work.

A.3. The Case of Federated Point Estimation

The point-estimation analysis models device parameters and observations in a one-dimensional linear setting, then characterizes Ditto’s optimal regularization for test performance and fairness. In both benign and adversarial settings, the analysis identifies λ choices that minimize relevant performance criteria.

The analysis considers one-dimensional federated point estimation as a special case of linear regression, with device-specific data-distribution parameters.
Each device has n data points, with independently Gaussian observation noise, and its empirical-loss minimizer is denoted by bwk.
Ditto’s device model bwk(λ) combines the global optimum w∗ with the device’s empirical minimizer bwk through λ-dependent weighting.
λ∗ minimizes test performance in the point-estimation setting.
Among Ditto’s solutions without adversaries, λ∗ minimizes the variance of test mean square error across devices, yielding the fairest performance.
With adversarial and benign devices modeled separately, the analysis extends the point-estimation results to poisoning settings and characterizes corresponding optimal λ solutions.

B. Algorithm and Convergence Analysis

The paper presents Ditto as a scalable personalization solver layered onto global federated learning and analyzes how personalized models converge under partial participation and local updating. Under stated assumptions, personalized models inherit the convergence rate of the global model, with λ linked to cross-device heterogeneity.

Algorithm: Algorithm 2 is the main experimental Ditto solver and is a special case of the more general solver in Algorithm 1.
Convergence: Personalized models can inherit the convergence rate of the global model for a general global objective G(·).
Assumptions: The convergence analysis assumes global-model convergence, strongly convex local objectives, bounded stochastic gradients, and bounded distance between local and global optima.
Convergence: The convergence discussion links λ to heterogeneity: λ scales inversely with the bound M on local-global deviation, consistently with the fairness and robustness analysis involving τ^2.
Convergence: For FedAvg, Corollary 1 specializes the convergence result to Algorithm 2 using the global model’s O(1/t) convergence rate.

C.1. Datasets and Models

The experiments evaluate Ditto across heterogeneous image and text federated benchmarks, alongside several personalization baselines and finetuning procedures. The section also describes validation-based λ selection and robust-baseline combinations used in the evaluations.

Datasets: The benchmark suite includes image datasets FEMNIST, CelebA, and Fashion MNIST, plus the text dataset StackOverflow, using convex and non-convex models.
Datasets: FEMNIST includes manually partitioned and naturally partitioned versions with different levels of statistical heterogeneity.
Baselines: Personalization comparisons include MOCHA, APFL, EWC, L2SGD, Mapper, Per-FedAvg, and Symmetrized KL.
Finetuning: Finetuning is a possible Ditto solver, but in non-convex settings a corrupted global model can produce inferior performance, and selecting a stopping point is difficult from training or validation data alone.
λ selection: The evaluation assumes the server does not know which devices are benign or malicious and lets devices select λ locally from candidate values using validation data.
Robustness: Ditto is also evaluated when combined with robust baselines, including learning a robust global model through robust aggregation.

D.4. Ditto Complete Results

The complete-results evaluation reports test performance across devices for all attacks, datasets, and defense baselines. It uses randomized train-validation-test splits and standardized dataset-specific training settings, with additional comparisons on Vehicle and multiple image and text benchmarks.

Evaluation: Results are reported for robustness and fairness across all attacks and datasets, compared with all defense baselines.
Evaluation: Local data are randomly split into 72% training, 8% validation, and 20% test sets, with results reported on test data.
Evaluation: The experiments use dataset-specific learning rates and batch sizes, then run the same number of communication rounds for all attacks on each dataset.
Defense baselines: Robust-baseline settings include coordinate-wise median, Krum variants, k-norm, k-loss, and gradient clipping, with some methods given the expected number of malicious devices.
Datasets: Vehicle results use a linear SVM and additionally compare Ditto with MOCHA on a convex problem.
Results tables: Complete results are provided for FEMNIST, Fashion MNIST, FEMNIST skewed, CelebA, and StackOverflow using average and standard deviation of test accuracy across devices.