PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Locally, Act Globally: Federated Learning with Local and Global Representations

Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B. Allen, Randy P. Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency

arXiv:2001.01523v3cs.LGcs.DCstat.ML

TL;DR

Federated learning must communicate model updates without exposing private device data, but large global models make this costly. The paper proposes LG-FEDAVG, which combines compact local representations with a global model, and reports strong performance alongside lower communication costs, heterogeneous-data handling, and fair representations.

  • Problem

    Federated learning communicates parameter updates instead of private device data, creating scalability challenges as global models become larger.

  • Method

    LG-FEDAVG jointly learns compact local representations on devices and a global model that operates on those representations.

  • Results

    LG-FEDAVG achieves strong performance while reducing communication costs, learning personalized models, handling heterogeneous data, and obfuscating protected attributes.

  • Takeaways & Limitations

    Local models support communication-efficient training, new-device heterogeneity, personalized prediction, and fair representations within federated learning.

  • Takeaways & Limitations

    The authors identify unresolved trade-offs among security, privacy, and performance and call for further quantification.

Abstract

from arXiv · show

Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalization analysis which shows that a combination of local and global models reduces both variance in the data as well as variance across device distributions. Empirically, we demonstrate that local models enable communication-efficient training while retaining performance. We also evaluate on the task of personalized mood prediction from real-world mobile data where privacy is key. Finally, local models handle heterogeneous data from new devices, and learn fair representations that obfuscate protected attributes such as race, age, and gender.

1 Introduction

LG-FEDAVG addresses federated learning’s communication challenge by jointly learning compact local representations and a global model. The approach targets efficiency, heterogeneous devices, and fair representations while maintaining performance.

  • LG-FEDAVG jointly learns compact local representations on each device and a global model across devices.
  • The analysis indicates that combining local and global models reduces variance in the data and across device distributions.
  • Efficiency: Smaller local representations reduce the global model’s parameters and communicated updates while maintaining performance on MNIST, CIFAR, and VQA.
  • Heterogeneity: Local representations process unseen device data with specialized encoders, improving handling of heterogeneous modalities and distributions.
  • Fairness: Local models can learn fair representations that obfuscate protected attributes including race, age, and gender.

2 Related Work

Related work situates LG-FEDAVG within federated learning research on scale, heterogeneity, communication efficiency, and fair representations. Its framework combines these concerns rather than targeting only one.

  • Federated learning addresses large-scale training across private, heterogeneous data sources and multiple learning objectives.
  • LG-FEDAVG combines heterogeneous-device handling, reduced communication complexity, and fair representation learning in one framework.
  • Prior work improves communication through sparsification, efficient gradients, and update compression, techniques described as complementary to LG-FEDAVG.

3 Local Global Federated Averaging

LG-FEDAVG trains local encoders and a shared global predictor end to end, so the global model operates on compact representations rather than raw device data. The design supports communication efficiency, flexible local learning, and new-device prediction.

  • Training Procedure: During training, devices update local models and local copies of global parameters, returning updated global parameters for server aggregation.
  • Local Representation Learning: Each device learns a low-dimensional representation that captures predictive features without overfitting its device-specific distribution.
  • Local Representation Learning: Local learning may be supervised, unsupervised, or self-supervised, with adversarial training available for fair representations.
  • Global Model: The global model predicts from representations across devices, allowing it to be substantially smaller than a model operating on raw inputs.
  • Communication: Communication occurs only for global-model training, and the global model can remain small when local representations are effective.
  • New Test: For unseen devices, averaging local-model logits provides an ensemble-based prediction procedure after one-time local-weight exchange.

4 Theoretical Analysis

The theory decomposes federated generalization into data and device variation, showing that local and global extremes address different sources of error. An intermediate local-global combination can outperform either extreme under nonzero variation.

  • Assumptions: The analysis models shared global features and device-specific features, with device variance represented by ρ^2 and target noise by σ^2.
  • Baseline Analysis: Local models control data variance using only device-local samples but do not suffer from device variance.
  • Baseline Analysis: Global models reduce data variance at rate d/(MN) but incur an O(ρ^2) device-variance term when devices differ substantially.
  • LG-FEDAVG Analysis: LG-FEDAVG interpolates between local and global models through α, balancing their respective generalization-error trade-offs.
  • LG-FEDAVG Analysis: When ρ^2 and σ^2 are positive, the optimal interpolation has lower generalization error than both purely local and purely global models.
  • LG-FEDAVG Analysis: Higher device variance favors local models, whereas higher data variance favors global models.
  • Connection to Deep Networks: The theory is derived for linear models, while experiments use deep networks and interpret α as the split between local and global layers.

5 Experiments

Experiments evaluate LG-FEDAVG across synthetic, image, multimodal, personalized, heterogeneous, and fairness settings. The results indicate that combining local representations with global models can preserve or improve performance while reducing communication and addressing device variation.

  • Verifying Theoretical Analysis: Synthetic experiments show α-interpolation of local and global models performs close to or better than either extreme across variance settings.Local models are favored at higher device variance, whereas global models are favored at lower device variance.
  • Verifying Theoretical Analysis: LG-FEDAVG consistently outperforms local-only and FEDAVG models on highly non-i.i.d. CIFAR-10 splits, with larger gaps under greater device variance.Devices receive examples from at most s∈{2, 3, 4, 5, 10} classes; s=2 represents the highest device variance.
  • Model Performance & Communication Efficiency: On CIFAR-10 new-device testing, LG-FEDAVG achieves similar performance to FEDAVG using around 50% of the total parameters, and better performance with the same parameter count.For local testing, it significantly outperforms FEDAVG because local models better capture device-specific distributions.
  • Model Performance & Communication Efficiency: LG-FEDAVG reaches 40% accuracy on VQA while requiring lower communication costs than FEDAVG.The evaluation uses local LSTM and ResNet-18 encoders with a global early-fusion model.
  • Heterogeneous Data in an Online Setting: LG-FEDAVG relieves catastrophic forgetting on rotated MNIST, achieving 93% online rotated-MNIST performance and 97% regular-MNIST performance without requiring fine-tuning.FEDAVG without fine-tuning reaches 92% on rotated MNIST but only 32% on regular MNIST.
  • Learning Fair Representations: LG-FEDAVG+Adv makes protected attributes unpredictable at approximately 50% adversary AUC, with only an approximately 4% drop in global accuracy.A post-fit classifier trained on local representations produces similarly random protected-attribute predictions.

6 Conclusion

The paper proposes LG-FEDAVG, combining local representation learning with federated global-model training. Its analysis and experiments report reduced variance, lower communication costs, personalization, improved handling of heterogeneous data, and fair representations that obfuscate protected attributes.

  • Conclusion: LG-FEDAVG combines local representation learning with federated training of global models.The theoretical analysis states that ensembles of local and global models reduce both data variance and device variance.
  • Conclusion: Across real-world datasets, LG-FEDAVG achieves strong performance while reducing communication costs, learning personalized models, handling heterogeneous data, and obfuscating protected attributes.

Broader Impacts

Federated learning supports large-scale learning from distributed private data, but communication, privacy, security, and social-bias concerns remain important broader-impact constraints. The paper presents local-global modeling as a flexible framework for balancing performance, communication, privacy, security, and fairness.

  • Applications: Federated learning has applications in mobile health, including monitoring mobile data for mental-health assessment and early intervention with participant permission.The paper motivates timely prediction of acute suicide risk as one mental-health application.
  • Risks and limitations: Federated learning still faces risks from consent, model-update leakage, external attacks, imbalanced data, and amplified social biases.The paper calls for further work quantifying trade-offs among security, privacy, performance, bias, and fairness.
  • Theory: The theoretical analysis decomposes generalization error into variance and bias terms, with variance arising from local and global parameter estimates.The analysis also distinguishes data variance from device variance in federated learning.
  • Theory: The proposed local-global ensemble interpolates between local and global models to reduce both data variance and device variance.The model’s interpretation depends on the balance between local and global parameter estimates.

A.1 Analysis of Federated Learning Baselines

The baseline analysis contrasts local and global federated models: local models avoid device variance but have limited data, whereas global models use more data but incur device variance.

  • Local models: Local models are trained separately on each device using only local data.The local baseline is represented by setting α=1 in the interpolation analysis.
  • Local models: Local models control data variance at rate d/N but do not suffer from device variance.Their local data may be limited in number and vary in quality and quantity across devices.
  • Global models: Global models update one joint model using data from all devices.The global baseline is represented by setting α=0.
  • Global models: Global models control data variance at rate d/(MN), but incur an additional O(ρ^2) device-variance term.Using all devices improves the data-variance rate, while one global model may not account for very different devices.

A.2 Analysis of LG-FEDAVG

LG-FEDAVG interpolates between local and global models, with a theoretically optimal balance that reduces both data and device variance. Its local representations can also be trained for fairness by removing protected-attribute information.

  • A.2 Analysis of LG-FEDAVG: LG-FEDAVG defines an α-interpolation between local and global models to target the trade-off between their different generalization errors.The interpolation parameter α lies between the local and global extremes.
  • A.2 Analysis of LG-FEDAVG: The interpolated model’s generalization error combines device-discrepancy, local and global variance, and covariance terms.The variance terms scale with the local and global model contributions.
  • A.2 Analysis of LG-FEDAVG: When both σ^2 and ρ^2 are nonzero, the optimal model has lower generalization error than either the local or global baseline.This comparison is stated by Corollary 1.
  • A.2 Analysis of LG-FEDAVG: Higher device variance favors local models, whereas higher data variance favors the global model.The optimal α increases with ρ^2 and decreases with σ^2.
  • B.1 Fair Training of Local Models: Fair local representations are trained so their conditional distributions are invariant to protected attributes.The paper uses adversarial training to make representations independent of protected attributes under the local device distribution.
  • B.1 Fair Training of Local Models: An adversarial model predicts protected attributes from local representations while the local model is trained against that objective.The local, global, and adversarial models are updated through a minimax procedure.
  • B.1 Fair Training of Local Models: Separating local and global models enables fair intermediate representations in the federated-learning setting.The paper describes this extension as analyzed both theoretically and empirically.
  • Representation learning choices: LG-FEDAVG also supports unsupervised, self-supervised, and supervised local representation-learning choices.Auxiliary local models may reconstruct inputs, predict auxiliary features, or share supervised classification branches.

B.2 Theoretical Analysis of Local Fair Representation Learning

The analysis shows that adversarially trained local representations can remain optimal for prediction while becoming invariant to protected attributes. The framework jointly optimizes local, global, and adversarial models through an alternating minimax procedure.

  • Optimization framework: The local, global, and adversarial models are jointly trained for prediction and adversarial objectives through a dual minimax optimization.The adversarial model controls the tradeoff between prediction and protected-attribute obfuscation through the dual objective.
  • Optimization framework: The minimax solution is computed by alternating updates to the local, global, and adversarial parameters until convergence.The losses use expected log likelihood through the inference networks and are optimized with gradient descent.
  • Theoretical result: Adversarial training makes the optimal local classifier invariant to protected attributes under the local data distribution.The result follows when the minimax lower bound is active, making the protected attributes and local representation independent.
  • Inference paths: Figure 3 traces inference through local, global-copy, and adversarial models for the two objectives.The global-model inference path also uses the local representation and protected attributes, motivating adversarial training of the global model.
  • Theoretical result: At the optimum, the local representation both minimizes prediction loss and maximizes the conditional entropy of protected attributes.Maximizing conditional entropy to the marginal entropy implies independence between the representation and protected attributes.

C.1 Synthetic Data

Synthetic and benchmark experiments examine how local and global models behave as data and device variance change. LG-FEDAVG maintains or improves performance while reducing communicated parameters across image, multimodal, and heterogeneous settings.

  • Synthetic data: As device variance decreases from ρ=0.5 to ρ=0.02 with data variance fixed at σ=1.5, the experiments span regimes favoring local or global models.The four settings range from local models near optimal to global models near optimal.
  • Synthetic data: α-interpolation of local and global models performs near the better extreme when one model is optimal and can outperform both extremes in intermediate regimes.Figure 4 reports this pattern for the four variance settings.
  • Image benchmarks: LG-FEDAVG maintains performance on MNIST and CIFAR while using fewer communicated parameters.The method consistently performs better on local tests and similarly on new tests across device splits, using around 50% of communicated parameters.
  • VQA: 41.30% VQA accuracy is achieved by LG-FEDAVG versus 40.22% for FEDAVG while using only 9.53% of the parameters.The convergence plot shows LG-FEDAVG surpassing FEDAVG after 20 rounds and continuing to improve.
  • Baselines: The multitask-learning baseline becomes infeasible at larger user counts or model sizes because it stores a p×M matrix of local parameters.In the CIFAR-10 setting, the authors reduce M from 100 to 10 as a relaxation.

C.3 Learning Personalized Mood Predictors from Mobile Data

The MAPS benchmark uses private, highly personalized mobile data to evaluate federated mood prediction. LG-FEDAVG’s local-global splits combine device-specific representations with information shared across devices and outperform local or global extremes.

  • Dataset: MAPS contains 572 samples from 14 participants, discretized into five mood classes for federated learning evaluation.The benchmark uses real-world mobile data with privacy concerns and high device variance.
  • Model and evaluation: Performance is averaged over 10 random 80/10/10 train, validation, and test splits.Within each iteration, the best model is selected on validation performance before reporting test accuracy.
  • Model and evaluation: The study evaluates personalized mood prediction from mobile text using a Bi-LSTM encoder and an MLP classifier.The Bi-LSTM has 128 hidden units, while the MLP has two hidden layers of size 512.
  • Results: α-splits across local and global models outperform both local-only and global-only extremes on the mobile mood task.The evaluated splits are α∈{0.2, 0.4, 0.6, 0.8}.

D Discussion and Future Work

The discussion presents LG-FEDAVG as a flexible framework for communication-efficient, heterogeneous, and fair federated learning. Future work focuses on combining compression methods and dynamically choosing the local-global model split for different test scenarios.

  • Future extensions: Existing sparsification, efficient-gradient, compression, distillation, and hashing methods could further reduce communication for LG-FEDAVG.These techniques are described as complementary to the local-global architecture.
  • Future extensions: The preferred local-global model split depends on whether test data comes from a known device or an unknown device.Known-device tests favor more accurate local models, whereas unknown-device tests require a more accurate global model.
  • Future extensions: Dynamic learning of how many layers belong to local versus global models is proposed as future work.Different devices could use dynamically learned local models for their differing data distributions.
  • Empirical discussion: MNIST results report higher local-test performance and similar new-test performance than FEDAVG while using around 50% of communicated parameters.The comparison covers non-i.i.d. device splits ranging from 2 to 10 classes per device.
Loading 2001.01523v3…