PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilled One-Shot Federated Learning

Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, Dapeng Wu

arXiv:2009.07999v3cs.LGcs.AIstat.ML

TL;DR

Federated learning needs lower communication costs while preserving useful global-model performance under heterogeneous private data. DOSFL distills each client’s data into synthetic artifacts, exchanges them in one round, and trains a global model. The paper reports up to 99.9% lower communication than FedAvg and at least 93% preserved accuracy across evaluated tasks, with security depending on the unknown initialization.

  • Problem

    Federated learning faces expensive communication and statistical heterogeneity, while existing approaches can still require hundreds of rounds or increased per-round information.

  • Method

    DOSFL has clients distill private datasets into synthetic data, labels, and learning rates for a shared initialization, then has the server train on their combined distilled data.

  • Results

    Up to 99.9% communication reduction versus FedAvg is reported, with at least 93% preserved accuracy across evaluated tasks and almost 99% accuracy for LP-DOSFL on IID MNIST.

  • Takeaways & Limitations

    DOSFL offers a weight-less and gradient-less route to quickly obtain a performant global model, after which clients can continue training or personalize locally.

  • Takeaways & Limitations

    The paper leaves defenses against security threats beyond eavesdropping, including poisoning attacks, beyond its scope.

Abstract

from arXiv · show

Current federated learning algorithms take tens of communication rounds transmitting unwieldy model weights under ideal circumstances and hundreds when data is poorly distributed. Inspired by recent work on dataset distillation and distributed one-shot learning, we propose Distilled One-Shot Federated Learning (DOSFL) to significantly reduce the communication cost while achieving comparable performance. In just one round, each client distills their private dataset, sends the synthetic data (e.g. images or sentences) to the server, and collectively trains a global model. The distilled data look like noise and are only useful to the specific model weights, i.e., become useless after the model updates. With this weight-less and gradient-less design, the total communication cost of DOSFL is up to three orders of magnitude less than FedAvg while preserving between 93% to 99% performance of a centralized counterpart. Afterwards, clients could switch to traditional methods such as FedAvg to finetune the last few percent to fit personalized local models with local datasets. Through comprehensive experiments, we show the accuracy and communication performance of DOSFL on both vision and language tasks with different models including CNN, LSTM, Transformer, etc. We demonstrate that an eavesdropping attacker cannot properly train a good model using the leaked distilled data, without knowing the initial model weights. DOSFL serves as an inexpensive method to quickly converge on a performant pre-trained model with less than 0.1% communication cost of traditional methods.

1 Introduction

Federated learning avoids centralizing private data but remains costly under statistical heterogeneity because conventional methods require repeated communication of model updates. DOSFL replaces bulky weights or gradients with distilled synthetic data exchanged in one round, reducing communication while retaining similar performance.

  • Statistical heterogeneity and expensive communication are the paper’s central federated-learning challenges.
  • Prior methods can still require hundreds of communication rounds while increasing information sent per round.
  • DOSFL uses one communication round in which clients upload distilled synthetic data, labels, and learning rates instead of gradients or weights.The server interleaves the distilled data and trains a global model; soft labels, soft reset, and random masking address poorly distributed client data.
  • Up to 99.9% communication reduction versus FedAvg is reported with similar accuracy, while IID performance retains 93% to 99% of centralized training.The distilled data are tied to specific model weights and become ineffective after model updates.
  • DOSFL communicates model-independent data artifacts rather than weights or gradients, which the authors present as a possible new direction for federated learning.

2 Related Work

Related work addresses federated-learning heterogeneity, communication, one-shot optimization, and dataset compression, but neural-network one-shot federated learning remains insufficiently established. Dataset distillation provides the compression mechanism DOSFL extends to federated settings and language tasks.

  • High statistical heterogeneity can slow and destabilize FedAvg because local-loss minimization may diverge from the global objective.
  • Several FedAvg successors improve convergence, but some at least double per-round communication costs.
  • Quantization, sparsification, asynchronous updates, and proximal methods reduce or manage communication and update inefficiencies without using one-shot distilled data.
  • Earlier one-shot federated methods used model ensembling, while theoretical one-shot optimization results rely on assumptions whose applicability to neural-network FL is unknown.
  • Dataset distillation compresses large datasets into a few synthetic examples optimized for rapidly training a neural network from specified parameters.
  • Soft labels extend dataset distillation beyond image classification to language tasks such as sentiment classification.

3 Distilled One Shot Federated Learning

DOSFL formulates federated learning as minimizing the weighted sum of client losses, then has clients distill private data for a shared initialization before the server trains on their combined sequences. Soft resets and random masking mitigate interference when client data are non-IID.

  • The federated objective minimizes a probability-weighted sum of client loss functions.
  • With a common loss function, each client’s loss is the average loss over its own private dataset.
  • The server initializes model parameters and distributes the initialization, after which clients distill their datasets into synthetic inputs, labels, and learning rates.Distilled inputs begin as random tensors, while labels and learning rates receive task-specific initialization.
  • Clients optimize distilled data by adapting the shared initialization through gradient descent and minimizing the adapted model’s loss on real-data minibatches.Distillation repeats sequences of synthetic examples over multiple epochs, increasing expressive power at the cost of computation.
  • The server merges clients’ distilled sequences and trains its model on the combined sequence.
  • Non-IID data can cause sequential distilled updates to interfere because later examples were optimized for an earlier model state.Soft resets sample nearby starting parameters, while random masking replaces portions of distilled data during training and restores them afterward.

4 Experiments

Experiments evaluate DOSFL across vision and language tasks, model architectures, participation settings, and communication-cost comparisons. Results show strong one-round performance, while non-IID and client-count conditions affect which distillation additions work best.

  • Image classification: Soft resets provide the largest non-IID performance improvement, followed by random masking and soft labels, but the additions are not additive.With all additions, non-IID DOSFL reaches approximately 79% test accuracy, and the best configuration changes with client count.
  • Experimental scope: DOSFL is evaluated on federated MNIST, IMDB, TREC-6, and Sent140 using CNN, Bi-LSTM, and other model settings.Language experiments represent distilled sentences as fixed-size matrices of pretrained GloVe embeddings.
  • Accuracy: At least 93% preserved accuracy is achieved across the evaluated tasks relative to baseline performance.The comparison includes DOSFL and non-IID FedAvg under an equivalent communication budget.
  • Low participation: LP-DOSFL uses serial client communication, because each subsequent client distills against the server model updated by the preceding client.This sequential design reduces total communication relative to DOSFL but delays the start of each subsequent client's distillation.
  • Low participation: LP-DOSFL reaches almost 99% accuracy on IID Federated MNIST, while its advantage over vanilla DOSFL largely disappears under non-IID data.The global model reaches final accuracy after about 15% of IID clients or 40% of non-IID clients finish distillation.

5 Discussion

DOSFL’s communication analysis compares its one-time distilled-data transfer with FedAvg across break-even and iso-accuracy settings, while discussing privacy, security, and practical deployment scope.

  • Communication cost: DOSFL’s communication cost counts a single client-to-server transfer of distilled data, with each data point’s element count and distilled batch size determining the expense.The server initialization cost is ignored because it can be distributed as a random seed.
  • Break-even analysis: Break-even rounds are independent of the number of clients N and vary across federated MNIST, IMDB, TREC-6, and Sent140.The MNIST break-even round is higher because LeNet has fewer parameters than TextCNN or Bi-LSTM.
  • Iso-accuracy comparison: Up to 3 orders of magnitude of communication savings are reported beyond MNIST, with larger models and more challenging datasets yielding greater savings.The iso-accuracy comparison uses the smallest FedAvg round reaching DOSFL’s reported accuracy across five trials.
  • Practical scope: DOSFL trades computation and a few-to-none percentage points of accuracy for substantial communication reduction, with later FedAvg or local training available for further improvement or personalization.The discussion identifies cross-silo FL with 2–100 organizations and larger models as a particularly suitable setting.
  • Privacy and security: The security analysis assumes the attacker does not know the model initialization; tested alternative initializations generally failed to reproduce the global model’s performance.The reported attacks mostly struggled to exceed 50% accuracy.
  • Privacy and security: DOSFL’s authors state that its privacy should be no worse than FedAvg, while data-poisoning risks remain and other attack defenses are outside the paper’s scope.The discussion suggests bounding gradients or using momentum against poisoning and points to differential privacy and secure multiparty computation for other attacks.

B.1 MNIST

The MNIST figures show first-step distilled images from individual clients under IID and non-IID data distributions, including a soft-label variant.

  • IID MNIST: Figure 4 shows first-step distilled images from one of ten clients for IID federated MNIST without soft labels, soft resets, or random masking.The figure describes the distilled images at the first step of the process.
  • Non-IID MNIST: Figure 5 shows first-step distilled images from one of ten clients for non-IID federated MNIST without soft labels, soft resets, or random masking.The figure provides the corresponding non-IID comparison to the IID setting.

B.2 IMDB

The IMDB examples illustrate distilled sentences for IID and non-IID federated settings, with logits and distillation learning rates reported for each example.

  • IID IMDB: The IID IMDB example from one of 100 clients assigns logit 1.63 to the positive class and 0 to the negative class, with learning rate 0.0272.The accompanying distilled sentence is visibly nonsensical text.
  • Related visual example: A separate MNIST figure uses soft labels and reports the three labels with the highest logits above each distilled image.This provides a visual comparison point for the label information used in distilled examples.
  • Non-IID IMDB: The non-IID IMDB example assigns logit 1.68 to the positive class and 0 to the negative class, with learning rate 0.0284.The accompanying sentence is likewise presented as fabricated text rather than ordinary language.

B.3 TREC6

The TREC6 examples show distilled sentences for IID and non-IID clients, reporting different selected classes, logits, and distillation learning rates.

  • IID TREC6: The IID TREC6 example from one of 29 clients assigns logit 1.96 to class 1 and 0 to the remaining classes, with learning rate 2.25.The accompanying distilled sentence is fabricated text.
  • Non-IID TREC6: The non-IID TREC6 example assigns logit 1.58 to class 2 and 0 to the remaining classes, with learning rate 1.87.The accompanying example is also presented as a distilled sentence composed of nonsensical text.

C Additional Results

LP-DOSFL results are reported for federated IMDB, TREC-6, and Sent140 tasks using the regular DOSFL methodology, except for serial rather than parallel distillation.

  • LP-DOSFL is evaluated on federated IMDB, TREC-6, and Sent140 tasks.The hyperparameters and methodology match regular DOSFL, except for the distillation order.
  • The implementation changes distillation order from parallel to serial relative to regular DOSFL.

C.2 Moderate Non-IID

DOSFL and LP-DOSFL retain IID performance down to a shard count of 10, but accuracy declines rapidly in more severely non-IID settings. On the listed language tasks, LP-DOSFL outperforms vanilla DOSFL in every case.

  • Moderate Non-IID: DOSFL and LP-DOSFL maintain IID performance as shard count decreases to 10.At this moderately non-IID setting, each client still averages examples from all 10 digits.
  • Moderate Non-IID: Beyond shard count 10, test accuracy degrades quickly for both DOSFL variants until their accuracies converge at shard count 2.
  • Moderate Non-IID: LP-DOSFL slightly declines as shard count decreases, whereas vanilla DOSFL remains flat through the moderate non-IID range.
  • Additional Results: LP-DOSFL outperforms vanilla DOSFL on federated IMDB, TREC-6, and Sent140 in every reported case.The TREC-6 evaluations use 2 and 29 clients rather than 10 and 100.
Loading 2009.07999v3…