Source-linked AI summary
Active Federated Learning
Jack Goetz, Kshitiz Malik, Duc Bui, Seungwhan Moon, Honglei Liu, Anuj Kumar
TL;DR
Federated Learning must reduce costly client-server communication while learning from heterogeneous private data. The paper introduces Active Federated Learning, which selects clients using model- and data-dependent valuations, and reports the same performance with 20-70% fewer epochs.
Problem
Federated Learning incurs high client-server transmission costs, while clients differ in how beneficial their data is for training.
Method
Active Federated Learning evaluates client data with a value function and uses the valuations to form a model- and data-dependent sampling distribution.
Results
20-70% fewer Epochs achieve models with the same performance as uniform random client selection.
Takeaways & Limitations
AFL provides a client-selection approach that can reduce training iterations and mimic established resampling techniques under suitable data conditions.
Takeaways & Limitations
Formal privacy guarantees remain future work, and the experiments use simplifying conditions that omit many practical Federated Learning problems.
Abstract
from arXiv · showhide
Federated Learning allows for population level models to be trained without centralizing client data by transmitting the global model to clients, calculating gradients locally, then averaging the gradients. Downloading models and uploading gradients uses the client's bandwidth, so minimizing these transmission costs is important. The data on each client is highly variable, so the benefit of training on different clients may differ dramatically. To exploit this we propose Active Federated Learning, where in each round clients are selected not uniformly at random, but with a probability conditioned on the current model and the data on the client to maximize efficiency. We propose a cheap, simple and intuitive sampling scheme which reduces the number of required training iterations by 20-70% while maintaining the same model accuracy, and which mimics well known resampling techniques under certain conditions.
1 Introduction
Federated Learning avoids centralizing private client data but incurs high communication costs. Active Federated Learning selects clients based on their usefulness to the current model to reduce training costs.
- Active Federated Learning preferentially trains on users that are more beneficial to the model during each iteration.
- Reducing communication costs is important because transmitting data between servers and clients is expensive.
2 Related Work
Prior Federated Learning work largely leaves user selection unchanged, while Active Federated Learning conditions selection on the model state and each user's data. AFL therefore resembles Active Learning but operates over client-level data subsets.
- Few existing Federated Learning techniques change how users are selected.
- AFL selects users according to the current model state and the data held by each user.
- Active Learning has unknown candidate labels, whereas AFL clients know both labels and covariates but return only a summary to the server.
- AFL trains on all data points belonging to each selected user rather than selecting arbitrary individual data points.
3 Background and Notation
Federated training distributes model updates across clients and aggregates them over iterations. Traditional Federated Learning selects a fixed-size client subset uniformly and independently at each iteration, whereas AFL seeks fewer iterations.
- Each iteration selects m clients, sends them the current model, and receives locally updated parameters.
- Clients train locally using their data before returning updated model parameters to the server.
- Federated ADAM aggregates client updates to produce the next model parameters.
- Traditional Federated Learning samples client subsets uniformly at random and independently at each iteration.
- AFL aims to obtain a good model with fewer training iterations by changing client-subset selection.
4 Active Federated Learning (AFL)
AFL evaluates how useful each client's data is for the current model, converts those valuations into a sampling distribution, and selects clients accordingly. Its loss-based valuation can reproduce familiar resampling behavior, while privacy introduces an unresolved trade-off.
- AFL selects an optimized client subset using a value function that reflects each user's usefulness during the training round.
- Selected clients return valuations to the server, which uses them to calculate the next iteration's sampling distribution.
- The sampling algorithm combines valuation-based selection with uniform random selection of a fraction of the clients.
- Loss valuation favors clients whose data currently produces higher loss and can mimic minority-class or margin-based resampling under suitable data structure.
- Differentially private valuations can protect client information but may mislead the server into selecting suboptimal clients.
5 Experimental Results
Experiments on Reddit and Sticker Intent compare AFL with uniform user selection and examine its relationship to server-side resampling. AFL reaches the same model performance with fewer training epochs, while user-level selection loses performance relative to arbitrary server-side selection.
- AFL versus uniform selection: 20-70% fewer Epochs achieve the same model performance with AFL as with uniform random user selection.An Epoch is enough training rounds to train on each client once in expectation under random sampling.
- AFL versus uniform selection: AFL and uniform selection were evaluated on Reddit and Sticker Intent, using mean and standard errors from 10 test-data repetitions.The Reddit task predicts whether comments are controversial, while Sticker Intent predicts whether messages receive sticker replies.
- User-level versus server-side selection: Server-side learning shows a larger difference between Random Sampling and Active Sampling than federated learning, suggesting loss from selecting users rather than arbitrary data subsets.The comparison used the Reddit dataset and tuned resampling and learning parameters separately for server and federated training.
6 Conclusion and Further directions
The paper concludes that Active Federated Learning adapts user-cohort selection to model state and client data, reducing iterations while preserving performance. It identifies privacy, availability bias, non-stationary data, valuation refresh, and broader problem settings as directions for further work.
- Conclusion: AFL actively adapts user-cohort selection to the model state and data on each client.The paper presents this as its first user cohort selection technique for Federated Learning.
- Conclusion: 20-70% fewer iterations achieve the same performance with AFL.The conclusion restates the experimental efficiency result.
- Further directions: Formal privacy guarantees remain vital future work, and the experiments used simplifying conditions that omit many practical Federated Learning problems.The paper also points to reliability-aware selection, valuation refresh under changing data, and more complex models as extensions.
- Further directions: Classification-focused experiments leave the behavior of AFL with more complex models as an open research direction.The loss value function may apply to any supervised problem, but the reported experiments and analyses focused on classification.