Source-linked AI summary
TiFL: A Tier-based Federated Learning System
Zheng Chai, Ahsan Ali, Syed Zawad, Stacey Truex, Ali Anwar, Nathalie Baracaldo, Yi Zhou, Heiko Ludwig, Feng Yan, Yue Cheng
TL;DR
Conventional FL is slowed and can lose accuracy under heterogeneous client resources and data. TiFL groups clients by training latency and adaptively selects tiers using accuracy signals; across heterogeneous settings, it reports faster training while maintaining comparable or better accuracy.
Problem
Resource, data-quantity, and non-IID heterogeneity in FL clients can substantially affect conventional training time and model accuracy.
Method
TiFL profiles client latency, groups clients into tiers, selects within tiers, and adaptively updates tiering using observed accuracy.
Results
TiFL outperforms conventional FL across heterogeneous settings, including 6× faster training under resource heterogeneity and 3× under data-quantity heterogeneity.
Takeaways & Limitations
Tier selection can improve training speed while preserving comparable accuracy and, in some settings, improving accuracy.
Takeaways & Limitations
The analysis assumes clients share the same privacy budget and therefore the same (ϵ, δ) values.
Abstract
from arXiv · showhide
Federated Learning (FL) enables learning a shared model across many clients without violating the privacy requirements. One of the key attributes in FL is the heterogeneity that exists in both resource and data due to the differences in computation and communication capacity, as well as the quantity and content of data among different clients. We conduct a case study to show that heterogeneity in resource and data has a significant impact on training time and model accuracy in conventional FL systems. To this end, we propose TiFL, a Tier-based Federated Learning System, which divides clients into tiers based on their training performance and selects clients from the same tier in each training round to mitigate the straggler problem caused by heterogeneity in resource and data quantity. To further tame the heterogeneity caused by non-IID (Independent and Identical Distribution) data and resources, TiFL employs an adaptive tier selection approach to update the tiering on-the-fly based on the observed training performance and accuracy overtime. We prototype TiFL in a FL testbed following Google's FL architecture and evaluate it using popular benchmarks and the state-of-the-art FL benchmark LEAF. Experimental evaluation shows that TiFL outperforms the conventional FL in various heterogeneous conditions. With the proposed adaptive tier selection policy, we demonstrate that TiFL achieves much faster training performance while keeping the same (and in some cases - better) test accuracy across the board.
1 INTRODUCTION
The paper shows that resource, data-quantity, and non-IID data heterogeneity constrain conventional FL performance, then proposes TiFL to reduce these effects through tiered, adaptive client selection.
- Motivation: Resource and data heterogeneity create stragglers and unequal round times, limiting synchronous FedAvg training performance.Slow clients and clients processing different sample quantities can bound or extend each training round.
- TiFL: TiFL groups clients into latency-based tiers and selects clients from the same tier during each training round.A lightweight profiler measures client training time, while adaptive selection controls throughput and accuracy.
- Adaptive selection: TiFL uses observed accuracy to infer non-IID data heterogeneity without directly measuring private class or feature distributions.The adaptive algorithm updates tiering online as heterogeneity characteristics change over time.
- Results: 6× faster training is achieved under resource heterogeneity without affecting accuracy, while data-quantity heterogeneity yields 3× speedup with comparable accuracy.The reported results also include 3× faster training and 8% higher accuracy on CIFAR10, plus 3× improvement on FEMINIST under LEAF.
2 RELATED WORK
Prior FL and distributed-learning approaches address privacy, scalability, or stragglers, but generally do not jointly account for FL’s broad resource and data heterogeneity.
- Federated learning: Existing FL approaches largely reduce communication overhead through model or weight compression rather than addressing resource and data heterogeneity.They are not designed to be straggler-aware despite potentially significant client latency differences.
- Federated learning: Asynchronous methods may handle delays, but cited comparisons favor synchronous FL for stronger security and privacy properties.Some asynchronous guarantees also rely on bounded delays or convex loss functions.
- Straggler handling: Randomly selecting extra devices tolerates 30% stragglers, but the threshold is arbitrary and discarding slow clients may exclude useful data distributions.This approach can therefore trade latency handling against data coverage.
- Straggler handling: FedProx mainly distinguishes stragglers from non-stragglers, whereas real FL environments contain a wide range of heterogeneity levels.Its partial training on stragglers may introduce bias and sub-optimal model accuracy, according to the paper.
- Data heterogeneity: Data-distribution clustering groups devices with similar datasets but does not consider clustering’s effects on training time or accuracy.Other proposed designs use work shedding and peer-to-peer communication to detect stragglers.
3 HETEROGENEITY IMPACT STUDY
The study analyzes how client heterogeneity affects cross-device FL: round latency is governed by the slowest selected client, while resource, data quantity, and non-IID distributions alter training time or accuracy.
- Heterogeneity: Cross-device FL clients differ in computing, communication, and data characteristics, potentially affecting both throughput and model accuracy.Resource heterogeneity reflects varying device capabilities, while data heterogeneity arises from differences among client datasets.
- Vanilla FL: Each global round randomly selects clients from the available pool, sends them the current model, and aggregates their returned updates.This describes the vanilla cross-device FL process used as the baseline.
- Heterogeneity impact: A global round’s latency equals the maximum selected-client latency, so one slow client can determine the round duration.The paper identifies this maximum-latency effect as the straggler problem.
- Heterogeneity impact: Random selection has a high probability of including at least one client from the slowest level when the client pool and selected set are large.The analysis concludes that vanilla cross-device FL can therefore suffer slow training performance.
- Experimental setup: In the testbed, 20 clients were divided into five groups with CPU allocations ranging from 4 to 1/5 CPU to emulate resource heterogeneity.CIFAR10 was trained with the vanilla cross-device FL process while client data sizes varied.
- Resource and data quantity: Increasing data from 500 to 5000 samples caused a near-linear increase in per-round training time at fixed CPU capacity.More CPU resources shortened training time, while larger datasets increased it at the same CPU allocation.
- Non-IID data: Reducing client classes from 10 to 5 to 2 progressively decreased accuracy under biased non-IID distributions.The study reports around 6% lower final accuracy for non-IID(10) than IID and a further 8% reduction for five classes per client.
4 TIFL: A TIER-BASED FEDERATED LEARNING SYSTEM
TiFL organizes clients into latency-based tiers and schedules clients within tiers to reduce straggler-driven training delays. Its adaptive scheduler uses tier accuracy to balance training time against data heterogeneity and model accuracy.
- Tier-based selection: Selecting clients with similar response latency reduces each round’s dependence on its slowest selected client, addressing the synchronous straggler problem.The global training time of a round is bounded by the slowest selected client.
- System architecture: TiFL adds a profiling-and-tiering module and a tier scheduler to an existing federated learning coordinator.The architecture can also support hierarchical master-child aggregators for scalability and fault tolerance.
- Profiling and tiering: A lightweight profiling process measures client latency, groups clients into tiers, and excludes clients that repeatedly exceed the timeout threshold.Clients responding within Tmax contribute actual training time, while timed-out clients contribute Tmax to their latency estimates.
- Static selection: Static tier selection exposes a trade-off: favoring faster tiers reduces training time but can bias training when heterogeneous data is distributed across tiers.Selecting only the fastest tier may affect accuracy because clients across tiers can hold diverse training data.
- Adaptive tier selection: The adaptive algorithm updates tier-selection probabilities from observed tier accuracies, increasing selection chances for lower-accuracy tiers and controlling tier credits over time.This automates the balance between training time and accuracy as system conditions change.
- Training-time estimation: TiFL also estimates total training time as the expected tier latency per round multiplied by the total number of rounds.The estimation uses tier latencies, tier-selection probabilities, and the total training-round count.
5 EXPERIMENTAL EVALUATION
TiFL is evaluated with both naive and adaptive tier-selection approaches across resource, data, and combined resource-plus-data heterogeneity scenarios.
- Evaluation scenarios: The evaluation covers resource heterogeneity, data heterogeneity, and combined resource-plus-data heterogeneity.Both the naive and adaptive selection approaches are tested.
5.1 Experimental Setup
The experiments use a 50-client TensorFlow CPU-cluster testbed and an extended LEAF framework, with five clients selected per round in the synthetic testbed.
- Synthetic testbed: The synthetic-data testbed deploys 50 clients on a CPU cluster, selecting five clients per round for local training and server aggregation.Each client uses exclusive CPU resources and sends trained weights to the server.
- LEAF extension: LEAF is extended to incorporate resource heterogeneity while retaining its non-IID data quantity and class distributions.The evaluation also uses LEAF’s FEMNIST configuration with 182 edge-device clients.
- Measurement comparison: The training-time estimation model’s predictions are compared with actual training times in Table 2.The table is used to assess the accuracy of the analytical training-time model.
5.2 Experimental Results
Experiments across image-classification benchmarks and heterogeneous settings show that TiFL reduces training time while generally preserving accuracy. Its adaptive selection policy improves the speed–accuracy trade-off when resource, data-quantity, and non-IID heterogeneity interact.
- Resource heterogeneity: 11× faster training than vanilla is achieved by fast under resource heterogeneity, while uniform exceeds 6× improvement.After 500 rounds, policy accuracy differences remain below 3.71%.
- Resource heterogeneity: Up to 6.19% better accuracy within a fixed wall-clock time is achieved by TiFL because its rounds complete faster.Different policies can require substantially different wall-clock times to finish 500 rounds.
- Data heterogeneity: Up to 3× speedup is observed under data-quantity heterogeneity, but aggressive fast-tier selection can reduce accuracy by using fewer training samples.Tier 1 contains 10% of the data, whereas Tier 5 contains 30%.
- Data heterogeneity: Non-IID heterogeneity does not affect training time in the reported setting but lowers accuracy through skewed features and increased class-training bias.The evaluation varies client holdings among 2, 5, and 10 classes.
- Combined heterogeneity: Under combined resource, data-quantity, and non-IID heterogeneity, uniform nearly matches vanilla accuracy, while fast suffers greater degradation from amplified class bias.TiFL’s shorter rounds improve wall-clock-time accuracy by allowing more iterations within the same time budget.
- Adaptive selection: Adaptive outperforms vanilla and uniform across Amount and Class settings, and nearly halves training time with comparable vanilla accuracy in Combine.On LEAF, adaptive reaches 82.1% accuracy, near vanilla’s 82.4% and uniform’s 82.6%, with 7× and 2× training-time improvements over those policies.
6 CONCLUSION
TiFL addresses resource and data heterogeneity in federated learning by grouping clients according to training latency and adaptively selecting tiers. Across heterogeneous scenarios, it outperforms conventional FL, achieving faster training and improved accuracy.
- TiFL groups clients into tiers by training response latency and selects clients from the same tier in each round.This tier-based approach targets resource and data heterogeneity in federated learning.
- An adaptive tier selection approach makes TiFL aware of data heterogeneity that cannot be directly measured under privacy constraints.
- TiFL outperforms conventional FL across resource, data quantity, non-IID, and combined heterogeneity scenarios.
- TiFL achieves up to 3× speedup in overall training time and a 6% accuracy improvement over conventional FL.