Source-linked AI summary
Latent Information Sharing for Accelerating Federated Learning
Seungjun Lee, Ensieh Khazaei, Dimitrios Hatzinakos, Baturalp Buyukates, Sunwoo Lee
TL;DR
Heterogeneous local data can cause client drift that hinders federated convergence. ASFL shares intermediate activations across clients; experiments report higher accuracy across non-IID benchmarks and up to 33% lower per-round training time than compared methods.
Problem
Heterogeneity among clients' local datasets leads to client drift, which significantly hinders federated convergence.
Source: §1 · p. 1Method
ASFL shares activation outputs from a critical hidden layer between clients alongside conventional model aggregation.
Source: §1 · p. 1, §4.1 · p. 3Results
ASFL achieves higher accuracy than compared state-of-the-art methods across non-IID benchmarks and reduces per-round training time by up to 33%.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that activation sharing is a practical approach for fast and accurate federated learning across a wide range of applications.
Source: §6 · p. 9Exchanging latent representations may still leak sensitive information, even though raw data are not shared.
Source: §4.4 · p. 6
Abstract
from arXiv · showhide
Federated learning (FL) is a communication-efficient distributed learning paradigm. However, client drift remains one of the most critical challenges, hindering the efficient training of a global model. In this study, we propose a novel latent information sharing scheme that directly mitigates data heterogeneity across clients. Our theoretical and empirical results show that sharing a small amount of hidden-layer activations significantly improves training efficiency while preserving convergence guarantees and data privacy. Furthermore, we compare our method with existing FL approaches designed to address client drift, including FedProx, SCAFFOLD, FedPVR, FedProto, and SplitFed, and demonstrate superior model accuracy under a fixed round budget without incurring excessive communication overhead. Overall, this work presents a promising new knowledge aggregation scheme and provides a comprehensive analysis of the impact of activation sharing on federated optimization.
1 Introduction
Client drift caused by heterogeneous local data hinders federated convergence, motivating collaboration beyond parameter-space coordination. ASFL shares intermediate activations and reports higher accuracy across non-IID benchmarks, alongside faster per-round training.
1 Introduction
- Severe heterogeneity among clients' local datasets leads to client drift, which significantly hinders federated convergence.The paper frames the challenge as addressing drift without sacrificing communication-efficient distributed learning.Source: §1 · p. 1
- Prior approaches use proximal regularization, gradient-variance reduction, or layer-wise aggregation, but the passage notes limitations in coordination, overhead, or explicit representation alignment.The discussed methods include FedProx, FedDyn, SCAFFOLD [3], FedVARP [4], FedMDVR, and FedLAMA.Source: §1 · p. 1
- ASFL adds exchange of intermediate activations from a critical hidden layer to conventional model aggregation to address client drift through representation sharing.The study asks which layer yields informative representations and how to reuse remote activations without prohibitive memory or communication overhead.Source: §1 · pp. 1–2
- The study analyzes activation sharing's effect on FedAvg convergence for non-convex smooth objectives and discusses reusing remote activations to limit additional communication.The remote activations are reused multiple times.Source: §1 · p. 2
- ASFL achieves higher accuracy than the compared state-of-the-art methods on non-IID benchmarks and reduces per-round training time by up to 33%.Under the same number of local steps per round, its privacy leakage under model inversion attacks is comparable to SplitFed.Source: §1 · p. 2
2 Background
The background describes federated optimization through local objectives and periodic aggregation, and explains how heterogeneous local updates produce client drift that slows global convergence.
2 Background
- FedAvg uses local SGD with periodic model aggregation to train a global model without explicitly sharing decentralized data, but can converge slowly under high heterogeneity.The passage identifies it as a pioneering federated optimization method.Source: §2 · p. 2
- Client drift arises because clients update on heterogeneous local data, biasing their updates relative to the global objective and creating a mismatch between expected local and global gradients.Greater data heterogeneity or more local update steps makes this deviation more pronounced and slows global-model convergence.Source: §2 · p. 2
- FedProx, SCAFFOLD, and SILVER are cited as approaches proposed to mitigate client drift.Source: §2 · p. 2
3 Related work
Related work addresses client drift through variance reduction, representation-based methods, and split learning. ASFL instead shares activations between clients without centralizing their data or requiring server-side computation.
3 Related work
- Variance-reduction methods can add memory and communication overhead; FedProto shares class-wise prototypes, while MOON adds auxiliary-model memory and contrastive computation.ASFL is compared with representative methods on accuracy under a fixed round budget and communication overhead.Source: §3 · p. 2
- Split Federated Learning sends cut-layer activations to a central server for remaining-layer computation, whereas ASFL requires neither client-data centralization nor server-side computation.The passage says server-side aggregation in SFL can mitigate client drift and reduce client-side computational burden.Source: §3 · p. 2
- Figure 1 depicts activation exchange, a tail round with backpropagation only within z using both clients' activations, and random client pairing.Source: Figure 1 · p. 3
4 Method
ASFL shares hidden-layer activations between paired clients, then uses local and remote activations to update the model’s output-side subnetwork. The method selects sharing and reuse strategies to balance variance reduction, system cost, convergence, and privacy.
4.1 Latent information sharing
- Paired clients exchange activations from a selected hidden layer, then train only the output-side subnetwork on mini-batches drawn from both clients’ activations.The model is partitioned into input-side y and output-side z; the exchange includes activations and corresponding labels.Source: §4.1 · p. 3
4.2 Controlling the degree of information sharing
- Sharing-layer choice and activation volume jointly determine how much remote information affects local training.Earlier sharing can extend variance reduction to more layers, but raises computational overhead because backpropagation traverses more layers.Source: §4.2.1 · p. 4, §4.2 · p. 4
4.2.1 Selecting the sharing layer via cost-variance trade-off
- The selected layer is the intersection of normalized system-efficiency and variance curves, balancing overhead against variance reduction.Figure 2 depicts cumulative variance for input-side y and system efficiency; the intersection marks the trade-off across benchmarks.Source: §4.2.1 · p. 4, Figure 2 · p. 5
4.2.2 Communication-efficient activation sharing
- A small subset of remote activations is sampled with replacement and reused across iterations to reduce communication while balancing local and remote representations.The augmented pool is sampled so the local and remote contributions are balanced.Source: §4.2.2 · p. 5
4.3 Convergence analysis
- Under the stated assumptions, ASFL converges to a stationary point or, with nonzero global variance, a neighborhood whose size depends on variance terms.With a diminishing step size, the result establishes convergence to a stationary point; the analysis also reports linear speedup under an appropriate step size and number of local updates.Source: §4.3 · pp. 5–6
4.4 Differential privacy for activation sharing
- Because shared activations may still leak sensitive information, PixelDP adds noise at y; downstream computation preserves the same differential-privacy guarantee.The text describes Laplace and Gaussian mechanisms for privatizing transmitted activations.Source: §4.4 · pp. 6–7
5 Experiments
Experiments compare ASFL across four benchmarks and examine its accuracy, communication cost, local computation time, privacy–utility trade-off, and activation-sharing layer selection. The results report higher accuracy across benchmarks, lower local computation time, and practical layer choices that reduce communication cost with marginal accuracy loss.
5 Experiments
- Experiments use CIFAR-10, CIFAR-100, SVHN, and FEMNIST, with 100 clients and 10 randomly activated per round; non-IID data use a Dirichlet distribution with α = 0.1.The models are ResNet18, MobileNetV2, ResNet20, and CNN, respectively.Source: §5 · p. 7
5.1 Comparative study
- ASFL attains higher accuracy across all benchmarks with only marginal additional communication cost; on CIFAR-10, it improves accuracy by approximately 7.5% over FedAvg while exchanging 8 MB more activations per round.The comparison evaluates ASFL without PixelDP to isolate activation sharing’s effect on optimization efficiency.Source: §5.1 · p. 7
- ASFL forms tighter, more coherent cross-client class clusters than FedAvg and SCAFFOLD in the t-SNE visualization.The study links this representation alignment to the higher validation accuracy in Table 1.Source: §5.1 · p. 8
5.2 Per-client round wall-clock time
- ASFL achieves the lowest local computation wall-clock time on all four datasets under the same total local update budget, including activation generation.The tail round reduces backward-pass computation by training only part of the model while preserving the total number of local updates.Source: §5.2 · p. 8, Table 2 · p. 8
5.3 ASFL with differential privacy
- Stronger PixelDP privacy guarantees increase injected noise and degrade performance, but ASFL still outperforms the non-private FL methods shown in Table 1 under moderate noise.Table 3 reports ASFL accuracy with PixelDP under Laplace and Gaussian noise at threat-model radius R = 0.1.Source: §5.3 · p. 8, Table 3 · p. 8
5.4 Ablation study
- Figure 4 compares communication cost and validation accuracy by layer index, with the proposed method selecting a practical layer that reduces communication cost while marginally degrading accuracy.For CIFAR-100, both cost and accuracy increase at layers 8 and 9.Source: §5.4 · pp. 8–9, Figure 4 · p. 9
- Additional experiments vary the number of exchanged activation batches, with results reported in Appendix A.5.2.The supplied passage does not report the outcomes of those experiments.Source: §5.4 · p. 9
6 Conclusion
The conclusion presents ASFL as a practical approach for fast and accurate federated learning, while identifying centralized activation aggregation and alternative activation-use strategies as future work.
6 Conclusion
- The authors conclude that ASFL is practical and effective for fast, accurate federated learning across a wide range of applications.Their theoretical analysis addresses convergence behavior, and experiments cover diverse benchmarks.Source: §6 · p. 9
- The study investigates privacy concerns from activation sharing using PixelDP and reports empirical insights.Source: §6 · p. 9
- Future work could extend the practical gossip-based communication pattern by aggregating activations at a central server and exploring alternative strategies for their use.Source: §6 · p. 9
A Appendix
The appendix collects experimental settings, theoretical proofs, communication-pattern details, and supplementary experimental analyses.
A.1 Experimental settings
- Appendix A.1 summarizes the experimental settings used for the main manuscript’s reported results.Source: A Appendix · p. 12
A.2 Proofs
- Appendix A.2 provides detailed proofs of the proposed lemma, theorem, and corollary.Source: A Appendix · p. 12
A.3–A.5 Additional details and analyses
- Appendices A.3–A.5 cover gossip-style latent-information transfer, additional privacy results, and supplementary analyses supporting the main empirical findings.Source: A Appendix · p. 12
A.1 Experimental details
This appendix section summarizes the experimental details needed to reproduce and interpret the main results.
A.1 Experimental details
- The section covers training settings, activation-sharing configurations, cost estimation, and gradient-variance measurement.Source: §A.1 · p. 12
A.1.1 Main comparison protocol
The main comparison uses a shared experimental protocol across dataset–backbone pairs and methods, with ASFL dividing its local computation budget between tail and standard updates.
A.1.1 Main comparison protocol
- Table 4 gives training settings by dataset–backbone pair, and Table 5 summarizes method-specific hyperparameters.FedProx uses µ; FedProto uses λ; and MOON uses µ and temperature T.Source: Table 4 · p. 12, Table 5 · p. 12, §A.1.1 · p. 13
- The main comparison matches client sampling, non-IID partitioning, optimizer settings, and total local-update budgets across methods.Each round samples 10 of 100 clients; data use a Dirichlet distribution with α = 0.1.Source: §A.1.1 · p. 12
- ASFL divides its local computation budget equally between tail-round and standard updates (τ ′ = τ), while other methods use it for standard training.Source: §A.1.1 · p. 12
A.1.2 Activation-sharing positions
ASFL selects activation-sharing positions from backbone-specific layer blocks, with the choice guided by a cost–variance trade-off.
A.1.2 Activation-sharing positions
- Candidate positions are indexed by the block where tail-side partial updating starts, and the selected block is included in that partial-training range.Source: §A.1.2 · p. 13
- ASFL forwards a mini-batch through preceding blocks and exchanges the activation at the selected block's input; the raw-input boundary is excluded.Source: §A.1.2 · p. 13
- Table 6 lists candidate positions by dataset–backbone pair, excluding the input boundary but including the final classifier; selection follows Figure 2's cost–variance trade-off.Source: Table 6 · p. 13, §A.1.2 · p. 13
A.1.3 Activation-sharing quantity
The activation-sharing quantity ablation varies the size of the peer activation pool while holding the selected sharing position fixed.
A.1.3 Activation-sharing quantity
- The ablation fixes each dataset–backbone pair's selected layer block and varies only the number of shared batches.Source: §A.1.3 · p. 13
- One shared batch means activations generated from one mini-batch, whose size follows Table 4.Source: §A.1.3 · p. 13
- The shared-batch count controls peer activation information exchanged through directed gossip, not the number of tail-round mini-batches used for updates.Received peer activations are reused to construct random mini-batches for the tail round.Source: §A.1.3 · p. 13
A.1.4 Communication and computational cost
This section analyzes activation-sharing costs, gradient variance, convergence conditions, gossip communication, and differential-privacy protections, while examining how design choices affect ASFL. It reports that benefits depend on data heterogeneity and the inclusion of informative remote activations.
A.1.4 Communication and computational cost
- Communication cost is the number of elements in the activation tensor entering a candidate block, while computational cost estimates backpropagation through its tail subnetwork’s convolutional and linear layers.The layer-wise estimates compare relative tail-side partial-training costs across candidate positions.Source: §A.1.4 · p. 13
A.1.5 Gradient variance measurement
- Gradient variance is estimated from per-sample, block-wise gradients, averaged across active clients and cumulatively summed over candidate blocks before min–max normalization.The block-wise variance is the average squared deviation of per-sample gradients from their mini-batch mean.Source: §A.1.5 · pp. 13–14
A.2 Convergence analysis of partial model updates with activation sharing
- Theorem 9 states a convergence condition for activation-sharing federated optimization under Assumptions 1–3 and a stated learning-rate restriction.The proof derives the condition from the preceding lemmas.Source: §A.2 · pp. 26–27
A.3.1 Discussion of gossip-style communication
- Each active client receives latent information from one uniformly sampled active peer per round; this directed gossip pattern avoids central-server blocking and can continue despite a client disconnection.The authors present it as suitable for distributed edge devices while retaining the standard federated training procedure.Source: §A.3.1 · p. 28
A.4 Additional discussion on privacy
- With post-processing, privatized activations sent to a paired client are (ϵ, δ)-DP, or (ϵ, 0)-DP under Laplace noise.The appendix also describes PixelDP [18] as providing privacy and certified robustness against adversarial examples under bounded input perturbations.Source: §A.4 · p. 29
A.4.1 Impact of activation sharing layer on performance
- On CIFAR-10, ASFL accuracy falls from 79.6% at an earlier partition to 66.0% deeper, while deeper layers reduce privacy-noise drops.Laplace Drop falls from 11.47% to 0.66% and Gaussian Drop from 12.47% to 0.73%; earlier layers transfer richer representations but need more careful noise calibration.Source: §A.4.1 · pp. 29–30, Table 7 · p. 30, Table 8 · p. 30
A.4.2 Model inversion attack on shared activations
- Under Gaussian perturbation (ϵ, δ) = (0.5, 0.05), reconstructed images do not visually reveal the private inputs in the optimization-based inversion attack.The attack matches shared intermediate features, and Figure 5 shows the perturbed activations and corresponding reconstructions.Source: Figure 5 · p. 31, §A.4.2 · p. 31
A.5.1 Ablation on activation pool construction
- On CIFAR-10, Random exceeds Replication and Default by 0.91 and 3.30 percentage points, respectively; it also has the highest average accuracy on FEMNIST.Random uniformly resamples received remote activations with replacement, helping preserve the intended balance of local and remote latent information when few activations are exchanged.Source: Table 11 · p. 32, §A.5.1 · p. 32
A.5.2 Effect of the number of exchanged activations
- Exchanging one or two high-loss batches can mitigate heterogeneity with marginal overhead, though gains vary and similar or uninformative remote data may fail to help or degrade accuracy.Accuracy generally improves as more batches are exchanged, but communication overhead also rises.Source: Table 12 · p. 32, §A.5.2 · p. 32
A.5.3 Effect of the Dirichlet Concentration Parameter
- At α = 0.05, ASFL reaches 67.14%, beating FedAvg by 8.21 and FedVARP by 6.00 percentage points; at α = 0.5, it remains highest at 87.60% with smaller method gaps.ASFL’s advantage is more pronounced under stronger heterogeneity, consistent with activation sharing’s aim of mitigating client drift.Source: §A.5.3 · pp. 32–33, Table 13 · p. 33
A.5.4 Isolated performance gain from activation sharing
- Replacing remote activations with local ones lowers accuracy from 75.27% to 66.23% (9.04-point drop), below FedAvg’s 67.70% by 1.47 points despite matched partial training.Both ASFL variants use the same partial training procedure and 16 tail update steps per tail round.Source: Table 14 · p. 33, §A.5.4 · p. 33