Source-linked AI summary
Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks
Zheng Lin, Guangyu Zhu, Yiqin Deng, Xianhao Chen, Yue Gao, Kaibin Huang, Yuguang Fang
TL;DR
Resource-constrained devices struggle with privacy-enhancing distributed learning because deep models make exclusive on-device training impractical, while existing split-learning schemes incur computing and communication latency. The paper proposes EPSL, combining parallel client-side training, last-layer gradient aggregation, and joint wireless resource and cut-layer optimization. Simulations report less time to reach target accuracy than existing benchmarks, while the optimization strategy is effective relative to its unoptimized counterpart.
Problem
Deep models make exclusive on-device FL training impractical for resource-constrained devices, and existing SFL/PSL schemes can incur substantial computing and communication latency.
Method
EPSL parallelizes client-side training, aggregates last-layer activation gradients, and jointly optimizes subchannel allocation, power control, and cut-layer selection.
Results
EPSL takes significantly less time to achieve target accuracy than existing benchmarks, and its tailored resource-management and layer-split strategy reduces latency compared with the unoptimized counterpart.
Takeaways & Limitations
EPSL provides a lower-latency split-learning framework for resource-constrained wireless edge systems within the evaluated settings.
Takeaways & Limitations
The resource-management formulation is NP-hard, so the paper uses an efficient suboptimal algorithm rather than a generally optimal polynomial-time solution.
Abstract
from arXiv · showhide
The increasingly deeper neural networks hinder the democratization of privacy-enhancing distributed learning, such as federated learning (FL), to resource-constrained devices. To overcome this challenge, in this paper, we advocate the integration of edge computing paradigm and parallel split learning (PSL), allowing multiple client devices to offload substantial training workloads to an edge server via layer-wise model split. By observing that existing PSL schemes incur excessive training latency and large volume of data transmissions, we propose an innovative PSL framework, namely, efficient parallel split learning (EPSL), to accelerate model training. To be specific, EPSL parallelizes client-side model training and reduces the dimension of local gradients for back propagation (BP) via last-layer gradient aggregation, leading to a significant reduction in server-side training and communication latency. Moreover, by considering the heterogeneous channel conditions and computing capabilities at client devices, we jointly optimize subchannel allocation, power control, and cut layer selection to minimize the per-round latency. Simulation results show that the proposed EPSL framework significantly decreases the training latency needed to achieve a target accuracy compared with the state-of-the-art benchmarks, and the tailored resource management and layer split strategy can considerably reduce latency than the counterpart without optimization.
1 INTRODUCTION
Resource-constrained devices make privacy-enhancing distributed learning difficult because on-device training is computationally heavy, while existing split-learning variants can incur substantial server and communication latency. EPSL addresses these constraints through parallel client training, last-layer gradient aggregation, and joint wireless resource and split-layer optimization.
- Motivation: FL avoids sharing raw data but places heavy computation burdens on edge devices, making deep models impractical for resource-constrained mobile and IoT devices.VGG16 is cited as requiring 138 million parameters and 15.5G MACs for forward propagation.
- Motivation: Existing SFL and PSL frameworks can incur significant computing and communication latency at resource-constrained edge servers.PSL involves large cut-layer data exchange, while SFL additionally involves substantial model exchange.
- Proposed framework: EPSL parallelizes client-side model training and aggregates last-layer activation gradients during back propagation to reduce server computation and communication overhead.The aggregation ratio ϕ controls the fraction of gradients aggregated and trades learning accuracy against latency reduction.
- Resource management: EPSL jointly optimizes subchannel allocation, power control, and cut layer selection to minimize per-round latency under heterogeneous wireless and computing conditions.The optimization is motivated partly by the straggler effect and the need to allocate resources appropriately across devices.
- Evaluation: Simulation studies show that EPSL reduces the training latency required to reach target accuracy compared with existing split-learning schemes.The tailored resource management and layer split strategy also reduces latency relative to the unoptimized counterpart.
2 RELATED WORK
Related work positions FL as privacy-preserving and parallel but resource-hungry, while SL targets resource-constrained devices by keeping raw data local. Prior research also studies privacy mechanisms and resource or model-splitting strategies.
- Federated learning: FL supports data privacy and parallel model training, but limited edge-device communication and computing capabilities hinder deployment.Prior work addresses client selection, resource allocation, hierarchical aggregation, and model compression.
- Split learning: SL targets resource-constrained devices by preserving raw data locally while enabling privacy-preserving deep-neural-network computation.Prior studies explored multiple end systems and gradient perturbation-based split learning.
- Resource management: Related resource-management work considers cut-layer selection, model placement, model splitting, and energy or time costs.These studies motivate tailored resource and layer-split strategies for split-learning systems.
3 SYSTEM MODEL
The system model represents wireless EPSL as cooperation between client devices and a central server, with a globally split model and local losses aggregated into a global objective. Resource-management variables include channel, power, and cut-layer decisions.
- System components: The wireless EPSL scenario consists of participating edge devices and a central server that coordinates server-side model training.The server gathers channel-status and device-computing information for resource management.
- System components: Each edge device owns a local dataset and performs forward and back propagation for its client-side model.The client-side parameters are denoted by Wc.
- Model partitioning: The global model concatenates server-side parameters Ws and client-side parameters Wc, with client activations passed to the server for prediction.For device i, activation si,k is computed locally and prediction ŷi,k is obtained from the server-side model.
- Optimization objective: The global loss is the weighted average of participating devices’ local sample-wise losses, and training seeks model parameters minimizing this global loss.The local loss depends on predicted values, labels, and model parameters.
- Resource variables: The model includes subchannels, transmit power, and cut-layer selection among the resource-management decision variables.The notation summary distinguishes computation workloads, activation sizes, and activation-gradient sizes at each cut layer.
4 EPSL FRAMEWORK
EPSL parallelizes client-side training and aggregates last-layer activation gradients to reduce server computation, back-propagation communication, and training latency. Its training procedure combines layer-split execution with wireless exchanges, while the aggregation ratio controls the accuracy–latency trade-off.
- Core idea: EPSL parallelizes client-side model training and aggregates last-layer activation gradients to reduce computation and communication overhead.The method also eliminates model exchange between clients and the server.
- Training procedure: Each round runs client-side forward propagation in parallel, transmits activations and labels, and concatenates them for server-side forward propagation.Each device samples a mini-batch of b local data samples before producing cut-layer activations.
- Gradient aggregation: During server-side back propagation, EPSL aggregates a fraction ϕ of last-layer activation gradients before continuing standard back propagation.The remaining b −⌈ϕb⌉ gradients are back-propagated without aggregation, and higher ϕ trades more aggregation for training latency and test accuracy.
- Efficiency mechanism: Last-layer aggregation reduces the back-propagated gradient dimension from Cb to ⌈ϕb⌉+ C(b −⌈ϕb⌉), lowering exchanged data volume and server workload.The server averages the first ⌈ϕb⌉ back-propagated gradients across edge devices.
- Evaluation: With ϕ = 0.5 or ϕ = 1, EPSL achieves similar learning accuracy to vanilla SL, SFL, and PSL while substantially reducing per-round latency.This comparison uses HAM 10000 under IID settings with ResNet-18 and C = 5 edge devices.
- Training procedure: The aggregated gradients are broadcast to all clients, while unaggregated gradients are sent to their corresponding devices for client-side back propagation and updates.Client models use the received activation gradients to complete their parameter updates.
5 RESOURCE MANAGEMENT FOR EPSL
EPSL computes one training round through seven client, server, and wireless transmission stages. Its latency is minimized by jointly selecting the cut layer and radio resources under heterogeneous device conditions.
- Resource management problem formulation: The per-round objective jointly optimizes subchannel allocation, power control, and cut layer selection to reduce latency caused by heterogeneous channels and computing capabilities.Subchannel constraints avoid co-channel interference, while cut-layer constraints ensure a unique model partition.
- Training latency computation: EPSL executes client-side forward propagation, activation transmission, server-side forward propagation, server-side back propagation, gradient broadcasting, unaggregated-gradient transmission, and client-side back propagation.The procedure analyzes these stages consecutively within one training round.
- Training latency computation: Client and server computation latency depends on device or server computing capability and the model workload determined by the selected cut layer.The device capability is represented by fi and κi, while server latency uses fs and κs.
- Training latency computation: Wireless latency is determined by activation and gradient data sizes, subchannel bandwidth and allocation, transmit power, and channel gains.EPSL considers frequency-division multiple access for activation transmission and shared wireless downlink transmission for gradients.
- Training latency computation: Last-layer activation-gradient aggregation reduces the gradients processed and communicated during server-side back propagation.Each device aggregates ⌈ϕb⌉ of its last-layer gradients, while the remaining gradients are directly back-propagated.
6 SOLUTION APPROACH
The solution approach decomposes the mixed resource-management problem into subproblems for allocation, power, cut-layer selection, and latency variables. It combines greedy allocation, convex or linear optimization, exhaustive search, and block-coordinate descent to obtain an efficient suboptimal solution.
- Subchannel allocation: A greedy subchannel allocation approach prioritizes stragglers with longer training latency and assigns favorable channels to less powerful devices.Remaining idle channels are iteratively allocated to the straggler at each iteration.
- Power control and cut-layer selection: Fixing subchannels leads to a power-control and cut-layer problem that is transformed using auxiliary variables T1, T2, and θ.T1 and T2 linearize the objective, while θ is introduced to address a nonconvex constraint.
- Power control and cut-layer selection: With selected variables fixed, the transformed subproblem in θ is convex and can be solved using toolkits such as CVX.The text identifies this as problem (30).
- Cut-layer selection: The cut-layer subproblem becomes a mixed-integer linear program solved by exhaustive search because neural networks typically have relatively few layers.AlexNet and GoogLeNet are cited as examples with 8 and 22 layers, respectively.
- Overall algorithm: A block-coordinate descent algorithm alternates among the decomposed subproblems and empirically converges within a few iterations to the same solution under various initial conditions.The paper notes that theoretical convergence is not guaranteed because the original problem is non-convex and mixed-integer.
7 SIMULATION RESULTS
Simulations evaluate EPSL’s learning performance and resource-management strategy under specified wireless, computing, dataset, and model settings. EPSL achieves comparable accuracy with lower latency, while joint resource optimization further reduces latency and remains robust to channel variation.
- Simulation Setup: The simulations use MNIST and HAM10000, under IID and non-IID settings, with ResNet-18 and a 64-sample mini-batch.MNIST has 60,000 training and 10,000 test samples; HAM10000 has 8,000 training and 2,015 test samples.
- Learning Performance: EPSL retains similar converged test accuracy to vanilla SL, SFL, and PSL while requiring the lowest time budget to reach target accuracy.Last-layer gradient aggregation reduces activation-gradient dimension and server computation workload.
- Learning Performance: EPSL has the lowest total training latency across dataset sizes, and its latency grows more slowly because of last-layer gradient aggregation and eliminated model exchange.Vanilla SL is consistently slowest because its client training is sequential.
- Resource Management: Approximately 50% lower training latency is achieved by the proposed resource-management solution than by the counterpart without optimization.Per-round latency decreases as total bandwidth increases, while power-control gains slow when data-exchange latency becomes small.
- Resource Management: Optimizing cut-layer selection provides greater performance improvement than power control and subchannel allocation when server computing capability varies.In this setting, latency is primarily limited by edge-device computing and data-exchange latency.
- Resource Management: Channel variation has little impact on EPSL performance, indicating robustness and reducing the need to frequently rerun the BCD-based algorithm.The layer-split decision remains unchanged until convergence in the simulations.
8 CONCLUSIONS
The paper proposes EPSL to accelerate split-learning training over wireless edge networks through parallel client training, gradient aggregation, and joint resource optimization. Simulations show lower time-to-accuracy than existing benchmarks, while label privacy, convergence analysis, and broader architecture generalization remain open issues.
- Contributions: EPSL parallelizes client-side training and aggregates last-layer activation gradients to reduce server computation and training latency.The framework jointly optimizes subchannel allocation, power control, and cut-layer selection under heterogeneous wireless and computing conditions.
- Results: Simulation results show that EPSL requires significantly less time to achieve target accuracy than existing benchmarks.The tailored resource-management and layer-split strategy also reduces latency compared with its unoptimized counterpart.
- Limitations and Future Work: Open issues include label privacy from label sharing in split learning, convergence analysis, and generalization to different neural-network architectures.The paper specifically highlights privacy-sensitive disease-diagnosis data as a setting where label privacy matters.