PaperlayerView exact revision on arXiv

Source-linked AI summary

Split Learning over Wireless Networks: Parallel Design and Resource Management

Wen Wu, Mushu Li, Kaige Qu, Conghao Zhou, Xuemin, Shen, Weihua Zhuang, Xu Li, Weisen Shi

arXiv:2204.08119v2cs.NI

TL;DR

The paper addresses the high latency caused by sequential device training in split learning. It proposes CPSL, which combines intra-cluster parallel device-side training with inter-cluster sequential training, and a two-timescale resource management algorithm. Simulations on non-IID data show reduced latency versus existing SL benchmarks while adapting to network dynamics.

  • Problem

    Sequential training across devices causes significant split-learning latency, particularly as the number of participating devices grows.

  • Method

    CPSL partitions devices into clusters for parallel device-side training and aggregation, while a two-timescale algorithm jointly selects the cut layer, clusters devices, and allocates radio spectrum.

  • Results

    Simulations on non-IID data show that CPSL with resource management greatly reduces training latency versus existing SL benchmarks while adapting to network dynamics.

  • Takeaways & Limitations

    Parallelizing device-side training and adapting resource decisions provides a low-latency split-learning approach for wireless networks with heterogeneous devices and dynamic conditions.

Abstract

from arXiv · show

Split learning (SL) is a collaborative learning framework, which can train an artificial intelligence (AI) model between a device and an edge server by splitting the AI model into a device-side model and a server-side model at a cut layer. The existing SL approach conducts the training process sequentially across devices, which incurs significant training latency especially when the number of devices is large. In this paper, we design a novel SL scheme to reduce the training latency, named Cluster-based Parallel SL (CPSL) which conducts model training in a "first-parallel-then-sequential" manner. Specifically, the CPSL is to partition devices into several clusters, parallelly train device-side models in each cluster and aggregate them, and then sequentially train the whole AI model across clusters, thereby parallelizing the training process and reducing training latency. Furthermore, we propose a resource management algorithm to minimize the training latency of CPSL considering device heterogeneity and network dynamics in wireless networks. This is achieved by stochastically optimizing the cut layer selection, real-time device clustering, and radio spectrum allocation. The proposed two-timescale algorithm can jointly make the cut layer selection decision in a large timescale and device clustering and radio spectrum allocation decisions in a small timescale. Extensive simulation results on non-independent and identically distributed data demonstrate that the proposed solutions can greatly reduce the training latency as compared with the existing SL benchmarks, while adapting to network dynamics.

I. INTRODUCTION

Existing split learning reduces communication and device computation costs but trains devices sequentially, creating high latency at scale. CPSL parallelizes device-side training within clusters and combines it with sequential inter-cluster training, while a two-timescale algorithm jointly manages key wireless resources.

  • Existing split learning: Split learning partitions the AI model at a cut layer between device-side and server-side models, exchanging smashed data and gradients during training.Devices execute the device-side model and send intermediate outputs to the edge server, which performs the server-side computation and returns gradients.
  • Existing split learning: 97.8% lower communication overhead and 93.9% lower device computation workload were reported for SL with cut layer POOL1 versus FL in the LeNet example.Communication decreased from 16.49 MB to 0.35 MB, while computation decreased from 91.6 MFlops to 5.6 MFlops.
  • Motivation: Sequential interaction across devices causes significant SL training latency, especially when many devices participate.The paper frames more efficient training as the motivation for parallelizing the process.
  • CPSL scheme: CPSL partitions devices into clusters, parallelly trains device-side models within each cluster, aggregates them, and then trains sequentially across clusters.Its operation is explicitly described as “first-parallel-then-sequential.”
  • CPSL scheme: CPSL addresses the sequential nature of SL by parallelizing device-side model training, and the paper develops mathematical models to analyze its training latency.The scheme is designed as a low-latency approach, with latency analysis established for CPSL.
  • Resource management: The resource management problem jointly optimizes cut layer selection, device clustering, and radio spectrum allocation under device heterogeneity and network dynamics.A two-timescale algorithm uses sample average approximation for the large-timescale cut-layer decision and Gibbs sampling for small-timescale clustering and spectrum allocation.
  • Evaluation: Extensive simulations on real-world non-IID data show that CPSL and its resource management algorithm greatly reduce training latency against state-of-the-art SL benchmarks while adapting to network dynamics.The contribution list also identifies the two-timescale algorithm as jointly determining cut layer selection, device clustering, and radio spectrum allocation.

II. RELATED WORK

Prior split-learning studies examine applications, privacy, cut-layer selection, and resource allocation, but generally do not exploit multi-device parallelism or jointly account for wireless dynamics and device heterogeneity. This paper addresses these gaps with a parallel SL solution using one shared server-side model and resource management for dynamic wireless networks.

  • Existing SL studies: Existing SL research covers empirical applications, privacy, communication overhead, and comparisons with federated learning across diverse settings.Applications include wireless power prediction and medical X-ray classification, while other studies examine security and communication overhead.
  • Existing SL studies: Prior SL designs study multiple cut layers and online cut-layer selection to reduce latency or avoid sharing data and labels.These works include two-cut-layer models, optimal cut-layer selection, and low-complexity selection of multiple cut layers.
  • Research gaps: Most existing studies do not incorporate network dynamics or device computing capabilities, limiting long-term cut-layer optimization under changing conditions.The cited works generally assume conditions that do not capture dynamic wireless channels and heterogeneous device computation.
  • Research gaps: Existing studies focus on one-device SL and therefore do not exploit parallelism among multiple devices, leaving long training latency when many devices participate.Sequential interaction across devices is especially costly as the number of participating devices grows.
  • Research gaps: A prior SL-FL hybrid parallelizes training with multiple server-side models, but requires abundant edge-server storage and memory, especially for many devices.The proposed work instead targets parallel SL with only one shared server-side model.
  • Paper direction: The paper proposes a resource management algorithm for parallel SL that accounts for network dynamics and device heterogeneity in wireless networks.The stated contribution combines a shared-server-side-model design with resource optimization for the proposed solution.

III. SYSTEM MODEL

The system consists of an access point with an edge server and multiple devices, each holding local data and a device-side model while jointly training a split AI model without sharing raw data. CPSL partitions devices into clusters, performs parallel intra-cluster training and aggregation, and then continues training across clusters sequentially.

  • System model: The wireless SL system contains an access point with an edge server and multiple devices equipped for device-side model training.The access point collects computing and channel information for resource-management decisions.
  • System model: Each device has a local dataset and device-side model, while the access point maintains the server-side model; together they form the whole AI model.The system collaboratively trains without sharing local device data.
  • Learning objective: The learning objective is to find model parameter w⋆ that minimizes the global loss, formed by weighting device losses according to dataset sizes.The global loss averages local losses with weights proportional to the number of samples in each dataset.
  • Vanilla SL: Vanilla SL trains the model sequentially across devices, causing latency that grows with participating-device count and is aggravated by limited computing capabilities.This motivates a parallel version of SL for training-latency reduction.
  • CPSL scheme: CPSL partitions devices into clusters, parallelly trains device-side models within each cluster, aggregates them, and then trains the whole model across clusters.The scheme operates through consecutive rounds with intra-cluster and inter-cluster learning stages.
  • Resource coordination: The access point uses real-time computing and channel information to partition devices into clusters and allocate radio spectrum during training.The procedure assumes participating devices remain within coverage with static channel and computing conditions during each round.
  • Intra-Cluster Learning: Within a cluster, devices execute their device-side models on local mini-batches, transmit smashed data to the access point, and receive gradients for parallel updates.The access point concatenates smashed data, executes and updates the server-side model, then aggregates device-side models.

C. Inter-Cluster Learning

CPSL transfers the aggregated device-side model from one cluster to the next, repeating intra-cluster learning until all clusters finish. Thus, devices train in parallel within clusters while clusters proceed sequentially, reducing training latency.

  • C. Inter-Cluster Learning: After a cluster finishes, the access point transfers its aggregated device-side model to the next cluster for continued training.The updated model is broadcast to devices in the next cluster, which then conduct intra-cluster learning.
  • C. Inter-Cluster Learning: Inter-cluster learning is sequential across clusters, while intra-cluster device training is parallel.This preserves sequential progression between clusters while exploiting parallelism among devices within each cluster.
  • C. Inter-Cluster Learning: Extensive simulations validate that CPSL can significantly reduce training latency.The reported validation concerns the proposed training process and its latency reduction.

V. TRAINING LATENCY ANALYSIS

The CPSL training-latency analysis identifies cut layer, device clustering, and radio spectrum allocation as key decisions. Cut layer selection is fixed for the full process, while clustering and spectrum allocation are updated per round.

  • CPSL determines cut layer selection, device clustering, and radio spectrum allocation as its training-latency decision variables.
  • The cut layer is selected beforehand from the available DNN layers using historical device data.
  • A cut layer at v = V produces an empty server-side model, reducing CPSL to FL with K_m devices.
  • Device clustering is determined from collected real-time device channel conditions and computing capabilities.
  • Radio spectrum allocation assigns subcarriers to devices using frequency-division multiple access, subject to total capacity C.

B. Training Latency

CPSL models each cluster’s training process as a sequence of starting, inner, and ending phases. Their latency accounts for device computation, transmissions, server execution and updates, model distribution, and synchronization across participating devices.

  • The AP waits for all participating devices before server-side execution and before aggregating device-side models.
  • Each cluster’s training process has L + 1 chronological phases: one starting phase, L − 1 inner phases, and one ending phase.No inner phase exists when the number of local rounds equals 1.
  • Starting-phase latency includes device-side model distribution, device-side model execution, smashed data transmission, server-side model execution, and server-side model update.
  • Device-side execution processes a mini-batch, with latency determined by computation workload, CPU capability f_k, and computing intensity κ.
  • Transmission latency depends on smashed-data size, allocated subcarriers, channel gain, transmission power, and subcarrier bandwidth.
  • Server-side execution and update latency includes forward and backpropagation processing for concatenated smashed data from participating devices.

2) Inner Phase:

Each inner phase captures the recurring exchange between server-side processing and device-side updates, while the ending phase completes gradient transmission, model transmission, and aggregation. Overall CPSL latency depends on cut layer, clustering, and spectrum allocation.

  • 2) Inner Phase:: Each inner phase contains smashed-data gradient transmission, device-side model update and execution, smashed-data transmission, server-side execution, and server-side update.
  • 2) Inner Phase:: Smashed-data gradients are transmitted back to devices after server-side execution and update using allocated radio spectrum.
  • 2) Inner Phase:: The cut layer shifts computation workload between device and edge server: shallow cuts burden the server, whereas deep cuts burden devices.
  • 3) Ending Phase:: The ending phase includes smashed-data gradient transmission, device-side model update, device-side model transmission, and model aggregation.
  • 3) Ending Phase:: Model aggregation latency is negligible because FedAvg aggregation has relatively low computational complexity.
  • CPSL training latency depends jointly on device clustering, radio spectrum allocation, and cut layer selection.
  • Cut layer selection affects both communication overhead and computation workload distribution between devices and the edge server.

VI. RESOURCE MANAGEMENT PROBLEM FORMULATION AND DECOMPOSITION

The resource-management problem is a stochastic mix-timescale optimization because device capabilities and channel conditions vary over time. The paper decomposes it into large-timescale cut-layer selection and small-timescale clustering and spectrum allocation, then solves it with a two-timescale algorithm.

  • A. Problem Formulation: The problem is stochastic because device computing capabilities and channel conditions exhibit temporal dynamics, and clustering and spectrum allocation are coupled.
  • A. Problem Formulation: The objective is to minimize overall CPSL training latency while jointly selecting the cut layer, device clustering, and radio spectrum allocation.
  • A. Problem Formulation: Cut layer selection is made for the entire training process, whereas clustering and spectrum allocation are determined for each training round.
  • B. Problem Decomposition: The formulation is decomposed using timescale separation into large-timescale cut-layer selection and small-timescale clustering and spectrum allocation.
  • B. Problem Decomposition: The small-timescale subproblem is combinatorial and NP-hard because its optimization variables are integers.
  • B. Problem Decomposition: The latency objective is non-convex because it minimizes the maximum latency across participating devices and stages.
  • VII. TWO-TIMESCALE RESOURCE MANAGEMENT ALGORITHM: The proposed two-timescale algorithm combines SAA-based cut-layer selection with Gibbs sampling-based joint clustering and spectrum allocation.
  • A. Large Timescale: Cut Layer Selection Algorithm: SAA estimates average per-round latency from historical samples, and exhaustive search evaluates all finite candidate cut layers offline.

B. Small Timescale: Joint Device Clustering and Radio Spectrum Allocation Algorithm

At the small timescale, device clustering and radio spectrum allocation are jointly optimized using real-time device and channel conditions. The problem is decomposed into clustering and per-cluster spectrum-allocation subproblems.

  • Small Timescale: Joint Device Clustering and Radio Spectrum Allocation Algorithm: The algorithm jointly determines clustering and spectrum allocation decisions in every training round.
  • Small Timescale: Joint Device Clustering and Radio Spectrum Allocation Algorithm: Real-time device computing capabilities and channel conditions determine clustering decisions that minimize instantaneous one-round training latency.
  • Small Timescale: Joint Device Clustering and Radio Spectrum Allocation Algorithm: A greedy-based subroutine is proposed for radio spectrum allocation after drawing historical samples of device capabilities and channel conditions.
  • Small Timescale: Joint Device Clustering and Radio Spectrum Allocation Algorithm: The joint optimization separates device clustering into an outer-layer subproblem and radio spectrum allocation into inner-layer subproblems for each cluster.Given clustering, spectrum allocation decisions are made independently across clusters.

1) Radio Spectrum Allocation Subproblem:

The inner-layer spectrum-allocation problem is an integer optimization with a non-convex objective, so the paper uses a diminishing-gain greedy procedure within a Gibbs-sampling clustering algorithm.

  • 1) Radio Spectrum Allocation Subproblem:: Radio spectrum allocation is optimized independently within each cluster because per-cluster decisions determine the accumulated training latency across clusters.
  • 1) Radio Spectrum Allocation Subproblem:: The spectrum-allocation problem is an integer optimization with a non-convex objective that cannot be solved by existing convex optimization methods.
  • 1) Radio Spectrum Allocation Subproblem:: The greedy allocation initializes each device with one subcarrier and repeatedly assigns the next subcarrier to the device yielding the largest latency reduction.
  • 1) Radio Spectrum Allocation Subproblem:: Device clustering is formulated as a binary optimization problem with cluster-capacity constraints and solved iteratively using Gibbs sampling.
  • 1) Radio Spectrum Allocation Subproblem:: The clustering algorithm embeds spectrum allocation, jointly determining both decisions while exploring updated assignments through a smooth-factor-controlled probability.When δ approaches 0, the algorithm converges to the global optima with probability 1.

VIII. SIMULATION RESULTS

Simulations evaluate CPSL under non-IID data using image-classification workloads and compare it with centralized learning, vanilla SL, and FL. CPSL reaches comparable accuracy with substantially lower training latency than SL.

  • VIII. SIMULATION RESULTS: The simulations use MNIST and Fashion-MNIST with non-IID device data, where each device contains three randomly selected classes and 180 samples.
  • VIII. SIMULATION RESULTS: The evaluation uses a 12-layer chain-topology LeNet with around 4.3 million 32-bit parameters and 5 devices per cluster.
  • VIII. SIMULATION RESULTS: CPSL achieves nearly the same accuracy as centralized learning and vanilla SL, but requires about twice as many training rounds to converge.
  • VIII. SIMULATION RESULTS: 600 seconds versus 1,400 seconds: CPSL requires less overall training time than vanilla SL to reach convergence.
  • VIII. SIMULATION RESULTS: 3.78 seconds, 13.90 seconds, and 33.43 seconds: CPSL, SL, and FL have these respective per-round training latencies.

2) Impact of Cluster Size:

CPSL reduces latency through parallel device-side training while preserving accuracy, and its two-timescale resource management adapts clustering, spectrum allocation, and cut-layer selection to heterogeneous wireless conditions.

  • 2) Impact of Cluster Size:: For 3 to 10 devices per cluster, CPSL converges faster than SL because device-side models are trained in parallel.
  • 2) Impact of Cluster Size:: CPSL achieves nearly the same final accuracy as the benchmarks without incurring accuracy loss while reducing training latency.
  • 2) Impact of Cluster Size:: POOL1 is selected as the optimal cut layer because it minimizes average per-round latency by reducing communication overhead and balancing device-server computation.
  • 2) Impact of Cluster Size:: At δ = 0.01, increasing the smooth factor may impede global-optimum identification and lead to inferior solutions.
  • 2) Impact of Cluster Size:: 80.1% and 56.9%: the proposed resource-management algorithm reduces latency on average versus heuristic and random clustering benchmarks, respectively.
  • 2) Impact of Cluster Size:: The performance gain is higher at 10 MHz than 60 MHz, highlighting the algorithm's role in alleviating CPSL stragglers under spectrum limitation.
Loading 2204.08119v2…