PaperlayerView exact revision on arXiv

Source-linked AI summary

Split Learning in 6G Edge Networks

Zheng Lin, Guanqiao Qu, Xianhao Chen, Kaibin Huang

arXiv:2306.12194v3cs.LGcs.DCcs.NI

TL;DR

Resource-limited IoT devices may not support the full on-device training required by federated learning, motivating split learning for 6G edge networks. The paper reviews split-learning advances, integrates them with 6G architecture and resource management, and identifies open problems including convergence analysis, asynchronous SL, and U-shaped SL.

  • Problem

    Federated learning can be unsuitable for resource-limited edge devices because it requires full model training locally, while split edge learning also faces wireless transmission and resource-coordination challenges.

  • Method

    The article reviews split-learning frameworks and designs for 6G edge architecture, resource-efficient learning, single-edge and multi-edge resource management, collaboration, placement, migration, and mobility.

  • Results

    The paper provides a comprehensive review of split edge learning and articulates its integration with 6G edge networks from learning and communication perspectives.

  • Takeaways & Limitations

    Split edge learning is presented as a promising approach for lowering on-device training resource demands and expanding machine learning across massive IoT devices.

Abstract

from arXiv · show

With the proliferation of distributed edge computing resources, the 6G mobile network will evolve into a network for connected intelligence. Along this line, the proposal to incorporate federated learning into the mobile edge has gained considerable interest in recent years. However, the deployment of federated learning faces substantial challenges as massive resource-limited IoT devices can hardly support on-device model training. This leads to the emergence of split learning (SL) which enables servers to handle the major training workload while still enhancing data privacy. In this article, we offer a brief overview of key advancements in SL and articulate its seamless integration with wireless edge networks. We begin by illustrating the tailored 6G architecture to support edge SL. Then, we examine the critical design issues for edge SL, including innovative resource-efficient learning frameworks and resource management strategies under a single edge server. Additionally, we expand the scope to multi-edge scenarios, exploring multi-edge collaboration and mobility management from a networking perspective. Finally, we discuss open problems for edge SL, including convergence analysis, asynchronous SL and U-shaped SL.

I. INTRODUCTION

The paper positions split learning as a way to support privacy-conscious model training across resource-constrained 6G edge devices. It reviews an architecture and resource-management approaches for integrating split learning with wireless edge networks.

  • Motivation: Cloud-based centralized training struggles with emerging IoT demands for bandwidth, latency, computing, and personalization.Edge learning addresses these demands through MEC-supported local training, reduced backhaul costs, low latency, and context awareness.
  • Motivation: Federated learning can be infeasible for resource-limited IoT devices because they must train the entire model locally.Split learning places part of the model at an edge server, allowing the server to handle most deep-network training while devices retain early layers and raw data.
  • Motivation: Split learning reduces device computing, storage, and memory requirements while retaining raw data locally for privacy preservation.The approach exchanges activations and gradients rather than requiring devices to train and store the complete model.
  • Design challenges: Wireless edge split learning must address frequent transmission of high-dimensional features and gradients over bandwidth-limited channels.The paper argues that learning design and communication-computing resource management must be jointly considered.
  • Scope: The article reviews 6G architecture, resource-efficient split-learning frameworks, and resource management for single-edge and multi-edge settings.Topics include model and activation compression, dynamic allocation, hierarchical or multi-hop learning, and model placement or migration for mobility.

II. BACKGROUND

The background contrasts centralized learning and federated learning with the communication and infrastructure pressures of edge deployment. Federated edge learning emerged to address the burden of repeated model exchanges at the mobile edge.

  • Federated learning: Standard centralized training sends data to remote cloud centers, creating severe privacy leakage.Federated learning keeps training data on mobile devices and uploads model updates for server aggregation.
  • Federated learning: Federated learning enables collaborative model training without accessing users’ raw data.Devices train locally, upload updates for aggregation, and download the aggregated model.
  • Federated learning: Federated learning’s large model uploads and downloads create substantial latency and burden telecommunications infrastructure.These pressures motivated communication-efficient federated learning deployment at the mobile edge, known as federated edge learning.

B. Split learning

Split learning partitions a model between clients and servers so servers share the training workload. Its variants address the latency and non-IID limitations of sequential training through parallel client-side training, with different synchronization requirements.

  • Split learning: Split learning partitions a model into client-side and server-side parts, enabling servers to share the training workload.This design targets resource limitations while preserving data privacy.
  • Vanilla split learning: Vanilla split learning trains one client at a time, which can cause excessive latency and poor performance under highly non-IID data.Sequential training may fit the model more strongly to the last client’s data distribution.
  • Parallel variants: Split federated learning and parallel split learning parallelize client-side sub-model training.Split federated learning periodically averages client models, whereas parallel split learning does not require client-side model averaging.

C. Federated Learning v.s. Split Learning

The paper frames the choice between federated and split learning around device capability and communication volume. It then motivates hierarchical 6G architectures that coordinate distributed computing, memory, communication, and mobility resources for split learning.

  • Choice between FL and SL: Split learning is favored when end devices lack the computing and memory needed to train a large model.Both federated and split learning preserve privacy, but split learning shifts more training workload to servers.
  • Communication perspective: Federated learning may be preferable for large client datasets, while split learning can be more communication-efficient when model data exceeds smashed-data volume.For ResNet-152 with batch size 32, the 37th-layer smashed data is approximately 0.49 MB versus a 230 MB model.
  • Resource coordination: Split edge learning requires joint communication and training design because sub-model computation and smashed-data transmission impose a fundamental tradeoff.Smashed data includes intermediate activations and back-propagated gradients.
  • Hierarchical architecture: Hierarchical systems distribute complete or partial models across heterogeneous devices, base stations, edge servers, and cloud resources.More powerful nodes farther from data sources can host larger sub-models, while constrained nodes host smaller ones.
  • Control and mobility: Centralized control and mobility management coordinate resources, provide QoS support, and enable service and model migration as users move.The architecture leverages network-wide distributed resources for diverse split-learning applications.

IV. RESOURCE-EFFICIENT SPLIT LEARNING FRAMEWORKS

Resource-efficient split learning frameworks target the edge’s limited spectrum and computing resources by reducing communication overhead and device-side workload. Feature compression offers a latency–accuracy trade-off, but its convergence behavior remains insufficiently characterized.

  • Edge SL must trade training accuracy against latency under limited networking and computing resources.
  • Auto-encoders compress smashed data before transmission and reconstruct it at the server, reducing split-layer communication overhead.The described approach uses cyclic convolution for compression and cyclic correlation for server-side decoding.
  • Direct activation pruning can compress smashed data without auto-encoders, but its impact on split learning still requires investigation.
  • Theoretical convergence bounds relating compression ratios to split-learning accuracy and latency remain an open requirement.

B. Split Learning with Weight Compression

Weight and gradient compression reduce split-learning resource demands across devices and servers. EPSL aggregates back-propagated gradients at the last layer, lowering server-side costs with a reported 0.46% convergence-accuracy deterioration.

  • Compressed models reduce device computing and memory costs, while servers can retain high-precision sub-models.
  • EPSL reduces back-propagation computing and communication costs from O(M) to O(1) by aggregating gradients at the last layer.M denotes the number of clients.
  • EPSL exposes aggregation ratio φ to balance communication and computing reduction against learning accuracy, with φ = 0 reducing it to PSL.
  • 0.46% deterioration is reported for EPSL when the model converges, without noticeably impacting learning accuracy.

V. RESOURCE MANAGEMENT FOR SPLIT LEARNING: THE SINGLE-CELL PERSPECTIVE

Single-cell split learning resource management must jointly address stragglers, model splitting, communication, and computation. Dynamic scheduling can exploit the split layer and batching to reduce latency.

  • Training latency in parallel split learning is determined by the slowest client, or straggler.
  • Mitigating stragglers requires jointly allocating wireless channels and server-side computing resources.
  • The split layer changes client and server workloads and communication overhead, distinguishing split learning resource allocation from federated learning.
  • On-demand scheduling can dynamically assign channel and computing resources to clients based on need, while batching controls split-learning overhead.

B. Client Selection

Multi-edge split learning addresses large-model deployment by distributing workloads across hierarchical edge and cloud resources. Deeper splits can narrow transmitted representations, while hierarchical cloud-edge-user SL outperforms two-tier counterparts in the described HAM10000 evaluation.

  • B. Client Selection: Client selection in resource-limited edge learning should jointly consider the number of selected clients or samples and data diversity.
  • Distributed edge servers can share workloads to overcome computing and memory constraints when deploying large AI models.
  • Multi-level server collaboration offers greater flexibility than two-level split learning for balancing communication and computing.
  • Deeper model splits reduce layer-output size in many CNNs, enabling narrower edge-cloud exchanges and lower communication costs.
  • The hierarchical cloud-edge-user architecture is superior to user-edge and user-cloud alternatives in the described five-client PSL comparison.The comparison concerns total training latency for achieving target accuracy on HAM10000.

B. Multi-hop Split Learning

Multi-hop split learning distributes compute-intensive training across interconnected edge servers or mobile devices. In mesh networks, model splitting and data routing must jointly account for bandwidth, computing, and memory constraints.

  • Multi-hop edge networks: Multi-hop split learning shares model-training workload across multiple edge servers to handle compute-intensive models.A general mesh can connect small and macro base stations for collaborative training.
  • System design: Mesh deployments require joint optimization of model splitting and data routing under bandwidth, computing, and memory constraints.Centralized smashed-data routing with SDN can coordinate sub-model placement and resource constraints using global knowledge.
  • System design: Centralized routing is expected to outperform distributed routing because an SDN controller provides global network knowledge.
  • Mobile ad hoc networks: Vehicular platoons and smartphone users can implement multi-hop split learning through V2V or D2D communications.These settings partition and share models across participating vehicles or devices.

C. Edge Model Placement and Migration

Edge model placement supports split learning by distributing neural-network components across users and servers, but it creates a tradeoff among storage, communication, and computing costs. The section also identifies convergence analysis as an unresolved issue for split edge learning.

  • Model placement: Partial model placement lets users and servers execute different neural-network portions, potentially improving caching performance for split learning and inference.
  • Model placement: Placing more of a model at an edge node trades greater edge storage against communication and computing costs.
  • Open problems: Convergence analysis remains necessary for resource optimization because it guides resource allocation to accelerate training.
  • Open problems: The convergence of parallel split learning lacks theoretical analysis, although empirical results suggest only a small learning-performance impairment.

B. Asynchronous Split Edge Learning

Asynchronous split learning can reduce waiting when clients have unequal training or transmission latencies, while U-shaped split learning keeps labels local when they contain sensitive information. The article concludes that split edge learning may reduce on-device resource demands in 6G.

  • Asynchronous split learning: Asynchronous parallel split learning updates the server-side model after training one or a specified number of clients instead of waiting for all clients.This design targets improved resource utilization under heterogeneous training and transmission latencies.
  • Asynchronous split learning: Harsh channel or training conditions can force faster clients to wait in the conventional synchronous framework.
  • U-shaped split learning: U-shaped split learning keeps the first and last layers on clients so output layers and sensitive labels remain local.The paradigm addresses cases where labels contain private information such as a patient’s disease.
  • Conclusion: Split edge learning is anticipated to lower on-device training resource demands and support machine learning across massive IoT deployments.
Loading 2306.12194v3…