PaperlayerView exact revision on arXiv

Source-linked AI summary

Concurrent Split Learning Through Stable Client Clustering

Mohammad Kohankhaki, Valentin Rentschler, Anke Schmeink

arXiv:2609.29395v1cs.DCcs.LG

TL;DR

A fixed global batch limits how many distributed clients can contribute data in one training step. GCPSL assigns clients to fixed clusters, runs a GPSL workload in each cluster concurrently, and periodically fuses model replicas. In measured runs, fixed assignment policies trade direct participation against target time, while more clusters can increase participation but reduce accuracy.

  • Problem

    When the client population greatly exceeds a fixed global batch, most clients cannot contribute data in a round despite available server capacity.

    Source: §I · p. 1

  • Method

    GCPSL runs a fixed-batch GPSL workload concurrently in each stable client cluster and periodically fuses the split-model replicas.

    Source: §I · p. 1

  • Results

    Size-balanced and random GCPSL reach the target in 5.70±0.50 and 5.66±0.18 minutes, respectively, while size balancing raises mean direct data participation from 56.96% to 60.21%.

    Source: D. Fixed-Assignment Results · p. 6

  • Takeaways & Limitations

    More clusters can raise direct participation but may reduce label diversity, and assignment choices trade participation and composition against workload outcomes.

    Source: A. Clustered Workloads · p. 3, A. Operating Region · p. 8

    The method applies when client assignments are stable and the server can run several moderate-batch workloads concurrently.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Training with a fixed global batch limits how many distributed clients can provide examples in any one step. We examine a way to use additional server workers without increasing the batch processed by an individual workload. Global Clustered Parallel Split Learning (GCPSL) assigns clients to fixed clusters, executes a Parallel Split Learning with Global Sampling (GPSL) workload for each cluster concurrently, and periodically fuses the client and server model segments. In simulations with 256 logical clients, dividing the population across more workloads improves direct data participation, while smaller clusters can incur an accuracy cost. A four-H100 implementation of label-aware GCPSL reaches 85% CIFAR-10 validation accuracy in $6.13 \pm 0.15$ minutes over three matched runs, versus $19.09 \pm 0.45$ minutes when the same workloads are serialized. Within the four-GPU allocation, size-balanced and random fixed affiliations reach the target in similar mean times (5.70 and 5.66 minutes); size balancing increases direct participation by 3.25 percentage points. These measurements characterize a trade-off among execution concurrency, assignment information, participation, and accuracy for stable-client split learning.

I. INTRODUCTION

A fixed global batch bounds per-step work but limits direct participation when the client population is large. GCPSL keeps each workload’s batch moderate while clustering clients into concurrent GPSL workloads and periodically fusing their split models.

I. INTRODUCTION

  • With K much larger than fixed batch size B, most clients cannot supply data in a round, even when server capacity is available.Increasing B can admit more clients, but changes memory use and the optimization regime.Source: §I · p. 1
  • GCPSL assigns clients to stable clusters, runs one original-batch GPSL workload per cluster concurrently, and periodically fuses split-model replicas.Separate clusters train without moving raw client data.Source: §I · p. 1, Figure 2 · p. 2
  • The paper measures participation and accuracy as cluster count grows and compares random, sample-count-balanced, and label-aware assignments.The evaluation also compares concurrent execution with serial execution of the same workloads.Source: §I · p. 1
  • The sequential schedule takes 3.11× as long as concurrent execution of the same workloads on four H100 GPUs.The paper also compares fixed assignments under the same allocation.Source: §I · p. 1

A. Split and Parallel Split Learning

  • Prior parallel and distribution-aware split-learning work does not directly study the participation limit caused by one fixed global GPSL batch.GPSL samples from global distributed data rather than concatenating independent local mini-batches, improving batch representativeness under non-IID data.Source: A. Split and Parallel Split Learning · p. 2

B. Participation and Workload Organization

GCPSL differs from federated grouping because client clusters control split-workload execution and fusion state as well as client grouping. The paper studies metadata-informed assignment and concurrency with stable clients and server execution slots.

B. Participation and Workload Organization

  • Client grouping in GCPSL controls server-workload length, activation traffic, and fusion state, unlike federated methods that aggregate client-computed model updates.The related work includes client selection, resource-based grouping, and clustered federated learning.Source: B. Participation and Workload… · p. 2
  • The study compares assignment metadata budgets from none, to per-client sample counts, to sample counts plus class histograms.Balanced sample counts can still yield a poor class mixture, while statistically balanced clusters can have longer schedules.Source: B. Participation and Workload… · p. 2

C. Concurrent and Distributed Training

  • The concurrency experiment holds per-workload batch, clustering rule, and training data fixed while measuring wall-clock time with less frequent fusion.GCPSL runs one cluster-local split-learning workload per server slot.Source: C. Concurrent and Distributed Training · p. 2

A. Entities and Assumptions

  • The system assumes stable client membership, locally retained raw examples, and a coordinator able to assign clients and collect metadata required by the chosen rule.The coordinator is trusted to create assignments and perform fusion; the model does not prescribe a processor or network mapping.Source: A. Entities and Assumptions · pp. 2–3
  • A system epoch ends only after every cluster finishes, so shorter workloads wait at the fusion barrier without taking extra updates.Clusters with different sample counts may require different numbers of local rounds.Source: A. Entities and Assumptions · p. 3
  • Direct data participation is distinct from model-update participation: clients with no example in a round may still apply their cluster’s shared client-model update.Such clients perform no client-side forward or backward computation that round.Source: A. Entities and Assumptions · p. 3

B. Communication and Assignment

GCPSL communicates locally within fixed client clusters and fuses model state across clusters at epoch boundaries. Its fixed-batch participation limit and clustering rules frame the trade-off between participation, sample balance, and class composition.

B. Communication and Assignment

  • Within each cluster, active clients exchange activations and cut gradients with its server replica; active gradients update every client replica there.At fusion, the coordinator combines client and server parameters plus server normalization state across clusters, then broadcasts the result.Source: B. Communication and Assignment · p. 3
  • Random clustering uses no data-derived metadata, size balancing uses sample counts, and label-aware clustering uses sample counts plus class histograms.Metadata determine fixed assignments before training and are not sent with each batch.Source: B. Communication and Assignment · p. 3, A. Clustered Workloads · p. 3

C. Fixed-Batch Participation Limit

  • For K=256, the no-participation lower bound is 50% at B=128 and 75% at B=64, independent of client or server speed.GCPSL reduces the client population served by each workload while preserving its batch size.Source: C. Fixed-Batch Participation Limit · p. 3

A. Clustered Workloads

  • Each cluster retains a fixed client map and sampler, optimizer, and model replicas across local rounds; clusters run in parallel and periodically fuse parameters and normalization state.Clustering occurs once before training, and the move search requires O(IKN) evaluations.Source: A. Clustered Workloads · pp. 3–4
  • Balanced clusters contain about K/N clients; increasing cluster count can raise direct participation but may reduce label diversity within each GPSL sampler.The assignment rule controls this participation-and-composition trade-off.Source: A. Clustered Workloads · p. 3
  • Label-aware assignment starts balanced, then accepts single-client moves that reduce a cost balancing cluster sample counts and class-distribution divergence from the global distribution.The search stops at a local optimum or fixed move limit, using incrementally maintained cluster statistics.Source: A. Clustered Workloads · p. 3

B. Cluster-Local GPSL

Within a cluster, active clients’ gradients produce a shared client-model update applied to every replica. Normalization is handled differently on client and server segments.

B. Cluster-Local GPSL

  • Active client gradients are averaged within each cluster, and every client replica there applies the average, including replicas whose owners supplied no data.Consequently, client replicas in a cluster share a common client model.Source: B. Cluster-Local GPSL · p. 4
  • Client-segment BatchNorm is replaced with GroupNorm, while server replicas retain BatchNorm running statistics and batch counters for fusion.The server normalization state comprises running mean, running variance, and batch counter.Source: B. Cluster-Local GPSL · p. 4

C. Fusion and Broadcast

GCPSL fuses cluster replicas at epoch boundaries and broadcasts a common model state, while retaining cluster-local optimizer states. Its schedule and participation measures reflect the longest workload, waiting at fusion barriers, and limits on direct data contribution.

C. Fusion and Broadcast

  • At each epoch boundary, GCPSL fuses model parameters and server normalization state using cluster sample weights, then broadcasts the common state to every replica.Validation and test evaluation use this post-fusion model.Source: C. Fusion and Broadcast · p. 4
  • Optimizer states remain local after fusion, so each cluster resumes with its own optimizer state despite receiving common parameters.GCPSL does not average momentum or other optimizer variables.Source: C. Fusion and Broadcast · p. 4

D. Schedule and Cost

  • Concurrent execution completes an epoch after the longest cluster workload, while shorter clusters wait at the fusion barrier without padded optimizer steps.Sequential execution of the same workloads requires the sum of their update slots.Source: D. Schedule and Cost · p. 4
  • Uneven client sample counts and repeated requests can leave observed inactivity above the lower bound on direct participation.At most B clients contribute in a cluster workload; GCPSL maintains N cluster-level split-model states and communicates O(NP) logical state at epoch boundaries.Source: D. Schedule and Cost · p. 4

V. EXPERIMENTAL METHODOLOGY

The experimental program combines simulator studies of cluster count, assignment, and fusion with a four-GPU comparison of execution and assignment regimes. CIFAR-10 and CIFAR-100 experiments use a fixed model and data protocol, with matched data and example budgets within simulator comparisons.

V. EXPERIMENTAL METHODOLOGY

  • The studies examine participation, target convergence, and accuracy in simulation, then compare concurrent and sequential workloads and three fixed-assignment rules on four H100 GPUs.Additional studies vary fusion interval and batch size, compare synchronization schemes, and evaluate CIFAR-100.Source: §V · p. 5

A. Model and Data

  • CIFAR-10 and CIFAR-100 use ResNet-18 split after its second residual stage, with 256 logical clients assigned support over two classes under the extended-Dirichlet protocol.The primary CIFAR-10 setting uses B = 64; CIFAR-10 at B = 128 and CIFAR-100 at B = 128 are also studied.Source: A. Model and Data · p. 5

B. Study Families

  • Simulator sweeps compare label-aware and size-balanced GCPSL at N = 2, 4, 8, and 16, with random GCPSL at N = 8, and vary fusion controls and batch size.Each run processes five million training examples; reported simulator means and standard deviations cover three configured runs.Source: B. Study Families · p. 5
  • Within each simulator comparison, the data partition, model split, optimization settings, and total example budget remain fixed as cluster count changes.The schedule-accounted view uses the longest cluster workload as the parallel critical path; processed samples distinguish shorter schedules from fewer examples.Source: B. Study Families · p. 5
  • The real-execution study uses four GPUs for CIFAR-10 with K = 256, N = 4, and B = 64 per cluster, with one H100 process and workload per cluster.Each execution regime has three matched runs; training and validation examples are split before client partitioning, and test examples are reserved for reporting.Source: B. Study Families · p. 5

C. Comparison Regimes

The comparison regimes vary concurrency, assignment, batch size, synchronization, or reassignment while specifying how timing, accuracy, participation, and communication are reported. The four-GPU implementation uses process-local cluster workloads and measures peak memory separately for matched workloads on one RTX A4000.

C. Comparison Regimes

  • The fixed-assignment comparison changes client assignments while holding the four-GPU schedule, per-workload batch, and fusion interval fixed; the sequential control runs the same four label-aware workloads in sequence.GPSL reference baselines vary batch size, synchronization, or client assignment.Source: C. Comparison Regimes · p. 5, D. Reporting Protocol · p. 5
  • GPSL references include one-workload batches of B = 64 or B = 256, sharded synchronous B = 64 across four ranks, data-parallel B = 64 per rank, and periodic reassignment.The synchronous baselines synchronize one global model every step; the repartitioned periodic baseline reassigns clients at each fusion interval.Source: C. Comparison Regimes · p. 5

D. Reporting Protocol

  • The 85% target is measured on validation accuracy, while test accuracy is reported at the best-validation checkpoint and epoch 100, so early target timing and final test accuracy are separate outcomes.If only some runs reach a target, tables report coverage and average crossing rounds among successful runs; a dash means the target was unreached.Source: D. Reporting Protocol · p. 5

E. Metrics and Execution Environment

  • The primary real-execution metric, t85, is wall-clock time to the first epoch reaching 85% validation accuracy and includes training, collectives, barriers, and validation.GPU-worker-minutes multiply t85 by allocated GPUs, including barrier waiting; direct data participation is the share of client–round slots with at least one supplied example.Source: E. Metrics and Execution Environment · pp. 5–6
  • Simulator Rq measures ideal schedule-accounted rounds to reach q% test accuracy, while processed samples provide a schedule-independent view and batch deviation measures class-distribution distance.Logical application payload is reported in decimal GB per phase and decimal TB accumulated to target crossing, not as measured physical network bytes.Source: E. Metrics and Execution Environment · p. 6
  • The four-GPU implementation assigns one PyTorch Distributed process, cluster, sampler, optimizer, and server replica to each H100, fusing parameters and normalization state at epoch barriers.A separate RTX A4000 control measures per-workload peak memory over ten epochs; it does not compare total memory across the four-GPU allocation.Source: E. Metrics and Execution Environment · p. 6

VI. RESULTS

GCPSL shortens schedule-accounted convergence at fixed workload batch sizes, but its accuracy and participation outcomes depend on cluster count and assignment rule. The simulator results also show that class-distribution balance does not consistently produce the fastest target crossing.

A. Cluster Count Trades Participation for Local Diversity

  • GCPSL reduces schedule length for CIFAR-10 targets at B = 64, while processed-sample trajectories are closer because each 100-epoch run processes the same five million examples.The main effect is better schedule use, not a uniform improvement in sample efficiency.Source: A. Cluster Count Trades Participation… · p. 6
  • At N = 4, label-aware GCPSL reduces R85 by 43.2% relative to GPSL and finishes 0.77 percentage points higher; size-balanced GCPSL reduces R85 by 47.1% with similar final accuracy.At N = 8, size-balanced GCPSL gives the lowest mean R85 and R88 using only sample-count metadata.Source: A. Cluster Count Trades Participation… · p. 6
  • At B = 128, GPSL reaches 85% in 4.69 × 10^3 rounds, while label-aware GCPSL at N = 4 takes 2.83 × 10^3 rounds and finishes 1.40 percentage points lower.The larger relative gain at B = 64 agrees with the fixed-batch bound.Source: A. Cluster Count Trades Participation… · p. 6

B. Assignment Information Changes the Accuracy Tradeoff

  • At N = 16, label-aware and size-balanced GCPSL lower inactivity below 5% but lose about 3.5–3.6 accuracy points relative to N = 4.Label-aware GCPSL gives the highest final accuracy at N = 2, while size-balanced GCPSL reaches targets earlier at several moderate cluster counts.Source: B. Assignment Information Changes the… · p. 6
  • At N = 8, mean batch deviation is 0.30 for label-aware, 0.43 for size-balanced, and 0.40 for random GCPSL, but label-aware clustering does not give the fastest CIFAR-10 crossing.In the CIFAR-100 control at N = 4, label-aware GCPSL reaches the 60% target more consistently than size balancing; across three CIFAR-10 runs, size balancing reaches 85% sooner while label-aware has lower mean R88.Source: B. Assignment Information Changes the… · p. 6

C. Concurrent Versus Sequential Execution

Concurrent execution reaches the CIFAR-10 validation target sooner than sequential execution of the same workloads; fixed-assignment and simulator controls report trade-offs in time, accuracy, participation, fusion, and batch size.

D. Fixed-Assignment Results

  • Label-aware GCPSL reaches the target in 6.13±0.15 minutes, with 59.43% mean direct data participation and 89.53% mean final test accuracy, but does not improve target time over random or size-balanced clustering.Source: D. Fixed-Assignment Results · p. 6
  • 89.32% versus 89.59% mean test accuracy: size-balanced GCPSL and the single-worker baseline, respectively; size balancing reaches the target 5.18 minutes sooner but uses 22.81 versus 10.88 mean GPU-worker-minutes.The reported accuracy, target time, and worker time are separate outcomes.Source: D. Fixed-Assignment Results · p. 7

E. Fusion, Batch Size, and CIFAR-100 Controls

  • Without fusion, label-aware GCPSL at N = 8 finishes at 20.59%; epoch fusion finishes 0.98 percentage points higher than 100-step fusion and reaches 88% in fewer rounds.Epoch fusion is used in the real-execution study.Source: E. Fusion, Batch Size, and CIFAR-100… · p. 7
  • 3.177 TB is the lowest displayed payload to 85% for size-balanced GCPSL at N = 4; 5.599 TB is the lowest payload to 88% for label-aware GCPSL at N = 4.These are logical endpoint payloads, not physical network traffic.Source: E. Fusion, Batch Size, and CIFAR-100… · p. 7
  • 1.47 × 10^3 versus 3.59 × 10^3 rounds: GPSL with B = 512 reaches 85% sooner than size-balanced GCPSL at N = 4; peak GPU memory is 4242 MiB versus 1640 MiB per workload.The memory measurements use the same server and software environment.Source: E. Fusion, Batch Size, and CIFAR-100… · pp. 7–8

C. Concurrent Versus Sequential Execution

  • 6.13±0.15 minutes versus 19.09±0.45 minutes: concurrent versus sequential label-aware GCPSL reaches 85% validation accuracy across three matched runs.Both schedules use the same four workloads; Fig. 5(a) reports wall-clock time to the target.Source: C. Concurrent Versus Sequential… · p. 6, Figure 5 · p. 8
  • 24.51 versus 76.35 mean GPU-worker-minutes: concurrent execution uses less allocated worker time than the sequential control to reach the target.The four-GPU allocation remains active in both schedules.Source: C. Concurrent Versus Sequential… · p. 6, Figure 5 · p. 8

E. Fusion, Batch Size, and CIFAR-100 Controls

  • On CIFAR-100, label-aware GCPSL at N = 4 reaches 60% in all three runs and finishes at 61.25%, while size-balanced GCPSL reaches 60% once and finishes at 59.43%.The passage attributes the class-histogram rule's preserved cluster composition to this setting.Source: E. Fusion, Batch Size, and CIFAR-100… · p. 8

VII. DISCUSSION AND LIMITATIONS

GCPSL’s operating region depends on balancing concurrent workloads against the client-to-batch ratio: more clusters increase participation but can narrow accuracy margins. Assignment and fusion choices also trade class composition against metadata use and coordination.

A. Operating Region

  • Increasing N from four to eight raises participation and reduces target rounds while narrowing the accuracy margin; at N = 16, informed assignments lose about 3.5–3.6 percentage points versus N = 4.When NB ≥ K, more clusters cannot improve the inactivity bound, while smaller clusters may reduce the label diversity seen by local samplers.Source: A. Operating Region · p. 8
  • Unlike random clustering, which requires no data-derived metadata, label-aware clustering uses class histograms; it better preserves composition on CIFAR-100, at the cost of disclosing more about each local dataset.Source: A. Operating Region · p. 8
  • Rq values are thousands of ideal schedule rounds, and the size-balanced CIFAR-100 R60 value represents the one run of three that reached 60%.A dash marks an unreached target.Source: A. Operating Region · p. 9
  • Without fusion, replicas trained on different cluster data remain uncoupled and perform poorly; the fusion interval trades coordination cost against replica drift.The controls compare one-epoch fusion, 100-step fusion, and no fusion.Source: A. Operating Region · p. 9, Table III · p. 9
  • GCPSL reaches the target sooner than a single-worker baseline but uses more GPU-worker time; globally synchronized and repartitioned references differ in synchronization and client-affiliation rules.Those references can also reach the target quickly, but do not maintain fixed client clusters with epoch-level fusion.Source: A. Operating Region · p. 9

B. Evidence Limits

The conclusion reports tradeoffs for stable-client split learning, while the real-execution evidence is limited to one model, dataset, client mapping, and four-GPU node. Physical edge conditions and several deployment settings remain untested, and the reported payload and memory measures have bounded scope.

B. Evidence Limits

  • The real-execution study covers one ResNet-18 split, CIFAR-10, 256 logical clients, and one four-H100 node; measured-run variation does not establish performance across other workloads.Multi-node communication, client churn, asynchronous fusion, other split points, larger models, and heterogeneous accelerators remain untested.Source: B. Evidence Limits · p. 9
  • The 256 clients are logical data owners mapped onto four GPU processes, not physical edge devices; latency, compute variation, and intermittent availability are absent from this implementation.These conditions limit how directly node-level times transfer to a deployed edge system.Source: B. Evidence Limits · p. 9
  • Schedule-accounted rounds omit deployed contention, payload counts application tensors rather than physical network traffic, and GCPSL provides no formal privacy guarantee.Activations, cut gradients, and model state can still leak information.Source: B. Evidence Limits · p. 9

VIII. CONCLUSION

  • GCPSL’s reported tradeoff applies to stable clients, fixed per-workload batches, and servers able to run several moderate-batch workloads concurrently.More clusters increase participation but can reduce accuracy as each workload sees fewer clients.Source: §VIII · p. 10
Loading 2609.29395v1…