PaperlayerView exact revision on arXiv

Source-linked AI summary

Latency-Aware Client Assignment for Parallel Split Learning With Global Sampling

Mohammad Kohankhaki, Valentin Rentschler, Anke Schmeink

arXiv:2609.32132v1cs.LG

TL;DR

GPSL forms representative pooled batches across clients with differing class distributions, but ignores delay differences when multiple clients can supply the same class. LB-GPSL fixes the integer class target before assigning examples to clients, using flow or greedy assignment and a planner that accounts for schedule-construction cost. In the reported evaluations, the variants reduce modeled training time, with accuracy and schedule-construction overhead relevant to the comparison.

  • Problem

    GPSL forms representative batches from non-IID client data but does not exploit predicted completion-time differences among clients holding the same class.

    Source: §I · p. 1

  • Method

    LB-GPSL fixes each pooled batch's integer class target, assigns its examples to eligible clients using calibrated delays, and selects between flow and greedy variants based on predicted schedule cost including construction overhead.

    Source: §I · pp. 1–2, §VIII · p. 8

  • Results

    Flow LB-GPSL reduces modeled training time 6.75% on CIFAR-10 with a 0.30 percentage-point decrease in final accuracy; Fast LB-GPSL reduces modeled time 16.87% on TIN-C20 and reaches all four validation targets earlier.

    Source: §IX · p. 9

  • Takeaways & Limitations

    The preferred assignment variant depends on whether its modeled client-time savings offset schedule-construction cost, especially in the tested larger-class settings.

    Source: §VIII · p. 8

    Reassignment can change gradients and learning trajectories even when class targets match, so learning quality must be assessed alongside modeled time.

    Source: §VIII · p. 8

Abstract

from arXiv · show

In cross-silo split learning, Parallel Split Learning with Global Sampling forms representative pooled batches when class distributions differ across clients, but ignores client delay when several clients can supply the same class. We introduce Latency Budgeted Parallel Split Learning with Global Sampling, which separates each pooled batch's integer class target from the choice of clients that supply its examples. The flow variant formulates this assignment as an integral network-flow problem and minimizes modeled client-side completion time for the current target. The fast variant uses a greedy next-completion rule to reduce schedule-construction cost. Both preserve the target stream and use every local example once per epoch. A planning rule selects between the variants while accounting for the cost of constructing both candidate schedules. On CIFAR-10, the flow variant reduces modeled training time by 6.75%, with a 0.30 percentage-point decrease in final accuracy. On Tiny ImageNet with 20 candidate classes per client, the fast variant reduces modeled time by 16.87% and reaches all four validation targets earlier than the latency-unaware baseline. Across 405 schedule comparisons, the planning rule stays within 2% of the lower realized cost in 96.54% of cases. In our evaluation, latency-aware provider assignment reduces modeled training time without changing the prescribed class targets, while the preferred variant depends on whether assignment savings outweigh schedule-construction overhead.

I. INTRODUCTION

GPSL forms representative pooled batches across non-IID clients but does not use predicted delays when choosing among clients that hold the same class. LB-GPSL preserves the class targets while assigning their examples to providers, using either a flow or greedy method and a cost-aware planner.

I. INTRODUCTION

  • GPSL forms representative global batches from non-IID client data, but does not use predicted completion-time differences when several clients hold the same class.This leaves provider choice as a separate opportunity while preserving the prescribed class target.Source: §I · p. 1
  • LB-GPSL derives an integer target from remaining pooled class counts, assigns its examples to eligible clients using calibrated delays, and leaves the PSL update rule unchanged.A fresh randomized schedule is constructed before each epoch.Source: §I · p. 1
  • Flow solves an integral assignment problem for modeled client-side time, Fast assigns each example to the eligible client with the earliest predicted next completion, and the planner includes schedule-construction overhead.The planner stays within 2% of the lower-cost algorithm in 96.54% of 405 schedule comparisons and requires no learning-curve information.Source: §I · pp. 1–2

A. Split Learning and Global Sampling

  • LB-GPSL changes how GPSL realizes an explicit class target rather than changing the PSL update, cut, client capacity, or pooled class target.Related split-learning systems instead modify learning, aggregation, execution, or resource-allocation choices.Source: §I · p. 1, A. Split Learning and Global Sampling · p. 2

I. INTRODUCTION

  • Flow LB-GPSL reduces modeled training time 6.75% on CIFAR-10; Fast LB-GPSL reduces it 16.87% on Tiny ImageNet with 20 candidate classes per client and reaches all four validation targets earlier.On CIFAR-10 and CIFAR-100, lower modeled time accompanies lower mean final accuracy.Source: §I · p. 2

B. Heterogeneity, Selection, and Load Balancing

The system model considers synchronous cross-silo PSL with heterogeneous client data and resources. Each pooled step assigns an integer class target to clients with remaining examples, while the standard PSL update uses a shared pooled loss and synchronized client participation.

B. Heterogeneity, Selection, and Load Balancing

  • LB-GPSL differs from participation- and capacity-selection approaches by assigning a fixed class target among eligible clients while preserving the complete epoch sample set.The cited related work includes methods that control participation, tiers, samples, pace, client capacity, or partition placement.Source: B. Heterogeneity, Selection, and Load… · p. 2

III. SYSTEM MODEL AND PROBLEM FORMULATION

  • The model is synchronous cross-silo PSL for supervised classification; clients report class counts and calibrated delay profiles while raw examples remain local.The pooled population is formed from the clients' local example counts.Source: §III · p. 2

A. One Synchronous PSL Update

  • At each PSL step, active clients send smashed activations and labels for a pooled loss, then receive cut-layer gradients and apply the same averaged client-model update.LB-GPSL retains this update rule rather than replacing it with periodic federated model averaging.Source: A. One Synchronous PSL Update · p. 2

B. Class–Client Allocation

  • The allocation represents each pooled batch with integral class–client counts that realize the integer target from remaining support and consume that support after assignment.A client remains eligible for a class only while it has unused examples of that class.Source: B. Class–Client Allocation · pp. 2–3

C. Calibrated Client-Side Timing Model

The timing model estimates client-side delay for each local batch and evaluates assignment cost under synchronous barriers. It also separates the modeled client-side objective from server computation, which is common for assignments with the same pooled batch size.

C. Calibrated Client-Side Timing Model

  • The predicted client-side time for a local batch includes forward computation, activation upload, gradient download, and backward computation.The calibrated model includes a batch-independent term, a batch-dependent coefficient, and a measured batch-size response.Source: C. Calibrated Client-Side Timing Model · p. 3
  • The modeled client-side objective accounts for separate forward–upload and downlink–backward synchronization barriers.For the same assignment, the two-barrier model is bounded by twice the maximum client-side time.Source: C. Calibrated Client-Side Timing Model · p. 3
  • The evaluation reconstructs the two-barrier model as a sensitivity check and adds server computation to complete modeled-time comparisons.Server computation is the same for assignments with the same pooled batch size.Source: C. Calibrated Client-Side Timing Model · p. 3

D. Optimization Problem

The optimization fixes each step's integer class target and global batch size, then chooses which eligible clients supply its examples. The integer target stream is rounded from pooled-class proportions without exceeding remaining support.

D. Optimization Problem

  • Flow LB-GPSL optimizes the assignment for the current target and remaining support, with the first constraint fixing the class target and global batch size.The decision is for one step under the current calibrated delay model.Source: D. Optimization Problem · p. 3

IV. LATENCY BUDGETED PARALLEL SPLIT LEARNING WITH GLOBAL SAMPLING

  • LB-GPSL constructs a pooled-batch class target first, then assigns its examples to eligible clients; Flow minimizes modeled client-side time and Fast uses earliest predicted next completion.Both methods use the same target stream.Source: §IV · p. 3

A. Integer Target Stream

  • The randomized target construction permutes classes and applies systematic rounding, returning each coordinate as the floor or ceiling of its fractional target.The target does not request more examples of a class than remain in pooled support, and fixing the permutation and offset fixes the stream independently of assignment.Source: A. Integer Target Stream · p. 4

B. Flow Assignment

The assignment methods preserve each integer class target while selecting client providers according to modeled completion time. Flow uses network flow, Fast uses next-completion choices, and a planning rule accounts for schedule-construction cost when choosing between them.

B. Flow Assignment

  • At a fixed budget, Flow tests target feasibility with a source–class–client–sink network and finds the smallest feasible completion-time budget.Integral capacities yield integral assignments; ties favor clients below their proportional epoch contribution.Source: B. Flow Assignment · p. 4

C. Fast Assignment

  • Fast assigns each requested class to an eligible client with the earliest predicted next completion, preserving the target and epoch sample set without flow or binary search.Ties favor clients that have contributed less relative to their pooled share, then lower client indices.Source: C. Fast Assignment · p. 4

D. Planning-Based Algorithm Selection

  • The planning rule selects Flow only when its predicted schedule cost is at least 1% lower; otherwise it selects Fast, accounting for schedule-construction costs.Both candidate planning schedules are constructed first; the schedule-only accounting also includes the unused schedule’s one-time construction cost over 100- and 200-epoch windows.Source: D. Planning-Based Algorithm Selection · p. 4

E. Offline Epoch Construction

Epoch schedules are constructed offline from class counts and targets, then sent to clients, which draw unused examples. The analysis establishes target-stream invariance and integral flow feasibility for assignments.

E. Offline Epoch Construction

  • At each epoch’s start, the server redraws schedule randomization choices, constructs the schedule before client communication, and sends it to clients to draw unused examples from requested classes.The schedule fixes class counts while sample identities remain random.Source: E. Offline Epoch Construction · p. 4

A. Target and Feasibility Guarantees

  • Any two feasible realization policies produce the same complete target stream when initial pooled counts and target-construction randomness are fixed.Each feasible realization removes the same target histogram from pooled counts, making later pooled states independent of provider identity.Source: A. Target and Feasibility Guarantees · p. 5
  • For a fixed budget, the flow network has a feasible assignment exactly when its maximum-flow value is Bt, and feasible flows yield integral assignments.The analysis also states that randomized systematic rounding produces integer target histograms of mass Bt with the stated expectation.Source: A. Target and Feasibility Guarantees · pp. 4–5

B. Current-Step Assignment Guarantees

Flow minimizes modeled client-side time for a fixed current state and target, while Fast is optimal under a sufficient common-support condition. The analysis also bounds assignment-related gradient shifts and establishes epoch sample conservation.

B. Current-Step Assignment Guarantees

  • For a fixed remaining state and target, Flow achieves the minimum modeled client-side time among feasible assignments.Its guarantee is stepwise: later support changes make the current-step and full-epoch optimization problems distinct.Source: B. Current-Step Assignment Guarantees · p. 5
  • Fast is current-step optimal when requested classes share the same eligible clients, no class-specific support binds, and client completion times are nondecreasing.Many providers per class alone do not suffice; the requested classes must be available on the same clients.Source: B. Current-Step Assignment Guarantees · p. 5

C. Mean Gradients and Epoch Conservation

  • When client-class expected gradients equal the class reference means for every class, all feasible target realizations have the same expected mean gradient.The bound concerns per-example gradients at one model state; batch normalization and optimizer history can also affect actual updates.Source: C. Mean Gradients and Epoch Conservation · p. 5
  • Every complete schedule uses each local example exactly once per epoch, including in the final partial batch.Assignments may change example order and which clients supply targets, but not the epoch sample set.Source: C. Mean Gradients and Epoch Conservation · pp. 5–6
  • Even with the same epoch sample set, different assignments can produce different learning trajectories and numbers of updates to reach a target accuracy.A lower modeled cost per update therefore does not necessarily mean an earlier accuracy crossing; the paper reports time to target alongside final accuracy and modeled time.Source: C. Mean Gradients and Epoch Conservation · p. 6

D. Scheduling Cost

The scheduling-cost analysis contrasts Flow’s candidate-generation and flow work with Fast’s avoidance of the flow solve. Experiments compare learning quality and modeled time against GPSL and use controls to isolate assignment and planning-rule effects.

D. Scheduling Cost

  • Flow generates completion-time candidates and uses a general-flow network whose vertex and arc counts depend on the class and client counts.For M ≤ 10, the implementation tests the 2^M − 1 nonempty subset conditions; larger class spaces use general maximum flow.Source: D. Scheduling Cost · p. 6
  • Fast LB-GPSL avoids the flow solve, while its stated scheduling bound includes maintaining class–client counts.The supplied complexity expression is incomplete, so no asymptotic bound is reported here.Source: D. Scheduling Cost · p. 6

VI. EXPERIMENTAL SETUP

  • Experiments compare learning quality and modeled time against GPSL, with a fixed-target control and schedule-only comparisons across class supports and delay profiles..Source: §VI · p. 6

A. Data and Training Protocol

The experiments compare latency-aware assignment methods with GPSL across non-IID client data and varied class support, using matched runs and calibrated timing profiles. Evaluation accounts for schedule construction and reports both training-time and learning outcomes.

A. Data and Training Protocol

  • 50 clients receive non-IID class distributions; studies vary provider support across CIFAR-10, CIFAR-100, and Tiny ImageNet, with the latter tested at 20 or 100 candidate classes per client.CIFAR-10 uses 45,000 training and 5,000 validation images; CIFAR-100 uses 45,000 training and 5,000 validation images, and Tiny ImageNet uses 90,000 training and 10,000 validation images.Source: A. Data and Training Protocol · p. 6

B. Compared Methods and Assignment Control

  • GPSL is the latency-unaware baseline; matched runs share data partition, initial model, and prescribed randomness, while the target-matched control fixes class targets and epoch sample set.Vanilla PSL and sequential split learning provide contextual comparisons under their native update schedules.Source: B. Compared Methods and Assignment… · p. 6

C. Timing Model and Cost Accounting

  • Timing profiles are calibrated from client computation measurements, and modeled client-side time sums forward computation, activation upload, gradient download, and backward computation.Learning curves include measured schedule-construction time; larger-class time-to-target comparisons also include H200 server computation.Source: C. Timing Model and Cost Accounting · pp. 6–7

D. Schedule-Selection Study

  • The schedule-selection study compares both algorithms across 81 settings and five seeds per setting, then evaluates predicted choices under realized delays with construction overhead included.The 81 settings yield 405 comparisons; the one-percent selection margin was fixed on separate development schedules.Source: D. Schedule-Selection Study · p. 7

E. Evaluation Metrics

  • Time to target is the first recorded validation point meeting a prespecified threshold, while algorithm-selection regret measures excess cost over the lower realized schedule cost, including overhead.Run-level quantities use means and sample standard deviations, with paired differences summarized using descriptive 95% t-intervals.Source: E. Evaluation Metrics · p. 7

VII. RESULTS

Latency-aware assignment reduces modeled training time, but the preferred algorithm and time-to-target gains vary by dataset, threshold, and schedule-construction cost. The planning rule usually stays close to the lower realized schedule cost, while contextual methods reach different accuracy ranges.

A. Flow and Fast Favor Different Operating Regimes

  • 6.75% lower modeled training time on CIFAR-10 (Flow); on TIN-C20, Fast lowers it 16.87%, while CIFAR-100 Fast lowers it 13.29% with final accuracy 0.59 percentage points lower.TIN-C100 complete-training results are reported in Supplement Sec. IV.Source: A. Flow and Fast Favor Different… · p. 7, Table I · p. 8

B. Time-to-Target Gains Vary by Threshold and Class Scale

  • 9.38% earlier at CIFAR-10's 85% target; Fast is 20.48–23.06% earlier across TIN-C20 targets, but 6.55% later at CIFAR-100's 70% target.On CIFAR-100, Fast reaches targets through 65% accuracy 7.93–17.99% earlier; the figure reports target-crossing reductions, with positive values indicating earlier crossings.Source: B. Time-to-Target Gains Vary by… · p. 7, Figure 4 · p. 8

C. Assignment Savings Must Offset Construction Cost

  • With targets fixed, total modeled epoch time falls 63.91% (Flow) and 77.13% (Fast) on CIFAR-100, versus 1.22% and 21.94% on TIN-C20.Schedule construction accounts for 38.97% and 32.43% of Flow's epoch time, versus 2.48% and 1.80% for Fast, in those respective settings.Source: C. Assignment Savings Must Offset… · p. 7

D. Planning Usually Selects the Lower-Cost Algorithm

  • The planning rule stays within 2% of the lower realized cost in 96.54% of comparisons and selects the lower-cost algorithm in 91.85%.Over 100 epochs, mean regret is 0.596%; it falls to 0.412% over 200 epochs as the unselected schedule's one-time construction cost contributes less.Source: D. Planning Usually Selects the… · p. 7

E. Contextual Methods Reach Different Accuracy Ranges

  • GPSL and Fast reach all displayed validation targets, while vanilla PSL reaches only lower thresholds and sequential split learning reaches fewer targets and takes substantially longer on CIFAR-100.A shorter complete run can reflect lower final accuracy rather than faster learning; Fig. 3 plots validation-accuracy curves against time including schedule construction.Source: E. Contextual Methods Reach Different… · p. 7, Figure 3 · p. 8

VIII. DISCUSSION AND LIMITATIONS

The discussion bounds the assignment and evaluation claims, while the conclusion reports dataset-specific time and accuracy tradeoffs alongside overhead-aware variant selection.

VIII. DISCUSSION AND LIMITATIONS

  • In tested larger-class settings, Fast's lower construction cost outweighs its extra modeled client-side time, and the rule chooses a variant before training.The comparison is made for a fixed class target.Source: §VIII · p. 8
  • Reassignment can change gradients and learning trajectories even when class targets match, and mean final accuracy was lower on CIFAR-10 and CIFAR-100.The authors therefore assess learning quality alongside modeled time.Source: §VIII · p. 8
  • Flow's exactness guarantee is one-step and applies to the current target and remaining support, while full-epoch cost and the physical two-barrier step are evaluated separately.Across evaluated settings, reconstructing the barriers changes epoch times by less than 0.012% and leaves method ordering unchanged.Source: §VIII · p. 8
  • The protocol builds a fresh randomized schedule each epoch; reusing a fixed schedule would reduce construction time, but its learning behavior remains untested.Source: §VIII · p. 8
  • Planning errors grow when realized delays differ from the planning profile; reported times use measured construction plus calibrated delays rather than end-to-end distributed measurements.Source: §VIII · p. 9
  • Evaluation covers three image datasets, two ResNet architectures, fixed partitions, and 50 clients; deployment requires class counts and delay profiles.Privacy-preserving estimation and broader model/network validation remain open.Source: §VIII · p. 9

IX. CONCLUSION

  • LB-GPSL preserves the integer target stream, global batch size, and complete epoch sample set while separating class composition from provider choice.Flow minimizes modeled client-side time for each current target; the planning rule compares variants with construction overhead included.Source: §IX · p. 9
  • On CIFAR-10, Flow LB-GPSL reduces modeled training time by 6.75% with a 0.30 percentage-point decrease in final accuracy; on TIN-C20, Fast reduces modeled time by 16.87% and reaches all four validation targets earlier.The planning rule usually selects the lower-cost algorithm, with its largest errors under strong unmodeled jitter.Source: §IX · p. 9, Table III · p. 9

Related papers

Loading 2609.32132v1…