PaperlayerView exact revision on arXiv

Source-linked AI summary

Resource-aware Sub-network Allocation for Heterogeneous Decentralized Learning

Zhuojun Tian, Ming Xiao, Mikael Skoglund

arXiv:2610.22015v1cs.DC

TL;DR

Synchronous decentralized learning can be constrained by resource-limited nodes, and deployment-time assignment of heterogeneous model capacities remains comparatively less explored. RASA-DL profiles subnetworks and allocates node-specific capacities by minimizing bottleneck latency subject to memory and aggregate-utility constraints. At the same modeled latency, it improves accuracy over the largest feasible uniform deployment in Moderate and Severe heterogeneity.

  • Problem

    Resource heterogeneity can bottleneck synchronous decentralized learning, while jointly configuring node-specific model capacities from deployment resources remains comparatively less explored.

    Source: §I · pp. 1–2

  • Method

    RASA-DL profiles sub-network hardware costs and deployment-aware learning utility, then allocates node-specific widths by minimizing bottleneck latency under memory and aggregate-utility constraints.

    Source: §I · p. 2, B. Resource Models · p. 3, §VI · p. 12

  • Results

    2.8017 and 2.4265 percentage points: RASA-DL-A improves accuracy over the latency-matched maximum-feasible uniform deployment in Moderate and Severe regimes, respectively.

    Source: B. Decentralized Learning Performance · p. 9

  • Takeaways & Limitations

    Controlled experiments show favorable accuracy–latency tradeoffs, and the heterogeneous deployments remain effective under Non-IID data distributions.

    Source: §VI · p. 12

    Evaluation uses a controlled simulated decentralized environment, so the reported system metric is modeled bottleneck round latency rather than measured end-to-end wall-clock latency.

    Source: A. Experimental Settings · p. 8

Abstract

from arXiv · show

System heterogeneity in computation, communication, and memory can substantially limit synchronous decentralized learning, where each training round is constrained by resource-limited nodes. Existing approaches provide limited support for jointly determining node-specific model capacities from deployment resources. We propose RASA-DL, a resource-aware sub-network allocation framework that treats the maximum sub-network width assigned to each node as a deployment variable. RASA-DL combines offline hardware profiling of computation and memory costs with deployment-aware profiling of learning utility under heterogeneous decentralized training. Using these profiles together with node- and link-level resource information, we formulate deployment as a bottleneck-latency minimization problem subject to per-node memory constraints and a required aggregate learning utility. Exploiting the monotone feasibility of the latency threshold, RASA-DL efficiently identifies the deployment through a binary-search-based allocation procedure. Experiments under controlled resource heterogeneity demonstrate favorable accuracy--latency tradeoffs. Under the Moderate and Severe regimes, the minimum-latency operating point improves accuracy over the largest globally feasible uniform deployment by 2.80 and 2.43 percentage points, respectively, at the same modeled round latency. The advantage is further amplified under strong Non-IID data heterogeneity.

I. INTRODUCTION

Heterogeneous resources make synchronous decentralized learning vulnerable to stragglers, while prior work leaves deployment-time assignment of node-specific model capacities comparatively less explored. RASA-DL uses nested sub-networks and resource-aware allocation to address this gap.

I. INTRODUCTION

  • Resource heterogeneity can make constrained nodes synchronization bottlenecks and cause memory infeasibility when every device receives the same architecture.Resource-rich devices may also be used inefficiently under a uniform deployment.Source: §I · p. 1
  • Prior decentralized methods improve peer-to-peer optimization or communication efficiency, but do not remove synchronization costs caused by heterogeneous resources.Work on heterogeneous model capacities in federated and decentralized learning exists, while deployment-resource-based capacity configuration in fully decentralized learning remains comparatively less explored.Source: §I · pp. 1–2
  • RASA-DL treats each node’s maximum sub-network capacity as a deployment variable and combines offline profiling with current node- and link-level conditions.Its allocation minimizes synchronous bottleneck latency subject to memory and aggregate learning-utility requirements.Source: §I · p. 2

A. Decentralized Learning

  • Candidate sub-networks are nested, so a node assigned a larger retention ratio contains the parameter blocks of a smaller assigned sub-network.Block-wise aggregation exchanges and aggregates each block only among nodes whose assigned sub-networks contain it.Source: A. Decentralized Learning · pp. 2–3
  • Each node trains an active width no larger than its maximum deployed width, and the synchronous round waits on participating nodes’ completion.The probability ρ controls activation of the maximum deployed sub-network; remaining probability is distributed uniformly among lower-width sub-networks.Source: A. Decentralized Learning · p. 3

B. Resource Models

The resource models estimate node computation and communication costs for each deployment and enforce memory feasibility using the maximum assigned sub-network.

B. Resource Models

  • Each node is characterized by computation capability and available memory, while each link has effective bandwidth and data-size-independent latency.The link latency includes propagation, queueing, and protocol-access delays.Source: B. Resource Models · p. 3
  • Per-step computation latency is profiled for each active sub-network, and expected local training latency accounts for the active-width sampling policy over local steps.The maximum deployed width sets the allowed active widths, while ρ controls their sampling distribution.Source: B. Resource Models · p. 3
  • The model neglects overlap-aware aggregation processing because measurements place it at approximately 1% of modeled round latency across evaluated deployments.Per-round computation latency is therefore approximated without this processing term.Source: B. Resource Models · p. 3
  • The communication model uses the maximum deployed sub-network’s parameter count and metadata to estimate each node’s transmitted data volume.Smaller active widths during local iterations do not change the model exchanged at the end of a round.Source: B. Resource Models · p. 3
  • Peak memory use is modeled from the profiled training-memory requirement of the node’s maximum deployed sub-network and must fit its memory budget.This feasibility check applies to the maximum assigned width, even when narrower sub-networks are active during local training.Source: B. Resource Models · p. 3

C. Resource-Aware sub-network Allocation

The allocation selects node-specific maximum retention ratios before training to balance synchronous bottleneck latency against learning capability under resource constraints.

C. Resource-Aware sub-network Allocation

  • Uniform model sizes can make constrained devices persistent stragglers while resource-rich nodes wait, motivating node-specific retention ratios.The allocation is made before decentralized training.Source: C. Resource-Aware sub-network Allocation · p. 4
  • Candidate-width utilities are profiled offline and used directly without assuming that learning capability increases monotonically with retention ratio.Nested architectures can have different optimization and generalization behavior after training.Source: C. Resource-Aware sub-network Allocation · p. 4
  • The allocation minimizes modeled synchronous round latency subject to per-node memory feasibility and a required weighted aggregate learning utility.The prescribed ρ is a system-level training parameter, not an allocation variable; the decision variables are the nodes’ maximum retention ratios.Source: C. Resource-Aware sub-network Allocation · p. 4
  • The formulation assigns smaller sub-networks to constrained nodes and retains larger ones on resource-rich nodes while targeting the slowest node’s round completion time.Its objective is bottleneck latency rather than average node latency.Source: C. Resource-Aware sub-network Allocation · p. 4

III. OFFLINE PROFILING FRAMEWORK

The offline framework constructs nested subnetworks, profiles their deployment-aware learning utility, and benchmarks hardware costs before node-specific allocation. The profiling is kept separate from deployment and training.

A. Overall Offline Profiling Procedure

  • The offline procedure builds a common sub-network family, measures candidate learning capability in a reference decentralized environment, and benchmarks latency and memory on reference hardware.These profiles are combined later with current node and communication conditions for allocation.Source: A. Overall Offline Profiling Procedure · p. 4
  • Offline profiling avoids repeating architecture construction and capability evaluation when system conditions change and avoids trialing every possible node-specific assignment.The number of such assignments grows combinatorially with participating nodes.Source: A. Overall Offline Profiling Procedure · p. 4

B. Nested Compact-sub-network Construction

  • A shared channel ranking creates nested compact networks whose common parameter alignment supports overlap-aware aggregation across widths.The channel scores are learned from labeled data while parent-network parameters and batch-normalization statistics are fixed; candidate networks retain prefixes of the same stage-wise orderings.Source: B. Nested Compact-sub-network… · pp. 4–5

C. Deployment-Aware Learning-Capability Profiling

  • Standalone validation accuracy may misestimate deployment-time performance because decentralized training couples heterogeneous neighboring models, width sampling, and overlap-aware aggregation.A lightweight reference environment includes these interactions to profile candidate widths under a closer deployment regime.Source: C. Deployment-Aware… · p. 5
  • The profiled utility is a configuration-level capability estimate, not a prediction of an individual node’s final accuracy.It is a scalar per candidate width, remains fixed during allocation and training, and does not require a monotonic relationship between width and utility.Source: C. Deployment-Aware… · p. 5

D. Hardware Resource Profiling

Hardware profiling characterizes candidate sub-networks’ training computation and peak memory, while communication latency can be calculated from their parameter counts and current link conditions. These profiles are reusable during allocation.

D. Hardware Resource Profiling

  • Candidate sub-networks are empirically profiled on reference hardware because parameter counts and FLOPs alone may not reflect wall-clock training cost.Execution cost depends on tensor dimensions, memory-access behavior, and hardware efficiency.Source: D. Hardware Resource Profiling · p. 5
  • Profiling measures complete training-step latency, including forward propagation, backward propagation, and the optimizer update.Measurements follow an initial warm-up period.Source: D. Hardware Resource Profiling · pp. 5–6
  • Peak training memory is measured in the same profiling process, and profiled values are stored for reuse during deployment.The resulting profiling library is combined with current node and link conditions.Source: D. Hardware Resource Profiling · p. 6
  • Communication latency is evaluated directly from each candidate’s parameter count and the current link bandwidth and propagation delay.The communication payload is known once the compact architecture is materialized.Source: D. Hardware Resource Profiling · p. 6

IV. ALGORITHM DEVELOPMENT

The algorithm prunes dominated node configurations, then tests whether each latency threshold can satisfy memory and aggregate-utility constraints. Monotone feasibility makes binary search sufficient to find the minimum feasible threshold.

A. Algorithm Derivation

  • A node configuration is removed when another cannot increase completion time or memory consumption and does not reduce profiled learning utility.The remaining configurations form node-specific Pareto sets.Source: A. Algorithm Derivation · p. 6
  • For a candidate threshold, feasibility requires nonempty feasible configuration sets for every node and sufficient maximum aggregate utility.The feasibility test uses each node’s utility-maximizing configuration under that threshold.Source: A. Algorithm Derivation · pp. 6–7
  • Binary search finds the minimum feasible latency threshold because feasibility is monotone and feasible thresholds form an upper interval.The search operates over an ordered candidate latency set.Source: A. Algorithm Derivation · p. 7
  • The allocation selects the minimum feasible threshold and its corresponding utility-maximizing node configurations, satisfying the utility constraint.The selected allocation achieves the minimum bottleneck latency.Source: A. Algorithm Derivation · p. 7

B. Resource-aware Sub-network Allocation

RASA-DL searches candidate latency thresholds using Pareto sets and configuration-level profiles, without running decentralized training during allocation. Its allocation procedure has complexity O(KP log KP).

B. Resource-aware Sub-network Allocation

  • Allocation uses offline profile values and current system conditions without executing decentralized training, and its online overhead is negligible compared with training.The profiling procedures are completed offline only once.Source: B. Resource-aware Sub-network Allocation · p. 7
  • O(KP log KP) is the allocation procedure’s complexity given node-specific Pareto sets.Constructing and sorting the candidate latency set takes O(KP log KP); feasibility tests scan at most KP configurations and binary search performs O(log KP) tests.Source: B. Resource-aware Sub-network Allocation · p. 7
  • The algorithm takes Pareto sets, node weights, memory budgets, target utility, and a fixed sampling parameter as inputs.It constructs and sorts the candidate latency set before testing feasibility.Source: B. Resource-aware Sub-network Allocation · p. 7

C. Nested Compact Decentralized Learning

Each node keeps its allocated maximum retention ratio throughout training, samples nested active sub-networks for local SGD, and exchanges its maximum deployed sub-network with neighbors. Aggregation uses only overlapping parameters.

C. Nested Compact Decentralized Learning

  • Each node is assigned a maximum retention ratio that remains fixed throughout decentralized training.The corresponding compact sub-network is initialized from the canonical parent model.Source: C. Nested Compact Decentralized Learning · p. 7
  • At each local iteration, a node samples an active retention ratio and updates only the sampled sub-network’s parameters.Parameters outside the active sub-network remain unchanged during that iteration.Source: C. Nested Compact Decentralized Learning · p. 7
  • After local updates, nodes exchange their maximum deployed sub-networks and aggregate only overlapping parameters.The training procedure uses nested candidate sub-networks.Source: C. Nested Compact Decentralized Learning · p. 7
  • The canonical parent model is used only for initialization; subsequent decentralized training operates exclusively on allocated compact sub-networks.The procedure returns the trained compact models.Source: C. Nested Compact Decentralized Learning · p. 8

A. Experimental Settings

The experiments use nested ResNet-18 sub-networks and profile their learning utility and hardware costs before evaluating deployment under controlled resource heterogeneity. System-level evaluation uses a simulated decentralized network, with latency reported as a model rather than physical end-to-end measurement.

A. Experimental Settings

  • Four nested CIFAR-10 ResNet-18 sub-networks retain 1.0, 0.75, 0.5, or 0.25 of the channels, with parameter counts from approximately 11.18 to 0.70 million.They share a common ImageNet-pretrained, CIFAR-10-fine-tuned parent and a fixed structured channel ranking.Source: A. Experimental Settings · p. 8
  • Learning-capability profiles use eight ring-connected virtual nodes, twice assigning each candidate width, and average round-30 validation accuracy across nodes and three seeds.Profiling uses IID CIFAR-10, 45,000 training samples, 5,000 validation samples, 30 rounds, and 100 local steps per round; these utilities are deployment estimates, not final test accuracies.Source: A. Experimental Settings · p. 8
  • On an NVIDIA GH200, measured training-step latency ranges from 15.42 ms at retention 1.0 to 11.42 ms at 0.25, while memory ranges from 1.629 to 0.471 GB.Measurements include forward and backward propagation and optimizer updates, with batch size 64.Source: A. Experimental Settings · p. 8
  • System-level evaluation simulates 20 nodes and 40 links because a sufficiently diverse physical testbed was unavailable; reported bottleneck round latency is modeled, not measured end-to-end wall-clock latency.Node compute costs derive from reference-hardware profiles; memory budgets and link bandwidths and delays are configured independently.Source: A. Experimental Settings · p. 8
  • Mild, Moderate, and Severe regimes vary resource dispersion while holding topology and mean resource levels fixed; the largest feasible uniform width falls from 0.5 in Mild to 0.25 in Moderate and Severe.The overall heterogeneity indices are H = 0.121, 0.388, and 0.471, respectively.Source: A. Experimental Settings · p. 8

B. Decentralized Learning Performance

Under Moderate and Severe resource heterogeneity, RASA-DL-A improves accuracy over the maximum-feasible uniform deployment at the same modeled bottleneck latency, while higher-utility allocations trade additional latency for learning performance. Deployment-aware profiling also selects higher-capacity deployments that reach higher accuracy than standalone profiling under the same utility target.

B. Decentralized Learning Performance

  • In Mild heterogeneity, maximum-feasible uniform deployments and DFedAvgM remain competitive because every node can accommodate width 0.5.RASA-DL offers a broader range of lower-latency and higher-capability choices, but does not uniformly dominate the strongest common-model deployment.Source: B. Decentralized Learning Performance · p. 9
  • Under Moderate and Severe heterogeneity, uniform methods are limited to width 0.25 although many nodes can support larger models; RASA-DL-B and C improve learning performance over A at added latency.The frontier therefore provides multiple non-dominated operating points rather than one fixed deployment.Source: B. Decentralized Learning Performance · p. 9
  • Across all three regimes, RASA-DL-A and C converge consistently to their final operating points, and their final accuracy differences emerge progressively during training.RASA-DL-C reaches higher final accuracy than A with its higher-capacity deployment.Source: B. Decentralized Learning Performance · p. 9
  • RASA-DL is most useful when the globally feasible common model is substantially smaller than models supported by many individual nodes, allowing unused capacity to shape the accuracy–latency tradeoff.This avoids requiring the entire system to operate at the capability of its least-capable participant.Source: B. Decentralized Learning Performance · pp. 9–10
  • 2.8017 and 2.4265 percentage points: RASA-DL-A exceeds the maximum-feasible uniform deployment in Moderate and Severe regimes, respectively, at matched modeled bottleneck round latency.Against DFedAvgM, the corresponding gains are 3.0505 and 2.6753 percentage points.Source: B. Decentralized Learning Performance · p. 9, Table I · p. 10

C. Comparison Between Standalone and Deployment-Aware Profiling

  • At the common 89% utility target, deployment-aware profiling selects higher-capacity deployments than standalone profiling in all three regimes, with higher modeled per-round latency and higher final accuracy.The training procedure is identical, so the reported difference follows from deployment selection rather than a change to the training algorithm.Source: C. Comparison Between Standalone and… · pp. 10–11

D. Generalization to CIFAR-100

On CIFAR-100, RASA-DL-A improves accuracy over uniform deployments at the same modeled latency, while higher-capacity points trade increased latency for higher accuracy. Its heterogeneous deployments also outperform fixed baselines under stronger Non-IID data, and the conclusion identifies time-varying resources and online adaptation as future work.

D. Generalization to CIFAR-100

  • The CIFAR-100 experiment retains the ResNet-18 backbone and Moderate resource configuration, but recomputes the deployment-aware utility profile for the changed task.It uses an IID partition and averages results over three seeds; no operating point is selected using final test accuracy.Source: D. Generalization to CIFAR-100 · p. 11
  • 8.26 and 8.65 percentage-point gains over Fixed Uniform and DFedAvgM, respectively, with RASA-DL-A reaching 58.27% accuracy at the same 3.959 s modeled round latency.The largest globally feasible uniform deployment is the 0.25-width sub-network; RASA-DL-A assigns larger sub-networks to resource-capable nodes.Source: Table II · p. 12, D. Generalization to CIFAR-100 · p. 12
  • RASA-DL-B reaches 63.51% accuracy at 4.393 s, while RASA-DL-C reaches 67.16% at 6.035 s, illustrating higher accuracy at increased modeled system cost.Their deployment compositions are and, respectively.Source: D. Generalization to CIFAR-100 · p. 12

E. Robustness to Statistical Heterogeneity

  • At the same 4.2358 s modeled latency, RASA-DL-A improves accuracy over Fixed Uniform-25 by 4.79, 4.42, 12.48, and 18.68 percentage points for α = 1.0, 0.5, 0.2, and 0.1, respectively.The improvement is positive for every training seed in all four settings, and grows as statistical heterogeneity increases.Source: E. Robustness to Statistical… · p. 12
  • With stronger label skew, retaining larger sub-networks on resource-capable nodes provides a more favorable accuracy–latency tradeoff when the globally feasible minimum-width deployment suffers a large accuracy loss.The utility profile and deployment allocations remain unchanged from the original design during this evaluation.Source: E. Robustness to Statistical… · p. 12

VI. CONCLUSION

  • RASA-DL formulates deployment as bottleneck-latency minimization subject to memory and aggregate-utility constraints, using monotone latency-threshold feasibility to avoid exhaustive search.Source: §VI · p. 12
  • Controlled-resource experiments show favorable accuracy–latency tradeoffs versus uniform deployments and DFedAvgM; future work targets time-varying resources, online capacity adaptation, and joint communication optimization.Source: §VI · p. 12

Related papers

Loading 2610.22015v1…