PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Facility LLM Pre-training on HPC: Elastic Aggregation, Data Leasing, and Queue-Aware Placement

Zarè Palanciyan, Thomas van Osch, Douwe van der Wal, Olivera Kotevska, Tim Kok

arXiv:2610.03457v1cs.DC

TL;DR

Fragmented academic allocations make it difficult to combine facilities into a single training run. The paper pools facilities with elastic two-loop training, data leasing, and queue-aware placement. Its three-site run reaches held-out perplexity 34.7 against 28.2 for a centralised baseline, while retaining data coverage across departures.

  • Problem

    Academic facilities award fragmented allocations with different hardware and queues, while training frontier-grade models requires more capacity than one public research supercomputer provides.

    Source: §1 · p. 1

  • Method

    The system combines elastic token-weighted Nesterov aggregation, heartbeat-backed leasing of disjoint corpus blocks, and queue-aware planning based on scheduler probes.

    Source: §1 · pp. 1–2

  • Results

    After 20,000 optimizer steps, the three-site run reaches held-out perplexity 34.7 against 28.2 for a centralised baseline, with no blocks lost or duplicated across four site departures.

    Source: §4 · p. 5, §5 · p. 6

  • Takeaways & Limitations

    Cross-site pre-training is operational rather than competitive: it turns fragmented allocations into one training run at a measured cost.

    Source: §5 · p. 6

    The proof of concept covers one small model scale, one epoch, and one site topology, without downstream benchmarking.

    Source: §5 · p. 6

Abstract

from arXiv · show

Academic compute is fragmented: allocations are granted per facility, and facilities differ in accelerators and software stacks, schedule jobs independently, and share neither a network nor a filesystem. We present a system that pools such allocations to pre-train a single language model across three supercomputers on two continents, up to 7,400km apart: Snellius (NVIDIA H100), LUMI and Frontier (both AMD MI250X). It combines (i) DiLoCo-style two-loop training with an elastic, token-weighted Nesterov outer step for which zero, one or many live sites are all normal states; (ii) DARL, a data-leasing protocol whose heartbeat-backed leases guarantee that, within an epoch, no sample is trained twice or lost under crashes, late joins and work stealing; and (iii) queue-aware placement driven by unprivileged sbatch --test-only probes. Training Qwen3-0.6B on C4 for 20,000 optimizer steps, the three-site run reaches a held-out perplexity of 34.7, against 28.2 for a centralised baseline. Per-round overhead (weight exchange and checkpointing) stays near 110 s regardless of the number of local steps H, so its share of wall-clock time falls from 32% at H=100 to 5.7% at H=1,000 and 3.1% at H=2,000. In a 23.8 h three-site run with four site departures, 1.2% of granted data blocks were reclaimed and none was duplicated or lost. In an idealized queue-model projection, queue-aware placement shortens time-to-target by 18-43% compared with waiting for all sites to be allocated. Cross-site pre-training is thus operational rather than competitive: it turns fragmented allocations into one training run at a measured cost.

1 Introduction

Public research allocations are fragmented across facilities with different hardware, queues, and expiry, yet training a single model requires combining them into one run. The paper presents a system addressing training logistics, metascheduling, and infrastructure across facilities.

1 Introduction

  • Fragmented allocations across facilities leave academic groups needing to combine capacity into one run to train models at industry scale.Facilities differ in hardware, queues, and allocation expiry.Source: §1 · p. 1
  • The system must coordinate corpus division and weight exchange without a shared filesystem or network, independent batch jobs, and heterogeneous infrastructure.The paper frames these as training logistics, metascheduling, and infrastructure problems.Source: §1 · p. 1
  • The experiment trains one 0.6B-parameter model on disjoint, non-repeating dataset shards across LUMI, Frontier, and Snellius.The paper also reports a comparison against centralised training.Source: §1 · p. 1
  • Elastic aggregation applies a token-weighted Nesterov outer step and treats zero, one, or multiple live sites as normal states.A solo round takes the full outer step; joiners adopt the current global model and schedule.Source: §1 · p. 2
  • DARL leases disjoint blocks of the pre-tokenised corpus under heartbeat-backed TTL contracts, guaranteeing no within-epoch duplicates or losses during site changes.The stated guarantee covers sites joining and leaving.Source: §1 · p. 2
  • Queue-aware planning uses scheduled sbatch --test-only probes and sacct wall-time statistics to select site membership, job shape, and accumulation.The planner simulates the run round by round.Source: §1 · p. 2

2 Related work

Prior work studies low-communication and federated optimization, geo-distributed training, and queue-aware placement, but the paper distinguishes its approach by treating cross-facility HPC placement as a scheduling problem. It presents a field-tested pre-training setup combining elastic aggregation, data leasing, and distributed job allocation.

2 Related work

  • DiLoCo [6] adapts local-step training to language models with inner AdamW and outer Nesterov updates; OpenDiLoCo [13] and INTELLECT-1 [12] demonstrate geo-distributed runs at increasing scale.The aggregation lineage includes FedAvg and server-side-momentum variants [10] [21] [11].Source: §2 · p. 2
  • Prior cloud and decentralized-GPU work studies cross-site training, but the cited systems do not model HPC batch queues.The passage describes latency, compute-to-communication ratios, and transfer fees as constraints in prior cloud studies.Source: §2 · p. 2
  • Queue-history research predicts waits and runtimes within one facility, while xFFL orchestrates cross-facility runs and FedQueue [14] absorbs admission delays into aggregation.FedQueue buffers late arrivals and down-weights stale updates.Source: §2 · p. 2
  • The paper separates its contribution as a field-tested LLM pre-training setup with elastic aggregation, dynamic data leasing, and queue-aware distributed job allocation.Source: §2 · p. 2

3 System

The system combines two-loop training with elastic aggregation, heartbeat-backed data leasing, and queue-aware scheduling. Its mechanisms address site churn and heterogeneous availability while coordinating training across facilities.

3.1 Two-loop training and elastic aggregation

  • Each site runs H=100 inner AdamW steps from the global model, then uploads its delta and consumed-token count for a token-weighted outer Nesterov step.The outer loop treats a single live contributor as a full step, rejects stale generations, and advances the global step by the maximum steps taken by any contributor.Source: §3.1 · pp. 2–3
  • The elastic update rejects stale-generation deltas so a requeued job trained from an old global model cannot affect the average.The global step advances by the maximum steps any contributor took.Source: §3.1 · p. 3

3.2 Distributed Allocation and Remote Leasing (DARL)

  • DARL keeps the authoritative block-state map at a central coordinator and leases contiguous corpus blocks under heartbeat-backed TTL contracts.Identical pre-tokenised datasets are partitioned into blocks and shuffled with a deterministic global permutation to avoid assigning sites different topical slices.Source: §3.2 · p. 3
  • Expired leases return untouched data to the pool, drained sites can steal an untouched lease tail, and grants scale with measured commit rates.Checkpointing before commit makes weights and data-consumption records fail together, supporting exactly-once coverage.Source: §3.2 · p. 3

3.3 Scheduling

  • Cross-facility placement requires queue-time estimates alongside walltime caps and per-GPU throughput, but production HPC sites often expose no scheduler API or cron access.The paper uses an unprivileged mechanism available at its three Slurm sites.Source: §3.3 · p. 3
  • For each job shape, scheduled sbatch --test-only probes estimate live queue placement without enqueuing jobs, with estimates discounted using realized walltime usage.The discount is derived from a trailing sacct window because Slurm assumes queued jobs use their full requested walltime.Source: §3.3 · p. 3
  • A waterfill planner assigns outstanding data units to soonest-available sites using historical per-GPU rates, walltime limits, and optional reserve time.Plans are recomputed periodically; a late-starting job leases no shards, so misprediction costs calendar time rather than lost or duplicated work.Source: §3.3 · p. 3

4 Results

At matched tokens and steps, centralised training outperformed two-site DiLoCo on perplexity and wall-clock time. The results also show that federated overhead can be amortized across longer local phases, while leasing recovers departed sites’ work without duplication.

Quality and time

  • At matched tokens and steps, centralised training reached perplexity 28.2 in 5 h 41 m, versus 37.5 in 19 h 48 m for two-site DiLoCo.Figure 1 plots training perplexity against training tokens and connected wall-clock time.Source: Figure 1 · p. 4, §4 · p. 4

Distributed overhead

  • Across a 20× range of H, overhead stayed essentially flat while its wall-clock share fell from 32.2% at H=100 to 3.1% at H=2000.Communication varied by under 10 s across the three runs; at H=1000, training took 95% of total round time.Source: Table 1 · p. 4, §4 · p. 4

Transport implementation

  • At H=100, blob/HTTP reduced average round overhead by 14.2 s versus inline/gRPC.The implementation’s gRPC path serialized the model as 310 separate per-tensor byte fields; the blob path streamed tensors in one sequential file.Source: §4 · pp. 4–5, Table 2 · p. 5

Distributed overhead

  • At H=100, calibrated accumulation (aSnellius=5) cut idle time from 75% to 8%, with sites finishing 13 s apart.Without calibration, the fast site finished 127 s early.Source: Figure 2 · p. 5

Data leasing

  • Over 23.8 h, the coordinator ledger reconciled 2,045 committed blocks; Frontier’s nine-block lease was regranted and committed without duplication.Departures cost exactly one in-flight lease, which the TTL returned to the pool.Source: Table 3 · p. 5, §4 · p. 5
  • Snellius delivered 32% and Frontier 173% of an equal third of the corpus, a 5.5× spread that would make a pre-split slow shard unfinishable.With identical loaders, all sites walked the same prefix; unique corpus coverage then equaled the largest site’s share, 1,181 of 2,045 blocks.Source: §4 · pp. 5–6

Scheduler

  • In the 100B-token LUMI–Snellius scheduler evaluation, Snellius processed more tokens than LUMI, reflecting its higher throughput.The allocation comparison covers three queue scenarios and assumes 0% barrier awaits.Source: Table 4 · p. 6, §4 · p. 6

5 Conclusion

The system combines elastic aggregation, DARL, and queue-aware planning for LLM pre-training across three heterogeneous supercomputers. The three-site run reached held-out perplexity 34.7 versus 28.2 for a centralised baseline, while data coverage held through site departures.

5 Conclusion

  • 34.7 held-out perplexity after 20,000 optimiser steps, versus 28.2 for the centralised baseline.The paper characterizes cross-site training as operational rather than competitive.Source: §5 · p. 6
  • Across four site departures, 1.2% of granted blocks were reclaimed, and no block was lost or trained twice.This was measured in a three-site run.Source: §5 · p. 6
  • At H = 1000, measured communication overhead was 6%.Source: §5 · p. 6
  • The proof of concept covers one small model scale, one epoch, and one site topology, without downstream benchmarking.The authors identify production-scale training, more facilities, stronger fault tolerance, and historical queue traces as future work.Source: §5 · p. 6

A HPC sites

The experiments used non-root privileges; Table 5 details the three systems' hardware, software, network connectivity, and batch configurations.

A HPC sites

  • All experiments were run under non-root privileges.Source: A HPC sites · p. 9
  • Table 5 details the hardware, software, network connectivity, and batch configurations of the three systems.Source: Table 5 · p. 9
  • Table 5 is the visual reference for comparing the systems' infrastructure and batch configurations.Source: Table 5 · p. 9

B Training configuration

Table 6 details the training configuration shared by all runs.

B Training configuration

C License

The paper identifies the licences for external assets and states that its released code is Apache-2.0.

C License

  • The paper lists licences for torchtitan, Flower, the OpenEuroLLM tokeniser, and the C4 corpus.Source: C License · pp. 9–10
  • The released code is Apache-2.0 licensed.Source: C License · p. 10
  • The C4 corpus is identified as ODC-BY, subject to the Common Crawl terms of use.Source: C License · p. 10

Related papers

Loading 2610.03457v1…