PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

Pengyu He, Yan Zhang, Ruien Li, Guangwen Yang

arXiv:2609.36662v1cs.LGcs.AI

TL;DR

Distributed LLM training faces growing communication constraints, while fixed synchronization intervals may not suit every training stage. AutoLoCo adapts local horizons and corrects outer updates; it reduces synchronization rounds by 27% relative to DiLoCo while achieving a lower mean training loss.

  • Problem

    Fixed local-update intervals do not adapt to changing training stages, although synchronization is a growing constraint in distributed LLM training.

    Source: §1 · pp. 1–2

  • Method

    AutoLoCo selects local intervals online from training statistics and adjusts outer momentum and learning rate according to accumulated inner learning rates.

    Source: §1 · p. 2

  • Results

    27% fewer synchronization rounds than DiLoCo while achieving a lower mean training loss.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AutoLoCo reduces communication frequency while preserving training performance in the reported pretraining and fine-tuning experiments.

    Source: §5 · p. 10

Abstract

from arXiv · show

The pre-training of Large Language Models (LLMs) is increasingly conducted across multiple data centers. As training scales to a larger number of accelerators, the fraction of time spent on computation decreases, while the fraction spent on communication increases. Therefore, frequent synchronization becomes a growing bottleneck. Local update methods reduce this cost by allowing workers to perform several optimizer steps between synchronizations. Most local update methods set the number of local optimizer steps between synchronizations before training and keep this interval fixed throughout the run. However, the best interval can change during the entire train process. If the interval and optimizer are adapted to the current training state, the communication frequency is reduced while maintaining the training performance. In this work, we introduce AutoLoCo, an adaptive training framework to reduce communication in LLM training. It adapts the local interval using scalar training statistics and corrects each outer update. Our method is motivated by two observations: 1) the appropriate local interval varies across training stages, and 2) changing the number of inner steps per interval creates a mismatch with an unchanged outer optimizer, requiring a correction to the outer update. We optimize this mismatch by correction of the outer optimizer for the momentum and the learning rate using the accumulated inner learning rate. Our experiments under communication constraints demonstrate that AutoLoCo reduces communication frequency by 27% relative to DiLoCo while maintaining training performance.

1 INTRODUCTION

Distributed LLM training across loosely connected resources makes communication a central constraint, while fixed local intervals can be inefficient as training progresses. AutoLoCo adapts the interval and corrects the outer optimizer to reduce synchronization while preserving training performance.

1 INTRODUCTION

  • Distributed training across datacenters and other loosely connected resources makes synchronization latency and bandwidth a central constraint.Each synchronization blocks and transfers updates for the entire model.Source: §1 · p. 1
  • A fixed DiLoCo interval can synchronize unnecessarily when short or permit excessive disagreement when long, while changing horizons can mismatch the outer optimizer.The local trajectory, accumulated inner learning rate, pseudo-gradient scale, and represented local optimization change with the interval.Source: §1 · pp. 1–2
  • AutoLoCo reduces synchronization rounds by 27% relative to DiLoCo while achieving a lower mean training loss.Its only additional communication consists of scalar summaries and decision metadata.Source: §1 · p. 2
  • AutoLoCo selects token-aligned local horizons online and adjusts outer momentum and learning rate using accumulated inner learning rates.It evaluates both compared intervals from the same position in the learning-rate schedule.Source: §1 · p. 2

2 RELATED WORK

Prior work reduces communication payload, reduces synchronization frequency, adapts intervals by schedules or online feedback, or adjusts outer optimization. AutoLoCo couples online interval selection with outer optimization adapted to the amount of local work represented by each interval.

2 RELATED WORK

  • Communication-payload methods reduce bytes exchanged through quantization, sparsification, or structured compression, while local-update methods reduce synchronization frequency.The cited foundation-model approaches also include payload reduction or exposed communication-time reduction.Source: §2 · p. 2
  • Prior outer-optimizer and federated-optimization work relates aggregated updates to the local progress they represent, and AutoLoCo uses accumulated inner learning rate to adjust momentum and learning rate.The related work includes methods that adapt server steps, preserve optimizer behavior, or normalize updates by effective local progress.Source: §2 · p. 3
  • Synchronization intervals are adapted either by schedules tied to learning-rate progress or by online feedback such as local-model deviation or worker variance.AutoLoCo extends interval selection while also adapting how the DiLoCo outer optimizer interprets the resulting pseudo-gradients.Source: §2 · p. 3
  • AutoLoCo treats synchronization timing and outer optimization as a coupled design problem, adapting the interval to training state and the outer update to represented local work.This approach is complementary to compression, quantization, and asynchronous communication architectures.Source: §2 · p. 3

3 METHOD

AutoLoCo replaces DiLoCo’s fixed local horizon with a token-aligned adaptive horizon and an outer correction that accounts for the optimization represented by each interval. Its interval-level feedback loop selects local work, updates the shared model with corrected optimizer settings, and uses completed-interval statistics to guide the next horizon.

3 METHOD

  • AutoLoCo preserves DiLoCo’s local-update and Nesterov outer-update loop while adding adaptive horizon selection and outer-optimizer correction.The components determine local steps before synchronization and align the outer optimizer with the represented local optimization.Source: §3 · p. 3

3.1 ADAPTIVE INTERVAL SELECTION

  • The interval controller evaluates keeping, enlarging, or reducing the next token-equivalent horizon using local-drift energy and aggregation coherence against an accepted-horizon reference profile.Token budgets are mapped to executable steps using global non-padding token counts and an estimate of tokens per step; horizons are rounded to admissible granularity.Source: §3.1 · pp. 3–5

3.2 OUTER OPTIMIZER CORRECTION FOR VARIABLE LOCAL HORIZONS

  • The outer correction compares the selected horizon with a base token interval at the same learning-rate schedule position, then uses their accumulated inner learning-rate ratio to normalize the pseudo-gradient and adjust momentum and outer-step scale.The same interval ratio controls normalization, momentum retention, and bounded outer-step scaling.Source: §3.2 · p. 5

3.3 AUTOLOCO: ADAPTIVE GRADIENT SYNCHRONIZATION MEETS OUTER OPTIMIZER CORRECTION

  • At each interval, workers execute the selected local horizon, average pseudo-gradients, apply the corrected Nesterov update, and use completed-interval statistics to update the profile, token mapper, and next horizon.The horizon component sets the local trajectory; the outer correction determines how its pseudo-gradient updates the synchronized model.Source: §3.3 · p. 6

4 EXPERIMENTS

Across C4 pre-training and UltraChat fine-tuning, AutoLoCo is evaluated against DiLoCo under matched budgets and worker configurations. In the main pre-training setup, it uses fewer synchronizations while improving training and validation loss.

4 EXPERIMENTS

  • The experiments use 215M-parameter and Llama-3.1-8B models, with C4 pre-training and UltraChat fine-tuning; the main pre-training setup uses 8 workers and 50,000 steps.The 8B model is full-parameter fine-tuned for one epoch, with 8- and 16-worker settings.Source: §4 · pp. 6–7

4.1 PRETRAIN QUALITY AND COMMUNICATION COST

  • With 73 rather than 100 synchronizations, AutoLoCo improves training loss from 2.8004 to 2.7843 and validation loss from 2.8020 to 2.7842, saving 27% of cumulative logical volume.The training-loss advantage over DiLoCo emerges late in training.Source: Table 1 · p. 6, §4.1 · p. 7, Figure 2 · p. 7

4.2 FINE-TUNING QUALITY AND COMMUNICATION COST

  • With 8 workers, AutoLoCo cuts synchronizations from 65 to 49 and payload by 24.6%, improving test NLL from 0.9112 to 0.8766; with 16 workers, test NLL is 0.8772 versus DiLoCo’s 0.9016 with the same savings.The comparisons use UltraChat fine-tuning for one epoch with a global batch of 32.Source: §4.2 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8
  • With 16 workers, AutoLoCo scores above DiLoCo on all five downstream tasks; with 8 workers, gains include about five points on MMLU-Pro and BBH and 7.32 points on HumanEval+.For 8 workers, IFEval improves by 0.18 points over DiLoCo and remains 4.25 points behind DDP.Source: §4.2 · pp. 8–9, Table 5 · p. 8, Table 6 · p. 8

4.3 ABLATION

  • AutoLoCo reaches training loss 2.8116, versus 2.8251 for DiLoCo and 2.8484 with momentum adjustment alone; adding outer-step adjustment reverses the loss increase.Adaptive intervals alone yield 2.8310; during the H = 750 phase, momentum adjustment makes median outer updates 36.45% smaller than with adaptive intervals alone.Source: §4.3 · p. 9, Table 7 · p. 9
  • Normalization reduces median outer-update magnitude by 47.50% during C4’s H = 750 phase but increases final training loss; in the reported SFT comparison, it slightly lowers test NLL.These observations motivate different normalization settings for the two evaluated training recipes.Source: §4.3 · p. 9, Table 8 · p. 9, Table 9 · p. 9

5 CONCLUSION

AutoLoCo adapts synchronization intervals and corrects the outer optimizer for variable local horizons, reducing communication while preserving training performance. Its current-run reference profile is calibrated from observations at the accepted horizon.

5 CONCLUSION

  • Communication frequency falls 27% in pretraining and 24.6% in fine-tuning relative to DiLoCo while training performance is preserved.The conclusion reports both reductions as outcomes of the pretraining and fine-tuning experiments.Source: §5 · p. 10

A.1 CURRENT RUN REFERENCE

  • The reference profile is calibrated after the 1,000-step inner-learning-rate warm-up, using observations from intervals at the initial horizon.The warm-up spans the first two 500-step intervals; two subsequent finite intervals set the reference statistics to geometric means.Source: §A.1 · p. 13
  • Three additional intervals at the same horizon estimate normal variation, with each residual family’s robust center taken as the median of up to 16 stored residuals.Residuals enter history only after increase-supporting or reference-consistent intervals.Source: §A.1 · p. 14
  • The accumulated-inner-learning-rate reference follows the latest accepted interval because the inner learning-rate schedule changes during training.Source: §A.1 · p. 14

A.2 INTERVAL EVALUATION

The interval controller monitors completed intervals against an accepted-horizon reference, then tests longer candidates or reduces the horizon in response to deviations. Token-aligned horizons are mapped to executable steps and updated using current-run token observations.

A.2 INTERVAL EVALUATION

  • A larger horizon is considered only after the post-reduction hold-off and a reference-consistent interval; moderate deviations must occur twice to reduce the horizon, while severe deviations trigger immediate reduction.A non-finite required statistic follows the reduction path and may invoke the minimum-horizon fallback.Source: §A.2 · p. 14
  • The controller selects the longest admissible candidate whose previewed accumulated inner learning rate is at most 1.15 times the accepted horizon’s exposure.Candidate horizons are mapped to steps before evaluation.Source: §A.2 · p. 14
  • While a rejected-horizon bound remains active, the next candidate is nearest the geometric midpoint between the accepted horizon and that bound; the bound retires when its preview falls to at most 0.90 of its recorded exposure.Learning-rate decay can change how much local optimization the same horizon represents.Source: §A.2 · p. 14
  • A first candidate interval that supports an increase is adopted immediately; a reference-consistent result requires one more interval, while moderate or severe deviation rejects the candidate and restores the prior horizon.After adoption or return, the reference profile is re-estimated before monitoring resumes.Source: §A.2 · p. 14, Figure 3 · p. 15

A.3 HORIZON UPDATES

  • The token mapper calibrates Mbase from the median global non-padding token counts of the first three recipe-horizon intervals and updates its tokens-per-step estimate after each interval with β = 0.9.The newly selected token horizon is mapped only after that update, so the next interval uses the latest observed token density; no historical trace or external fixed-horizon run supplies Mbase.Source: §A.3 · p. 14
  • A confirmed moderate deviation reduces the token-equivalent horizon by 1.20, while a severe deviation reduces it by 1.50; after reduction, the controller recalibrates before proposing another increase.The reduced horizon is selected from admissible multiples of qH; one interval initializes its reference, three estimate variability, and one monitoring interval must finish before another increase proposal.Source: §A.3 · p. 15

B IMPLEMENTATION DETAILS FOR THE OUTER OPTIMIZER CORRECTION

Variable local horizons change the optimization represented by each DiLoCo outer update, so AutoLoCo compares each interval with a scheduler-aligned base interval and corrects the outer update. Its algorithm combines pseudo-gradient normalization, momentum and learning-rate adjustment, and selection of the next horizon.

B.1 WHY VARIABLE HORIZONS CHANGE THE OUTER UPDATE

  • Longer intervals accumulate more inner learning rate, change averaged pseudo-gradient scale, and let incoming momentum persist across more inner optimization, making a fixed outer optimizer mismatched across variable horizons.The shared horizon is constant within an interval but may change between successive intervals.Source: §B.1 · p. 15

B.2 SCHEDULER-ALIGNED INTERVAL COMPARISON

  • The correction compares the selected interval’s accumulated inner learning rate with a base token interval evaluated from the same scheduler position; the resulting interval ratio is 1 when exposure does not exceed the base comparator.A scheduler preview configures the outer correction, while the completed measurement supports the next horizon decision.Source: §B.2 · p. 16

B.3 PSEUDO-GRADIENT NORMALIZATION

  • After workers average pseudo-gradients, AutoLoCo normalizes the outer-optimizer input by the interval ratio; at ρt = 1 normalization is identity, and larger ratios reduce the input proportionally.The implementation adds ϵ to the denominator for numerical protection, and normalization is enabled independently of the momentum and learning-rate adjustments.Source: §B.3 · p. 16

B.4 MOMENTUM RETENTION AND BOUNDED OUTER-STEP SCALING

  • For longer intervals, the momentum correction lowers retention according to the interval ratio and limits the momentum-adjusted outer-step scale to κmax, without rescaling the stored momentum state.The momentum exponent extends repeated base-momentum retention to non-integer interval ratios.Source: §B.4 · pp. 16–17

B.5 THE AUTOLOCO ALGORITHM

  • The outer-interval algorithm sets up the horizon and correction, runs local loops in parallel, synchronizes pseudo-gradients for the corrected outer update, then assesses interval statistics and selects the next horizon.Source: §B.5 · p. 17

C THEORETICAL ANALYSIS OF AUTOLOCO

The analysis relates worker-endpoint disagreement to local drift and aggregation coherence, then derives interval-aware outer-optimizer corrections from accumulated inner learning-rate mass. It also shows that AutoLoCo recovers DiLoCo under the base settings.

C.1 INTERPRETATION OF LOCAL-DRIFT ENERGY AND AGGREGATION COHERENCE

  • Dt and Ct jointly characterize endpoint disagreement, which is nonnegative and no greater than the total drift energy Dt.For fixed worker count and drift energy, larger Ct implies smaller disagreement.Source: §C.1 · p. 19

C.2 LEARNING-RATE MASS AND SCHEDULER-ALIGNED INTERVAL RATIOS

  • The learning-rate-mass ratio approximates the selected interval’s displacement relative to the base prefix when effective updates vary little across the compared steps.The relation is exact when appended effective updates equal the base-prefix mean.Source: §C.2 · pp. 20–21

C.3 MOMENTUM RETENTION AND BOUNDED OUTER-STEP SCALING

  • AutoLoCo adjusts momentum retention and outer-step scale using the interval ratio, with bounded scaling and a correction accounting for momentum’s fixed-input response.The displacement interpretation of learning-rate mass requires effective updates to remain sufficiently close over the compared steps.Source: §C.2 · p. 20, §C.3 · p. 21

D.1 DILOCO PRELIMINARIES

  • DiLoCo’s loop performs local AdamW updates before workers average pseudo-gradients for a Nesterov outer update; AutoLoCo keeps this loop but varies the executable horizon and scalar outer settings.The base DiLoCo interval is fixed, with constant outer learning rate and momentum.Source: §D.1 · p. 22

D.2 DILOCO AS A SPECIAL CASE

  • When the selected horizon and outer optimizer stay at their base settings, AutoLoCo reduces exactly to DiLoCo.This equivalence assumes the mapper operates at the reference token density.Source: §D.2 · p. 22

E.1 TRAINING AND EVALUATION PROTOCOL

The protocol evaluates distributed pre-training and supervised fine-tuning under matched comparison settings. It specifies the data preparation, evaluation measures, and baseline synchronization schedules.

E.1 TRAINING AND EVALUATION PROTOCOL

  • The main pre-training budget covers 52.43B input token positions, while four-worker ablations cover 26.21B; SFT uses one pass over retained UltraChat-200k training data.C4 documents are truncated or padded to 1,024 tokens without packing; SFT comparisons use shared shuffled global batches of 32.Source: §E.1 · p. 23
  • Final synchronized models are evaluated with token-averaged negative log-likelihood on 4,096 C4 validation documents or 1,000 valid SFT test conversations.SFT loss is averaged over assistant targets.Source: §E.1 · p. 23

E.2 BASELINES AND AUTOLOCO CONFIGURATION

  • DiLoCo uses local AdamW and a Nesterov outer optimizer with fixed horizons, while QSR and Linear-Interval provide additional local-update baselines.DiLoCo uses 500 local steps for C4 and 100 for SFT; Linear-Interval schedules 100 synchronizations across its 50,000-step budget.Source: §E.2 · p. 23

E.3 ULTRACHAT FINE-TUNING AND EVALUATION CONFIGURATION

The UltraChat fine-tuning setup compares distributed training configurations and evaluates downstream tasks with task-specific metrics. The ablation varies horizon adaptation and outer-optimizer correction.

E.3 ULTRACHAT FINE-TUNING AND EVALUATION CONFIGURATION

  • The fine-tuning comparison uses Llama-3.1-8B, global batch 32, and 6,490 optimizer steps, including an evaluation with 16 local workers.AutoLoCo includes pseudo-gradient normalization and adaptive momentum and learning-rate correction.Source: §E.3 · p. 24
  • Downstream evaluation reports task-specific accuracy, exact match, or pass@1 on IFEval, IFBench, MMLU-Pro, BBH, and HumanEval+.The passage specifies prompt-level strict or loose accuracy, few-shot settings, and plus pass@1 across 164 HumanEval+ tasks.Source: §E.3 · p. 24

E.4 ABLATION CONFIGURATION

  • The ablation compares adaptive horizons alone, horizons plus momentum adjustment, and full AutoLoCo with bounded outer-step adjustment.These four-worker C4 runs share initialization, data, inner AdamW recipe, and seed; DiLoCo’s outer momentum is set to 0.7.Source: §E.4 · p. 24

F.1 TRAINING TRAJECTORIES AND THRESHOLDS

AutoLoCo reaches the reported training-loss threshold earlier than DiLoCo in the aggregated trajectory comparison. The executed schedule uses changing horizons and includes a final remainder.

F.1 TRAINING TRAJECTORIES AND THRESHOLDS

  • 37,336 steps: AutoLoCo reaches loss 2.85, compared with 38,217 for DiLoCo.The comparison aggregates each threshold crossing using the first crossings reported in Table 2; at 100% budget, AutoLoCo’s loss minus DiLoCo’s is −0.0161.Source: §F.1 · p. 24, Figure 4 · p. 25

F.2 EXECUTED INTERVALS AND OUTER SETTINGS

  • AutoLoCo executes 73 intervals totaling 50,000 steps, growing from a 500-step horizon to 750 steps before a 250-step remainder.Its listed sequence includes horizons from 500 to 750 steps and exercises horizon growth and operation at the upper bound.Source: §F.2 · p. 25

G ADDITIONAL EXPERIMENTS OF LOW-PRECISION COMMUNICATION

The appendix pairs AutoLoCo’s interval-based communication reduction with INT8 pseudo-gradient communication. In the 16-worker UltraChat experiment, the combination reduces logical payload relative to DiLoCo with BF16 communication while preserving test NLL closely to BF16 results.

G ADDITIONAL EXPERIMENTS OF LOW-PRECISION COMMUNICATION

  • Lower-precision communication and less frequent synchronization are presented as complementary ways to reduce communication overhead.The appendix combines AutoLoCo and INT8 in a 16-worker UltraChat experiment.Source: G ADDITIONAL EXPERIMENTS OF… · p. 25

G.1 LOW-PRECISION COMMUNICATION CONFIGURATION

  • Each tensor part is communicated in blocks of up to 4,096 elements using scaled, rounded, clipped INT8 values, with FP32 decoding and aggregation.The codec uses no error feedback; the mean is encoded once for distribution.Source: §G.1 · p. 25
  • INT8 communication applies to pseudo-gradients, while model computation remains in BF16 and master parameters and optimizer states remain in FP32.The controller measures local pseudo-gradients before quantization and computes the aggregate statistic from the decoded aggregate.Source: §G.1 · p. 26

G.2 COMPATIBILITY WITH LOW-PRECISION COMMUNICATION

  • 24.6% lower total logical payload than DiLoCo with INT8 communication accompanies test NLL improving from 0.9016 to 0.8772.Combining AutoLoCo with INT8 reduces total logical payload by 62.3% relative to DiLoCo with BF16 communication; both methods’ INT8 test NLL differs from its BF16 result by less than 0.0001.Source: Table 14 · p. 26, §G.2 · p. 26
Loading 2609.36662v1…