ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training

• Distributed LLM training is limited by optimizer-state memory, gradient-synchronization overhead, and slow or heterogeneous interconnects.Sharding reduces memory pressure but can increase synchronization and expose straggler effects.Source: §1 · p. 1
• Local-update methods reduce communication frequency but require extra hyperparameters and substantially more memory, preventing efficient optimizer sharding.CO2, for example, requires memory overhead equivalent to four model copies.Source: §1 · pp. 1–2
• Delayed parameter updates overlap GPU computation with CPU optimization but introduce one-step staleness that harms convergence and diverges from standard LLM training dynamics.This limitation is especially relevant for sharded training, where materializing full optimizer states defeats memory savings.Source: §1 · p. 2
• ACCO overlaps gradient computation and parameter updates in sharded optimizers, removes outer loops and warm-up, and compensates for delayed updates without new hyperparameters.The method is designed to preserve AdamW DDP training dynamics while retaining memory efficiency.Source: §1 · p. 2
• ACCO achieves standard SGD convergence guarantees and empirically delivers substantial wall-clock speedups across pretraining and fine-tuning, including heterogeneous environments.The reported results emphasize effective communication hiding without compromising stability or memory efficiency.Source: §1 · p. 2

Local optimization methods for reducing communications

• Local-update algorithms reduce communication and can adapt computation to heterogeneous hardware, but their local states prevent optimizer sharding and introduce extra control variables.Different local update counts can also make models progress at different speeds.Source: §2 · pp. 2–3

Overlapping communications and computations

• Asynchronous overlap methods study delayed communication or computation, but generally omit memory-efficient sharding and adaptive optimizers such as Adam or AdamW.Much of the delay literature focuses on plain SGD rather than optimizers commonly used for LLM training.Source: §2 · p. 3
• DPU enables memory-efficient CPU optimization but introduces one-step delay, remains communication-bound, and may require sensitive warm-up tuning for convergence.The related-work discussion contrasts these limitations with ACCO’s removal of warm-up requirements.Source: §2 · p. 3

Memory-efficient distributed training of LLMs

• Pipeline and model-parallel approaches can leave workers idle or increase memory overhead, while delayed gradients and accumulation mitigate idleness with additional trade-offs.These techniques are often combined into three-dimensional parallelism.Source: §2 · pp. 3–4
• ZeRO-DP and Fully-Sharded Data Parallelism reduce replicated model-state memory by sharding optimizer states, gradients, and parameters, at the cost of increased synchronization and communication.CPU offloading can reduce memory further but requires expensive CPU-GPU transfers.Source: §2 · p. 4

3.1 Background and Notations

• In DDP, gradient averaging and optimization are strictly sequential, creating a communication barrier that limits communication–computation overlap.Optimizer state synchronization can be especially costly when states are sharded.Source: §3.1 · p. 4
• DPU overlaps computation and communication by accumulating gradients until communication finishes, but applies optimizer updates to gradients computed on parameters delayed by one step.This delayed update is inherent to the parallel execution and can substantially affect convergence.Source: §3.1 · pp. 4–5

3.2 ACCO: a structured approach to Communication-Computation overlap.

• ACCO splits each mini-batch into two successive stages so communication and gradient computation overlap without delayed gradients.The first half estimates the next communication-side parameters, while the full mini-batch computes the next computation-side parameters.Source: §3.2 · p. 5
• The ACCO analysis uses a nonnegative Lyapunov function combining objective gaps for both parameter copies with their squared distance.This function is used to establish the method’s convergence behavior.Source: §3.2 · p. 5
• ACCO’s two parameter streams are organized so both stochastic gradients are computed synchronously with respect to the current parameters, avoiding gradient staleness.The method uses separate computation and communication streams while preserving the same parameter version for the relevant calculations.Source: §3.2 · pp. 5–6
• ACCO recovers standard GD and SGD convergence guarantees, and its training curves are nearly indistinguishable from DDP rather than showing the delayed dynamics observed for DPU or WP.The stated guarantees apply under the supplied smoothness, step-size, unbiasedness, independence, and bounded-variance conditions.Source: §3.2 · pp. 5–6

4.1 Empirical motivation for ACCO

• Communication can exceed computation by more than 4×, leaving GPUs idle for most of the time beyond 24 workers.The comparison uses Llama-2 7B gradient computation and All-Reduce timing on A100 GPUs.Source: §4.1 · pp. 6–7

4.2 Experimental setup

• ACCO uses parallel computation and communication streams with AdamW, mixed precision, and sharded optimizer states against DDP with ZeRO-1.The implementation was profiled to verify concurrent streams, and local methods were excluded because of memory requirements.Source: §4.2 · p. 7

4.3 ACCO on TinyStories

• Delayed updates substantially hurt convergence, whereas ACCO matches the DDP training curve and avoids the warmup sensitivity observed for DPU.The tested DPU warmup settings included 0, 40, and 500 steps; the 40-step setting did not recover the baseline behavior.Source: §4.3 · pp. 7–8
• ACCO matches DDP throughout TinyStories training, while the WP compensation method performs worse, especially early in training.The comparison motivated dismissing WP for the remaining experiments.Source: §4.3 · p. 8

4.4 Training GPT-Neo on OpenWebText

• 25% speedup was achieved by ACCO over DDP with ZeRO on GPT-Neo pre-training, while both methods produced similar perplexity on LAMBADA and OpenWebText.The experiments covered 125M-parameter GPT-Neo models on OpenWebText across H100 and A100 clusters.Source: §4.4 · p. 8

4.5 ACCO for instruction fine-tuning

• 87% speedup was achieved for instruction fine-tuning, with ACCO reaching low validation loss faster than DDP on both single-node and multi-node configurations.The advantage became more pronounced when workers were distributed across multiple nodes.Source: §4.5 · p. 9, Figure 8 · p. 9

4.6 Experiment Using Heterogeneous Devices

• With one worker four times slower, ACCO lets other workers accumulate gradients during the wait, producing substantially smaller wall-clock time than DDP.Under DDP, the three faster workers were idle for three quarters of the time.Source: §4.6 · p. 9, Figure 9 · p. 9

• ACCO overlaps gradient computation and communication while partitioning optimizer states to reduce communication overhead memory-efficiently.The method uses a two-stage compensation mechanism for delayed updates.Source: Conclusion · p. 10

• The paper’s ACCO claims are supported by theoretical analysis in Section 3 and experiments in Sections 4.3–4.6.Source: NeurIPS Paper Checklist · p. 16
• Section 4.2 and appendices disclose model, hardware, optimizer, precision, batch-size, dataset, metric, and compute details for reproducibility.Source: NeurIPS Paper Checklist · pp. 17–18
• The paper states that code will be released publicly and that all datasets used are publicly available.Source: NeurIPS Paper Checklist · p. 17
• The checklist materials state that datasets and baselines are cited with sources and licenses where applicable.Source: NeurIPS Paper Checklist · p. 18
• The paper reports no human-subjects or crowdsourcing studies and no new datasets or models are released.Source: NeurIPS Paper Checklist · p. 19

• Removing DPU gradient accumulation halves the batch size by taking an optimizer step at each communication.The study uses the TinyStories configuration and reports this comparison in Figure 10.Source: §A.1 · p. 19
• Two accumulation steps on 8 GPUs double optimizer steps per minibatch while matching the standard batch size.The effective batch size is divided by 2 when using only one accumulation step.Source: Figure 10 · p. 20
• Faster early convergence with two accumulation steps is accompanied by worse final training loss.Source: Figure 10 · p. 20

• ACCO’s gradient-descent convergence is established for L-smooth objectives with a global minimizer and a bounded step size.The proof defines a Lyapunov potential and derives the result by summing its changes over iterations.Source: §A.2 · p. 20
• The proof combines function values and a regularizer into a Lyapunov change, then sums over t = 0 to T − 1 to obtain the bound.Source: §A.2 · p. 22
• The stochastic-gradient result assumes unbiased conditionally independent gradients and bounded variance.Source: §A.2 · p. 22
• For stochastic gradients, the convergence proposition holds for any T > 0 when η ≤ 1/(2L).The derivation applies the full-gradient approach with L-smoothness and additional variance terms.Source: §A.2 · pp. 22–23
• The analysis notes that analogous bounds can be derived for non-increasing step sizes using a Lyapunov potential monotonicity property.Source: §A.2 · p. 23

A.3 Pre-training on OpenWebText

• OpenWebText pre-training uses GPT-Neo 125M with max_position_embeddings changed from 2048 to 1024, and additional H100 results are reported in Figure 11.The dataset and model configuration are sourced from Hugging Face.Source: §A.3 · p. 23, Figure 11 · p. 24

A.4 Instruction Fine-Tuning

• Instruction fine-tuning uses the pre-trained GPT-Neo 2.7B and its tokenizer on the Alpaca dataset, with further results shown in Figure 12.The fine-tuning configuration is provided separately in Table 5.Source: §A.4 · p. 23, Figure 12 · p. 24, Table 5 · p. 26
• The Nsight profile shows ACCO’s computation and communication streams running in parallel on an 8-worker, 36M-parameter TinyStories experiment.At this small scale, computation takes longer than communication; the passage states that this changes for larger models and more workers.Source: Figure 13 · p. 25

B.2 Algorithm Pseudo-Code

• The presented algorithm supports a time-varying batch size N(t).Source: §B.2 · p. 26
• The GPT-Neo 125M configuration and OpenWebText dataset are referenced through Hugging Face resources.Source: §B.2 · p. 26
• The GPT-Neo 2.7B model and Alpaca dataset used for fine-tuning are referenced through Hugging Face resources.Source: §B.2 · p. 26

B.3 Slurm script to reproduce our results

• The reproducibility materials include a listing for the SLURM and ACCO fine-tuning configuration.Source: §B.3 · p. 27
• The SLURM script requests two nodes with one GPU and one task per node.Source: §B.3 · p. 27
• The fine-tuning configuration uses batch size 4, four gradient-accumulation steps, learning rate 1e-5, mixed precision, and the ACCO method.It specifies 50,000 total steps and a cosine scheduler.Source: §B.3 · p. 27
• Algorithm 1 trains each worker with separate computation and communication streams that alternate through readiness-controlled stages.The pseudocode initializes delayed gradients, executes two stages, and advances the training step after updating the parameter buffer.Source: §B.3 · p. 28
• The communication stream executes Stage 1 and Stage 2 with readiness flags controlling when each stage begins.Source: §B.3 · p. 28