PaperlayerView exact revision on arXiv

Source-linked AI summary

ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork

Tim Beringer, Patrick Diem, Felix Wolf, Arya Mazaheri

arXiv:2608.23840v1cs.DCcs.AI

TL;DR

Large-scale sharded and decentralized training has a difficult configuration space shaped by coupled compute, communication, memory, and hardware factors. ShardMeter models these interactions analytically to predict runtime and guide deployment choices. In a case study, it reduced expenses by 21% while retaining 99% of maximum throughput.

  • Problem

    Choosing efficient configurations for sharded and geographically distributed training is difficult because workloads scale differently across models, datasets, and hardware, making trial-and-error costly.

  • Method

    ShardMeter combines regression models for compute and communication with execution-graph dependencies to predict runtime and optimize feasible deployments from model and hardware specifications.

  • Results

    21% lower expenses while retaining 99% of maximum throughput was achieved in the decentralized-training case study.

  • Takeaways & Limitations

    ShardMeter identifies compute-communication bottlenecks and supports selecting deployments that balance wall-clock time and GPU-hour cost.

  • Takeaways & Limitations

    The current scope excludes dynamic factors such as GPU throttling, power capping, node failures, and I/O, and does not yet cover tensor parallelism.

Abstract

from arXiv · show

Training large-scale AI models often outgrows a single data center, demanding sharded, multi-cluster, and decentralized training. However, the huge space of resource allocations makes exhaustive benchmarking and manual tuning impractical, while performance depends on tightly coupled factors like model size, GPU memory, batch size, bandwidth, and sharding strategy. We introduce ShardMeter, a lightweight analytical performance model that predicts the end-to-end runtime of transformer-based workloads across arbitrary sharded, distributed, and even decentralized training. Given a model's characteristics and a target hardware topology, ShardMeter estimates per-GPU and per-island throughput, training cost, total wall-clock time, and identifies performance bottlenecks. Our analysis reveals diminishing-return regimes as island size increases, quantifies transitions between compute- and communication-bound scaling, evaluates hyperparameter trade-offs, and models cost-throughput for large-scale decentralized training. ShardMeter exposes these insights to quickly explore the configuration space, choose near-optimal deployment plans, and avoid costly trial and error.

I. INTRODUCTION

Large-scale transformer training spans heterogeneous, sharded, and geographically distributed systems, making configuration difficult to predict and tune. ShardMeter addresses this gap with an analytical model, bottleneck analysis, and cost-aware deployment optimization.

  • Motivation: Distributed training configurations couple model size, GPU memory, batch size, hardware, network fabric, and parallelization strategy.These factors determine whether workloads become compute-bound or communication-bound.
  • Research gap: Existing performance models generally cannot represent heterogeneous sharded or decentralized transformer workloads and their coupled communication-computation dependencies.Many also require profiling a specific model before making predictions.
  • ShardMeter: ShardMeter predicts end-to-end training time, identifies bottlenecks, and suggests near-cost-optimal deployment schedules across centralized and geographically distributed workloads.It scales analysis from single-node locally sharded setups to large geographically distributed training.
  • Contributions: Median prediction error is <15% for a lightweight model capturing intertwined compute-communication dynamics in FSDP and decentralized training.The model is intended to support runtime prediction without profiling new models on specific hardware.
  • Contributions: Training expenses can fall by up to 21% while preserving near-optimal throughput through heterogeneous deployment exploration and Pareto-aware optimization.The optimizer exhaustively explores feasible heterogeneous GPU deployments.

II. BACKGROUND

Modern training uses multiple parallelization strategies to overcome model-size and hardware constraints. This section focuses on data parallelism, especially FSDP’s memory savings and its added per-step communication, illustrated through execution timelines and dependencies.

  • Deep-learning training fundamentals: Training repeatedly performs forward and backward passes on mini-batches, while parallelism improves throughput and accommodates large models.The paper examines data parallelism in detail among data, pipeline, context, tensor, and expert parallelism.
  • Distributed data parallelism: DDP replicates the full model on every GPU and synchronizes gradients with an all-reduce after backward computation.Its applicability is limited when model parameters and associated training state exceed single-GPU memory.
  • Execution structure: Figure 1 depicts overlapped communication and computation through a timeline and a dependency graph of all-gather, reduce-scatter, forward, and backward operations.FSDP units execute their own layer-group communication and computation to reduce memory usage.
  • Fully-sharded data parallelism: FSDP partitions model parameters, optimizer states, and activation buffers across participating ranks when the model no longer fits on one GPU.This reduces per-GPU memory usage.
  • Fully-sharded data parallelism: FSDP gathers parameter shards before forward computation and performs reduce-scatter after backward computation, adding communication inside each training step.These collectives occur within communication islands, typically over fast local networks.

B. Decentralized training

Decentralized training distributes computation across geographically separated islands, combining local sharding with delayed model-update exchange. ShardMeter models these execution structures to predict cost and optimize deployments.

  • Decentralized training: Decentralized training uses geographically dispersed islands that compute gradients locally and exchange model updates only after a predefined interval.Communication remains a significant bottleneck in this setting.
  • DiLoCo: DiLoCo replicates the model within each island while sharding it across GPUs with FSDP, keeping collective operations on fast intra-island networks.After local steps, each island forms an outer model and computes a pseudo-gradient.
  • Streaming DiLoCo: Streaming DiLoCo partitions pseudo-gradients into fragments sent incrementally, interleaving exchanges with computation across training iterations.This spreads communication over time while increasing the number of communication events.
  • ShardMeter approach: ShardMeter uses regression models for logical compute and communication operations, then composes them into execution graphs for FSDP and DiLoCo.The graphs capture higher-level dependencies for end-to-end prediction and configuration optimization.
  • ShardMeter approach: The workflow first collects benchmark data, then builds a performance model that respects execution-graph dependencies.Representative workloads and fine-grained traces support the modeling process.

Benchmark generation:

ShardMeter builds its predictors from benchmarked transformer configurations and profiled communication and computation primitives. It uses compact regression models over static workload features for rapid end-to-end analysis.

  • Benchmark generation: Benchmark generation uses a parameterized catalog of transformer configurations spanning architectural and hyperparameter diversity observed across language models.The benchmark runs on target hardware to populate primitive-latency models.
  • Runtime collection: Profiling measures all_gather, reduce_scatter, and decentralized all_reduce operations, while forward and backward durations are obtained by summing kernels within each logical phase.Phase kernels are identified using naming patterns and execution order.
  • Benchmark generation: Batch size is increased until an out-of-memory error occurs in the benchmark parameter sweep.Table I lists the static model parameters and typical hyperparameter values used for benchmarking.
  • Runtime collection: PyTorch profiler traces record GPU-kernel timestamps, device and stream identifiers, and triggering call stacks for each benchmark training step.Call-stack parsing associates kernels with higher-level forward and backward phases.
  • Regression modeling: Linear regression predicts primitive runtime from static parameters, providing a compact model that can be rapidly evaluated during scheduling.Static parameters are extracted from architecture and remain independent of runtime hardware or network conditions.
  • Communication modeling: Communication modeling represents latency and bandwidth as path-specific coefficient sets and uses the slowest selected interconnect coefficients.The model accounts for the narrowest link limiting ring-based collective throughput.

Computation modeling:

ShardMeter models computation time from static architectural features and embeds those estimates in task graphs that represent sequential execution, concurrency, dependencies, and compute–communication overlap.

  • Computation modeling:: Static model parameters are selected from benchmark-derived data to predict forward and backward computation time without runtime-dependent features.The parameters are extracted from the architecture and are independent of hardware or network conditions.
  • Computation modeling:: Linear regression predicts computation time Tfwd/bwd from selected architectural features.
  • Computation modeling:: Each feature contributes through a tunable coefficient, with additive coefficient–feature products and no interaction terms.The model assumes each parameter independently scales underlying GPU-kernel dimensions approximately linearly.
  • Computation modeling:: Runtime functions from computation and communication models are inserted into task graphs to simulate end-to-end training steps and resource dependencies.
  • Computation modeling:: Task graphs divide execution into sequential streams while allowing concurrent cross-stream tasks when dependencies are satisfied.This structure reflects interleaved compute and communication operations in execution traces.

1) Execution graph for decentralized training:

ShardMeter extends local FSDP execution graphs into decentralized training graphs and searches constrained island configurations to produce near-optimal runtime or cost plans.

  • 1) Execution graph for decentralized training:: DiLoCo adds a second graph level in which each task is a nested FSDP training-island graph with blocking inter-island gradient exchange.
  • 1) Execution graph for decentralized training:: Streaming DiLoCo uses an additional asynchronous communication stream that sends fragments after their gradients are computed and synchronizes them before later use.
  • 1) Execution graph for decentralized training:: ShardMeter enumerates admissible island configurations, estimates maximal memory-fitting micro-batches, and simulates throughput and monetary cost for each island.
  • 1) Execution graph for decentralized training:: Global batch sizes are initially allocated in proportion to island throughput and then iteratively refined while enforcing minimum step sizes and constraints.
  • 1) Execution graph for decentralized training:: The search returns a near-optimal deployment plan that minimizes runtime, minimizes cost, or realizes a user-specified runtime–cost trade-off.

IV. EXPERIMENTAL RESULTS

The experiments validate ShardMeter’s compute and communication components and compare predictions with measured runtimes across transformer and distributed-training workloads.

  • IV. EXPERIMENTAL RESULTS: The evaluation spans individual kernels, single-island and multiple-island strategies, and decentralized-training variants.
  • IV. EXPERIMENTAL RESULTS: Experiments use A100 GPU systems, with single-node tests on four 80 GB GPUs and distributed tests on HPC nodes with InfiniBand networking.
  • IV. EXPERIMENTAL RESULTS: The computation dataset varies transformer hyperparameters and derives execution times from ten profiled training iterations per configuration.
  • IV. EXPERIMENTAL RESULTS: Prediction accuracy is measured using mean absolute percentage error and compared directly against measured runtime on the target hardware.
  • IV. EXPERIMENTAL RESULTS: 14.99% MAPE was achieved for all_gather, 13.1% for reduce_scatter, and 2.14% for all_reduce.

Computation modeling accuracy:

ShardMeter predicts transformer computation and end-to-end FSDP behavior across model sizes and batch regimes, while exposing transitions between communication- and compute-bound execution.

  • Computation modeling accuracy:: 5% to 29% MAPE was observed across forward and backward predictions for transformer models from 360M to 6B parameters.The median error for every operation and target device stayed below 13%.
  • Computation modeling accuracy:: All end-to-end FSDP runtime predictions exhibited error rates below 13% across the tested models and platforms.The reported end-to-end accuracy reflects whether each step is compute-bound, communication-bound, or mixed.
  • Computation modeling accuracy:: FSDP reduces per-GPU memory use by distributing parameters across ranks but adds communication overhead within every training step.
  • Computation modeling accuracy:: Hardware, architecture, hyperparameters, and system configuration jointly influence the transition between compute-bound and communication-bound execution.ShardMeter is designed to detect and report these interdependencies.
  • Computation modeling accuracy:: At small batch sizes, gradient synchronization dominates performance; increasing batch size shifts execution toward compute-bound operation.
  • Computation modeling accuracy:: Figure 3 shows low-batch compute-stream gaps caused by communication waits, which shrink as batch size grows.

Model size impact:

ShardMeter shows that model size, batch size, memory capacity, interconnects, and GPU count jointly determine whether training is compute- or communication-bound. Increasing batch size can improve throughput in communication-bound regimes, but memory limits and compute saturation create diminishing returns.

  • Model size impact:: Larger models can remain compute-bound at relatively small batch sizes, whereas smaller models may remain communication-bound across broader ranges.Dense matrix operations make large transformers computationally intensive, while smaller models perform fewer computations per step.
  • Model size impact:: GPU memory limits maximum per-device batch size and can force additional gradient-accumulation steps to maintain a fixed global batch size.This overhead is especially important for FSDP because each accumulation step requires repeated all_gather operations.
  • Model size impact:: Faster interconnects shift training toward compute-bound behavior at smaller batch sizes, while frequent inter-node communication can dominate FSDP runtime.DiLoCo can reduce this effect when communication is limited to outer optimization steps.
  • Model size impact:: In communication-bound regimes, larger batch sizes can sharply increase throughput with marginal runtime growth, but compute-bound regimes produce proportional runtime increases and diminishing throughput returns.Figure 5 compares these throughput-runtime trade-offs across model sizes, batch sizes, and 8- versus 16-GPU configurations.
  • Model size impact:: ShardMeter can evaluate batch sizes beyond measured VRAM capacity to estimate how larger-memory hardware could change performance, while accounting for FSDP accumulation overhead.The analysis highlights that VRAM constraints can limit throughput even when computational and communication resources remain available.
  • Model size impact:: Increasing batch size reduces compute-stream idle gaps while making communication-related gaps more prominent as the workload shifts regimes.ShardMeter’s dependence graphs expose communication-induced waiting at low batch sizes and reduced compute idle time as batch size grows.

D. Simulating communication bottlenecks in asynchronous decentralized training

ShardMeter simulates asynchronous Streaming DiLoCo execution to expose when outer-step communication stalls training. It evaluates fragment counts and synchronization delays as competing ways to remove bottlenecks while preserving viable synchronization.

  • D. Simulating communication bottlenecks in asynchronous decentralized training: Streaming DiLoCo adds an asynchronous outer-step communication stream whose fragments are launched after gradients are computed and must synchronize before later use.ShardMeter extends the FSDP execution graph to represent these dependencies and user-defined delays.
  • D. Simulating communication bottlenecks in asynchronous decentralized training: In a normal configuration, six training steps with an outer step every two iterations introduce no noticeable slowdown.The simulated trace separates forward/backward computation from all-gather, reduce-scatter, and outer-step communication.
  • D. Simulating communication bottlenecks in asynchronous decentralized training: When communication exceeds the time available before a blocking receive, training halts and overall runtime increases significantly.The bottleneck appears when the outer step progresses more slowly than concurrent inner-step training.
  • D. Simulating communication bottlenecks in asynchronous decentralized training: Doubling the number of fragments removes the communication bottleneck but increases iterations from 6 to 12, potentially affecting loss.Increasing the step size instead provides more buffer before results are needed, but also lengthens full gradient exchange.
  • D. Simulating communication bottlenecks in asynchronous decentralized training: Synchronization that is too infrequent can cause replica drift, so ShardMeter enforces a lower bound that selects viable delays and fragment counts.The selected settings aim to prevent inner-step waiting while minimizing delay until the pseudo-gradient is exchanged.

V. CASE STUDY: MODELING DECENTRALIZED TRAINING

The Llama-8B case study enumerates feasible decentralized deployments across heterogeneous clusters and maps their runtime-cost trade-offs. The resulting Pareto frontier shows no universally optimal deployment and strong diminishing returns at the highest throughputs.

  • V. CASE STUDY: MODELING DECENTRALIZED TRAINING: ShardMeter evaluates up to 10 training islands for Llama-8B with a fixed global batch size of 2 million tokens.The study balances throughput and cost across heterogeneous GPU clusters under hardware and cost constraints.
  • V. CASE STUDY: MODELING DECENTRALIZED TRAINING: Each feasible deployment plan is evaluated for one-epoch wall-clock time and monetary cost in GPU-hours while modeling DiLoCo-FSDP communication overhead.The reported metrics come from the end-to-end performance model rather than isolated microbenchmarks.
  • V. CASE STUDY: MODELING DECENTRALIZED TRAINING: Feasible deployments form a Pareto frontier, so improving runtime requires sacrificing cost or vice versa rather than selecting one universally optimal configuration.The preferred point depends on whether the objective prioritizes throughput, runtime, or cost.
  • V. CASE STUDY: MODELING DECENTRALIZED TRAINING: Throughput gains quickly diminish among the 20 highest-throughput frontier configurations, while their costs vary substantially.This indicates room for cost reduction even when targeting near-maximal throughput.
  • V. CASE STUDY: MODELING DECENTRALIZED TRAINING: Accepting 99% of maximum achievable throughput reduces cost by 21.69% relative to the throughput-optimal deployment.The case study demonstrates a concrete cost-throughput trade-off rather than a single best deployment.

VII. CONCLUSION AND OUTLOOK

ShardMeter predicts transformer training behavior across sharded, distributed, and geographically decentralized regimes without prior profiling, while diagnosing compute-communication bottlenecks and supporting cost-aware deployment choices. Its case study retains 99% of maximum throughput while reducing expenses by 21%, but the authors identify dynamic runtime factors and additional parallelization and environmental costs as future extensions.

  • VII. CONCLUSION AND OUTLOOK: ShardMeter predicts per-GPU and per-island runtime and throughput from model specifications and hardware resources without prior profiling.Its scope includes modern transformer workloads in large-scale distributed and geographically decentralized settings.
  • VII. CONCLUSION AND OUTLOOK: By modeling execution streams and idle periods, ShardMeter identifies bottlenecks and the batch-size regimes where training changes between compute- and communication-bound behavior.It also predicts asynchronous outer-step bottlenecks in Streaming DiLoCo and suggests mitigations.
  • VII. CONCLUSION AND OUTLOOK: A cost-aware deployment spectrum lets practitioners select configurations meeting runtime or cost objectives without exhaustively exploring the design space.In the case study, expenses fell by 21% while retaining 99% of maximum throughput.
  • VII. CONCLUSION AND OUTLOOK: Current planned extensions include modeling GPU throttling, power capping, node failures, I/O, tensor parallelism, energy use, and CO2 emissions.These additions would broaden the runtime, scalability, and deployment-impact factors represented by the model.
Loading 2608.23840v1…