PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Effective Training Time for Large-Scale Recommendation Systems

Mingming Ding, Ruilin Chen, Yuzhen Huang, Hang Qi, Menglu Yu, San Tan, Damian Reeves, Boris Sarana, Kevin Tang, Satendra Gera, Gagan Jain, Sahil Shah, Vishwa Karia, Fuzail Khan, Yashasvi Makin, Edward Z. Yang, Oguz Ulgen, Jia Chen Ren, Laith Sakka, Mayank Garg, Meet Vadakkanchery, Aici Lin, Wei Sun, Mengjiao Zhou, Shuai Yang, Junqing Zhou, Max Leung, Apoorv Purwar, Musharaf Sultan, John Bocharov, Zhenyu Tang, Vivek Trehan

arXiv:2610.02057v1cs.IR

TL;DR

Large-scale recommendation jobs lose wall time to lifecycle overhead, and the paper addresses the need to measure and localize those losses. It uses ETT% to guide optimizations across the training stack and evaluates them on paired models and in the fleet. ETT% improves on all six benchmark models, with fleet-wide offline ETT% also rising after deployment.

  • Problem

    Recommendation workloads spend substantial wall time outside productive training, and existing metrics do not both capture lifecycle losses and localize them to an owning system component.

    Source: §I · pp. 1–2

  • Method

    The paper uses ETT% and its two-level decomposition to attribute losses to owned infrastructure components and guide fleet-scale optimizations.

    Source: §I · pp. 1–2

  • Results

    ETT% improved on all six benchmark models from 59–85% to 80–93%, with a mean gain of 15.5 percentage points, while fleet-wide offline ETT% rose from about 80% to above 90%.

    Source: B. End-to-End ETT% Results · p. 8, D. Fleet-Wide Longitudinal Results · p. 9

  • Takeaways & Limitations

    The fleet-wide longitudinal measure shows offline ETT% gains across more than 500 models, including models that received no optimization work during the period.

    Source: D. Fleet-Wide Longitudinal Results · p. 9

Abstract

from arXiv · show

Lifecycle overhead silently consumes accelerator capacity across large-scale recommendation training fleets. Our largest recommendation workloads process tens of billions train- ing examples per day on thousands of GPUs. Before this work, only 50-60% of their end-to-end wall time advanced training on new data. We present a fleet-scale study of this lifecycle overhead and a set of optimizations spanning the full training stack. We use Effective Training Time (ETT%) as an operational framework to instrument lost time, localize it to independently owned infrastructure components, and expose work repeated across job restarts. This analysis guides optimizations like communication elimination and pipeline overlap during trainer initialization; dynamic-shape handling, autotuning pruning, and reusable Py- Torch 2 compilation caches; asynchronous checkpointing; stan- dalone model publishing; and reductions in recovery cost. We evaluate the optimizations on representative models and measure their impacts in our training fleet. ETT% improves on every benchmark, by 15.5% on average, and reaches 85% on our largest workload. Fleet-wide ETT% rose from about 80% to above 90% after deployment.

I. INTRODUCTION

Recommendation jobs spend substantial wall time outside productive training, while existing metrics do not both measure the full lifecycle and localize losses to an owner. The paper introduces ETT% and a fleet-scale optimization process, reporting improved benchmark and fleet results.

I. INTRODUCTION

  • MFU covers training-loop efficiency, while Goodput covers the job lifecycle but lacks the phase and ownership granularity needed in this environment.The paper argues optimization needs both an end-to-end objective and a diagnostic path to an owned component.Source: §I · p. 1
  • ETT% decomposes lifecycle losses into start, recovery, and failure terms, then maps phase-level losses to infrastructure systems and owning teams.The framework is used to guide optimizations across initialization, compilation, checkpointing, publishing, and failure handling.Source: §I · pp. 1–2
  • The study evaluates seven recommendation models on H100 clusters from 8 to 2,000 GPUs and tracks deployed results across hundreds of models.The authors describe the contribution as a fleet-scale process for selecting methods, measuring end-to-end returns, and preserving gains as systems evolve.Source: §I · p. 1
  • ETT% improved on six paired benchmarks from 59–85% to 80–93%, with a mean gain of 15.5 percentage points.Recovery-path reductions accounted for 58% of total time saved.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

  • Recommendation training restarts for new data windows, tuning, infrastructure failures, or preemption, repeating initialization work such as compilation and data warm-up.These workloads differ from LLM pre-training over a largely static corpus.Source: §II · p. 2

A. A Measured Time Breakdown

A measured 128-GPU recommendation job spends about 40% of its wall time outside productive training, and restarts can repeat cold-start phases. Existing metrics either omit lifecycle time or do not localize it to an owned system component.

A. A Measured Time Breakdown

  • 90.0 of 151.5 minutes advanced training on new data in the representative job, while the remaining 40% went to non-training phases.PT2 compilation took 30.0 minutes and trainer initialization took 15.0 minutes.Source: A. A Measured Time Breakdown · p. 2
  • Each restart can repeat cold-start phases, adding time outside the training loop beyond the initial lifecycle overhead.With C = 51 minutes and F = 3, the upper bound is 204 minutes of allocated GPU time, or approximately 3,400 GPU-hours per day on 1,000 GPUs; warm restarts may skip phases.Source: A. A Measured Time Breakdown · p. 2

B. Why Existing Metrics Do Not Localize the Problem

  • MFU measures training steps but excludes lifecycle time, while Goodput spans the job lifetime but does not necessarily identify the infrastructure phase causing a decline.A short recommendation job can spend 34% of wall time in cold start, unlike the 0.2% reported for a three-week LLM run.Source: B. Why Existing Metrics Do Not… · pp. 2–3
  • ETT% retains Goodput’s lifecycle scope and adds phase-level components mapped to infrastructure systems and owning teams.This granularity better fits an environment where compiler, scheduler, storage, and trainer framework teams are separate.Source: B. Why Existing Metrics Do Not… · p. 3

C. Why Recommendation Workloads Differ

Recommendation workloads amplify lifecycle overhead through short jobs, complex dynamic-shape models, and streaming-data warm-up. ETT% measures the fraction of wall time spent consuming new data using idle-time and failure-related components.

C. Why Recommendation Workloads Differ

  • Short jobs make a roughly fixed cold-start cost a larger share of runtime: 51 minutes is 34% of a 151.5-minute recommendation job.The same cold start is 0.2% of a three-week LLM run.Source: C. Why Recommendation Workloads Differ · p. 3
  • Heterogeneous recommendation architectures create large dynamic-shape computation graphs, and cold-cache PyTorch 2 compilation takes more than one hour on the largest models.Dynamic shapes can trigger recompilation.Source: C. Why Recommendation Workloads Differ · p. 3
  • Streaming recommendation data requires schema validation, partition negotiation, and first-batch buffering, adding 3 to 5 minutes of warm-up.This warm-up is absent from static-dataset workloads.Source: C. Why Recommendation Workloads Differ · p. 3

A. Definition

  • ETT% is the fraction of end-to-end wall time spent consuming new training data, formulated through idle time and failure-induced overhead.Its idle-time factors are Time to Start, Number of Failures, and Time to Recover.Source: A. Definition · p. 3

B. L1 Hierarchical Decomposition

ETT% separates per-attempt startup and recovery overhead from interruption frequency, then decomposes losses into components with clear ownership. Initialization changes reduce repeated work, including replacing metadata collectives with local construction from an available sharding plan.

B. L1 Hierarchical Decomposition

  • TTS measures cold-start idle time before the first new-data step, while TTR measures per-failure idle time until productive training resumes; NoF captures interruption frequency.Cumulative recovery cost is NoF × TTR.Source: B. L1 Hierarchical Decomposition · p. 3

C. L2 Hierarchical Decomposition

  • The Level-2 decomposition divides Level-1 metrics into six components, each with ownership, measurable baselines, and targeted improvement strategies.The paper describes the job overhead as 40% of wall time across five overhead areas.Source: C. L2 Hierarchical Decomposition · pp. 3–4, Table II · p. 4

A. Trainer Initialization Optimizations

  • Trainer initialization comprises seven sequential sub-stages and takes 15.0 minutes on the referenced job; the reported comparison uses TTS because the techniques move work off the critical path.On Model D, baseline TTS is 31.8 minutes.Source: A. Trainer Initialization Optimizations · p. 4

1) Communication Optimizations:

  • Replacing per-table all_gather collectives with local metadata construction from the broadcast sharding plan reduced Model D TTS from 31.8 to 30.2 minutes, or 5%.The sharding phase fell from 110 to 24 seconds on 128 GPUs.Source: 1) Communication Optimizations · p. 4

2) Pipeline Optimizations:

Initialization optimizations remove unnecessary sequential work by overlapping independent stages and reducing compilation cost. Together, the techniques cut TTS by 41%, and their benefits also recur when a restart repeats startup phases.

2) Pipeline Optimizations:

  • A synthetic fast-batch lets DPP warm-up and PT2 compilation run alongside independent model creation, reducing TTS from 30.2 to 25.4 minutes.The critical path becomes the longest of the three branches rather than their sum; PT2 compilation is the longest branch.Source: 2) Pipeline Optimizations · p. 4

3) Parallel Component Initialization:

  • Starting trainer components alongside DPP warm-up reduces TTS from 25.4 to 23.1 minutes.These remaining initialization sub-stages are independent and had previously run one after another.Source: 3) Parallel Component Initialization · p. 4

4) Combined Effect:

  • TTS falls from 31.8 to 23.1 minutes with the three initialization techniques, then to 18.6 minutes after compilation optimizations, a total reduction of 41%.The three techniques together reduce TTS by 27%; the per-technique values come from an earlier report that enabled them one at a time.Source: 4) Combined Effect · p. 4
  • Because a restart repeats TTS phases, reducing startup time also reduces Time to Recover; the restart effect is measured across restarts.The techniques’ benefits are not assumed to be independent.Source: 4) Combined Effect · p. 4

B. PyTorch 2.0 Compilation

PT2 compilation can consume substantial time before productive training and can recur during recovery. The section reduces compilation work through dynamic-shape handling, autotuning pruning, and reusable artifacts, with controlled cold- and warm-cache reductions and a stated scope caveat.

Avoiding unnecessary graph recompilation

  • Dynamic-shape annotations let one graph cover varying input dimensions, while avoiding selected module traces reduces unnecessary graph breaks and guard overhead.Without an explicit dynamic-shape contract, TorchDynamo can specialize for an observed shape and compile another graph when the shape changes.Source: B. PyTorch 2.0 Compilation · p. 5

Reducing autotuning search

  • Pruning Triton autotuning search removes repeated configuration searches from the critical path while retaining runtime performance.The team profiled kernels consuming the most autotuning time and committed configurations that preserved performance as static defaults.Source: B. PyTorch 2.0 Compilation · p. 5

Making compilation artifacts reusable

  • Stabilized symbolic hashes and Mega-Cache make compilation artifacts reusable across attempts and jobs, potentially removing most compilation work from recovery.The archive packages compilation state for Dynamo, AOTAutograd, Inductor, Triton, and autotuning.Source: B. PyTorch 2.0 Compilation · p. 5

Compilation results

  • Cold-cache compilation falls from 2,050 to 254 seconds (87.6%), while warm-path compilation falls from 1,089 to 114 seconds (89.5%).Table III is a cumulative cold-cache ablation on a representative model using 32 NVIDIA H100 GPUs; each row includes the changes above it.Source: Table III · p. 5, B. PyTorch 2.0 Compilation · p. 5
  • The ablation uses a scaled-down model and controlled cache state; compilation also varies with model changes, GPU count, and cache availability.The experiment attributes reductions, while separate measurements establish the realized operating point.Source: B. PyTorch 2.0 Compilation · p. 5

C. Checkpoint Optimization

Checkpointing trades cumulative save-blocking overhead against training progress lost after failure, so interval tuning balances both costs. Asynchronous staging and other save-time reductions shorten interruptions, with checkpoint save blocking time reduced fleet-wide by 77%.

Tuning Checkpoint Frequency

  • A checkpoint interval balances unsaved training time after failure against cumulative blocking overhead from more frequent saves.Given a known failure rate, the interval can be tuned to minimize wasted training time; the reported relationship has a minimum before wasted time rises again.Source: C. Checkpoint Optimization · p. 6, Figure 3 · p. 6

Reducing Checkpoint Save Blocking Time

  • Asynchronous checkpointing copies checkpoint data into CPU memory so training resumes while a background process completes the upload.This decouples checkpoint I/O from the training critical path.Source: C. Checkpoint Optimization · p. 6
  • PyTorch-native staging uses a bulk device-to-host copy into pinned buffers to remove upload back-pressure from the critical path, at the cost of moderately increased trainer host memory.The earlier fixed-size shared-memory staging path stalled the trainer whenever its buffer filled.Source: C. Checkpoint Optimization · p. 6
  • Checkpoint save blocking time falls by 77% fleet-wide, allowing training to return to the productive loop sooner.Figure 4 presents the reduction in checkpoint save blocking time in seconds.Source: Figure 4 · p. 6, C. Checkpoint Optimization · p. 6

D. Shutdown and Model Publishing

Standalone publishing separates GPU training from CPU snapshot generation, while scheduling changes target preemptions that add recovery cycles. These approaches address GPU-held shutdown time and lifecycle overhead from job placement and interruption.

D. Shutdown and Model Publishing

  • Standalone publishing releases GPUs after an anchor checkpoint and moves snapshot generation to a separate CPU job, reducing GPU-held shutdown time by approximately 30 minutes per job.For models using the approach, shutdown time fell from more than 25 minutes per job to 3.3 minutes or less.Source: D. Shutdown and Model Publishing · p. 6, Figure 5 · p. 7

E. Scheduling Optimization

  • Under fleet congestion, preemption kills running jobs and forces a restart from the last checkpoint, adding re-queuing, re-initialization, and data reprocessing without productive training.The added recovery overhead enters ETT as the NoF x TTR term.Source: E. Scheduling Optimization · p. 7
  • Stage-aware preemption can favor jobs that have just checkpointed, while tenant wait-time knobs trade queue latency for fewer preemptions and higher completion rates.Crash-loop detection also prevents repeated allocate-preempt-requeue cycles, though attributing these scheduling changes to ETT% is difficult.Source: E. Scheduling Optimization · p. 7

V. EVALUATION

Across six paired benchmark models, ETT% improved after optimization, with different baseline efficiency and bottlenecks across models. The evaluation used controlled pairs and phase-level measurements, while its single-run design and lack of isolated technique ablations limit what can be attributed to individual optimizations.

A. Experimental Setup

  • The seven-model set targets the fleet's largest GPU consumers, not all workloads, and spans more than two orders of magnitude in GPU count on shared H100 clusters.Configurations range from 8 to 2,000 GPUs; shared-cluster queueing delay and load were not controlled.Source: A. Experimental Setup · p. 7
  • Paired jobs held model code, GPU count, checkpoint interval, and two induced failures fixed; phase-boundary timestamps count only steps advancing the new-data cursor as productive.There was one baseline and one optimized job per model pair, with jobs running 10–12 hours.Source: A. Experimental Setup · p. 7
  • The evaluation reports single measurements without confidence intervals and lacks leave-one-out ablations, so individual technique effects were not isolated.The authors describe the paired design as attributing measured differences to the optimizations, but Section V-C reports a bound on trainer-start contribution rather than an isolated measurement of each technique.Source: A. Experimental Setup · pp. 7–8

B. End-to-End ETT% Results

  • ETT% improved on all six benchmark models, from baseline values of 59–85% to optimized values of 80–93%, with a mean gain of 15.5 percentage points.Figure 6 panels use separate vertical scales, so heights should be compared within—not across—panels.Source: Figure 6 · p. 8, B. End-to-End ETT% Results · p. 8
  • Model B began at 85.2% ETT%, while Model F began at 59.2%; per-component measurements identify where work paid off, unlike the scalar alone.The largest bottleneck differs between the models.Source: B. End-to-End ETT% Results · p. 8
  • Baseline lost time was 15–41% of wall time; recovery accounted for 43–74% of lost time, while trainer start accounted for 8–19%.Recovery is multiplied by the number of failures, whereas TTS is paid once per job.Source: B. End-to-End ETT% Results · p. 8
  • Recovery contributed 58% of total savings, and TTR fell by 1.8–8.6 times the corresponding TTS reduction.Recovery repeats hardware setup, trainer initialization, and compilation, and restores a checkpoint and warm compilation cache.Source: B. End-to-End ETT% Results · p. 8
  • Where standalone publishing was active, shutdown fell from 25.6–54.6 minutes per job to 3.3 minutes or less; Models E and F were unchanged.Standalone publishing was not enabled for Models E and F.Source: B. End-to-End ETT% Results · p. 8

C. Interaction and Cascade Analysis

The analysis shows that overlapping startup phases alter how optimization gains are attributed, while restart effects compound reductions across repeated jobs. Recovery-path savings include inherited startup work, so direct attribution understates the trainer-start contribution.

C. Interaction and Cascade Analysis

  • The four cumulative reductions sum to 13.2 minutes, or 41% of TTS, but depend on enablement order and do not establish that techniques are independent.Source: C. Interaction and Cascade Analysis · p. 8
  • With NoF = 2, trainer-start work accounted for 24.8% of recovered minutes across six models, versus 8% by direct attribution—a factor-of-three difference.Recovery repeats hardware setup, trainer initialization, and compilation; the inherited TTS contribution is bounded by NoF multiplied by the TTS reduction.Source: C. Interaction and Cascade Analysis · p. 8
  • The remaining 41.3% of recovered minutes belonged to the recovery path alone; its warm compilation cache helps explain why TTR reductions were 1.8–8.6 times the corresponding TTS reductions.Source: C. Interaction and Cascade Analysis · p. 8
  • Fast-batch overlaps preprocessing warm-up with compilation, saving the 3–5-minute warm-up while compilation exceeds five minutes and capping the reduction once compilation is shorter.Only the compilation time exposed on the critical path contributes to TTS reduction, explaining why the cumulative compilation contribution was 4.5 minutes despite a larger standalone compilation reduction.Source: C. Interaction and Cascade Analysis · pp. 8–9
  • Across techniques, reductions are additive when they remove distinct critical-path segments; across restarts, each reduction carries a factor of (1+NoF), so the effects compound rather than multiply across techniques.Source: C. Interaction and Cascade Analysis · p. 9

D. Fleet-Wide Longitudinal Results

Over six months, fleet-wide ETT% rose as teams used component-level attribution to identify and address regressions. The paper reports released GPU capacity alongside the gains, while noting that migration and changing fleet composition limit fixed-cohort comparisons.

D. Fleet-Wide Longitudinal Results

  • Fleet-wide ETT% rose from about 80% to above 90% over six months across more than 500 models; models receiving no optimization work and models already at high ETT% made the gain smaller than the six-benchmark mean.Source: D. Fleet-Wide Longitudinal Results · p. 9
  • The dashboard traced a Level-1 NoF loss to preemption growth and queue wait, rejecting an application-failure hypothesis; normalized attempts per job stayed flat and the application failure rate fell.Teams acted on the attributed components, and ETT% recovered over time.Source: D. Fleet-Wide Longitudinal Results · p. 9
  • The fleet gain returns GPUs to other workloads sooner, although the paper does not disclose the absolute fleet-compute figure.Source: D. Fleet-Wide Longitudinal Results · p. 9
  • The longitudinal trend measures the fleet as operated, not a fixed cohort, because platform migrations destabilized some ETT% readings and fleet composition changed during the window.Source: D. Fleet-Wide Longitudinal Results · p. 9

VI. CONCLUSION

  • The conclusion describes ETT% as decomposed into components mapped to infrastructure systems and owners, alongside optimizations in initialization, PT2 compilation, checkpointing, shutdown, and scheduling.Source: §VI · p. 9

Related papers

Loading 2610.02057v1…