PaperlayerView exact revision on arXiv

Source-linked AI summary

Execution-Feasible Dispatch of LLM Pre-training Loads for Renewable-Dominant AI Data Centers

Xi Tong, Ershun Du, Haiyang Jiang, Long Chen, Qin Wang

arXiv:2610.13047v1cs.DC

TL;DR

Aggregate or delay-tolerant AIDC models do not resolve checkpoint-safe pauses, discrete resumed power states, or deadline-feasible recovery for LLM pre-training. The paper maps LLM jobs to checkpoint-domain workloads and feasible 3D GPU modes for dispatch; on a 2048-GPU AIDC, this reduces operating cost by 20.02% and curtailment from 1.077 MWh to 0.014 MWh versus fixed continuous training.

  • Problem

    Existing AIDC load abstractions may overestimate LLM pre-training flexibility because they do not capture checkpoint-constrained pauses, discrete resumption modes, and deadline-feasible recovery.

    Source: A. Background and Motivation · p. 1, B. Literature Review · p. 2

  • Method

    The framework maps LLM descriptors to checkpoint-domain workloads and feasible 3D-parallel GPU modes, then embeds execution-feasible states in day-ahead dispatch.

    Source: C. Contributions and Paper Organization · p. 2, §VI · p. 9

  • Results

    Operating cost falls by 20.02% and renewable curtailment falls from 1.077 MWh to 0.014 MWh versus fixed continuous training on a 2048-GPU AIDC.

    Source: §VI · p. 9

  • Takeaways & Limitations

    Checkpoint records alone do not create flexibility; in the studied case, useful flexibility combines checkpoint waiting with feasible 3D-mode compensation, while workload reshaping dominates and BESS performs residual balancing.

    Source: §VI · p. 9

Abstract

from arXiv · show

AI data centers (AIDCs) are emerging as large power-system loads, making their operational flexibility increasingly valuable. Existing studies often treat AIDC demand as continuously adjustable or generically delay-tolerant, which overstating the flexibility of large language model (LLM) pretraining. This paper proposes an execution-feasible dispatch framework for LLM pre-training in renewable-dominant AIDCs. LLM descriptors are first converted into checkpoint-domain daily workloads, then mapped to feasible 3D-parallel GPU modes with explicit throughput, GPU-capacity, memory, and IT-power limits. The resulting checkpoint- and mode-constrained load states are embedded into a day-ahead power dispatch model. Case studies on a 2048 GPUs AIDC with two LLaMA pretraining tasks show that checkpoint-aware 3D dispatch reduces operating cost by 20.02% and renewable curtailment from 1.077 to 0.014 MWh relative to fixed continuous training. Related results also show that checkpoint records alone do not create load flexibility,practical grid-interactive flexibility arises from checkpoint waiting combined with execution-feasible 3D-mode compensation.

I. INTRODUCTION

LLM pre-training flexibility must be modeled through checkpoint-safe pauses and feasible GPU execution modes, rather than generic continuous or delay-tolerant demand. The paper develops an execution-feasible load model and dispatch framework for renewable-dominant AIDCs.

A. Background and Motivation

  • AIDCs could support power-system flexibility, but LLM pre-training demand is constrained by checkpoint states, discrete GPU modes, memory, capacity, throughput, and deadlines.These constraints make it nontrivial to convert computing-side flexibility into load states usable for power dispatch.Source: A. Background and Motivation · p. 1

B. Literature Review

  • Prior work spans data-center energy, green operation, and training systems, but lacks dispatch models resolving safe pauses, resumed power states, and deadline-feasible recovery.Source: B. Literature Review · pp. 1–2

C. Contributions and Paper Organization

A. Framework of Green-Power Direct Supply AIDC System

  • Local wind and photovoltaic generation serve the AIDC, while grid purchases, curtailment, and BESS operation are coordinated with its computing load.The computing-side load includes base IT power and schedulable LLM pre-training power; PUE maps internal IT power to external electric load.Source: A. Framework of Green-Power Direct… · p. 2

B. Flexibility Description of LLM Pre-Training Loads

LLM pre-training flexibility consists of safe waiting at checkpoint states and resumption in discrete, feasible 3D GPU modes. The coordination framework maps task descriptors through workload and resource layers into dispatchable load states.

B. Flexibility Description of LLM Pre-Training Loads

  • LLM pre-training can wait or resume only at checkpoint states, and resumed power is set by feasible, discrete 3D GPU modes rather than a continuous command.Waiting lowers instantaneous training power but does not reduce remaining computation; feasible modes provide the capability to recover delayed workload.Source: B. Flexibility Description of LLM… · p. 2

C. Coordination Framework and Dispatch Problem

  • The coordination framework converts task descriptors into checkpoint progress and feasible GPU modes, then dispatches those load states with renewables, grid purchases, curtailment, and BESS.Dispatch is subject to checkpoint continuity, completion, GPU and AIDC capacity, BESS, and power-balance constraints.Source: C. Coordination Framework and Dispatch… · p. 3

A. LLM Training Workflow and Task Descriptor

  • The modeled workflow schedules iterative pre-training and periodic checkpointing, using task descriptors that include model size, token target, batch size, sequence length, architecture parameters, tranche ratio, and deadline.Data preparation, validation, model export, and post-training are outside the scheduling scope.Source: A. LLM Training Workflow and Task… · p. 3

B. Task-to-Computing Mapping and Checkpoint-Domain Progress

The mapping converts LLM training descriptors into computational workloads and tracks daily-tranche progress through checkpoint segments. Checkpoint waiting pauses training without reducing the computation that must be completed.

B. Task-to-Computing Mapping and Checkpoint-Domain Progress

C. Computing-to-Resource Mapping and 3D-Parallel Execution Mode

The computing-to-resource mapping represents execution through 3D-parallel GPU modes. Each mode links its parallel configuration to GPU occupation, throughput, compute per interval, and IT power, subject to memory feasibility.

C. Computing-to-Resource Mapping and 3D-Parallel Execution Mode

M GPU

The framework screens execution modes for GPU memory, engineering divisibility, and IT-power attributes, then prunes dominated candidates. The retained modes are feasible under adopted hardware and model assumptions, not asserted to be proprietary production configurations.

M GPU

  • The memory screen accounts for mixed-precision model-state memory, activation recomputation, calibrated activation coefficients, and runtime overheads.A mode is memory-feasible only if it passes the stated memory condition.Source: M GPU · pp. 4–5
  • Engineering divisibility conditions ensure valid layer assignment, tensor/head partitioning, and batch splitting.Source: M GPU · p. 5
  • Mode IT power includes equivalent active training power per GPU and communication or mode-dependent overhead.In the case study, communication overhead is absorbed into the equivalent mode power coefficient.Source: M GPU · p. 5

D. Feasible Mode Set and Dominance Pruning

  • The feasible-mode set records each mode's GPU occupation, throughput, interval workload, IT power, and memory requirement; dominated modes are removed.A dominated mode uses no fewer GPUs or less IT power while providing no higher throughput; retained candidates reflect adopted hardware and model assumptions, not proprietary production configurations.Source: D. Feasible Mode Set and Dominance… · p. 5, M GPU · p. 5

IV. CHECKPOINT-AWARE DAY-AHEAD GREEN-POWER DISPATCHING MODEL

The dispatch model represents each training job as a path through time-and-checkpoint states. Waiting is allowed at checkpoints, while training edges use feasible 3D modes and must preserve continuity and completion within the deadline.

IV. CHECKPOINT-AWARE DAY-AHEAD GREEN-POWER DISPATCHING MODEL

  • The proposed mechanism contrasts fixed-throughput continuous training with dispatch that changes state only at checkpoint boundaries and selects feasible 3D modes.Source: A. Mode-Pruned Checkpoint-State Graph · p. 5, Figure 6 · p. 5
  • Each job's graph contains time-and-checkpoint states, with an artificial sink; a state records the checkpoint reached at the beginning of a time state.Source: A. Mode-Pruned Checkpoint-State Graph · p. 5
  • Waiting edges advance one dispatch interval without changing the checkpoint, while training edges execute checkpoint segments using a selected 3D-parallel mode.Incompatible mode transitions can be excluded during edge generation.Source: A. Mode-Pruned Checkpoint-State Graph · p. 5
  • Selected edges form one feasible source-to-sink path, enforcing checkpoint-state continuity and daily-tranche completion while excluding interruption between checkpoints.Training edges must fit within the deadline; the ceiling operation counts a partially used final slot as occupied.Source: A. Mode-Pruned Checkpoint-State Graph · p. 6

B. Edge-to-Power Decoding and Resource Constraints

Selected training edges encode which 3D-parallel modes run and how many checkpoint segments they complete. The active edges determine training IT power and GPU use.

B. Edge-to-Power Decoding and Resource Constraints

P AIDC

The model decodes selected 3D modes into AIDC load and couples training with green-power, grid, and battery constraints. Decoded progress is for reporting, not feasibility.

P AIDC

  • Selected edges determine active 3D modes, training IT power, and total AIDC load after PUE conversion.Source: P AIDC · p. 6
  • GPU and AIDC power capacity constrain the decoded load, while normalized job progress is reported from completed training edges rather than required for feasibility.Path constraints already ensure each job reaches its terminal checkpoint before its deadline.Source: P AIDC · p. 6

C. Green-Power Bus and BESS Operation Constraints

  • Green-power bus constraints coordinate renewable curtailment and grid purchase with AIDC demand; BESS dynamics include power, state-of-charge, and charge/discharge exclusiveness limits.Source: C. Green-Power Bus and BESS Operation… · p. 6
  • Algorithm 2 builds checkpoint-state graphs, adds waiting and feasible training edges, then solves the dispatch model for training paths, load, grid purchase, curtailment, and BESS trajectories.In no-BESS scenarios, charging is fixed at zero and BESS state equations are removed.Source: C. Green-Power Bus and BESS Operation… · pp. 6–7

D. Objective Function and Solution Procedure

The day-ahead model minimizes grid-purchase, renewable-curtailment, and BESS-throughput costs. It is a MILP whose execution-feasible modes and load variables are decoded from checkpoint-state paths.

D. Objective Function and Solution Procedure

V. NUMERICAL CASE STUDY

In a 2048-GPU renewable-direct-supply AIDC case with two LLaMA pre-training tasks, executable checkpoint-aware 3D dispatch substantially reduces cost and curtailment versus fixed training. Scenario comparisons distinguish this flexibility from checkpoint recording alone and idealized mode variation.

A. Basic Data and Case Settings

  • The case uses a 2048-GPU AIDC dedicated to LLM pre-training, CAISO renewable and price profiles, and two representative LLaMA tasks.Task descriptors include architecture, token targets, nominal batch, and GPU-hours; implemented batch, baseline modes, checkpoint segments, and daily tranches are case settings.Source: A. Basic Data and Case Settings · p. 7, Figure 7 · p. 7

1) Validation Scenarios:

  • S1 and S2 are intentionally identical because S2 records checkpoint states without waiting; S3 lowers no-storage cost by 21.99% and eliminates curtailment but omits checkpoint continuity.S3 is an optimistic benchmark, not a physically executable policy.Source: B. Overall Scenario Comparison · p. 8

B. Overall Scenario Comparison

  • 20.02% lower operating cost and renewable curtailment reduced from 1.077 to 0.014 MWh in S4 versus fixed-training S1.S4 enforces checkpoint-state continuity.Source: B. Overall Scenario Comparison · p. 8

C. Source–Load–Storage–Grid Dispatch Comparison

In the studied AIDC, checkpoint-aware dispatch lowers cost and curtailment relative to fixed continuous training. Its load changes reflect feasible checkpoint waits and 3D modes, with storage providing complementary balancing.

C. Source–Load–Storage–Grid Dispatch Comparison

  • 1.077 MWh of renewable curtailment in S1 is eliminated in S6, while grid purchases fall from 8.965 to 8.422 MWh and green-power share rises from 63.20% to 66.17%.S6 reshapes the AIDC load between 455.0 and 1335.2 kW; BESS performs a brief residual charge–discharge adjustment.Source: Figure 8 · p. 8, C. Source–Load–Storage–Grid Dispatch… · p. 8

D. Mechanism Analysis of Checkpoint-Aware 3D Training Dispatch

  • 135 feasible configurations for LLM-A and 111 for LLM-B are pruned to 12 and 21 effective modes, with S6 activating only four and three, respectively.The retained modes provide distinct GPU–throughput–power combinations for recovering delayed training work without arbitrary continuous load scaling.Source: Figure 9 · p. 8, D. Mechanism Analysis of… · pp. 8–9
  • 100% of both profiles’ daily compute completes by hour 24 with zero boundary idle under the execution-feasible S6 trajectory.LLM-B waits at a checkpoint for three hours and resumes at hour 22; LLM-A continues in a 160-GPU low-power mode during the late evening.Source: Figure 10 · p. 8, Figure 11 · p. 9, D. Mechanism Analysis of… · p. 9

E. Sensitivity Analysis

  • 12 checkpoint segments capture most available flexibility, while the final cost improvement is only 1.51%; capacities of 768–1152 GPUs cannot complete both tranches.The supplied sensitivity passage identifies 1280 GPUs as a threshold but does not state the outcome beyond that point.Source: E. Sensitivity Analysis · p. 9

VI. CONCLUSION

  • 20.02% lower operating cost and renewable curtailment reduced from 1.077 MWh to 0.014 MWh with checkpoint-aware 3D dispatch versus fixed continuous training.The numerical studies use a 2048-GPU AIDC with two representative LLaMA-derived profiles. The profiles and generated modes are constructed from public traces and data, rather than proprietary production traces.Source: §VI · p. 9
Loading 2610.13047v1…