LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models

Ahmad Faiz; Sotaro Kaneda; Ruhan Wang; Rita Osi; Prateek Sharma; Fan Chen; Lei Jiang

Source-linked AI summary

LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models

Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, Lei Jiang

arXiv:2309.14393v2Sep 25, 2023cs.CLcs.AIcs.CYcs.LG

TL;DR

LLM carbon footprints span training, inference, experimentation, and storage, but existing prediction tools do not support the needed LLM architectures and carbon components. LLMCarbon provides end-to-end projections for dense and MoE LLMs, and its predictions differ from Google’s published footprints by ≤8.2% while outperforming mlco2.

Problem
Existing carbon-footprint prediction lacks comprehensive support for dense and MoE LLMs, critical architectural parameters, and embodied emissions across LLM lifecycle phases.
Method
LLMCarbon models operational and embodied carbon for dense and MoE LLMs across training, inference, experimentation, and storage using LLM, hardware, and data-center parameters.
Results
LLMCarbon’s predictions differ from Google’s published LLM carbon footprints by ≤8.2% and are more accurate than mlco2.
Takeaways & Limitations
LLMCarbon enables quantitative exploration of carbon-footprint and test-loss trade-offs and comparisons among LLM configurations.

Abstract

from arXiv · show

The carbon footprint associated with large language models (LLMs) is a significant concern, encompassing emissions from their training, inference, experimentation, and storage processes, including operational and embodied carbon emissions. An essential aspect is accurately estimating the carbon impact of emerging LLMs even before their training, which heavily relies on GPU usage. Existing studies have reported the carbon footprint of LLM training, but only one tool, mlco2, can predict the carbon footprint of new neural networks prior to physical training. However, mlco2 has several serious limitations. It cannot extend its estimation to dense or mixture-of-experts (MoE) LLMs, disregards critical architectural parameters, focuses solely on GPUs, and cannot model embodied carbon footprints. Addressing these gaps, we introduce \textit{\carb}, an end-to-end carbon footprint projection model designed for both dense and MoE LLMs. Compared to mlco2, \carb~significantly enhances the accuracy of carbon footprint estimations for various LLMs. The source code is released at \url{https://github.com/SotaroKaneda/MLCarbon}.

1 INTRODUCTION

LLM adoption has intensified machine-learning emissions, creating a need to project LLM carbon footprints before resource-intensive training. LLMCarbon addresses gaps in prior tools by modeling dense and MoE LLMs across lifecycle phases and including operational and embodied emissions.

LLM adoption has increased environmental impacts, with one 213-million-parameter transformer likened to the lifetime emissions of five cars.
Carbon projections should incorporate parameter count, hardware configuration, data-center efficiency, and both operational and embodied emissions before training.
Prior work mainly reports training emissions, while mlco2 is the only cited predictive tool but cannot adequately cover LLM carbon footprints.
LLMCarbon predicts carbon footprints for dense and MoE LLMs during training, inference, experimentation, and storage using LLM, hardware, and data-center parameters.
Against Google’s published footprints, LLMCarbon’s predictions differ by ≤8.2% and are more accurate than mlco2.

2 BACKGROUND

LLM carbon footprint includes operational emissions from hardware energy use and embodied emissions from hardware manufacturing. The background also covers scaling laws, carbon-reduction strategies, and parallelism used to process large models efficiently.

LLM carbon footprint comprises operational emissions from hardware energy consumption and embodied emissions from hardware manufacturing.
The Neural Scaling Law relates test loss to model parameters, training-data scale, and computational resources across architectures and downstream tasks.
Carbon-reduction efforts include sparse MoE architectures and specialized hardware such as TPUs, alongside data-center optimization.
LLM processing commonly uses data, tensor, pipeline, and expert parallelism, with hardware efficiency measured as actual throughput relative to peak throughput.
Data parallelism replicates the full model across devices, partitions input data, and periodically aggregates gradients to synchronize model weights.
Pipeline parallelism distributes an LLM’s layers across multiple devices, with each device handling an equal number of layers.

3 RELATED WORK

Prior work has largely measured and reported carbon footprints from actual ML-model training. The cited comparison identifies mlco2 as the only previous model capable of predicting carbon footprint before training.

Prior studies predominantly measure and report carbon footprints associated with the actual training phase of machine-learning models.
The comparison categorizes those prior training-footprint studies as “others.”
mlco2 is identified as the only previous model with capability to predict carbon footprint before physical training.

4 LLMCARBON

LLMCarbon predicts an LLM’s carbon footprint from architectural, data-center, and hardware specifications through linked parameter, operational, and embodied-carbon models. Its parameter modeling incorporates architectural attributes, while scaling-law modeling supports comparisons across models and datasets.

LLMCarbon takes an LLM’s architecture, data-center specification, and hardware configuration as inputs for carbon-footprint prediction.
The framework’s figures cover its overview, dense and MoE parallelism settings, computing-device counts, and hardware efficiency.
The framework combines operational and embodied carbon footprints to compute the LLM’s overall carbon footprint.
The parameter model uses hidden size, layer count, vocabulary size, and expert count, or accepts parameter count directly to reduce projection errors.
For dense and MoE models, the parameter-count formulations distinguish dense parameters from expert-related contributions using architectural quantities and the expert fraction.
LLMCarbon adopts the Chinchilla scaling law to relate test loss to parameter count and training-dataset size using fitted constants and irreducible loss.

4.4 FLOP MODEL

The FLOP model estimates dense and MoE LLM processing costs from parameter and token counts, then incorporates parallelism and device-count effects through hardware efficiency.

Dense LLM training and inference FLOPs are estimated from parameter count and processed-token count, while MoE processing uses the dense base model’s parameter count.
Hardware efficiency is actual throughput divided by peak throughput and depends on data, tensor, pipeline, and expert parallelism plus device count.
Optimal parallelism is represented as (p, t, d, e), denoting pipeline, tensor, data, and expert parallelism degrees.
Tensor parallelism is increased within interconnected-device limits before pipeline parallelism is scaled for larger models, subject to a bound on t·p.
Execution time is derived from FLOP count, device count, peak throughput, and efficiency, while hardware energy sums peak power, efficiency, device count, and execution time across hardware units.
Efficiency decreases when the device count differs from t·p·d, and Equation 6 models this effect using fitting constants and the highest-efficiency device count.

4.6 OPERATIONAL CARBON MODEL

The operational carbon model converts hardware energy into operational energy using data-center PUE and then into emissions using carbon intensity, while the broader model also accounts for embodied hardware carbon.

Hardware energy is aggregated across CPUs, LLM devices, memories, SSDs, and other units using power, efficiency, device count, and execution time.
PUE scales computing-hardware energy into operational energy by accounting for data-center auxiliary consumption such as cooling.
Operational carbon footprint equals operational energy multiplied by the data center’s carbon intensity, which decreases as carbon-free energy use increases.
Embodied chip carbon is calculated from chip area and carbon emitted per unit area, with values specified for hardware units and fabrication sources.
The total LLM carbon footprint combines operational and embodied carbon footprints.

5 VALIDATION

LLMCarbon is validated across training, inference, and storage carbon-footprint predictions, with close agreement to reported operational and energy measurements. Its training predictions are substantially closer to actual data than mlco2’s.

Training Phase: LLMCarbon’s training operational-footprint projections differ from actual data by ≤8.2%, whereas mlco2’s disparities exceed 69%.The larger mlco2 error is attributed to assuming peak computing throughput and peak power for all devices.
Inference Phase: LLMCarbon’s GPT-3 inference prediction differs from the measured operational carbon footprint by no more than +3.3%.The evaluated batch used 16 A100 GPUs, batch size 32, and 128 input tokens; predicted latency was 3.1 seconds versus 3 seconds actual.
Storage Phase: LLMCarbon’s six-month Noor storage-energy estimates have an error margin below 3.6%.Predicted storage-data energy was 1.596 MWh versus 1.69 MWh actual, and data-transfer energy was 1.77 MWh versus 1.8 MWh actual.

6 CASE STUDIES USING LLMCARBON

LLMCarbon case studies examine embodied emissions, parallelism choices, accelerator selection, and training-footprint scaling. They show that embodied carbon can be substantial and that hardware and parallelism choices materially affect operational emissions.

Large Embodied Carbon Footprint: Embodied carbon contributes approximately 24%–35% of total LLM life-cycle carbon footprints, rising to 92%–95% in a Meta data center powered 97% by renewable energy.The case study includes training, inference, experimentation, and storage emissions in the life-cycle total.
Optimal Parallelism Setting: Using optimal data, tensor, pipeline, and expert parallelism settings decreases operational carbon footprints by 16%–39%.The validated LLMs used suboptimal parallelism settings, motivating the configuration search.
New Accelerators: Relative to V100 GPUs, H100 and TPUv4 reduce operational carbon footprints by 71% and 41%, respectively, while embodied footprints remain similar.The similar embodied footprints are attributed to SSD and DRAM dominating total embodied emissions.
Training Carbon Footprint Scaling: LLMCarbon is used to explore training-carbon scaling across validated models and additional dense and MoE LLMs.The analysis includes models such as PaLM, Gopher, Chinchilla, LaMDA, Jurassic-1, MT-NLG, Bloom, YaLM, GLM, GLaM, and FB-MoE.

7 CONCLUSION

LLMCarbon is an end-to-end model for estimating operational and embodied carbon across dense and MoE LLM life cycles. It supports design-space exploration by relating carbon footprint to test loss and comparing configurations.

Conclusion: LLMCarbon models operational and embodied carbon for dense and MoE LLMs across training, inference, experimentation, and storage.The model is presented as an end-to-end carbon-footprint modeling tool.
Conclusion: LLMCarbon enables design-space exploration through carbon-footprint and test-loss trade-offs and quantitative comparisons among LLM configurations.The conclusion frames these comparisons as supporting carbon-footprint reduction measures.

A MORE ON THE LLM PARAMETER MODEL

The parameter model derives parameter counts for GPT-3-like dense, encoder-decoder, decoder-only, and MoE LLM architectures. It expresses counts using architectural quantities such as hidden size, layer count, vocabulary size, and expert count.

GPT3-like Dense LLMs: A GPT-3-like dense LLM has 12h^2l + Vh parameters under conventional n_head d_head = h and d_ff = 4h assumptions.The formula combines per-layer attention and feed-forward parameters with token-embedding parameters.
Dense LLM Variants: The parameter-model appendix covers dense architecture variants, including encoder-decoder and decoder-only LLMs.The encoder-decoder discussion separates encoder and decoder contributions, while LaMDA-like models use decoder-only layers.
MoE LLMs: The MoE architectural details used for validation and exploration include GShard, Switch, GLaM, FB-MoE, ST-MoE, and PR-MoE.These models are listed as the selected MoE LLMs in the architectural-details table.
MoE LLMs: For MoE LLMs, expert parameters are P_exp = 2hd_ffN_e and self-attention parameters are P_att = 4hN_head d_head.The formulation accommodates cases where n_head d_head ≠ h and d_ff ≠ 4h.

B PARAMETER MODEL VALIDATION

The paper validates parameter-count calculations for dense and mixture-of-experts LLMs using architectural specifications and model-specific equations. It identifies substantial prediction disparities for several MoE models and attributes them to positional-encoding usage.

Dense LLMs: Dense LLM parameter counts are calculated from architectural parameters using Equation 1, while Noor and YaLM cannot be evaluated because their original papers lack architectural specifications.Noor was used to validate training operational energy, and YaLM was used in LLM scaling, but their parameter counts cannot be determined with LLMCarbon from the available specifications.
MoE LLMs: MoE parameter counts use Equation 2 for GLaM and FB-MoE and Equation 16 for Gshard, Switch, ST-MoE, and PR-MoE.PR-MoE includes MoE layers with either 64 or 128 experts.
MoE LLMs: Gshard, GLaM, and PR-MoE show the largest disparities between predicted and actual parameter counts, attributed to positional-encoding usage.