PaperlayerView exact revision on arXiv

Source-linked AI summary

DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs

Zeyu Cao, Xuan Guo, Cheng Zhang, Cheuk Hang Lau, Ilia Shumailov, Yiren Zhao

arXiv:2608.14614v1cs.LGcs.AIcs.AR

TL;DR

Retired GPUs offer a potentially accessible alternative for modern LLM inference, but evidence on their economic and environmental viability remains limited. This paper builds and operates a second-hand 128-GPU cluster with specialized serving software, finding competitive inference performance under appropriate workload, energy, and market conditions.

  • Problem

    Retired functional GPUs could improve AI infrastructure accessibility and reduce pressures from embodied carbon, electronic waste, high prices, and constrained supply chains, but their productive reuse remains underexplored.

  • Method

    The authors built and operated a 128-GPU V100 cluster from entirely second-hand components for one year and developed a pipeline-parallel LLM serving engine.

  • Results

    Second-hand GPUs achieved throughput competitive with cutting-edge hardware for inference when paired with specialized software optimizations and appropriate deployment conditions.

  • Takeaways & Limitations

    GPU afterlife is viable for expanding AI inference capacity when workload characteristics, energy infrastructure, and market conditions are favorable, especially with clean energy deployment.

  • Takeaways & Limitations

    The demonstrated value is workload-specific: second-hand GPUs are better suited to inference than training because bandwidth and reliability constraints make training more costly.

Abstract

from arXiv · show

As AI datacenters retire functional GPUs, vast quantities of still capable accelerators enter secondary markets. This paper investigates whether these retired GPUs can find a productive afterlife to form a DumpsterCluster that can serve modern LLM inference, and under what conditions such repurposing is economically viable and environmentally sustainable. We physically built a 128-GPU DumpsterCluster from scratch using only second-hand components and ran it for one year. At current market prices (\$22K for the DumpsterCluster vs. \$600K for an 8-GPU B200 system), the economic advantages are substantial. Through pipeline-parallel optimizations, our V100 based DumpsterCluster achieves competitive LLaMA-70B throughput, validating production viability. However, our deployment reveals critical context dependencies. Older GPUs consume significantly more energy per token, making total cost of ownership favorable only in regions with inexpensive electricity. Under grid-average carbon intensity, second-hand systems can produce approximately 4x higher total carbon emissions per token for 8B models, and over 40x for 70B models, compared to current-generation hardware. These findings show that GPU afterlife is not universally sustainable - hardware repurposing must be strategically coupled with low carbon energy sources. When deployed in regions with favourable energy economics and clean electricity, second-hand GPUs offer a viable pathway for expanding AI capacity while advancing affordability, energy security, and environmental responsibility.

1 Introduction

Rapid AI-model growth and accelerated GPU release cycles are causing datacenters to retire functional hardware, raising sustainability, supply-chain, and accessibility concerns. The paper responds by building and operating a 128-GPU DumpsterCluster entirely from second-hand components for one year on real LLM inference workloads.

  • Motivation: Accelerated GPU release cycles create a growing secondary market as datacenters retire still-functional GPUs for newer hardware.Each generation improves performance while requiring priority access to constrained semiconductor manufacturing capacity.
  • Motivation: Continual accelerator manufacturing and premature retirement impose substantial embodied carbon costs and contribute to mounting electronic waste.These practices motivate sustainability concerns alongside the need to expand access to computing infrastructure.
  • Motivation: Concentrated advanced-GPU production has made frontier hardware constrained, expensive, slow to procure, and dependent on fragile supply chains.The passage identifies supply-chain security and accessibility as additional concerns created by current-generation systems.
  • Contribution: 128 GPUs formed the DumpsterCluster, built entirely from second-hand components and deployed for one year serving real LLM inference workloads.The reused components included GPU accelerators, CPU processors, memory modules, and motherboards; each rack-level node used NVIDIA V100 GPUs.

2 Method

The study evaluates second-hand NVIDIA V100 GPUs for LLM serving by analyzing their economic, embodied-carbon, operational, and reliability trade-offs and constructing a 128-GPU DumpsterCluster. It combines hardware binning, redundant capacity, and device-level pipeline-first parallelism to address aging hardware and reduced inter-GPU bandwidth.

  • Economic motivation: $60 batch-market V100 16GB cards were more than 100× cheaper than B200 GPUs, while the used 128-V100 pod was assembled entirely from second-hand hardware.The comparison used an 8x B200 system priced around $600K.
  • Price depreciation: β = 2.43 for V100 and β = 4.66 for A100 indicated accelerating depreciation in fitted stretched-exponential price curves.The curves used second-hand market prices relative to GPU release dates and modeled nonlinear value decay as devices aged.
  • Embodied carbon: Reusing GPUs extends their lifetimes, reducing annualized embodied carbon and potentially reducing demand for manufacturing equivalent new GPU capacity.The paper frames this reduction as a circular-economy benefit that could also help alleviate GPU shortages.
  • Operational trade-offs: Older GPUs may incur higher operational carbon emissions and costs because V100 uses TSMC 12nm and HBM2, whereas H100 uses TSMC N4 and HBM3.The method identifies older silicon technology and lower-bandwidth memory as sources of reduced energy efficiency.
  • Cluster construction: 128 used V100 GPUs formed the DumpsterCluster, with each server node containing two SXM baseboards and four-GPU groups connected by 300GB/s bidirectional NVLink.Each node used an Intel Xeon 6138 processor, 256GB DRAM, four 100Gbps NICs, and a 100Gbps RDMA fabric.
  • Inference parallelism: Device-level pipeline-first parallelism assigns each GPU as an individual pipeline stage to address reduced inter-GPU bandwidth in second-hand systems.This differs from traditional node-level pipelining and is designed for older GPUs that are less memory-bound than newer accelerators.

3 Results

The results show that a 128-V100 DumpsterCluster can provide competitive LLM inference throughput, including for 70B models that cannot fit on a single V100. Its lower acquisition cost can make deployment financially viable, especially with carefully selected energy supplies, despite higher operational costs.

  • Inference throughput: 224K TPS: the 128-V100 cluster achieves this throughput on the 8B Prefill-heavy workload.This demonstrates that scaled-out second-hand hardware can deliver competitive throughput despite individual device limitations.
  • Inference throughput: 1.5K TPS: the 128-V100 cluster achieves this on the 70B Prefill-heavy workload, versus 37K TPS for a B200 8-GPU pod and 20K TPS for an H100 8-GPU pod.The 128-V100 capital expenditure is 78× less, and device-level pipeline parallelism enables a model that cannot fit on one 16GB V100.
  • Scaling efficiency: 7.6×: scaling from 4 to 128 V100 GPUs improves 8B throughput by approximately this factor.Scaling benefits are more pronounced for 70B models, where the deeper pipeline is essential rather than optional.
  • Economic viability: 3.6× throughput: a 128-V100 pod achieves approximately this relative to a single B200 GPU, while costing roughly 10× less in hardware acquisition.The 128-V100 pod costs $7.68K in GPU cost, compared with $75K for a single B200 GPU.
  • Economic viability: 6× more cost-effective: under the China Energy Mix, a 128-V100 pod can reach this advantage over an H100 counterpart for an 8B model.Reduced capital expenses can outweigh higher V100 operational costs over the evaluated deployment lifetimes.

4 Discussion and Limitations

Second-hand GPUs can deliver competitive, cost-effective LLM inference when specialized software and suitable deployment conditions are available, but reliability, operational overhead, market dynamics, and energy infrastructure constrain viability. Sustainable deployment therefore depends on coupling older hardware with inexpensive, low-carbon electricity and recognizing its limits for training workloads.

  • Deployment viability: Second-hand GPUs provide competitive LLM inference performance and cost-effectiveness when paired with specialized software optimizations and appropriate deployment conditions.The paper frames these conditions as central to productive second-hand GPU deployment.
  • Reliability and operations: 13.8% of 58 incidents were true hardware failures requiring replacement, while 86.2% were recoverable software-level incidents.The 128-GPU V100 cluster experienced 8 hardware failures over one year, corresponding to an approximately 6.25% annual device failure rate and motivating 10% spare-capacity provisioning.
  • Reliability and operations: Second-hand deployments require greater monitoring, maintenance, replacement logistics, and in-house hardware expertise because vendor support is lacking.These trade-offs remain economically favorable when hardware capital costs dominate, as with V100 clusters at current market prices.
  • Energy and sustainability: Carbon footprints range from sub-1 g/MT with 100% renewable energy to approximately 7 g/MT on fossil-fuel-dominated grids for 8B models, making clean-energy co-location essential.The paper identifies electricity cost and carbon intensity as fundamental determinants of second-hand GPU viability and sustainability.
  • Limitations and scope: Second-hand GPU economics depend on current market prices that could rise with adoption, while limited NVLink bandwidth and stricter reliability requirements make training less suitable than inference.V100 NVLink bandwidth is 300 GB/s versus 900 GB/s for H100, and a single failure during multi-week training can trigger checkpoint rollback and recomputation.

5 Conclusion

Second-hand GPUs can expand AI inference capacity when paired with specialized software optimizations and deployed under appropriate conditions. A year-long production deployment of a 128-GPU V100 cluster achieved competitive LLM serving performance despite older hardware limitations.

  • 5 Conclusion: Second-hand GPUs offer a viable pathway for expanding AI inference capacity when paired with specialized software optimizations and appropriate deployment conditions.The conclusion frames viability as conditional on both software optimization and deployment context.
  • 5 Conclusion: 128 V100 GPUs built entirely from second-hand components were deployed in production for one year.The cluster was constructed from entirely second-hand components and operated in a production setting.
  • 5 Conclusion: A pipeline-first parallelization strategy enabled competitive LLM serving performance despite older devices’ hardware limitations.The strategy was developed specifically to address the limitations of older hardware.

Appendix A Regional Energy Supply Characteristics · Appendix B GPU Price Calculation

Appendix A derives regional energy prices, carbon intensities, and renewable-energy shares from industrial-sector statistics and 2023 regional grid data. Appendix B estimates hardware costs in USD using vendor quotations and secondary-market valuations, including a $600,000 eight-GPU B200 system quotation.

  • Appendix A Regional Energy Supply Characteristics: Energy prices are derived from industrial-sector energy prices reported by various statistical sources.The appendix uses industrial-sector pricing as the basis for all regional energy-price estimates.
  • Appendix A Regional Energy Supply Characteristics: Regional electricity costs use EIA statistics for the USA, CEIC data for China, and statistics derived from source for Brazil.Carbon intensity and renewable-energy penetration are sourced from 2023 regional grid emission factors and energy-mix data.
  • Appendix A Regional Energy Supply Characteristics: China (Non-renewable) represents coal-dominated grids with 30% renewable penetration and high carbon intensity.This regional profile reflects differences in energy infrastructure and policy environments.
  • Appendix A Regional Energy Supply Characteristics: The United States represents a mixed grid with 20% renewable penetration from natural gas, nuclear, and renewable sources.The stated energy mix reflects a moderate-renewable regional deployment environment.
  • Appendix A Regional Energy Supply Characteristics: Brazil has 90% renewable content, primarily hydroelectric, resulting in low carbon intensity despite higher electricity costs.Its high renewable share distinguishes Brazil from the coal-dominated China profile.
  • Appendix A Regional Energy Supply Characteristics: China (Renewable only) represents deployment co-located with 100% renewable energy sources, achieving minimal operational carbon emissions.This profile models renewable-only deployment rather than the coal-dominated Chinese grid.
  • Appendix B GPU Price Calculation: $600,000 is the quoted full-system price for an NVIDIA B200 system, including peripheral components and networking hardware.The quotation was obtained from Scan.co.uk, and the individual GPU cost is calculated by dividing the aggregate price by eight.

Appendix C Workload Characteristics

The evaluation uses three workload configurations to cover distinct inference profiles, all with a baseline 17-token system prompt. They span prompt-heavy processing, autoregressive decoding, and variable real-world conversations.

  • Appendix C Workload Characteristics: All workloads use a baseline 17-token system prompt.
  • Appendix C Workload Characteristics: Prefill-Heavy uses 1,259 input tokens dominated by tool definitions and caps generation at 128 tokens to test prompt-processing bottlenecks.
  • Appendix C Workload Characteristics: Decode-Heavy uses 199 input tokens and caps generation at 1,024 tokens to assess autoregressive generation efficiency.
  • Appendix C Workload Characteristics: ShareGPT samples 500 conversations with seed=42, chunked to 2,048 tokens under FastChat protocols, to simulate real-world variability.

Appendix D Additional Experiment Result · D.1 Decode Heavy Experiment Results

The decode-heavy experiment evaluates LLaMA3.1-8B serving on new and second-hand GPUs under a long-generation workload, measuring throughput, operational and combined costs, and carbon footprint. Results distinguish profiling software versions and report V100 carbon footprints with extended and zero-embodied accounting.

  • D.1 Decode Heavy Experiment Results: The workload uses a concise 199-token input and a 1,024-token generation window, emphasizing autoregressive decoding throughput.It represents conversational AI applications where output length dominates.
  • D.1 Decode Heavy Experiment Results: The experiment compares new and second-hand GPUs while serving LLaMA 3.1-8B.The table evaluates this model under the decode-heavy workload.
  • D.1 Decode Heavy Experiment Results: The reported performance metric is Token Per Second (TPS), capturing serving throughput for the decode-heavy workload.The workload specifically emphasizes autoregressive decoding throughput.
  • D.1 Decode Heavy Experiment Results: Operational cost is reported in USD per million Tokens using unit electricity prices from different energy mixes.This isolates electricity-driven operating cost across energy-mix scenarios.
  • D.1 Decode Heavy Experiment Results: Combined capital and operational costs are evaluated over 3-year, 5-year, and 8-year life-times.The table includes multiple lifetime assumptions for total cost assessment.
  • D.1 Decode Heavy Experiment Results: Carbon footprints are reported for V100s using extended and zero-embodied accounting values.The table labels these alternatives as “Carbon Footprint (extended)/Carbon Footprint (zero embodied)”.
  • D.1 Decode Heavy Experiment Results: B200 results were profiled with vLLM v0.17, whereas H100, A100, and V100 benchmarks used vLLM v0.6.The profiling-version distinction applies across the reported GPU benchmarks.

D.2 ShareGPT Experiment Results · Appendix E Carbon Calculation

The ShareGPT evaluation measures serving performance, operational cost, combined lifecycle cost, and carbon footprint for LLaMA 3.1-70B and 8B on new and second-hand GPUs. Appendix E attributes inference emissions to operational and embodied carbon, whose sum defines total attributed LLM carbon emissions.

  • D.2 ShareGPT Experiment Results: The ShareGPT workload samples 500 real-world conversations with approximately 262 mean input tokens and a 180-token maximum output.It is designed to capture conversational variability in production chatbot deployments.
  • D.2 ShareGPT Experiment Results: Table 7 evaluates the decode-heavy LLaMA 3.1-70B workload across new and second-hand GPUs.The reported measures include Token Per Second, operational cost per million tokens, combined capital and operational costs over 3-, 5-, and 8-year lifetimes, and carbon footprints.
  • D.2 ShareGPT Experiment Results: B200 results were profiled with vLLM v0.17, while H100, A100, and V100 benchmarks used vLLM v0.6.The implementation note applies to the reported benchmark results.
  • D.2 ShareGPT Experiment Results: V100 carbon-footprint values are reported as extended/zero embodied.This reporting convention is specified for the V100 results in the experiment tables.
  • D.2 ShareGPT Experiment Results: Table 8 evaluates the ShareGPT LLaMA 3.1-8B workload across new and second-hand GPUs.It reports Token Per Second, operational cost per million tokens under different energy mixes, combined capital and operational costs over 3-, 5-, and 8-year lifetimes, and carbon footprints.
  • Appendix E Carbon Calculation: Inference-stage LLM carbon emissions comprise operational carbon from energy consumption and embodied carbon from producing the required hardware.Appendix E denotes these components Copt and Cem, respectively.
  • Appendix E Carbon Calculation: Total attributed LLM carbon emission is defined from the operational and embodied carbon components.The appendix introduces this total after identifying Copt and Cem as the two sources.

E.1 Operational carbon estimation

Operational carbon is determined by data-center runtime energy consumption and the carbon intensity of its energy supply. The analysis models infrastructure overhead with PUE 1.2 and considers four representative energy mixes, including fully renewable supplies with residual lifecycle emissions.

  • Operational carbon basis: Operational carbon (Copt) depends on data-center runtime energy consumption and the energy supply’s carbon intensity.Data-center energy includes servers, cooling, and other infrastructure components; carbon intensity I is measured in gCO2/kWh.
  • Data-center energy: Data-center energy modeling includes server operation, cooling, and other infrastructure components within a shared data center.The model assumes all server components operate in the same data center.
  • Data-center energy: PUE is fixed at 1.2 across hardware components to capture cooling and other infrastructure overhead.The value is averaged across high-efficiency and high-density configurations reported in prior literature.
  • Carbon intensity and energy supply: Four representative energy mixes span fossil-fuel-dominated grids to fully renewable wind or hydroelectric supplies.The configurations follow reported energy-mix data and use regional industrial electricity tariffs.
  • Carbon intensity and energy supply: Approximately 10 g/kWh of residual carbon intensity remains even for fully renewable energy supplies because of renewable-infrastructure lifecycle emissions.Energy supply also affects the operational costs reported elsewhere in the analysis.

E.2 Embodied carbon estimation

The paper estimates embodied carbon by summing logic-die, VRAM, and server-peripheral emissions, then amortizes the total over deployment lifespan. For reused V100s, it incorporates reliability overhead and reports extended- and zero-embodied accounting perspectives, while excluding end-of-life emissions.

  • E.2 Embodied carbon estimation: Embodied carbon is decomposed into logic-die, VRAM, and peripheral server emissions.Logic-die carbon uses silicon area and process-node carbon intensity; VRAM carbon uses HBM capacity and a unit embodied-carbon factor.
  • E.2 Embodied carbon estimation: 150 kg CO2e per server is adopted for peripheral embodied carbon.The shared server components include the CPU, RAM, and chassis.
  • E.2 Embodied carbon estimation: 10% additional devices are deployed as spare capacity to cover in-service failures in second-hand systems.This factor is based on an approximately 90% survival rate over a 5-year service lifespan and extends previously deployed V100s to 8 years total.
  • E.2 Embodied carbon estimation: Two carbon-accounting perspectives are reported for reused V100s: extended, which amortizes full embodied carbon, and zero embodied, which counts only operational carbon.These perspectives form upper and lower bounds on attributable embodied carbon; new hardware receives one standard annualized estimate.
  • E.2 Embodied carbon estimation: End-of-life disposal emissions and upstream material-sourcing emissions are excluded from the lifecycle analysis.The estimates primarily account for manufacturing carbon because comprehensive GPU lifecycle data remain limited.
Loading 2608.14614v1…