Hard Data on Soft Errors: A Large-Scale Assessment of Real-World Error Rates in GPGPU

Imran S. Haque; Vijay S. Pande

Source-linked AI summary

Hard Data on Soft Errors: A Large-Scale Assessment of Real-World Error Rates in GPGPU

Imran S. Haque, Vijay S. Pande

arXiv:0910.0505v2Oct 3, 2009cs.ARcs.GR

TL;DR

GPU reliability in error-intolerant scientific applications was largely unproven, and memory reliability without ECC had not been quantified at scale. The paper introduces MemtestG80 and surveys more than 20,000 GPUs on Folding@home. About two-thirds showed pattern-sensitive faults, with error rates strongly associated with board architecture.

Problem
GPU memory reliability in error-intolerant GPGPU applications was largely unproven, and the impact of lacking ECC had not been quantified at scale.
Source: §I · p. 1
Method
The authors developed MemtestG80 and used it to assess GPU error rates across more than 20,000 NVIDIA GPUs on Folding@home.
Source: §I · p. 1, §III · p. 2
Results
Two-thirds of tested GPUs exhibited pattern-sensitive memory faults after controlling for overclocking and time of day; GT200 rates were nearly an order of magnitude below G80/G92.
Source: §I · p. 1
Takeaways & Limitations
The installed consumer-GPU base raised reliability concerns, while controlled negative results suggest—but do not prove—that GPGPU can be reliable with environmental control and dedicated hardware.
Source: §VII · p. 9
The measurements covered NVIDIA GPUs, and the controlled negative results do not conclusively prove reliability under environmental control with dedicated-GPGPU hardware.
Source: §VII · p. 9

Abstract

from arXiv · show

Graphics processing units (GPUs) are gaining widespread use in computational chemistry and other scientific simulation contexts because of their huge performance advantages relative to conventional CPUs. However, the reliability of GPUs in error-intolerant applications is largely unproven. In particular, a lack of error checking and correcting (ECC) capability in the memory subsystems of graphics cards has been cited as a hindrance to the acceptance of GPUs as high-performance coprocessors, but the impact of this design has not been previously quantified. In this article we present MemtestG80, our software for assessing memory error rates on NVIDIA G80 and GT200-architecture-based graphics cards. Furthermore, we present the results of a large-scale assessment of GPU error rate, conducted by running MemtestG80 on over 20,000 hosts on the Folding@home distributed computing network. Our control experiments on consumer-grade and dedicated-GPGPU hardware in a controlled environment found no errors. However, our survey over cards on Folding@home finds that, in their installed environments, two-thirds of tested GPUs exhibit a detectable, pattern-sensitive rate of memory soft errors. We demonstrate that these errors persist after controlling for overclocking and environmental proxies for temperature, but depend strongly on board architecture.

I. INTRODUCTION

GPUs offer substantial computational performance for scientific applications, but their reliability in error-intolerant GPGPU work—particularly memory reliability without ECC—was largely unproven. This paper presents a large-scale assessment and reports pattern-sensitive faults across the installed GPU base, with rates varying by architecture.

I. INTRODUCTION

GPUs’ high throughput has enabled scientific applications, while the lack of ECC in GPGPU memory systems has raised reliability concerns.The paper measures GPU memory reliability using a custom tester on more than 20,000 Folding@home GPUs.Source: §I · p. 1
Two-thirds of tested GPUs exhibited pattern-sensitive memory faults after controlling for overclocking and time of day, with GT200 rates nearly an order of magnitude below G80/G92.The study covered more than 20,000 GPUs worldwide.Source: §I · p. 1

II. BACKGROUND AND PREVIOUS WORK

The study addresses the absence of a large-scale GPGPU reliability assessment by applying soft-error testing methods to an installed base of GPUs.Prior estimates of memory soft error rates required very large-scale testing to accumulate statistically meaningful data.Source: §II · p. 2

III. DESIGN AND VALIDATION OF MEMTESTG80

MemtestG80 is a CUDA-based tester for memory errors on NVIDIA GPUs, built from established memory-test patterns and a custom logic test. Its test iteration runs every implemented test once, with runtime scaling to the tested memory size.

III. DESIGN AND VALIDATION OF MEMTESTG80

MemtestG80 implements most Memtest86 memory tests in CUDA and adds a custom logic test designed for the GPU implementation.The inherited patterns include randomized tests and tests for pattern-sensitive errors.Source: §III · p. 2
One MemtestG80 iteration executes each implemented test once, including all rounds except one stated exception.The exception is described in Section IV-A.Source: §III · p. 2
Testing 64 MiB typically takes 1–5 seconds, and runtime scales linearly with the amount of memory tested.Source: §III · p. 2

A. Offload and Parallelization Scheme

MemtestG80 performs pattern generation, memory access, and verification in parallel on the GPU, and uses a custom logic test to distinguish logic faults from memory faults. Validation found that modulo-20 testing was especially sensitive to overclocking-induced errors, while controlled negative tests detected none.

A. Offload and Parallelization Scheme

Pattern generation, memory access, and verification run in parallel on the GPU, assuming the tester code is less vulnerable than tested memory and logic errors are less frequent than memory errors.A custom logic test checks the latter assumption using similar architectural paths.Source: A. Offload and Parallelization Scheme · p. 3

B. Logic Testing

The logic test compares runs of a zero-started linear congruential generator for k and 4k cycles; error counts that scale with operations indicate logic faults rather than memory faults.Nonzero outputs can indicate either kind of fault, while scaling with operation count points to logic.Source: B. Logic Testing · p. 3

C. Validation

Negative controls detected no errors across over 93,000 MemtestG80 iterations over 700MiB on a GeForce 8800 GTX and an aggregate 1.48 million iterations over 320 MiB on eight Tesla C870 boards.Controls ran in controlled-temperature environments with known-good power sources; a true negative control guaranteeing no errors was not possible.Source: C. Validation · p. 3
The positive control overclocked a GeForce 9500GT from its default 400 MHz memory clock through 530 MHz, resetting and cooling the board between frequencies.Tests used 20 iterations per frequency, except 10 at 530 MHz.Source: C. Validation · p. 3
Moving-inversions tests were insensitive to overclocking-induced errors, motivating the all-zero pattern for the logic test because it appears insensitive to memory errors.Source: C. Validation · p. 3
The modulo-20 test detected overclocking-induced errors at lower clock rates than other tests, and its sensitivity likely reflects a stride that does not align with memory architecture.Its pattern sensitivity persisted in real-world testing and demonstrates that error likelihood depends on access and data patterns.Source: C. Validation · pp. 3–4

IV. EXPERIMENTAL METHODOLOGY

MemtestG80 was deployed on Folding@home, where test-region sizes and settings varied with GPU capability and donor responsiveness. The study recorded whether each iteration detected any errors, a binary approach that avoids false positives but can underestimate error rates.

IV. EXPERIMENTAL METHODOLOGY

Folding@home testing used 32–128 MiB regions, with only two modulo-20 rounds per iteration to limit donor-computer responsiveness costs.The remaining rounds ran in later iterations.Source: A. Testing Procedure · p. 4
GPU capability determined test-region size and the logic-test LCG period, while device information included card name, memory size, and shader-domain clock speed.The tested memory region was considered large enough to span all or a substantial fraction of physical memory chips.Source: A. Testing Procedure · p. 4
Each iteration was classified only as having detected errors or not, avoiding false positives but allowing false negatives and therefore possible underestimation of error rates.An error could toggle an error flag off, but the binary decision cannot overestimate the error rate.Source: A. Testing Procedure · p. 4

B. Test Device Statistics

The study tested more than 20,000 GPUs worldwide, with most results collected using 64 MiB regions and an LCG period of 512. Cards at stock and overclocked frequencies were both well represented, while the sample emphasized consumer GeForce boards.

B. Test Device Statistics

More than 20,000 GPUs completed at least one test, with over 4.6 billion test iterations executed; 96.6% of data used 64 MiB regions and an LCG period of 512.The remaining data used 32 MiB regions with period-256 LCGs (3.2%) or 128 MiB regions with period-1024 LCGs.Source: B. Test Device Statistics · p. 4
The sample covered North America and Europe well, but Africa was poorly sampled apart from South Africa.Source: B. Test Device Statistics · p. 4
Shader-clock specifications were used to classify cards as stock or overclocked, but some identically named models could not be classified because they had multiple possible stock rates.The study treated shader overclocking as a proxy for memory overclocking because the two were expected to covary.Source: B. Test Device Statistics · p. 4
For most iteration-count cutoffs, overclocked and stock cards were comparable in number, although a slight majority of tested cards were overclocked.Cards with indeterminate overclocking status were also shown separately.Source: B. Test Device Statistics · p. 4, Figure 2 · p. 5
The dataset was strongly biased toward consumer GeForce cards, with only a few professional Quadro and GPGPU-dedicated Tesla boards sampled.Table I counts tested NVIDIA product families with at least 300,000 iterations.Source: B. Test Device Statistics · p. 4, Table I · p. 5

V. RESULTS

Both moving-inversions tests reported no errors in the overclocking positive control experiment.The figure plots word error rate against clock rate for each memory test type.Source: Figure 1 · p. 5

TEST ITERATIONS

Card failure probabilities formed a strongly bimodal distribution: one population never failed a test, while most remaining cards clustered near a low nonzero failure probability. At a 300,000-iteration cutoff, about one-third of cards had no observed memory errors.

Nearly all cards outside the zero-error population had mean P (fail) around 2 × 10^-5, while a very small population exceeded 1 × 10^-4.The bimodal pattern remained visible at the largest iteration cutoff, and zero-error fractions were similar at thresholds of 50,000, 300,000, and 1,000,000 iterations.Source: TEST ITERATIONS · p. 5
At a cutoff of at least 300,000 iterations, approximately one-third of cards never exhibited a memory error, and only about 2% had P (fail) above 10^-4.Nearly all remaining cards had failure probabilities between 0 and 10^-4.Source: TEST ITERATIONS · p. 6

VI. ANALYSIS

The analysis uses information gain to test whether an indicator variable explains variation in card failure-probability distributions. Failure probabilities are histogrammed per card, and the indicator's explanatory value is assessed through entropy-based comparisons.

VI. ANALYSIS

The analysis section examines mutual information between distributions and information gain for data-partitioning attributes.These are presented as established statistical techniques.Source: §VI · p. 6

A. Hypothesis testing by information gain

Information gain tests whether a partitioning indicator V explains variability in a labeled dataset D, using the distributions of D and its subsets.The paper uses the measure as a heuristic criterion for decision-tree partitioning.Source: A. Hypothesis testing by information… · p. 6
A large I(D; V) relative to H(D) indicates that V explains a significant portion of D's distribution.The measure is also described as mutual information between D and V.Source: A. Hypothesis testing by information… · p. 6
Per-card failure probabilities are histogrammed across the full 0-to-1 range to estimate distributions for hypothesis tests.Although high-probability bins have few counts, most are zero-valued and do not affect entropy calculations.Source: A. Hypothesis testing by information… · p. 6

B. Bimodality of P(fail)

The analysis tests whether overclocking, time of day, or board architecture can explain the bimodal failure-probability distribution. The reported information-gain results indicate that overclocking and local time of day explain little of its shape.

B. Bimodality of P(fail)

The bimodality prompted tests of overclocking, thermal effects, and architectural variability as possible explanations for cards with nonzero failure probabilities.Source: B. Bimodality of P(fail) · p. 6

1) Shader Overclocking:

Overclocking status had information gain of 7.5 × 10^-2 bits, far below the 0.8896-bit perfect-indicator reference, making it unlikely to cause the second mode around P (fail) = 2 × 10^-5.The comparison used cards with known status, partitioned into stock and known-overclocked groups.Source: 1) Shader Overclocking · p. 6, B. Bimodality of P(fail) · p. 6

2) Time of day:

Local time of day had information gain of 0.0413 bits and was not an adequate explanation for the error-distribution shape.Day was defined as 6am–6pm and night as 6pm–6am local time; only tests conducted entirely within one interval were compared.Source: 2) Time of day · p. 6

3) Board architecture:

Board architecture strongly accounts for the bimodal failure distribution: GT200 cards were much less likely to fail than G80/G92 cards. The observed error-rate difference was not explained by coalescing alone.

3) Board architecture:

The GT200-versus-non-GT200 partition explained 0.453 bits of information about the error distribution, indicating that architecture substantially accounts for its structure.The analysis partitions all GeForce cards by whether they use GT200.Source: 3) Board architecture · p. 6
About 90% of GT200 cards never reported errors, and error-producing GT200 cards clustered at 2.2 × 10−6, an order of magnitude below the overall-dataset mode.GT200-based boards were about one-fifth to one-fourth of the dataset, depending on iteration threshold.Source: 3) Board architecture · pp. 6–7

4) Bimodality conclusions:

The authors’ data suggest that GT200 has an apparent soft error rate nearly tenfold lower than G80, but improved coalescing support is insufficient to explain the difference.Coalescing combines multiple memory reads or writes into single transactions.Source: 4) Bimodality conclusions · p. 7
On the modulo-20 test, G80 generated 16.7% more memory traffic by bytes than GT200, which the authors say does not explain a tenfold error-probability reduction; card age remains a possible factor.The test used the same number of transactions on both architectures, while GT200 could use smaller transactions.Source: 4) Bimodality conclusions · p. 7

C. Impact of errors on molecular dynamics

The measured relationship between MemtestG80 errors and Folding@home simulation failures was weak, and the authors could not determine the errors’ true impact on simulations.

C. Impact of errors on molecular dynamics

0.131 bits of mutual information linked MemtestG80 errors with Folding@home early unit ends, suggesting the two likely do not correlate well.The comparison counted work units in which at least one MemtestG80 iteration ran.Source: C. Impact of errors on molecular… · p. 7
The measured relationship may understate simulation impact because early unit ends have causes unrelated to board errors and some simulation errors were not logged.Source: C. Impact of errors on molecular… · p. 7
The authors conclude that their results are inconclusive about the true impact of observed errors on scientific simulations.Source: C. Impact of errors on molecular… · p. 8

D. Failure modes of tests

The test-failure analysis uses mutual information to compare failure patterns across MemtestG80 tests. It indicates that Modulo-20 probes a distinct pattern, while Random Blocks and the logic tests reveal overlapping and distinct failure modes.

D. Failure modes of tests

The analysis histograms each test’s per-execution failure counts into 10 bins, normalizes them into empirical distributions, and compares tests using mutual information.The paired vectors correspond to the same MemtestG80 executions.Source: D. Failure modes of tests · p. 8
Modulo-20 covaried weakly with every other test, consistent with its greater sensitivity and a failure mechanism distinct from those probed by other tests.Source: D. Failure modes of tests · p. 8
Figure 5’s row-to-column entries give the fraction of the column test’s entropy explained by the row test; brighter cells indicate a larger explained fraction.Source: Figure 5 · p. 8, D. Failure modes of tests · p. 8
Random Blocks captures errors measured by the LCG logic tests, while the four-iteration logic tests expose a distinct failure mode that can also be triggered by soft errors in shared memory.The logic tests covary strongly among themselves, and memory tests explain L4 better than LS4.Source: D. Failure modes of tests · p. 8

VII. CONCLUSIONS

The first large-scale GPGPU study found pattern-sensitive faults in the installed GPU base, while controlled negative controls suggest—but do not prove—that reliable GPGPU is possible. The authors conclude that consumer-grade GPGPU can be reliable with close attention to hardware characteristics.

VII. CONCLUSIONS

Approximately two-thirds of tested cards exhibited pattern-sensitive susceptibility to soft errors, and overclocking or time of day did not adequately explain the nonzero rate.Error rates correlated strongly with GPU architecture; newer GT200 boards had much lower rates than older G80/G92 boards.Source: §VII · p. 9
Controlled negative controls suggest—but do not conclusively prove—that GPGPU can be reliable, while installed consumer hardware raises concerns for distributed computing and GPU-accelerated software.The study collected data only from NVIDIA GPUs.Source: §VII · p. 9
MemtestG80 was released under an open-source LGPL license for GPU stress testing or self-checking.Source: §VII · p. 9
Eight Tesla boards had no observed memory or logic errors across 1.48 million iterations, suggesting greater reliability, though environmental or unusually good-batch effects remain possible.The sample was too small to provide a conclusive answer about professional-level and dedicated-GPGPU boards.Source: Is dedicated GPGPU hardware the answer? · p. 9
Developers may need active testing, memory parity or ECC, and redundancy or software error detection because transient GPU logic errors also require consideration.Memory testing and ECC can guard against memory errors, but do not address all GPU logic faults.Source: What can be done? · p. 9
The authors conclude that reliable GPGPU computing on consumer-grade hardware is possible, but requires close attention to hardware characteristics.Source: What can be done? · p. 9