Source-linked AI summary
Silent Data Corruptions at Scale
Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, Sriram Sankar
TL;DR
Silent data corruptions can evade CPU error reporting while propagating into application failures and data loss. The paper combines defect analysis, a real datacenter case study, debugging workflows, and fleet-scale detection and mitigation, finding hundreds of affected CPUs across hundreds of thousands of machines. It concludes that reducing SDC risk requires hardware resiliency, production detection, and fault-tolerant software architectures.
Problem
Silent data corruptions evade CPU error reporting, allowing incorrect computations to propagate through datacenter applications and cause data loss and extended debugging efforts.
Method
The paper analyzes silicon defect categories, presents a real application case study, traces faulty instructions, and surveys fleet-scale detection and mitigation strategies.
Results
Hundreds of CPUs with real silent data corruption were identified through detection techniques in a fleet of hundreds of thousands of machines.
Takeaways & Limitations
Reducing SDC risk requires hardware resiliency, production detection mechanisms, and robust fault-tolerant software architectures.
Takeaways & Limitations
The paper omits detailed performance-versus-cost evaluation and skips several hardware and debugging implementation details.
Abstract
from arXiv · showhide
Silent Data Corruption (SDC) can have negative impact on large-scale infrastructure services. SDCs are not captured by error reporting mechanisms within a Central Processing Unit (CPU) and hence are not traceable at the hardware level. However, the data corruptions propagate across the stack and manifest as application-level problems. These types of errors can result in data loss and can require months of debug engineering time. In this paper, we describe common defect types observed in silicon manufacturing that leads to SDCs. We discuss a real-world example of silent data corruption within a datacenter application. We provide the debug flow followed to root-cause and triage faulty instructions within a CPU using a case study, as an illustration on how to debug this class of errors. We provide a high-level overview of the mitigations to reduce the risk of silent data corruptions within a large production fleet. In our large-scale infrastructure, we have run a vast library of silent error test scenarios across hundreds of thousands of machines in our fleet. This has resulted in hundreds of CPUs detected for these errors, showing that SDCs are a systemic issue across generations. We have monitored SDCs for a period longer than 18 months. Based on this experience, we determine that reducing silent data corruptions requires not only hardware resiliency and production detection mechanisms, but also robust fault-tolerant software architectures.
1 INTRODUCTION
Large datacenter CPUs can silently miscompute without hardware error signals, allowing incorrect values to propagate into applications. The paper examines real application impact, debugging, and detection and mitigation strategies for these corruptions.
- Silent data corruptions can make CPUs return incorrect results without system-event or error-log indications.The paper gives 2x3 producing 5 as an example under certain microarchitectural conditions.
- These undetected miscomputations can propagate through services and create application-level failures.The paper focuses on datacenter CPU scenarios and their application consequences.
- The study presents a real application case, debugging processes for faulty CPU instructions, and detection and mitigation strategies.It reports one case study while noting multiple observed scenarios and architectural blocks.
- Prior work largely studied radiation-induced soft errors and synthetic fault injection, whereas this paper examines repeatable corruptions caused by device characteristics.The paper contrasts its observations with probabilistic environmental models.
- The investigation began in 2018 and included multiple detection strategies and their performance costs.The paper omits the detailed performance-versus-cost evaluation for brevity.
2 RELATED WORK
Earlier studies emphasized soft errors from radiation, environmental effects, and synthetic injection. The paper argues that datacenter SDCs also require scale-oriented debugging, detection, and mitigation practices.
- Prior studies evaluated radiation-induced soft errors, environmental effects, and synthetic fault injection in processors and HPC systems.These studies examined error rates, processor vulnerability, MPI correction, and benchmark effects.
- ECC can reduce SRAM error rates, but it does not protect every datapath within a datacenter CPU.The paper distinguishes SRAM protection from broader CPU datapath coverage.
- CPU robustness models based on soft-error probabilities do not directly characterize the higher-order SDCs observed in datacenters.The paper motivates separate investigation of SDC debugging, detection, and mitigation at scale.
- The related-work gap motivates best practices for debugging, detecting, and mitigating silent data corruptions across large infrastructure fleets.
3 DEFECT CATEGORIES
SDCs can arise from design corner cases, timing errors, manufacturing variation, early-life weaknesses, degradation, and silicon wear-out. Their likelihood increases with CPU population, density, and datapath width.
- Design corner cases, timing-path errors, and manufacturing variation can produce functional errors or erroneous bit flips in CPUs.Examples include cache-controller limitations and unreliable transistor operating thresholds.
- Some devices pass manufacturing tests but reveal electrical weaknesses only after weeks, months, or later field workloads.
- Frequently used computational blocks can degrade faster than other CPU regions, creating uncommon usage-related failures.The paper compares these degradation failures with early-life failures and cites Rowhammer as an example in another server device.
- Silicon wear-out emerges after devices operate beyond their rated life and affects most components.This period commonly coincides with the duration of CPU failure-analysis or firmware support.
- All four failure modes can lead to SDCs, with higher likelihood in larger CPU populations and increased density or datapath width.The paper extends this observation to special-function accelerators and other wide-datapath devices.
4 APPLICATION LEVEL IMPACT OF SILENT CORRUPTIONS
The paper traces SDC propagation from distributed Spark workloads and compression pipelines to missing database files and application-visible data loss. A representative corruption turns a valid file-size computation into zero.
- 4.1 Spark: Spark processes large applications by mapping data blocks, reducing results across parallel RDDs, and collecting the output.Wordcount illustrates this flow across worker nodes and shuffle-reduce aggregation.
- 4.2 FB Compression Application: Facebook-scale infrastructure runs millions of compression and decompression operations daily, with compressed files stored in a database.The case study focuses on decompression and checks that file size is greater than zero before decompression.
- 4.2 FB Compression Application: A Scala power-function computation returned zero for a file known to have a non-zero decompressed size.The faulty computation occurred inside the decompression pipeline.
- 4.2 FB Compression Application: Because the computed size became zero, the file was not written to the decompressed output database.
- 4.2 FB Compression Application: Repeated occurrences caused missing files, unrecoverable compressed data, application failure, and critical data-loss reports.The issue appeared occasionally across cluster machines, making reproduction and debugging non-deterministic.
5 DEBUGGING SILENT DATA CORRUPTIONS AT SCALE
The debugging flow reduces a sporadic, data-dependent corruption from a distributed workload to a faulty instruction on one CPU core. It converts the reproducer across language, bytecode, JIT assembly, and debugger stages, then minimizes the assembly-level test case.
- 5 DEBUGGING SILENT DATA CORRUPTIONS AT SCALE: Logging across worker machines narrowed the issue to a host that showed clean system and kernel logs despite sporadic incorrect results.The machine returned zero when non-zero results were expected.
- 5 DEBUGGING SILENT DATA CORRUPTIONS AT SCALE: Single-threading reduced the multi-machine reproducer to consistent failures for certain data values on one particular core.The multi-threading-related sporadic behavior disappeared, while data-value dependence remained.
- 5 DEBUGGING SILENT DATA CORRUPTIONS AT SCALE: Core 59 consistently returned zero for a math.pow input under the faulty data pattern, whereas a different input value set produced an accurate result.Repeated iterations were used to confirm the corruption's data dependency.
- 5 DEBUGGING SILENT DATA CORRUPTIONS AT SCALE: Because Scala workloads cannot be stepped through directly in GDB, the reproducer was converted from Scala to Java, Java bytecode, JIT assembly, and then machine-level instructions.This conversion preserves the reproducer while exposing the executed instructions needed for root-cause triage.
- 5 DEBUGGING SILENT DATA CORRUPTIONS AT SCALE: HotSpot PrintAssembly exposed the JIT-generated assembly, but the initial output contained 430K lines and required parsing, sanitization, and reverse engineering.The disassembly listed methods in the call stack rather than clearly showing the executed instruction sequence.
6 REVISITING APPLICATION FAILURES
Silent data corruptions can produce unlogged, application-level failures including incorrect results, data loss, and difficult root-cause analysis. Fleet detection and debugging practices help quantify and mitigate these systemic errors, but protections add cost and complexity.
- Machines operating the application had no logs or system-level health information indicating the corruption failure mode.
- Incorrect computations affected non-zero operands, positive and negative powers, zero results, and values with varying precision.
- A defective CPU could cause decompressed files to have incorrect sizes, become truncated, and leave dangling file nodes or missing data.
- Targeted reproducers are needed because intrinsic core data dependencies and inputs make these corruptions difficult to detect and root-cause.
- Best practices from reproducers enable faster root-cause and sensitivity analysis, while fleet mechanisms integrate those lessons into detection.
- Silent data corruptions are systemic rather than rare, and mitigation strategies can extend beyond CPUs to ASICs and wider datapath devices.
7 DETECTION MECHANISMS
Fleet detection executes targeted computations and compares their outputs with known references. The paper outlines opportunistic, scheduled, and concurrent testing, each trading coverage, production availability, overhead, and coordination requirements.
- Detection workloads execute specific computations and compare their results with known reference values to validate instruction accuracy.
- Opportunistic testing uses machines in maintenance or transition states with randomized inputs, but coverage depends on how often those states occur.
- Periodic scheduling provides planned coverage but forces machines out of production and carries high testing overhead.
- Minimal tests can run in parallel with production workloads and report pass or fail to a collector, requiring coordination to avoid adverse workload impact.
8 SOFTWARE FAULT TOLERANT MECHANISMS
Software fault tolerance complements hardware detection by adding redundancy and checkpoint verification, while libraries can incorporate algorithmic protections. These approaches improve resilience but consume resources and require cost evaluation.
- Robust infrastructure software abstractions are needed to address silent errors.
- 8.1 Redundancy: Software-level redundancy and periodic checkpoint verification can prevent application-level failures by checking computation accuracy.
- 8.1 Redundancy: Redundant architectures increase resource-pool requirements but provide probabilistic fault tolerance to applications.
- 8.2 Fault Tolerant Libraries: Fault tolerance in libraries such as PyTorch can reduce application exposure to silent corruptions, with added application overhead and potentially negligible performance drop.
- 8.2 Fault Tolerant Libraries: Facebook infrastructure implemented multiple hardware-detection and software-fault-tolerance variants over 18 months, using benefit and cost quantification to improve reliability.
- 8.2 Fault Tolerant Libraries: Statistical trade-offs across detection strategies, coverage scenarios, and fault-tolerant libraries are deferred to a subsequent publication.
9 CONCLUSIONS
The paper presents silent data corruption as a real, scalable datacenter phenomenon that is difficult to debug and can propagate into application failures. Detection and mitigation improve resilience but add cost and complexity.
- Silent data corruptions are real phenomena in datacenter applications operating at scale.
- The case illustrates data-dependent, difficult-to-debug errors and connects their behavior to silicon characteristics, instruction flows, compilers, and software architectures.
- Multiple detection and mitigation strategies contribute to fault-tolerant software architecture and help mitigate corruption costs at Facebook scale.