Source-linked AI summary
Let the Neurons Die: Exploiting ReLU-Induced Model Degradation
Kexin Li, Wenjun Qiu, Joshua Abraham, Aditi Maheshwari, David Lie
TL;DR
ReLU networks can lose gradient flow through units whose pre-activations remain negative, motivating attacks on learning availability. This paper uses data ordering and gradient-inverted poisoning to target that failure mode without directly modifying victim parameters. On MNIST, ordering 100 of 60,000 examples reduces test accuracy from 96% to 95% after five epochs, while 200 single-class poisons leave most conditions at approximately 86–88% versus approximately 96% under clean training.
Problem
ReLU units with persistently negative pre-activations produce zero outputs and block gradient flow, and widespread inactivity can reduce representational capacity and complicate optimization.
Source: §1 · p. 1Method
The paper introduces greedy DOA ordering and IG-DOA/IG-SKA, which synthesize class-conditioned samples by matching reference gradients at adverse states constructed through ordering or soft knockout.
Source: §1 · pp. 2–3Results
On MNIST after five epochs, ordering 100 of 60,000 examples reduces test accuracy from 95.93% to 95.04%, while 200 single-class poisons leave most conditions at approximately 86–88% versus approximately 96% under clean training.
Source: §1 · p. 2Takeaways & Limitations
The MNIST results show that ReLU-targeted attacks can impair learning through training data and its presentation order without directly modifying victim parameters.
Source: §6 · p. 11The evaluation covers one MNIST multilayer perceptron over five epochs; variability across initializations and poison-budget sensitivity remain future work, and an accuracy deficit does not establish failure to recover with longer training.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Rectified linear unit (ReLU) networks can suffer from dying neurons, where units with persistently negative pre-activations produce zero outputs, blocking gradients through their activations. To exploit this failure mode, we present three training-time availability attacks based on data ordering and poisoning. We begin with the basic dynamic data-ordering attack (DOA), which greedily constructs a training prefix by selecting the next example that minimizes the target layer's post-update weight sum, aiming to push ReLU units toward negative pre-activations without modifying training samples or labels. We then develop two poisoning attacks, IG-DOA and IG-SKA, which use gradient inversion to synthesize class-conditioned samples by matching reference gradients in adverse model states constructed through data ordering or soft knockout, respectively. Soft knockout rearranges weights across adjacent layers to concentrate negative contributions. On a fully connected ReLU network trained on MNIST, ordering 100 of 60,000 training examples reduces test accuracy from 96% to 95% after only five epochs. Adding 200 poisoned samples from a single class reduces test accuracy to approximately 86-88% after five epochs in most evaluated conditions, compared with approximately 96% under clean training. These results demonstrate that ReLU-targeted data ordering and poisoning can impair learning without directly modifying the victim model's parameters.
1 Introduction
The paper targets ReLU inactivity as a training-time availability vulnerability and investigates attacks using sample order and synthetic data, without directly modifying the victim’s parameters. On MNIST, ordering causes modest degradation, while gradient-inverted poisoning produces larger, class-dependent losses under shuffled training.
1 Introduction
- The attacks target dying ReLU: persistently negative activations yield zero gradient flow, and widespread inactivity can reduce representational capacity and complicate optimization.The stated attacker has white-box access; order-only control preserves samples and labels, while poisoning adds synthetic labelled examples to a shuffled set.Source: §1 · pp. 1–2
- Adding 200 single-class synthetic examples leaves most evaluated conditions at approximately 86–88% accuracy, versus approximately 96% under clean training; ordering 100 of 60,000 examples yields 95.04% versus 95.93%.Both comparisons are after five epochs; some poisoning conditions recover within the evaluation period.Source: §1 · p. 2
- DOA greedily orders existing examples by minimizing a target layer’s post-update weight sum, leaving samples and labels unchanged.It also provides one route to constructing adverse model states for poison synthesis.Source: §1 · p. 2
- IG-DOA and IG-SKA synthesize labelled poisons by matching class-specific reference gradients at adverse states reached through DOA or adapted soft knockout.Soft knockout rearranges adjacent-layer weights to concentrate negative contributions; it is performed on a local model, not the victim.Source: §1 · pp. 2–3
2 Background and Related Work
The background situates the work at the intersection of ReLU inactivity, training-time poisoning, data ordering, and gradient inversion. The paper repurposes these ideas to target availability rather than a specific misclassification.
ReLU failure modes
- Dying ReLU arises because negative pre-activations have zero derivative; prior weight permutation attacks exploit this but assume control over initial weights.This paper instead studies whether training sequence and gradients from harmful states can be used to construct malicious samples.Source: §2 · p. 3
Training-time attacks
- Availability poisoning seeks to reduce learning effectiveness, unlike clean-label attacks aimed at specific errors or backdoors; prior work also studies convex settings and gradient-based deep-network poisoning,,.Certified filtering guarantees are described as conditional on assumptions of the learning setting [14].Source: §2 · p. 3
Data ordering
Gradient inversion
- Gradient inversion can reconstruct inputs by matching observed gradients; this paper repurposes it to generate poisoned samples at deliberately harmful model states.The cited example combines cosine-based gradient matching with image priors.Source: §2 · p. 4
3 Design
The design assumes a white-box attacker who can reorder existing data or add synthetic labelled data, but cannot directly alter the independently initialized victim. It builds adverse states through ordering or soft knockout, then uses gradient inversion to synthesize poisons.
3.1 Threat Model and Assumptions
- The attacker has white-box access to the model, loss, and training data, and can either reorder unchanged correctly labelled examples or add synthetic labelled examples to a shuffled set.The victim is independently initialized, and the poisoning attacker does not directly alter its parameters.Source: §3.1 · p. 4
3.2 Dynamic Data Ordering
- DOA evaluates each unused candidate after replaying the current prefix from a common checkpoint, then greedily selects the candidate with the smallest target-layer post-update weight sum.The ordering is presented in Algorithm 1.Source: §3.2 · p. 4, §3.3 · p. 5
3.3 Soft-Knockout Weight Construction
- Soft knockout sorts adjacent-layer weight matrices and places a fraction of small entries into selected rows of A and columns of B to concentrate negative contributions.Unlike DOA, it directly constructs an adverse state and presumes control of, or an offline copy of, model weights during training.Source: §3.3 · p. 4
3.4 Repurposing Gradient Inversion Attacks
- At a DOA- or SKA-constructed adverse state, the poisoning back-end matches a class-specific reference update to a synthetic input, assigns that class label, and adds the sample to a fresh victim’s training set.The objective uses gradient inversion with an image prior; the variants are IG-DOA and IG-SKA.Source: §3.4 · p. 5
4 Evaluation
The evaluation tests data ordering and gradient-inverted poisoning on a fully connected ReLU network trained on MNIST. Poisoning often leaves substantial accuracy deficits after five epochs, while ordering produces a more modest deficit and the models continue learning.
4.1 Experimental Setup
- The evaluation uses MNIST’s 60,000 training and 10,000 test images with a fully connected ReLU classifier; each condition is one seeded run without confidence intervals or significance claims.Accuracy is measured on the unmodified ten-class test set.Source: §4.1 · pp. 5–6
Effect of Data Ordering
- All four DOA conditions reduce accuracy relative to clean training by epoch 4, but attacked models improve over the five-epoch period rather than cease learning.The experiment reorders existing examples without changing their contents or labels.Source: Figure 1 · p. 7, Effect of Data Ordering · pp. 7–8
Effect of Gradient-Inverted Poisoning
- At epoch 4, nine of ten single-class IG-DOA conditions remain at approximately 86–88% accuracy versus approximately 96% under clean training, a 10% deficit despite shuffling.Each condition adds 200 synthetic examples assigned one digit label.Source: Figure 2 · p. 8, Effect of Gradient-Inverted Poisoning · p. 9
- At epoch 4, eight single-class IG-SKA conditions remain around 87% accuracy, approximately 9% below clean, while C9 and All finish close to clean accuracy.C6 rises to approximately 94–95% at epoch 4 after remaining around 86–87% through epoch 3.Source: Figure 3 · p. 9, Effect of Gradient-Inverted Poisoning · p. 9
5 Limitations and Discussion
The evaluation is limited to a single MNIST multilayer perceptron over five epochs, and some poisoning conditions recover toward the clean baseline. The authors identify unmeasured variability, poison-budget sensitivity, and attack defenses as open questions.
Evaluation scope and generalization
- The evaluation covers MNIST on one multilayer perceptron over five epochs, so it does not establish generalization to other datasets, architectures, or training configurations.The authors also note that a five-epoch accuracy deficit does not establish that a model cannot recover with further training.Source: §5 · p. 10
- Some poisoning conditions recover toward the clean baseline during evaluation, while others remain substantially below it.The authors say longer training horizons are needed to characterize the persistence of the effects.Source: §5 · p. 10
- Variability across initializations and sensitivity to poison budget remain unmeasured; the authors propose repeated runs and further evaluation.They also identify alternative activation choices as an untested direction for determining whether they mitigate the observed degradation.Source: §5 · p. 10
Threat model and defenses
- The threat model assumes full knowledge of the model, loss, and training data, plus control over sample order or synthetic-sample addition; proposed defenses are unevaluated.Detection accuracy, cost, and ability to reduce the attacks’ effects remain open questions.Source: §5 · p. 10
6 Conclusion
The paper presents training-time availability attacks that exploit ReLU inactivity through data ordering and poisoning. In the MNIST evaluation, ordering and poisoned samples reduce accuracy, with the poisoning effects varying by class condition.
6 Conclusion
- Adding 200 poisoned samples leaves most single-class conditions at approximately 86–88% accuracy, versus approximately 96% under clean training, after five epochs.The augmented dataset is shuffled and the victim independently initialized.Source: §6 · p. 10
- Ordering 100 of 60,000 training examples reduces test accuracy from 96% to 95% after five epochs.The attacks use training-data presentation and poisoning without directly modifying victim parameters.Source: §6 · pp. 10–11
- Some poisoning conditions recover toward the clean baseline, so the effect varies across poison classes.The authors frame the findings as a proof of concept for studying activation behavior and training-pipeline integrity.Source: §6 · pp. 10–11
Impact Statement
The work frames its attacks as a basis for studying training-pipeline reliability and investigating safeguards, while acknowledging potential misuse. Its experiments are restricted to a controlled MNIST proof-of-concept setting.
Impact Statement
- The authors identify checks on sample order, activation and gradient monitoring, and poison detection as possible safeguards to investigate.These directions could support earlier identification of compromised runs and reduce risks of deploying degraded models or spending resources on retraining.Source: Impact Statement · p. 11
- Publishing the methods could help adversaries manipulate training order or construct poisons, although experiments use controlled MNIST and no private or sensitive data.The results do not establish a general vulnerability of deployed ReLU models or the effectiveness of proposed safeguards.Source: Impact Statement · p. 11
- The work acknowledges NSERC funding and institutional support, and the authors declare no competing interests.Named support includes an NSERC Discovery Grant, a Canada Research Chair, the Viola Carless Smith Research Fellowship, and the Queen Elizabeth II Graduate Scholarship in Science and Technology.Source: Impact Statement · p. 11