PaperlayerView exact revision on arXiv

Source-linked AI summary

Maximal Brain Damage Without Data or Optimization: Disrupting Neural Networks via Sign-Bit Flips

Ido Galil, Moshe Kimhi, Ran El-Yaniv

arXiv:2502.07408v2cs.LGcs.AIcs.CV

TL;DR

The paper addresses whether DNNs can be severely disrupted when an attacker has only write access to stored parameters and no data or optimization. It introduces DNL and 1P-DNL to locate critical sign bits, then demonstrates cross-domain failures and a targeted defense that protects vulnerable bits.

  • Problem

    The paper studies how much access and computation an attacker needs to induce severe DNN failure when models are deployed in important systems.

  • Method

    DNL uses lightweight heuristics to locate critical parameters without data or extra passes, while 1P-DNL refines selection with one forward and backward pass on random inputs.

  • Results

    The vulnerability spans classification, detection, segmentation, and reasoning language models, with severe disruption from only a few targeted sign-bit flips.

  • Takeaways & Limitations

    Selective protection of the most vulnerable sign bits substantially improves robustness against these attacks.

  • Takeaways & Limitations

    DNL assumes the adversary can directly modify a small number of stored parameters; partial writability or addressability may reduce effectiveness.

Abstract

from arXiv · show

Deep Neural Networks (DNNs) can be catastrophically disrupted by flipping only a handful of parameter bits. We introduce Deep Neural Lesion (DNL), a data-free and optimizationfree method that locates critical parameters, and an enhanced single-pass variant, 1P-DNL, that refines this selection with one forward and backward pass on random inputs. We show that this vulnerability spans multiple domains, including image classification, object detection, instance segmentation, and reasoning large language models. In image classification, flipping just two sign bits in ResNet-50 on ImageNet reduces accuracy by 99.8%. In object detection and instance segmentation, one or two sign flips in the backbone collapse COCO detection and mask AP for Mask R-CNN and YOLOv8-seg models. In language modeling, two sign flips into different experts reduce Qwen3-30B-A3B-Thinking from 78% to 0% accuracy. We also show that selectively protecting a small fraction of vulnerable sign bits provides a practical defense against such attacks.

1 Introduction

The paper introduces DNL, a lightweight, data-free attack that identifies critical sign bits and severely disrupts DNNs across vision and language tasks. It also characterizes vulnerable parameters, evaluates defense evasion, and proposes selective protection of those parameters.

  • Extensive Evaluation Across Domains: The vulnerability spans image classification, object detection, instance segmentation, and reasoning large language models.Evaluation covers 60 classifiers, including 48 ImageNet models, across diverse tasks and datasets.
  • Security Implications: One or two carefully selected sign-bit flips can critically degrade perception, reasoning, and downstream decision-making while imposing minimal computational overhead.The paper frames this low-footprint attack as difficult to detect and mitigate in deployed systems.
  • The DNL Attack: DNL identifies critical parameters whose sign-bit flips can catastrophically degrade DNN performance without training or domain-specific data.The attack requires access to stored weights and uses either no additional passes or one forward and backward pass on random inputs.
  • Characterization of Critical Parameters: Critical parameters tend to have large magnitudes and occur in early layers, although some targeting heuristics are architecture-specific.The paper distinguishes generic heuristics such as early-layer targeting from architecture-specific choices such as one flip per convolutional kernel.
  • Defense Evasion: DNL fully or largely bypasses binarization, redundancy coding, and weight-scaling defenses.The evaluation motivates protective strategies beyond these existing defenses.
  • Defense Mechanisms: Selective protection of the most vulnerable parameters substantially improves model resilience to sign-flip attacks.The proposed defense protects only a small targeted subset rather than all parameters.

2 Problem Setup

The problem setup models an attacker who can modify stored model parameters but lacks training or validation data and, for the pass-free threat model, cannot evaluate the model. The attack seeks a large performance drop from only a few bit flips, motivating lightweight heuristics instead of exhaustive search.

  • Threat Model: The attacker has write access to stored parameters but no training or validation data and no forward or backward model evaluations.A relaxed 1-Pass DNL setting permits one forward pass and one backward pass on a single random input.
  • Bit-Flip Attacks: A k-bit-flip attack selects k distinct memory-bit indices and produces modified parameters by flipping those stored bits.The setup uses the bit representation of all model parameters, such as IEEE-754 floating-point encoding.
  • Mechanisms Enabling Parameter Bit Flips: The threat model is enabled by software, firmware, and hardware-level exploits that provide access to model parameters.Examples include rootkits, memory-level attacks, and voltage or frequency manipulation.
  • Formal Objective: The paper frames bit selection as a discrete optimization problem and evaluates vulnerability primarily at small flip counts such as k ≤10.The formulation measures performance degradation caused by flipping selected parameter bits.
  • Adversarial Objective: The attacker’s objective is to induce a significant performance drop while flipping only a handful of bits for stealth and practicality.Small flip counts are emphasized because practical hardware attacks often produce only sporadic bit upsets.
  • Search Challenge: Exhaustive search over millions of parameters is computationally infeasible in real time, so lightweight heuristics are used to locate critical parameters.The setup motivates heuristics based on inductive insights into information flow through the network.

3 Locating Models’ Most Critical Parameters

The paper locates critical parameters by combining sign-bit targeting, magnitude and gradient-based saliency, early-layer selection, and a one-flip-per-kernel constraint. These heuristics identify a small set of parameters whose sign flips can severely disrupt diverse neural networks, while multiple flips within one kernel may partially cancel.

  • Sign-bit flips are targeted because changing the FP32 sign bit reverses a parameter’s sign without changing its magnitude, producing a clean and often strong failure mode.The paper notes that exponent flips alter magnitude, whereas sign-bit flips switch positive and negative parameters directly.
  • Random sign flips are usually ineffective, with experiments showing that even 100,000 flips often fail to reduce accuracy consistently.This motivates identifying critical parameters rather than perturbing arbitrary weights.
  • Magnitude-Based Strategy: Magnitude-based scoring selects high-absolute-value parameters because the paper hypothesizes that reversing large weights causes greater disruption.The score is defined as S(θ_i) = |θ_i|, and top-k magnitude attacks significantly disrupt most evaluated models.
  • One-Flip-Per-Kernel Constraint: In convolutional models, flipping one sign bit per kernel spreads disruption across features and avoids cancellation from multiple flips within the same kernel.Opposite-signed kernel lobes and positively correlated patch entries can make the cross-term negative, partially offsetting a second flip.
  • Layer Selection: Early-layer targeting is often more damaging because perturbations in fundamental feature detectors propagate through subsequent layers.The paper reports that early manipulations are disproportionately damaging, although architectures such as ShuffleNetV2 may require redirecting selection away from their largest late-layer parameters.
  • Hybrid Importance Score: The one-pass variant combines magnitude and second-order or gradient-based saliency, using random-input forward and backward information to refine critical-parameter selection.The hybrid score combines |θ_i| with second-order information, while 1P-DNL uses Gauss–Newton-style estimates related to |θ_i g_i|.
  • Hybrid Importance Score: With 10 sign flips, DNL and 1P-DNL cause most evaluated ImageNet models to collapse, with 43 of 48 models exceeding 60% accuracy reduction.The paper states that 1P-DNL is more damaging than purely magnitude-based methods while remaining substantially simpler than iterative optimization attacks.

4 Results Across Domains

DNL and 1P-DNL remain effective across vision and language tasks, with targeted sign-bit attacks causing severe degradation using very few flips. The results also indicate that vulnerability extends across model classes and is driven by strategically selected parameters rather than random corruption.

  • Language models: Targeted sign flips reduce all three evaluated reasoning models, while the strongest layer scope differs by model.Qwen3-30B-A3B and Nemotron Nano are strongest in the first five blocks, whereas Qwen3-4B is more vulnerable when all layers are considered.
  • Language models: Two sign flips collapse Qwen3-30B-A3B accuracy, while 1P-DNL reaches full collapse at four flips.1P-DNL already produces a 71.8% reduction with one sign flip.
  • Language models: Two sign flips into different Qwen3-30B-A3B experts reduce accuracy from 0.78 to 0.00.The attacked weights are in different expert down-projections, suggesting corrupted expert outputs can propagate through latent representations.
  • Language models: Across nine encoder-task pairs, mean relative accuracy reduction ranges from 69.99% to 83.07%.RoBERTa on MRPC is the most robust tested setting at mAR(10)=69.99%.
  • Cross-domain findings: The attacks generalize across encoder-based NLP models and vision architectures, with model size showing no clear correlation with attack susceptibility.The evaluation spans multiple model families and domains.
  • Image classification: DNL and 1P-DNL sharply collapse accuracy on DTD, FGVC-Aircraft, Food101, and Stanford Cars after one or two sign flips.DNL reaches AR(5) ≥85% across shown model-dataset combinations, while 1P-DNL reaches AR(4) ≥90%.
  • Detection and segmentation: Object detection and segmentation are similarly brittle: one backbone sign flip drives Mask R-CNN box AP to about 0.01 and mask AP to 0.00.YOLOv8-seg loses over 77% of both detection and segmentation performance with one or two flips.

5 Comparison to Other Weight Attacks

Compared with earlier bit-flip attacks, DNL and 1P-DNL target sign bits without data, iterative optimization, or extensive computation. Despite this restrictive threat model, they achieve severe accuracy reductions with fewer flips across image-classification models.

  • Prior methods: Prior attacks typically require multiple forward and backward passes and access to data samples for gradient computation.Earlier approaches therefore fall outside the paper’s restrictive threat model.
  • Prior methods: Pseudo-sample and partial-statistics methods reduce data requirements but still rely on model feedback or approximate gradients.DNL instead uses a data-agnostic sign-bit strategy.
  • Comparison: DNL and 1P-DNL operate without data or optimization and can match or exceed prior accuracy reductions with fewer bit flips.The comparison covers ImageNet-1K INT8 quantized models.
  • Comparison: 1P-DNL collapses ResNet-50 by 99.4% with a single sign flip.This is reported as a representative ImageNet-1K comparison result.

6 Defenses and Counter-Measures

The paper evaluates redundancy, coding, scaling, and selective protection as defenses against sign-flip attacks. Selectively protecting DNL-identified critical parameters offers the strongest lightweight option among the reported approaches.

  • Defenses: Replicating sign bits and majority-voting at inference improves robustness but multiplies memory and bandwidth.An attacker must corrupt most replicas simultaneously.
  • Defenses: ECC can detect and correct isolated flips, but protecting many parameters requires stronger and costlier codes.The paper proposes applying protection only to high-scoring weights identified by DNL.
  • Selective defense: Shielding 0.001% of parameters already halves BFA’s impact on ResNet-18 and ResNet-50.The experiment allows up to ten flips across three runs per flip budget.
  • Selective defense: Guarding 1% of parameters nullifies BFA on every tested model.These results also indicate that DNL identifies the parameters exhaustive BFA seeks to corrupt.
  • Existing defenses: DNL and 1P-DNL either fully or largely bypass representative existing defenses, including redundancy-based coding schemes.The evaluation includes DeepNcode and other proposed bit-flip defenses.
  • Existing defenses: Weight scaling has no effect on DNL because sign flips remain sign inversions after rescaling.The unchanged accuracy reduction is reported empirically.

7 Concluding Remarks

The paper concludes that data-free, optimization-free bit flips can severely damage DNNs across domains, while targeted protection of vulnerable sign bits improves robustness. Effectiveness depends on attackers being able to modify a sufficiently broad set of stored parameters.

  • A cheap, heuristic, data-free attack can severely disrupt classifiers, detection and segmentation models, and reasoning language models by targeting critical parameters.
  • Selective protection of vulnerable sign bits substantially improves model robustness against the attack.
  • DNL assumes direct modification of a small number of stored parameters; partial writability or addressability can reduce effectiveness because global weight search is unavailable.
  • Both DNL and 1P-DNL are simultaneously optimization-free and data-agnostic while often achieving equal-or-higher accuracy reductions with fewer flips than comparison attacks.

B One Flip Per Kernel Constraint Examples

The examples show that flipping multiple high-magnitude weights within one convolutional kernel can partially offset damage, motivating constraints that distribute flips across kernels. Sign-bit flips are generally more destructive than exponent flips in vision at low budgets.

  • AR(2) = 81.31 for MobileNetV3-Large, but adding a same-kernel flip raises accuracy to AR(3) = 46.97, partially offsetting degradation.
  • AR(6) = 74.2 for RegNet-Y 16GF, while adding a seventh flip in the same kernel improves accuracy to AR(7) = 66.5.
  • Sign-bit flips typically cause greater disruption per flip than most-significant exponent-bit flips in representative vision models.

C.1 Language Models: Exponent-Bit Flips

Exponent-bit attacks are especially destructive in reasoning language models: targeted flips can immediately reduce accuracy to zero, and exponent changes can induce extreme rescaling absent from sign flips. The authors therefore identify floating-point format as an important scope consideration.

  • A single targeted exponent flip reduces all three reasoning LLMs to 0% accuracy in the first-five-block setting under both DNL and 1P-DNL.
  • A single rank-check exponent flip in Qwen3-30B-A3B yields 0% accuracy, one non-finite parameter, and gibberish from the first generated tokens.
  • Exponent attacks can induce extreme rescaling by altering the exponent field, whereas sign flips merely negate the stored value.

D Weight Score Ablation

The ablation compares parameter-scoring functions across 48 ImageNet models and finds mixed architecture-specific vulnerabilities. A hybrid score combining magnitude and gradient information provides the most reliable performance drops across the tested models.

  • mAR10 is compared across different parameter weight-scoring functions on 48 ImageNet models.
  • GraSP approximates the Hessian with the square of first-order gradients, while OBD uses a second-order saliency approximation.
  • The hybrid score is defined with and without a second-order term as the proposed scoring variant.
  • Some models are vulnerable to OBD despite resisting magnitude-based attacks, while others show the reverse pattern.
  • Combining magnitude and gradient terms consistently identifies critical weights and delivers the most reliable performance drop across tested models.

E Additional Datasets Evaluation

Additional evaluations show that targeted sign-bit flips severely degrade diverse image encoders and remain effective across architectures, while binarization alone provides negligible protection.

  • Accuracy drops steeply with very few detected sign-bit flips across DTD, Food101, FGVC Aircraft, and Stanford Cars classifiers.
  • The results indicate severe degradation across differing architectures and capacities rather than dependence on one specific network or dataset.
  • DNL remains deterministic across repeated runs, while 1P-DNL’s random-input scoring produces only 0.02 standard deviation in accuracy reduction across 10 seeds.
  • Binarized ResNet-18 remains vulnerable to sign-bit attacks, confirming that binarization alone offers negligible protection.

H Selective Defense Against Sign-Flips: Additional Setups

Additional defense and attack evaluations show that vulnerability varies across model families, while protecting the most vulnerable sign bits substantially improves robustness under random flips.

  • Model vulnerability: Model size alone does not mitigate sign-flip attacks across five model families of varying capacities under 1P-DNL.
  • Selective defense: Protecting selected vulnerable sign bits substantially improves robustness under 100k random sign flips, even with partial coverage.
  • ImageNet evaluation: Across 48 ImageNet classifiers, many MobileNet, MnasNet, ViT, and VGG models reach AR ≈100% with fewer than 5–10 targeted flips.
  • Selective defense: Randomly protecting 20% of sign bits barely mitigates attacks, unlike shielding a small fraction of the most vulnerable bits.
  • Attack variants: Single-pass 1P-DNL strengthens attacks at low flip budgets compared with pass-free DNL, with many models nearly collapsing after a few sign-bit flips.
Loading 2502.07408v2…