GrayShield: Bit-Level Sanitization for Transformer Model Supply-Chain Security

Introduction

• GrayShield targets covert malware payloads in model weights and addresses the gap in explicit channel-erasure arguments for complete replacement of declared payload-bearing bits.Existing defenses include detection that does not remove payloads, PTQ that overwrites low-order information as a side effect, and fine-tuning that requires trusted data.Source: Introduction · p. 1

Background and Related Work

• Gray codes provide one-bit transitions between adjacent codewords, while balanced masked periods and phase rotation diversify overwrite patterns.The zero-capacity result follows from payload-independent complete overwrite, not from Gray coding or phase secrecy.Source: Background and Related Work · p. 2
• Prior work includes coded weight payloads, while compression and perturbation defenses lack guarantees against structured payloads or complete erasure of the declared channel.The paper distinguishes this goal from detecting compromised checkpoints and from inference-failure defenses against sparse malicious flips.Source: Background and Related Work · p. 2

Algorithm

• GrayShield applies a keyed per-tensor phase and Gray-code overwrite while preserving sign and exponent bits; its procedure is vectorizable in O(N) time.The phase varies patterns across tensors and releases, but secrecy is not required for target-channel erasure.Source: Algorithm · p. 2

Formal Guarantees

• Complete payload-independent replacement makes the sanitized target bits independent of the payload, giving the declared channel zero capacity; balanced bits additionally yield chance-level recovery.When pre-defense recovery is effectively one, chance-level bit accuracy corresponds to RR near 50 percentage points.Source: Formal Guarantees · p. 3

Threat Model

• The threat model places a white-box payload embedder before post-training sanitization, with the defender lacking training data; experiments target FP32 weights.The stated scope excludes higher-bit payloads, executable serialization, metadata side channels, and post-sanitization reinsertion; reduced-precision utility remains unvalidated.Source: Threat Model · p. 4

Target parameter set and hiding capacity

• The target set is encoder-block attention and MLP weight matrices, excluding embedding layers and task-specific classifier heads by default.Attacker knowledge ranges from Tier 1 naive substitution through Tier 2 redundancy or interleaving to Tier 3 knowledge of GrayShield but not its seed.Source: Target parameter set and hiding capacity · p. 4, Three-tier attacker knowledge hierarchy · p. 4

Models and datasets

• Four pretrained Transformer presets cover text and vision: BERT-base and RoBERTa-base on SST-2, and ViT-Base/16 and Swin-Base on CIFAR-10.Evaluations use held-out subsets of 512 examples, providing approximately 0.195 percentage-point accuracy resolution.Source: Experimental Setup · p. 4

Malware payloads

• The payload set contains a 134.3 KB low-entropy JavaScript sample and an 8,971.4 KB high-entropy compiled binary.Their entropies are 0.603 and 0.994, respectively; the low-entropy payload tolerates partial corruption, while the high-entropy payload requires near-lossless reconstruction.Source: Experimental Setup · p. 4

Defenses

• Seven post-training defenses are evaluated at x = 19, including GrayShield, fixed-pattern overwrite, INT8 quantize–dequantize, fine-tuning, noise, selective perturbation, and random bit flips.The chosen depth covers stealthy embeddings at x′ ≤ x; deeper embeddings show detectable accuracy and geometric degradation.Source: Experimental Setup · p. 4

Attacker variants

• Five Tier-2 attacker variants are tested: Naive, Interleave, Repeat3, Repeat5, and RS(255,127).The defense depth is set to the largest utility-preserving value identified in RQ1.Source: Experimental Setup · p. 4

Metrics and statistical protocol

• Recovery Reduction is reported in percentage points, and near 50 pp indicates post-defense bit accuracy near binary chance when pre-defense recovery is effectively one.The protocol also reports accuracy change, W1, runtime, RR consistency, and bit-level Hamming distance; experiments use run seed 1 and evaluation seed 42.Source: Experimental Setup · p. 4

RQ1: Injection Feasibility

• Payload injection remains stealthy through x = 16 and enters a transition regime at x = 19, where high-entropy payloads first show measurable accuracy degradation.The worst completed high-entropy case remains below 1% accuracy impact.Source: RQ1: Injection Feasibility · pp. 4–5

RQ2: Defense Effectiveness under a Naive Attacker

• GS, PM, and PTQ are the only defenses near chance under the naive attacker, but GS combines low RR variance with the smallest W1 shift in this group.GS’s average W1 is 2.6×10−5, approximately 25× smaller than PM’s and 42× smaller than PTQ’s.Source: RQ2: Defense Effectiveness under a… · p. 5

RQ3: Robustness under Adaptive Attackers

• 49.96 ± 0.66 pp aggregate RR keeps GS within 49.5–50.4 pp across all five implemented attacker variants.The variants are Naive, Interleave, Repeat3, Repeat5, and RS(255,127).Source: RQ3: Robustness under Adaptive Attackers · p. 5

RQ4: Pareto Trade-off and Deployment Profile

• 49.96±0.66 pp RR at x = 19 makes GS the recommended operating point, meeting the stated accuracy, variance, runtime, and distribution-shift criteria.GS is the only defense reported to satisfy all four deployment constraints simultaneously.Source: RQ4: Pareto Trade-off and Deployment… · p. 5

Theory–empirics alignment

• Complete payload-independent replacement erases the declared target channel, while balance is the additional condition for chance-level bit accuracy.Across five attackers, GS reports 49.96 ± 0.66 pp RR; PM has higher dispersion and 25× larger W1 despite similar mean RR.Source: Discussion · p. 5

Deployment interpretation

• GS alone satisfies the stated operating envelope of RR in [48, 52] pp, σRR < 2 pp, |∆Acc| < 1%, runtime below 500 ms, and low distributional drift.PTQ is slower and more distortive, PM is less stable, and FT is slow and requires trusted data.Source: Discussion · p. 5

Limitations

• The evaluation covers FP32 and four 86M–125M presets; reduced-precision utility remains unvalidated, and the guarantee is limited to the bit-local scope.Higher-bit payloads, executable serialization, metadata side channels, and post-sanitization reinsertion are outside scope.Source: Discussion · p. 6

• Payload-independent overwrite completely replaces the declared weight-LSB payload channel and gives that scoped channel zero capacity.The low-transition Gray structure and keyed per-tensor phase provide regularity and pattern diversity.Source: Conclusion · pp. 6–7
• GrayShield maintains recovery reduction around 50 pp across adaptive attackers, with sub-1% accuracy impact and lower W1 than evaluated near-chance baselines.It also combines low variance and runtime with that lower distribution shift.Source: Figure 4 · p. 7, Conclusion · p. 7
• Figure 4 reports stable recovery reduction across attacker variants and resilience across the implemented adaptive strategies.Its panels show attacker-wise RR, defense-level distributions, the effectiveness–stability trade-off, and an attacker-by-defense heatmap.Source: Figure 4 · p. 7

Pre-specified Hypotheses and Outcomes

• Table 3 consolidates 13 pre-specified RQ1–RQ4 hypotheses with their tests, effect sizes, and decisions; the supplied results include validation across coding strategies and a critical chance-channel test.The table caption specifies two-tailed tests at α = 0.05 unless marked one-tailed, with a Bonferroni correction for H(4b).Source: Pre-specified Hypotheses and Outcomes · p. 10, Table 3 · p. 10

Information-theoretic framing

• The information-theoretic framing treats pre-defense extraction as noiseless and seeks a sanitization map that removes payload information; the analysis assumes PRF security, a secret seed, bounded attackers, and a memoryless channel.The seed is specified as 128 bits and stored securely, for example in an HSM.Source: Information-theoretic framing · p. 10, Assumptions · p. 10

Perfect Channel Erasure

• Under the stated assumptions, complete payload-independent overwrite makes the declared channel’s effective capacity zero for any payload distribution.The proof uses a keyed pseudorandom overwrite pattern independent of payload bits when the seed is hidden.Source: Perfect Channel Erasure · p. 11

Coding-Theoretic Limits

• At a BSC(1/2), no positive-rate error-correcting code can achieve reliable decoding, including the discussed Reed–Solomon, LDPC, and Polar codes.The passage invokes the channel coding converse and gives code-specific decoding consequences.Source: Coding-Theoretic Limits · p. 11

Accuracy Stability Bound

• The Lipschitz accuracy bound relates loss change to weight displacement; the reported worst-case Drel is 5.4×10^-3 at x = 19, consistent with sub-1% accuracy changes.The passage relates this consistency to a spectral-norm analysis and main-paper results.Source: Accuracy Stability Bound · p. 11

Robustness under Imperfect Randomness

• For crossover probability 1/2 + ϵ, effective capacity is O(ϵ^2), so modest deviations from perfect uniformity still yield near-zero capacity.This is presented as robustness beyond the idealized PRF model.Source: Robustness under Imperfect Randomness · p. 11

Threat Model

• The threat model places an attacker embedding binary payloads in FP32 mantissa LSBs before data-free post-training sanitization of Transformer encoder attention and MLP weights.Embedding layers and task-specific classifier heads are excluded by default.Source: Threat Model · p. 12

Target parameter set and hiding capacity

• The implemented evaluation targets FP32, while FP16, BF16, and INT8 are discussed as deployment-format extensions rather than empirically validated formats.For BERT-base encoder blocks, the passage estimates about 53 MB capacity at x = 5 in FP16; its INT8 example describes x = 1 as having negligible stealth cost.Source: Target parameter set and hiding capacity · p. 12, Multi-precision threat surface · p. 12

Notation and Abbreviations

• The notation table groups definitions for information-theoretic symbols, precision formats, coding theory terms, defenses, models, and datasets.It identifies BSC(p), binary entropy, BSC capacity, and mutual information among the information-theoretic entries.Source: Table 4 · p. 12, Symbol / Abbrev. Definition · p. 13

Symbol / Abbrev. Definition

• FP32, FP16, BF16, and INT8 denote the listed numeric formats, with FP32 specified as 1 sign, 8 exponent, and 23 mantissa bits.FP16 is listed as 1-5-10 and BF16 as 1-8-7.Source: Symbol / Abbrev. Definition · p. 13
• The appendix expands ECC, RS, LDPC, and Polar, and lists defense abbreviations alongside the evaluated models, datasets, and task pairings.The listed presets pair BERT and RoBERTa with SST-2 and ViT and Swin with CIFAR-10.Source: Symbol / Abbrev. Definition · p. 13

Definition 11–12

• The covert channel encodes payload bits in target-weight mantissa LSBs and decodes those same positions in the noiseless pre-defense setting.Storage capacity counts embeddable bits, whereas effective Shannon capacity measures mutual information after sanitization.Source: Formal Definitions · p. 13

Definition 13

• A post-training sanitization map is data-free, modifies weights only, and has runtime linear in the number of target parameters.The map is defined over model weights and a key space.Source: Formal Definitions · p. 13

Definition 15–16

• Effective channel capacity takes the supremum of payload–recovery mutual information over payload distributions; Proposition 4 gives GrayShield zero capacity under the idealized PRF model.The recovery reduction compares pre-defense and post-defense bit accuracy; pre-defense recovery is effectively 1.0 for aligned injection at x ≤ 16.Source: Formal Definitions · p. 14

Definition 17

• Stealth jointly bounds perturbation magnitude, directional preservation, and utility change; the study confirms its stated boundary at x = 16 and reports degraded stealth at x = 23.The reported tuples are (10^-4, 2.6×10^-7, 0.01) at x = 16 and (8.5×10^-2, 3.5×10^-3, 0.01) at x = 23.Source: Formal Definitions · p. 14

Definition 18–19

• The perturbation field records overwrite values by flattened tensor position, while total-variation smoothness characterizes how slowly those perturbations vary.The text relates lower ETV to lower high-frequency spectral energy and contrasts GrayShield’s bound with PatternMask’s period-2 pattern.Source: Formal Definitions · p. 14

Definition 20–21

• The memoryless-channel definition requires conditional independence across extracted positions; chance-level recovery corresponds to GrayShield’s idealized p → 1/2 result and empirical RR near 50%.The empirical near-chance range is Recpost ∈ [0.49, 0.51].Source: Formal Definitions · pp. 14–15

Extended Trade-off Analysis

• Figure 6 distributes recovery reduction and accuracy degradation across evaluated model, payload, and attacker configurations, complementing the main RQ3 analysis.The full configuration-level visualization is placed in the appendix because it is visually dense relative to the main-paper figures.Source: Extended Trade-off Analysis · p. 15
• Across the configurations, the figure confirms stable high-level ordering while exposing differences in variance, operating-point consistency, and deployment robustness.Source: Extended Trade-off Analysis · p. 15

Multi-Precision Threat Extension

• FP16 and BF16 have 10 and 7 mantissa bits, respectively, and the capacity estimate for BERT-base at x = 5 in FP16 is approximately 53 MB.The paper states this is sufficient for both evaluated payloads.Source: Multi-Precision Threat Extension · p. 15
• The Gray-code overwrite proposition extends unchanged to FP16 and BF16 when x is clamped to the format’s mantissa depth.Source: Multi-Precision Threat Extension · p. 15
• For INT8, each weight supports x ≤ 2 bits, with the paper describing the quantization error at x = 1 as negligible.The stated error condition is δ_i = q · scale_ℓ with q ≤ 1/128.Source: Multi-Precision Threat Extension · p. 15
• Empirical validation of reduced-precision formats is left for future work.Source: Multi-Precision Threat Extension · p. 15

Bit Manipulation in FP32 Weights

• All attack and defense operations act exclusively on mantissa LSBs, preserving the sign and exponent bits unchanged.A float32 value comprises one sign bit, eight exponent bits, and 23 mantissa bits.Source: Bit Manipulation in FP32 Weights · p. 15
• The setup defines extraction as u & m, overwrite as (u & ∼m) | b, and random flip as XOR with independently sampled masked bits.Here u is the uint32 view of the weight and m is the LSB mask.Source: Bit Manipulation in FP32 Weights · p. 15
• The operations are fully vectorizable on contiguous FP32 parameters, taking O(N) time and O(1) auxiliary state per tensor.Source: Bit Manipulation in FP32 Weights · p. 15

Budget Definition

• The bit-level budget fixes the total modified bits B, so changing x changes the number of modified floats while keeping the bit count fair across x values.Under a bit-level budget, K = ⌊B/x⌋ floats are modified; a float-level budget instead touches K · x bits.Source: Budget Definition · p. 15

Additional Fidelity Metrics

• GrayShield’s average W1 is about 25× smaller than PM and about 42× smaller than PTQ, despite PM having a negligibly higher average Hamming distance.The comparison concerns evaluated near-chance defenses.Source: Additional Fidelity Metrics · p. 15

Reproducibility

• The replication package includes code, configuration grids, payload data, metadata, result artifacts, and provenance information for reproducing the main figures and tables.Its results bundle separates the refreshed RQ1 rerun from completed RQ2–RQ4 results and traces published results to source runs.Source: Reproducibility · pp. 15–17

Additional Fidelity Metrics

• GrayShield occupies the upper-left security–utility region, achieving near-chance recovery reduction with minimal accuracy degradation and low variance.Figure 6 also reports a weak relationship between recovery reduction and accuracy drop (R2 = 0.08).Source: Figure 6 · p. 16