PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifying a Training-Set Attack's Target Using Renormalized Influence Estimation

Zayd Hammoudeh, Daniel Lowd

arXiv:2201.10055v2cs.LGcs.CR

TL;DR

Targeted training-set attacks are difficult to analyze because existing influence estimators can miss the influential instances responsible for specific test errors. The paper introduces renormalized influence estimation and FIT for target and adversarial-instance identification, finding strong improvements across attack settings while supporting targeted mitigation.

  • Problem

    Existing influence estimators can systematically overlook groups of highly influential, low-loss training instances, limiting identification of attack-related examples.

    Source: §1 · p. 2, §4 · p. 4

  • Method

    The paper removes influence estimators’ low-loss penalty through renormalization, uses GAS to identify influential training instances, and extends it into FIT for anomalous target detection.

    Source: §1 · pp. 1–2

  • Results

    Renormalized estimators consistently outperform original estimators, with GAS often detecting 100% of malicious training instances with no clean-data false positives.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Target identification supports forensic analysis and targeted mitigation across poisoning and backdoor attacks, modalities, and model types.

    Source: §8 · p. 14

    Influence estimators can overemphasize globally influential, high-loss instances and underrepresent locally influential instances affecting only a few targets.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Targeted training-set attacks inject malicious instances into the training set to cause a trained model to mislabel one or more specific test instances. This work proposes the task of target identification, which determines whether a specific test instance is the target of a training-set attack. Target identification can be combined with adversarial-instance identification to find (and remove) the attack instances, mitigating the attack with minimal impact on other predictions. Rather than focusing on a single attack method or data modality, we build on influence estimation, which quantifies each training instance's contribution to a model's prediction. We show that existing influence estimators' poor practical performance often derives from their over-reliance on training instances and iterations with large losses. Our renormalized influence estimators fix this weakness; they far outperform the original estimators at identifying influential groups of training examples in both adversarial and non-adversarial settings, even finding up to 100% of adversarial training instances with no clean-data false positives. Target identification then simplifies to detecting test instances with anomalous influence values. We demonstrate our method's effectiveness on backdoor and poisoning attacks across various data domains, including text, vision, and speech, as well as against a gray-box, adaptive attacker that specifically optimizes the adversarial instances to evade our method. Our source code is available at https://github.com/ZaydH/target_identification.

1 INTRODUCTION

The paper introduces target identification and adversarial-instance identification for targeted training-set attacks, using influence anomalies to connect malicious training instances to affected test predictions. Renormalized influence estimators address a low-loss bias, enabling attack-agnostic detection and targeted mitigation across modalities.

1 INTRODUCTION

  • Target identification finds attacked test instances, while adversarial-instance identification finds the malicious training instances that constitute the attack.Together, these tasks provide information about attacker goals and support attack mitigation.Source: §1 · p. 1
  • Limited attack budgets require malicious instances to be highly influential, so targets exhibit unusually many strong training-example influences compared with non-targets.This insight links influence estimation with anomaly detection to solve both identification tasks jointly.Source: §1 · p. 1
  • Existing influence estimators can overlook influential low-loss instances because their low-loss penalty favors confidently predicted training examples as uninfluential.The paper removes this penalty through a simple renormalization.Source: §1 · pp. 1–2
  • GAS, the paper’s renormalized influence estimator, often detects 100% of malicious training instances with no clean-data false positives.Renormalized estimators consistently outperform their original versions in adversarial and non-adversarial settings.Source: §1 · p. 2
  • FIT mitigates attacks by removing exceptionally influential training instances associated with detected targets, leaving training outliers that help non-targets unaffected.The target-driven strategy has a positive or neutral effect on clean data while remaining effective against adversarial data.Source: §1 · p. 2
  • GAS and FIT are attack agnostic across poisoning and backdoor attacks and apply to text, vision, and speech models, including against an adaptive attacker.The approach relies on influence estimation rather than attack-specific or modality-specific properties.Source: §1 · p. 2
  • Target identification can reduce the broad clean-data damage of defenses that affect all predictions, which can cause up to 10× more clean-data errors.The motivation is to focus countermeasures on the few attacked targets rather than limiting every training instance’s influence.Source: §1 · p. 2

2 PROBLEM FORMULATION

The paper formalizes training examples, model outputs, optimization iterations, per-example losses, and predicted labels for a priori unknown test instances. These definitions establish the notation used to compute training-set influence.

2 PROBLEM FORMULATION

  • The training set Dtr contains n examples z_i = (x_i, y_i), while model f maps feature vectors to parameterized output activations.The initial parameters θ_0 may be random, pretrained, or both.Source: §2 · p. 2
  • At each iteration, a first-order optimizer updates parameters using the loss and a batch B_t of training examples.The per-example gradient is g_i^(t) = ∇_θL(z_i; θ_t).Source: §2 · p. 2
  • An unknown test instance uses the model’s final predicted label as its available label, which may differ from the instance’s true label.The notation distinguishes examples and gradients evaluated with predicted rather than true labels.Source: §2 · p. 2

Threat Model

The threat model separates adversarial training instances from clean data and allows poisoning or backdoor attacks that mislabel selected targets while preserving clean-data performance. The defender analyzes potential targets without assuming a modality, perturbation type, or clean validation set.

Threat Model

  • The attacker crafts D_adv within D_tr, with D_cl = D_tr \ D_adv denoting the clean training set.Only successful attacks are considered.Source: Threat Model · p. 2
  • Poisoning attacks perturb adversarial training instances while targeting benign test features, whereas backdoor attacks perturb both training and targeted test features using a trigger.Backdoor attacks are generally multi-target, and adversarial labels may also be changed.Source: Threat Model · p. 2
  • The attacker may know pretrained parameters, training hyperparameters, and the clean dataset but not the training procedure’s random seed.Consequently, attacks must remain robust to batch-ordering and initialization randomness.Source: Threat Model · p. 2
  • The defender seeks to identify attack targets and remove associated adversarial instances without assumptions about modality, domain, or perturbation, and without clean validation data.The analyzed test set contains potential targets selected for concern by the defender.Source: Threat Model · pp. 2–3

3 RELATED WORK

Existing defenses generally lack target identification and often assume particular attacks, modalities, architectures, or training procedures. The paper instead builds on influence estimation, comparing training instances’ responsibility for test predictions and introducing renormalized estimators for identifying adversarial instances.

3.1 Defenses Against Training-Set Attacks

  • Existing certified and empirical defenses differ in guarantees and side effects, while no known defense provides target identification for training-set attacks.Related backdoor defenses may ignore poisoning attacks or assume clean data, attack types, or training methodologies.Source: §3.1 · p. 3
  • Data sanitization faces a threshold trade-off: removing too little leaves attacks successful, while removing too much degrades clean-data performance.Target identification can provide information for tuning removal thresholds.Source: §3.1 · p. 3
  • Most existing defenses are not attack agnostic, whereas influence estimation offers a general basis for handling different modalities, architectures, optimizers, and training paradigms.The paper’s defense is largely empirical rather than certified.Source: §3.1 · p. 3, §3 · p. 3

3.2 Training-Set Influence Estimation

  • Influence estimation quantifies each training instance’s relative responsibility for a test prediction and is often interpreted through the counterfactual effect of removing that instance.The paper reviews static and dynamic estimators, including influence functions, representer points, TracIn, and TracInCP.Source: §3.2 · p. 3
  • Static estimators use final parameters, whereas dynamic estimators aggregate effects across training iterations and parameter updates.TracIn traces gradient-descent changes, while TracInCP evaluates all training examples at selected checkpoints.Source: §3.2 · pp. 3–4
  • Renormalized influence estimators outperformed their unnormalized versions when ranking MNIST adversarial training instances by influence on MNIST test instances, improving AUPRC by up to 25×.Results used 10,000 clean and 150 adversarial instances and were averaged across 30 trials.Source: §3.2 · p. 4
  • TracInCP is more computationally expensive than TracIn, with slowdown linear in the number of checkpoints per epoch.TracIn and TracInCP otherwise require only the iteration subset T as a hyperparameter.Source: §3.2 · p. 4

4 WHY INFLUENCE ESTIMATION OFTEN FAILS AND HOW TO FIX IT

The paper shows that conventional influence estimators overlook influential low-loss training groups because of a loss-driven bias, then introduces renormalization to correct this weakness. Renormalized estimators improve adversarial and non-adversarial influence identification, with layerwise variants and GAS providing strong performance across attack settings.

4.1 A Simple Experiment

  • Dynamic estimators outperformed static counterparts, yet no estimator consistently ranked the 150 adversarial MNIST instances among the five most influential despite the deliberately easy attack.The experiment used CIFAR10 frog-versus-airplane classification with 150 MNIST-zero instances labeled airplane; TracInCP was the strongest original estimator.Source: §4.1 · pp. 4–5

4.2 Why Influence Estimation Performs Poorly

  • Influence estimators overemphasize high-loss, high-gradient instances, systematically overlooking influential low-loss examples and favoring globally over locally influential data.Static estimators underperformed dynamic ones by up to an order of magnitude, while adversarial instances often had losses several orders of magnitude below clean data.Source: §4.2 · p. 5

4.3 Renormalizing Influence Estimation

  • Renormalization replaces each gradient with its unit vector, removing the low-loss penalty so influence reflects gradient or feature similarity rather than loss magnitude.The method applies broadly to existing estimators; GAS is renormalized TracInCP and computes weighted gradient cosine similarity across training.Source: §4.3 · p. 6
  • Renormalized estimators improved AUPRC by up to 25× over standard influence estimation, with GAS the strongest renormalized method and its top five instances consistently drawn from MNIST.Dynamic rescaling by the test-gradient norm also upweights confident-training iterations and limits domination by a single checkpoint.Source: §4.3 · p. 6

4.4 Renormalization & More Advanced Attacks

  • Layerwise renormalization addresses uneven gradient allocation: Conv1 and the final linear layer comprised over 50% of target gradient norm despite only 0.04% of parameters, and the dominant layer depended on the attack.Layerwise normalization preserves the low-loss correction without changing asymptotic complexity; dynamic estimators can detect adversarial alignment during training under the stated randomness assumptions.Source: §4.4 · p. 7, Figure 3 · p. 7

4.5 Renormalization & Non-Adversarial Data

  • On non-adversarial CIFAR10 filtering, layerwise renormalization was best across setups except at large filtering percentages, where GAS slightly surpassed it; GAS(-L) retained a consistent advantage over TracIn variants.Renormalized influence was at least as good as the original estimator across filtering fractions and outperformed random removal.Source: §4.5 · pp. 7–8, Figure 4 · p. 8

5 IDENTIFYING ATTACK TARGETS

FIT identifies attack targets by finding test instances whose renormalized influence vectors have anomalously heavy upper tails, then removes influential training instances associated with those targets. Its target-driven mitigation stops when the target is correctly classified, while renormalization addresses influence estimators’ tendency to overlook confidently predicted, low-loss instances.

5 IDENTIFYING ATTACK TARGETS

  • FIT identifies targets as test instances with an unusual number of highly influential training examples, then ranks their influence vectors by anomalous upper-tail heaviness.Its pipeline computes influence vectors, scores anomalies, selects the most anomalous test example, and invokes mitigation.Source: §5 · p. 8

5.1 Measuring (Renormalized) Influence

  • Renormalization removes the low-loss penalty that causes existing influence estimators to overlook groups of highly influential, low-loss training instances.GAS and GAS-L are used because they combine the renormalized estimators’ empirical performance with computational efficiency and simplicity.Source: §5.1 · p. 8

5.2 Identifying Anomalous Influence

  • Target influence distributions in vision poisoning and speech backdoor attacks contain extreme adversarial mass, whereas non-targets lack these exceptionally high-influence instances.Training-example gradient norms alone can be poorly correlated with adversarial status, motivating renormalization.Source: §5.2 · p. 8, Figure 5 · p. 9
  • FIT uses robust median-centered statistics and the Q estimator because adversarial instances are outliers and influence anomalies are one-sided.The median has a 50% breakdown point, while Q avoids MAD’s symmetry assumption and has better Gaussian efficiency.Source: §5.2 · p. 9
  • The upper-tail cutoff κ controls detectability: attacks with fewer than κ adversarial instances are much harder to detect.The cutoff is the κth largest anomaly score value.Source: §5.2 · p. 9

5.3 Target-Driven Attack Mitigation

  • Target-driven mitigation iteratively filters anomalously influential training instances, remeasures influence after removal, and stops when the target’s misprediction is eliminated.Larger cutoffs remove less clean data but may require more iterations; slowly annealing the cutoff also reduces clean-data removal.Source: §5.3 · pp. 9–10
  • Successfully identifying a target guarantees attack success rate 0 on analyzed targets, while mitigation can be adapted to multiple targets or constrained by a maximum sanitization threshold.A threshold can trigger additional human or forensic analysis if an attacker induces excessive sanitization.Source: §5 · p. 8, §5.3 · p. 10

6 EVALUATION

The evaluation spans four poisoning and backdoor attacks across text, vision, and speech, testing adversarial-set identification, target identification, and targeted mitigation. Renormalized influence methods generally outperform original estimators, identify targets accurately, and neutralize attacks while removing little clean data.

6.1 Training-Set Attacks Evaluated

  • Across four published attacks, the evaluation covers two single-target poisoning attacks and two multi-target backdoor attacks across text, vision, and speech domains.The attacks include speech and vision backdoors, natural-language poisoning, and vision poisoning on pretrained or randomly initialized models.Source: §6 · p. 10, §6.1 · p. 10
  • The evaluation is not exhaustive, and the method’s broad applicability is supported when attacks contain highly influential training examples targeting specific instances.The authors describe the selected attacks as diverse and representative rather than comprehensive.Source: §6.1 · p. 11

6.2 Identifying Adversarial Set Dadv

  • GAS and GAS-L were top performers across adversarial-set identification, nearly perfectly identifying the speech-backdoor and text-poisoning adversarial instances.For vision backdoor and poisoning, the renormalized estimators identified most adversarial instances and outperformed the four original estimators.Source: §6.2 · p. 11, Figure 6 · p. 11
  • Renormalization improved static influence estimators by up to 600×, while layerwise renormalization improved influence-function adversarial-set AUPRC by up to 3.5×.Layerwise normalization addresses cases where large-magnitude layers dominate influence and GAS estimates.Source: §6.2 · p. 11, Figure 8 · p. 12

6.3 Identifying Attack Targets

  • FIT with GAS or GAS-L achieved near-perfect target identification for both backdoor attacks and natural-language poisoning, with GAS-based FIT exceeding 0.8 mean AUPRC on vision poisoning.For the single-target vision-poisoning attack, mean AUPRC above 0.8 corresponds to an average target rank better than 1.25.Source: Figure 7 · p. 11, §6.3 · p. 12
  • FIT’s natural-language target-identification AUPRC changed only 0.2% with GAS-L and 2.1% with GAS across κ values from 1 to 25.This indicates stable performance across a broad range of upper-tail cutoff thresholds.Source: §6.3 · p. 12

6.4 Target-Driven Mitigation

  • Target-driven mitigation removed generally less than 0.2% of clean training data while preserving or improving clean test accuracy for three of four attacks.For Weber et al.’s vision backdoor, accuracy degradation was 0.1%; on Zhu et al.’s vision poisoning, at most 0.03% of clean data was removed.Source: §6.4 · p. 12, Table 1 · p. 12
  • Sanitizing one target sharply reduced attack success on other targets, lowering speech-backdoor ASR from 100% to 4.7% with GAS and 6.5% with GAS-L.For the vision backdoor, ASR fell from 90.5% to 11.9% with GAS and 6.7% with GAS-L.Source: §6.4 · p. 12

7 ADAPTIVE ATTACKS

The paper evaluates FIT against attackers that jointly optimize attack effectiveness and GAS influence, finding that the defense remains effective despite adaptive optimization. These experiments also quantify computational and setup constraints affecting the adaptive evaluations.

7 ADAPTIVE ATTACKS

  • The adaptive attack jointly minimizes adversarial loss and estimated GAS influence using surrogate models, with λ balancing the two objectives.The poison perturbations remain constrained to an ℓ∞ ball, and the full optimization details are deferred to Section D.Source: §7 · p. 13
  • Adaptive evaluation reduced the poison size to 40 and surrogate checkpoints to four because jointly optimizing surrogate GAS had much higher GPU memory cost.These adjustments reduced the attacker’s success rate from 77.9% to 64.3%.Source: §7 · p. 13

F.2.1 Adversarial-Set Identification of the Jointly Optimized Poisoning Attack

  • 0.68 AUPRC remained for GAS adversarial-set identification, down 7% from 0.73 after the attacker added a surrogate GAS objective.The jointly optimized attack reduced GAS similarity by 7%.Source: §7 · p. 13, Figure 9 · p. 13

F.2.2 Target Identification of the Jointly Optimized Poisoning Attack

  • 0.78 AUPRC remained for FIT with GAS after joint optimization, down 9% from 0.86 while the target’s average rank changed from 1.16 to 1.28.Because Zhu et al.’s attack is single-target, the rank remains high despite adaptation.Source: Figure 10 · p. 13, §7 · p. 14

F.2.3 Target-Driven Attack Mitigation of the Jointly Optimized Poisoning Attack

  • Target-driven mitigation removed at most 0.05% of clean data on average, while post-mitigation test accuracy improved or stayed unchanged except for one 0.1% decrease.The authors summarize the adaptive experiments as successfully identifying both adversarial instances and the target before mitigation.Source: §7 · p. 14, Table 2 · p. 14

8 DISCUSSION AND CONCLUSIONS

The discussion frames target identification and adversarial-instance identification as complementary tasks for understanding and stopping training-set attacks. It presents renormalized influence as the key enabling improvement while leaving certified integration open.

8 DISCUSSION AND CONCLUSIONS

  • Target identification supports risk analysis, forensic reasoning about attacker identity, and defenses tailored to the attacker’s objective.The paper notes that combining target identification with certified guarantees remains an open question.Source: §8 · p. 14
  • The paper treats target identification and adversarial-instance identification as related tasks within protection against training-set attacks.The two tasks respectively identify the attacked test instances and the malicious training instances.Source: §8 · p. 14
  • Renormalized influence addresses influence’s low-loss penalty and improves estimation in adversarial and non-adversarial settings, often by an order of magnitude or more.FIT depends on identifying groups of highly influential training instances.Source: §8 · p. 14

B Proof

The supplied material is an appendix table of contents spanning proof material and experimental-configuration subsections. It identifies dataset-size and hyperparameter sections but does not provide their substantive contents.

B Proof

  • The appendix lists a section titled “B Proof” followed by detailed experimental setup material.The supplied passages contain headings rather than theorem statements or proof arguments.Source: B Proof · p. 17, §C.1.1 · p. 17
  • The supplied appendix excerpts do not state numerical training-set sizes, target-set sizes, or proof conclusions.Only section titles and page references are present in these passages.Source: B Proof · p. 17, §C.1.1 · p. 17, §C.2.1 · p. 17

C.1.1 Training Set Sizes

C.2.1 Model Training

The supplied appendix material covers experimental setup, algorithm summaries, additional results, runtime complexity, and a theorem proof. It also records the software environment and evaluation resources used by the paper.

F.1.1 Speech Recognition Backdoor Full Results

  • The appendix lists full results for speech backdoors, vision backdoors, natural-language poisoning, vision poisoning, and adaptive attacks.Additional sections cover target identification, mitigation, ablations, poisoning-rate sensitivity, and execution time.Source: §F.1.1 · p. 17, §C.2.1 · p. 17, §F.2.1 · p. 17

A ADDITIONAL ALGORITHMS

  • GAS and TracInCP produce influence vectors from training parameters, training data, batches, and a test example, while TracIn uses parameter checkpoints and iteration subsets.The appendix presents these procedures for side-by-side comparison.Source: A ADDITIONAL ALGORITHMS · p. 19
  • FIT ranks test examples by influence-distribution heaviness and sanitizes the model and training set for each identified target.The implementation takes a training set, test examples, and upper-tail count κ as inputs.Source: A ADDITIONAL ALGORITHMS · p. 19

A.1 Runtime Complexity of FIT

  • Computing FIT influence over all training and test examples costs O(|θ|n|T|m), which is its slowest practical component and grows linearly with both dataset sizes.The authors identify heuristic pruning of training and test examples as a future acceleration direction.Source: §A.1 · p. 19
  • Mitigation repeats model training and influence estimation for selected targets until the predicted label changes, with re-estimation cheaper than the initial all-test computation because l < m.The stated costs are O(|θ|nTl) for retraining and O(|θ||T|l) for influence re-estimation.Source: §A.1 · p. 19

Proof of Theorem 4.1

  • Theorem 4.1 assumes a twice-differentiable, strictly convex loss that is either even or monotonically decreasing.The proof handles the monotonically decreasing and even cases separately.Source: Proof of Theorem 4.1 · p. 20

C DETAILED EXPERIMENTAL SETUP

  • The detailed experimental setup covers datasets, hyperparameters, architectures, automatic differentiation, and Python 3.6.5.The experiments use PyTorch, and the paper provides source-code references for the implementation and a poisoning benchmark.Source: C DETAILED EXPERIMENTAL SETUP · p. 20

C.1 Dataset Configurations

The experiments span image, text, and speech attacks, with configurations covering datasets, target sets, model training, mitigation, baselines, and architectures. Hyperparameters and evaluation settings are specified for each experimental component.

C.1 Dataset Configurations

  • The evaluation covers CIFAR10 vision classification, SST-2 sentiment analysis, and speech digit-pair backdoors across multiple attack settings.Vision experiments use CIFAR10 binary class pairs; language experiments use eight SST-2 reviews and 50 poisons per trial; speech experiments use fixed backdoor digit-pair datasets.Source: §C.1 · pp. 20–21, §C.1.1 · p. 21, §C.2.3 · p. 24

C.1.2 Target Set Sizes.

  • Target-identification experiments define target and non-target sets separately, with class imbalance determining the random-label AUPRC baseline.The target and non-target set sizes are detailed for Section 6.3, and the class imbalance ratio determines the baseline AUPRC for random labeling.Source: §C.1.2 · p. 21, Table 6 · p. 21

C.2 Hyperparameters

  • Model-training, influence-estimation, attack, and mitigation hyperparameters are specified in separate tables, including the upper-tail cutoff κ and filtering cutoff ζ.The mitigation cutoff is initialized and decreased after a specified number of iterations until the attack is mitigated.Source: §C.2 · p. 22, §C.2.2 · p. 22, Table 7 · p. 22

C.2.3 Target-Driven Mitigation Hyperparameters.

  • The attack configurations include Weber et al.’s three CIFAR10 trigger patterns, Liu et al.’s white-noise speech trigger, and Wallace et al.’s bilevel natural-language poisoning.The vision triggers are one-pixel, four-pixel, and blend patterns; the speech dataset is used unchanged; and the language attack iteratively replaces tokens using poison gradients.Source: §C.2.3 · pp. 23–24

C.2.5 Baselines.

  • Baselines include influence functions, TracIn, k-nearest-neighbor methods, and Deep k-NN, with neural-model-specific methods selected for comparison.Influence functions require tuning to avoid numerical instability, while HyDRA is excluded from the main comparison because its O(n|θ|) memory complexity is impractical for large models and datasets.Source: §C.2.5 · pp. 24–25

C.3 Network Architectures

  • The evaluated architectures are ResNet9 for CIFAR10, RoBERTaBASE for language poisoning, and a modified convolutional network for speech classification.The speech model replaces dropout with batch normalization and halves convolutional kernel counts for single-GPU training.Source: §C.3 · p. 25, Table 15 · p. 26, Table 16 · p. 27

D CONVEX POLYTOPE POISONING & GAS JOINT OPTIMIZATION

This section examines convex-polytope poisoning and its adaptive joint optimization against GAS, including practical constraints, attack-success trade-offs, examples, and evaluations across speech, vision, and language settings. It also assesses GAS(-L) under a gray-box adaptive vision-backdoor attacker, noting the exclusion of a speech attack because its dataset provided author-selected clean seed instances.

D CONVEX POLYTOPE POISONING & GAS JOINT OPTIMIZATION

  • The convex-polytope attack constructs poison instances whose feature representations surround a target, succeeding under the stated linear-classifier assumptions or when penultimate representations form a convex hull.Zhu et al. optimize this representation iteratively across surrogate models because unknown training randomness and model-parameter feedback create a cyclic dependency.Source: D CONVEX POLYTOPE POISONING & GAS… · p. 28
  • Optimizing against renormalized GAS requires Θ(|θ|^2) automatic-differentiation memory, increasing GPU VRAM requirements by ≥12× and limiting joint optimization to four surrogate checkpoints.With adversarial sets larger than 40, joint optimization exceeded the 11.5GB GPU capacity, making the resulting GAS estimate coarser.Source: D CONVEX POLYTOPE POISONING & GAS… · pp. 28–29

F.1 Full Experimental Results

  • Full experiments report that GAS and GAS-L always outperformed baselines for speech backdoor adversarial-set identification, while GAS-L perfectly identified all poison in all but one language-poisoning trial.For vision poisoning, GAS and GAS-L outperformed every baseline across all class pairs using only initial parameters and five subepoch checkpoints.Source: Table 18 · p. 32, Table 24 · p. 36, Table 27 · p. 38

F.2.1 Adversarial-Set Identification of the Jointly Optimized Poisoning Attack.

F.2.3 Target-Driven Attack Mitigation of the Jointly Optimized Poisoning Attack.

  • Target-driven mitigation removed only 0.05% of clean data with GAS and 0.03% with GAS-L, while average test accuracy improved or stayed unchanged in all but one case.These results were comparable to Zhu et al.’s [84] standard attack, which removed 0.02% and 0.03% of clean data with GAS and GAS-L, respectively.Source: §F.2.3 · p. 42, Table 32 · p. 42

F.3 Adaptive Adversarial-Instance Selection

  • GAS(-L) remained highly effective against adaptive seed selection, achieving 0.925–0.947 AUPRC despite a 5–8% decline versus the non-adaptive baseline.The attacker selected the 150 least influential instances from 2,000 candidates using a gray-box surrogate matched to the target’s architecture, hyperparameters, initialization, and clean data.Source: §F.3 · p. 43
  • GAS(-L) still achieves consistently high adversarial-set identification performance against adaptive vision backdoor attacks that conceal Dadv using clean or backdoored surrogate models.Results are averaged across related setups with at least five trials per setup, using uniform-random selection as the baseline.Source: Table 33 · p. 43
  • Liu et al.’s speech recognition attack was excluded because its dataset supplied clean seed instances pre-selected by the authors.The attack also uses a fixed adversarial trigger.Source: §F.3 · p. 43

F.4 An Adversarial Attack on Target-Driven Mitigation

The paper evaluates whether a reformulated backdoor attack can exploit target-driven mitigation as an availability attack. The reformulation can trigger harmful sanitization, motivating intervention thresholds and additional analysis.

F.4 An Adversarial Attack on Target-Driven Mitigation

  • A targeted vision backdoor becomes an availability attack when its trigger is inserted into y_adv rather than y_targ test instances.The reformulated attack leaves D_adv and training unchanged but increases confidence in correct predictions, potentially causing mitigation to flip them incorrectly.Source: §F.4 · p. 44
  • GAS(-L) identifies adversarial training instances as influential whether the trigger affects y_targ or y_adv test instances, with larger differences for the one-pixel pattern.The comparison uses auto as y_targ, dog as y_adv, 150 backdoor instances, and more than 15 trials per setup.Source: §F.4 · p. 44, Figure 25 · p. 45, Table 34 · p. 45
  • False target identification occurs in 5–20% of cases depending on attack pattern, allowing a persistent adversary to retry with different backdoored y_adv instances.For the blend pattern, auto instances had higher anomaly scores by 1.0Q for GAS and 1.3Q for GAS-L on average, while some dog instances exceeded them by up to 1.2Q and 5.3Q.Source: §F.4 · p. 44
  • Misclassifying a reformulated attack instance as a target can remove multiple orders of magnitude more clean data and reduce average clean test accuracy by up to 21%.The attack therefore exploits the mitigation procedure rather than merely evading target identification.Source: §F.4 · p. 44, Table 35 · p. 45
  • The proposed safeguard is to cap sanitization and initiate human or forensic intervention when mitigation would exceed a domain-specific threshold.An exponential-search variant reduces execution time, but its results may underestimate the reformulated attack’s severity.Source: §F.4 · p. 44

F.5 Poisoning-Rate Ablation Study

This section examines the stability and effectiveness of GAS and GAS-L across poisoning rates and settings, including adversarial-instance identification, target identification, and mitigation. It also considers robustness to concealed influential instances, computational costs, gradient aggregation, tail-heaviness, and low-influence attacks.

F.5 Poisoning-Rate Ablation Study

  • GAS and GAS-L remained remarkably stable across tested poisoning rates for adversarial-set identification, target identification, and attack mitigation, with evaluation restricted to attacks achieving ≥50% ASR where specified.The ablations covered four attacks for adversarial-set identification, emphasizing Zhu et al.’s [84] vision poisoning and Liu et al.’s [47] speech backdoor attacks.Source: §F.5 · p. 46, §F.5.1 · p. 47, §F.5.2 · p. 48

F.5.1 Adversarial-Set Identification Ablation Study.

  • Across Figures 26–29, GAS(-L) stably identified adversarial sets across poisoning rates, often outperforming Section 6 results, while TracInCP varied substantially for natural-language and vision poisoning.The reference tables report means and standard deviations for speech and vision backdoors, Wallace et al.’s [73] natural-language poisoning, and Zhu et al.’s [84] vision poisoning.Source: §F.5.1 · p. 47, Figure 28 · p. 47, Table 38 · p. 51

F.5.2 Target-Identification Ablation Study.

  • Target identification remained stable across poisoning rates for Liu et al.’s speech backdoor and Zhu et al.’s [84] vision poisoning attacks.For speech with fewer than 10 adversarial instances, κ was set to min{10,|Dadv|}; for vision poisoning, increasing κ improved performance as poisoning rate increased.Source: §F.5.2 · p. 48, Figure 30 · p. 48, Figure 31 · p. 48

F.5.3 Target-Driven Attack Mitigation Ablation Study.

  • For Zhu et al.’s [84] vision poisoning, mitigation removed ≤0.26% of clean data and reduced post-sanitization test accuracy by no more than 0.2%.Across poisoning rates, the fraction of clean data removed was slightly larger when the adversarial set was small; speech and vision attacks were neutralized with few removals and little accuracy change.Source: §F.5.3 · p. 49, Table 36 · p. 49, Table 37 · p. 50

F.6 Upper-Tail Heaviness Hyperparameter 𝜅Sensitivity Analysis

  • GAS and GAS-L target-identification AUPRC varied only 0.2% and 2.1% for natural-language poisoning and 1.8% and 5.8% for vision backdoor across tested κ ranges.The reported ranges were κ∈[1,25] for natural-language poisoning and κ∈[2,50] for vision backdoor.Source: §F.6 · p. 54, Figure 32 · p. 54

F.7 Alternatives to Renormalization

  • Renormalizing TracInCP using loss-only or full-gradient norms produced comparable backdoor performance, while loss-only normalization led joint-training experiments and GAS led natural-language poisoning.Loss-only normalization achieved near-perfect 0.998 mean AUPRC in the CIFAR10-and-MNIST joint-training experiment.Source: §F.7 · p. 55, Figure 33 · p. 55

F.8 Why Aggregate Training Gradients?

  • Aggregating gradients across checkpoints outperformed the single best checkpoint on average, because adversarial instances can become influential at different training points.The final checkpoint was better for two of three attacks, whereas the initial checkpoint was better for vision poisoning; this was unrelated to pretraining.Source: §F.8 · p. 56, Figure 34 · p. 56, Table 44 · p. 57

F.9 Evaluating on the Tail of the Attack’s Influence Distribution

  • GAS(-L) remained highly effective on the least influential 10% of attack instances, whereas baseline estimators’ AUPRC generally fell nearly to zero.The evaluation halved maximum perturbation distances and used 1,000 adversarial instances, simulating concealment of the top 90% most influential instances.Source: §F.9 · p. 58, Figure 35 · p. 58
  • Even after concealing the top 90% of influential instances, GAS(-L) remains highly effective on the least influential 10% for natural-language poisoning and speech backdoor attacks.The concealed sets comprise the top 45/50 natural-language poisoning instances and top 27/30 speech backdoor instances, leaving the bottom 5/50 and 3/30 for identification.Source: Figure 36 · p. 58, Figure 37 · p. 58

F.10 Adversarial-Set Identification Execution Time

  • GAS(-L) has essentially TracInCP’s execution time, while amortization can substantially accelerate analysis but is constrained by GPU memory and available parallel test examples.The timing comparison analyzes a single test instance across four attacks; more modern GPUs with larger memory can provide further speed-ups.Source: §F.10 · p. 59, Table 47 · p. 59, Table 48 · p. 59
  • For natural-language poisoning of RoBERTaBASE, the two-phase target-identification procedure avoids a meaningful performance drop despite the model’s 125M parameters.The streamlined approach is used because large models can be slow to analyze, even with amortization.Source: §F.10 · p. 59
  • The representer point estimator was fastest because it analyzes only the final linear classification layer, but it also had the worst performance.Peri et al.’s [55] Deep k-NN defense had a mean execution time of 242s with a 1.1s standard deviation.Source: §F.10 · p. 59

Related papers

Loading 2201.10055v2…