Source-linked AI summary
Blind Backdoors in Deep Learning Models
Eugene Bagdasaryan, Vitaly Shmatikov
TL;DR
The paper addresses whether compromised training code can inject effective backdoors without access to data, training execution, or the resulting model. It compromises loss computation, synthesizes poisoning inputs during training, and balances conflicting tasks with multi-objective optimization. The blind attack demonstrates several powerful backdoor classes and evades known defenses, while motivating trusted computational-graph defenses and exposing code-review and overhead-detection limitations.
Problem
The problem is whether training code alone can inject backdoors when the attacker cannot access training data, observe execution, or inspect the resulting model.
Method
The attack synthesizes poisoning inputs during loss computation and uses multi-objective optimization to balance conflicting main-task, backdoor-task, and defense-evasion objectives.
Results
The attack demonstrates single-pixel and physical ImageNet backdoors, covert alternative functionality, semantic backdoors without inference-time input modification, and evasion of known defenses.
Takeaways & Limitations
Trusted computational graphs and review of code changes are proposed as defenses against code-poisoning backdoors.
Takeaways & Limitations
Overhead-based detection requires stable baselines for particular hardware and training configurations, which may not be available.
Abstract
from arXiv · showhide
We investigate a new method for injecting backdoors into machine learning models, based on compromising the loss-value computation in the model-training code. We use it to demonstrate new classes of backdoors strictly more powerful than those in the prior literature: single-pixel and physical backdoors in ImageNet models, backdoors that switch the model to a covert, privacy-violating task, and backdoors that do not require inference-time input modifications. Our attack is blind: the attacker cannot modify the training data, nor observe the execution of his code, nor access the resulting model. The attack code creates poisoned training inputs "on the fly," as the model is training, and uses multi-objective optimization to achieve high accuracy on both the main and backdoor tasks. We show how a blind attack can evade any known defense and propose new ones.
1 Introduction
The paper introduces code poisoning as a blind backdoor attack that compromises loss computation, synthesizes poisoning inputs during training, and balances conflicting objectives. It demonstrates powerful backdoors and defense evasion while identifying code review and testing as defenses against the attack code.
- Threat and contribution: Code poisoning injects backdoors by compromising machine-learning training code rather than directly modifying training data or model weights.The attack targets code incorporated through open-source, proprietary, build, and integration components.
- Threat and contribution: The attack is blind because the attacker cannot access the training data, observe code execution, or inspect the resulting model or training outputs.This constrains the attacker to code prepared before training begins.
- Attack method: Poisoning inputs are synthesized on the fly during loss computation, while multi-objective optimization balances main-task and backdoor-task accuracy.The method uses Multiple Gradient Descent Algorithm with the Franke-Wolfe optimizer to find a self-balancing loss function.
- Demonstrated capabilities: The attack injects single-pixel and physical ImageNet backdoors, covert privacy-violating functionality, and semantic backdoors requiring no inference-time input modification.Examples include covertly recognizing individuals and classifying reviews containing a particular name as positive.
- Defense implications: A blind attacker can evade previously proposed backdoor defenses by incorporating defense evasion into the loss computation.The paper analyzes defenses based on input perturbation, behavioral anomalies, and outlier suppression.
- Defense implications: The paper reports attack overhead and discusses defenses including certified robustness and trusted computational graphs.The proposed defenses target robustness or deviations from trusted computation rather than only model behavior.
2 Backdoors in Deep Learning Models
This section defines deep-learning backdoors as models that retain their main task while supporting an attacker-injected task, potentially without inference-time input modification. The broader formulation allows non-universal triggers and backdoor behavior beyond a single attacker-chosen label.
- Model and training setup: A trained neural network computes loss from model outputs and labels, then updates parameters using backpropagated gradients.The formulation applies to supervised learning over inputs and labels, including criteria such as cross-entropy and mean square error.
- Backdoor formulation: A backdoored model supports an adversarial task alongside its original task and switches between them when a backdoor feature is present.The two tasks use the same output space, so backdoor labels form a subdomain of the main label space.
- Backdoor formulation: The paper broadens prior backdoors by allowing triggers that need not be applied at inference time and backdoors that need not be universal.Thus, trigger presence can determine behavior without forcing one identical output across all triggered inputs.
- Notation: The notation section introduces the symbols used to describe tasks, inputs, labels, and backdoor behavior.The supplied notation passage is a table caption without the table contents.
2. Backdoor task m∗: θ∗(x∗) = y∗, ∀(x∗,y∗) ∈(X ∗,Y ∗)
The paper characterizes backdoor triggers through input domains, predicates, and synthesizers, covering modified pixel or physical inputs, semantic triggers, and multiple task-specific backdoors. This formulation extends prior work beyond universal, single-label attacks and beyond inference-time input modification.
- Trigger domains and synthesizers: A backdoor-trigger predicate identifies inputs containing the feature, while a synthesizer can create modified inputs for pixel-pattern or physical backdoors.Semantic triggers instead occur naturally in inputs already belonging to the original input domain.
- Inference-time modification: Conventional pixel-pattern backdoors require the attacker to modify the input at inference time, even when the modification is visually small.The transformation may flip, swap, squeeze, or color pixels before inference.
- Multiple backdoors: A single model can support multiple backdoors and switch among their corresponding tasks according to the backdoor features present in an input.The paper represents these backdoors with separate synthesizers and task mappings.
- Physical and semantic triggers: Physical backdoors can be triggered by real objects in a scene without modifying the digital input.This distinguishes them from physical adversarial examples involving artificially generated objects.
- Physical and semantic triggers: Semantic backdoor features can occur in unmodified digital or physical inputs, but their domain must remain small because the main and backdoor tasks conflict there.Examples include word combinations in sentences and rare object colors in images.
- Beyond universal single-label attacks: The broader definition permits backdoors that behave differently across classes or switch the model to an entirely different functionality.A synthesizer maps an input and its correct label to the backdoor label when the trigger predicate holds.
3 Blind Code Poisoning
The paper defines blind code poisoning as compromising loss computation without access to the specific training data, execution, or resulting model. It synthesizes backdoor inputs during training and uses multi-objective optimization to balance conflicting main-task, backdoor, and defense-evasion objectives.
- 3.1 Threat model: Industrial ML pipelines incorporate open-source, commercial, and proprietary code, making compromised training code a realistic attack vector.The paper notes that recent high-visibility attacks demonstrated the realism of compromised code as a threat.
- 3.1 Threat model: Loss-value computation is a viable poisoning target because it receives the model, inputs, labels, and task-specific training information.The loss normally compares model predictions with expected labels using a criterion such as cross-entropy.
- 3.1 Threat model: A blind attacker knows the task, possible architectures, and general data domain, but not the specific data, hyperparameters, execution, or resulting model.The attack leaves the rest of the codebase unchanged and cannot apply gradients through the unavailable training optimizer.
- 3.3 Backdoors as multi-task learning: The attack trains one single-output model for conflicting main and backdoor tasks, where fixed loss coefficients cannot be selected in advance.The coefficients depend on the data and model, and no fixed combination necessarily optimizes both objectives.
- 3.4 Learning for conflicting objectives: The malicious code synthesizes backdoor inputs and labels during training, then uses MGDA with a Franke-Wolfe optimizer to compute a self-balancing loss.The resulting loss can combine main-task, backdoor, and optional evasion terms while the original optimizer performs the model update.
- 3.3 Backdoors as multi-task learning: Complex synthesizers support input-specific backdoor labels, task switching, semantic triggers, and universal pixel-pattern backdoors.Universal image backdoors overlay a trigger pattern and assign the resulting input a fixed target class.
4 Experiments
The experiments use blind loss-value attacks to implant diverse backdoors while preserving strong main-task performance. Results span ImageNet, multiple computational backdoors, covert facial identification, semantic triggers, and attack-overhead measurements.
- ImageNet backdoors: ImageNet backdoors preserved 65.3% full-training main-task accuracy and achieved 99% backdoor-task accuracy across pixel-pattern, single-pixel, and physical triggers.For fine-tuning, main-task accuracy changed from 69.1% to 68.9% with a single-pixel trigger and 68.7% with a physical trigger.
- Multiple backdoors: 96% main-task accuracy was maintained while summation and multiplication backdoors reached 95.17% and 95.47% accuracy, respectively.The same model supports two triggered computational behaviors for two-digit numbers.
- Covert facial identification: The facial-identification backdoor preserved 87% face-counting accuracy and recognized four targeted individuals with 62% accuracy.The trigger switches the model from counting faces to identifying specific people.
- Semantic backdoor: The semantic backdoor preserved 91% main-task test accuracy and achieved 98% accuracy while classifying unmodified reviews containing “Ed Wood” as positive.The attacker-chosen name triggers the behavior without inference-time input modification.
- Optimization and overhead: MGDA produced at least 3% better main-task accuracy than fixed loss coefficients, while a constrained attack significantly reduced overhead.The constrained attack modified 10% of batches and replaced half the inputs in each attacked batch.
- Optimization and overhead: Overhead-based detection requires stable hardware- and hyperparameter-specific baselines that are unavailable for many popular frameworks.Batch size strongly affects memory and training time, limiting the reliability of resource-usage detection.
5 Previously Proposed Defenses
Previously proposed defenses target untrusted data or training and primarily detect pixel-pattern backdoors through input perturbations, behavioral anomalies, or gradient shaping. Their coverage is limited for semantic backdoors and for attacks that do not provide backdoored inputs to the defender.
- Input perturbation: NeuralCleanse searches for a small mask and pattern that force inputs toward an attacker-chosen label.It treats the mask and pattern as differentiable layers and minimizes the mask’s l1 size while preserving targeted misclassification.
- Input perturbation: NeuralCleanse and similar methods frame backdoor discovery as finding the smallest adversarial patch.The underlying intuition is that patches in non-backdoored models are empirically larger than backdoor triggers.
- Behavioral analysis: SentiNet identifies image regions important to classification and tests whether transferring those regions causes a consistent label.It assumes backdoored models focus on the backdoor feature and uses Grad-CAM to locate candidate regions.
- Behavioral analysis: Other anomaly defenses inspect logits, intermediate activations, or spectral representations on normal and backdoored training inputs.Unlike SentiNet, these methods require many normal and backdoored examples to train an anomaly detector.
- Gradient shaping: Gradient shaping suppresses outlier gradients with mechanisms such as DPSGD, assuming backdoor inputs are underrepresented in training data.This strategy computes clipped gradients and adds Gaussian noise to reduce the influence of such inputs.
6 Evading Defenses
The attack incorporates defense-evasion objectives into the compromised loss computation, allowing it to preserve the main task while defeating input-perturbation, attention, anomaly, and gradient-shaping defenses. Evasion can be achieved with small main-task accuracy costs, while the attack retains high backdoor accuracy.
- Overview: The attack evades defenses by adding defense-evasion objectives to the loss computation while preserving the backdoor and main tasks.The paper evaluates evasion against input perturbation, model-attention, anomaly, and gradient-shaping defenses.
- NeuralCleanse: NeuralCleanse evasion makes its synthesized mask at least as large as masks for non-backdoored models by improving robustness to adversarial perturbations.The attack trains the model to output the correct label on NeuralCleanse-style modified inputs, reducing the defense’s ability to find a small trigger.
- NeuralCleanse: NeuralCleanse cannot distinguish the evasion-trained backdoored model from a non-backdoored model, with only a 0.9% main-task accuracy drop.Without evasion, NeuralCleanse identifies the small trigger; with evasion, its mask is large, even larger than for the non-backdoored model.
- SentiNet: SentiNet evasion diverts model attention from the backdoor feature by making highlighted regions on backdoored inputs resemble those on normal inputs.This defeats SentiNet at a 0.3% main-task accuracy cost.
- Model anomalies: Representation-based anomaly defenses can be similarly evaded by enforcing similarity between backdoored and normal representations.These defenses inspect embeddings or activations, so the evasion objective targets the representation difference they rely on.
- Gradient shaping: 99% backdoor-task accuracy is achieved under gradient shaping while maintaining main-task accuracy.The attack adds its backdoor loss to every batch, so every resulting gradient contributes to backdoor injection, unlike sparse data poisoning.
7 Mitigation
The paper proposes defenses based on certified robustness and trusted computational-graph verification. Certification covers only backdoors equivalent to universal adversarial perturbations, while graph verification targets extra loss-computation operations under explicit trust assumptions.
- Certification: Certified robustness can defend against backdoors that are equivalent to universal adversarial perturbations by certifying output stability under bounded input changes.The relevant change may be measured with l0, l1, or l2 metrics.
- Certification: Certification does not cover semantic or physical backdoors and can fail when the attack uses a different metric than the defense.The paper also notes that certification can break models when some small input changes should legitimately alter outputs.
- Graph verification: Graph verification detects additional loss terms by comparing the training loss computational graph with a trusted graph.The graph captures model operations used by backpropagation, and extra forward passes for attack objectives can change it.
- Graph verification: The graph-verification defense assumes the attacker can modify only loss-computation code and cannot tamper with a trusted computational graph.The graph must be built and checked by an expert, serialized, signed, and compared during every training iteration or unit test.
- Boundary: Graph verification may be evaded if loss-computation code can update the model without changing the computational graph.The authors are not aware of an efficient way to do this while preserving main-task accuracy.
8 Related Work
Related work includes data poisoning, model poisoning, trojaning, task-switching attacks, and adversarial examples. The paper distinguishes its blind code-poisoning attack through smaller triggers, broader functionality, and backdoors that need not modify inference-time inputs.
- Backdoor attacks: Data-poisoning attacks add mislabeled samples, apply patterns to existing inputs, or add correctly labeled inputs containing backdoor patterns.These approaches assume influence over the training data.
- Backdoor attacks: Model-poisoning and trojaning attacks assume the attacker can modify the model during training and observe the result.Trojaning may analyze the model to obtain a trigger or implant a malicious module, while model-reuse attacks seek persistence through fine-tuning.
- Adversarial examples: Pixel-pattern backdoors resemble universal adversarial perturbations because both can force arbitrary inputs toward an attacker-chosen label.Unlike UAPs, pixel-pattern backdoors require changing the model, making their threat model strictly inferior in that comparison.
- Novel capabilities: Backdoors need not require inference-time input modifications, unlike the previously proposed backdoors discussed by the paper.The paper identifies limited exceptions in federated learning and subpopulation-based poisoning.
- Novel capabilities: The paper demonstrates that blind attacks can introduce single-pixel ImageNet backdoors and trigger complex or privacy-violating functionality.Prior adversarial examples that switch tasks may cover almost 90% of the image, whereas the paper emphasizes smaller backdoors.
- Adversarial examples: Both backdoors and adversarial examples add features that convince a model to produce a targeted output, but their access patterns differ.Adversarial-example attackers analyze the model to find features, whereas backdoor attackers introduce features into the model during training.
9 Conclusion
The paper demonstrates a blind code-poisoning backdoor attack that operates before training data and training outputs are available. It synthesizes poisoning inputs during loss computation, uses multi-objective optimization, and achieves powerful backdoors while evading known defenses and motivating a new defense.
- The attack compromises training code before training begins and operates without access to training data, execution, model weights, or other training outputs.Its blindness is a central property of the attack model.
- It synthesizes poisoning inputs on the fly during training and uses multi-objective optimization to maintain high accuracy on both main and backdoor tasks.
- The attack injects single-pixel, physical, covert-functionality, and inference-input-independent backdoors, evades known defenses, and motivates detection through trusted-computational-graph deviations.
A Example of a Malicious Loss Computation
The example compromises RoBERTA’s loss computation in the HuggingFace Transformers repository through a model-specific forward method while passing the repository’s unit tests.
- Algorithm 2 demonstrates an attack that compromises RoBERTA’s loss-value computation in the HuggingFace Transformers repository.
- Because Transformers uses a separate class for each model, the loss is computed inside the model’s forward method.
- The malicious code commit passes all unit tests from the Transformers repository.