Selective Channel Restoration for Backdoored Vision-Language Models

• VLMs are vulnerable to backdoors implanted through poisoned fine-tuning, while existing defenses can require extensive parameter updates or per-query computation.Applying image-classifier defenses directly to VLM projection layers can also be inefficient or harm clean utility.Source: §1 · p. 1
• PSR learns robustness scores, selects a fixed fraction of low-scoring output channels per layer, and restores them to corresponding pretrained weights and biases.The layer-wise repair is described as reducing computation and better preserving clean vision-language performance than direct pruning.Source: §1 · p. 2
• Backdoored projectors become more sensitive than clean projectors as perturbation budgets increase, a pattern termed projection fragility.Figure 1 reports loss increases under bounded projector perturbations on clean samples for COCO and VQAv2.Source: Figure 1 · p. 2, §1 · p. 2
• Experiments across multiple tasks show that PSR reduces attack success rates to near zero while preserving clean-task performance.Source: §1 · p. 2

• VLM backdoors span patch, blended, warped, sample-specific, and VLM-specific triggers, motivating defenses that suppress backdoor behavior within the model.TrojVLM preserves clean-image caption quality, while VLOOD uses out-of-distribution auxiliary data.Source: §2 · p. 2
• VLM defenses include training-time sample filtering or regularization and test-time input or computation interventions, with adaptation intervention or per-query overhead as trade-offs.Examples include BYE and RobustIT for training-time defenses and PurMM, SRD, and CleanSight for test-time defenses.Source: §2 · p. 2
• General post-training defenses use neurons, channels, representations, or weight dynamics, but direct application to VLMs may yield suboptimal performance.Examples include Fine-Pruning, Neural Cleanse, ANP, CLP, RNP, TSBD, and Spectral Signatures.Source: §2 · pp. 2–3

3.1 Threat Model

• The attacker poisons projection-interface fine-tuning data so the model behaves normally on clean images but produces a target response for triggered images.Source: §3.1 · p. 3
• The defender is assumed to have the corresponding pretrained projection parameters, a small clean calibration set, and a restoration-ratio hyperparameter, but not the original fine-tuning data or complete adaptation recipe.Source: §3.1 · p. 3

3.2 Projection Interface and Channel Representation

• PSR targets selected linear layers in the projection interface, whose output channels correspond row-wise to associated parameters.The passage identifies each output channel by an index and describes its row-wise parameter correspondence.Source: §3.2 · p. 3

4 Methodology

• PSR probes channel sensitivity, selects a fixed fraction of low-scoring channels in each target layer, restores them from the pretrained model, and leaves other adapted parameters unchanged.The method keeps model parameters frozen during scoring and concentrates repair on fragile channels while preserving task-relevant adaptation elsewhere.Source: §4 · p. 3

4.1 Channel-wise Parameterization

• Channel-wise retention coefficients θ preserve outputs near 1 and suppress them near 0, while bounded auxiliary perturbations δ are used to learn θ and then discarded.Only these channel-wise variables are optimized; model parameters remain frozen.Source: §4.1 · p. 4

4.2 Robustness Score Optimization

• Using only clean calibration data, PSR alternates between maximizing clean loss over bounded perturbations and minimizing a regularized clean-loss objective over channel scores.More perturbation-sensitive retained outputs pressure the outer minimization to lower their scores, while the unperturbed clean-loss term discourages indiscriminate suppression.Source: §4.2 · pp. 4–5

4.3 Channel Selection and Parameter Restoration

• PSR selects a fixed fraction of the lowest-scoring channels independently per layer, restores their pretrained weight rows and bias entries, and leaves unselected adapted parameters unchanged.The procedure is summarized as per-layer channel selection followed by restoration from the pretrained base VLM.Source: §4.3 · p. 5

5.1 Experimental Setup

• Experiments cover Qwen3-VL and LLaVA-1.5 on COCO captioning and VQAv2 question answering, six attacks, and clean-data and channel-selection baselines.Evaluation reports ASR and task-specific clean utility; models are adapted through their projection interfaces.Source: §5.1 · pp. 5–7

5.2 Main Results

• ASR falls to at most 1.70% in every evaluated setting across two VLM architectures, two tasks, and six attacks, while clean utility remains close to undefended models in most settings.Tables 1 and 2 report COCO CIDEr and VQAv2 accuracy alongside ASR.Source: Table 1 · p. 6, Table 2 · p. 6, §5.2 · p. 7

5.3 Ablation Studies

• At r = 100%, ASR is 0.00% for all four attacks, but clean CIDEr falls to 68.84 from 123.03–126.09 at r = 30%.Across ratios from 10% to 50%, increasing restoration consistently lowers ASR while clean CIDEr declines gradually.Source: §5.3 · p. 7, Figure 3 · p. 8
• Across all four attacks, restoring both groups reduces ASR to 0.04–1.70%, indicating that the backdoor signal is distributed across the merger interface rather than concentrated in a single layer.Source: §5.3 · p. 8
• Restoring selected weights and biases from ϕ0 yields 0.04–1.70% ASR and 123.03–126.09 CIDEr, while zeroing them yields 0.00% ASR and 97.37–118.28 CIDEr.For BadNet, restoration from ϕ0 gives CIDEr 124.42 versus 107.89 when setting the parameters to zero.Source: Table 3 · p. 7, §5.3 · p. 8

5.4 Robustness to Adaptive Attacks

• Against perturbation-aware, persistence-augmented, and restoration-aware training, PSR reduces ASR to near zero in most settings; resisting the probe does not necessarily resist purification.All adaptive models are purified using the standard PSR configuration.Source: §5.4 · p. 8, Table 4 · p. 9

6 Conclusion

• PSR restores selected projection rows and biases to pretrained values using clean-data robustness scores, without intervention during adaptation or inference.The scores are based on projection fragility: backdoored models show larger clean-loss increases under bounded projection perturbations.Source: §6 · p. 9
• Across two VLM architectures, two multimodal tasks, and six attack types, PSR substantially suppresses attack success while generally retaining clean-task utility.The conclusion frames these findings as support for clean-data robustness scoring and restoration to pretrained values under the evaluated threat model.Source: §6 · p. 9

Appendix Contents

• The appendix covers detailed experimental settings, method details and supporting experiments, and limitations and scope.Its listed sections include models, datasets, attacks, defense baselines, the projection-fragility probe, and hyperparameters.Source: Appendix Contents · p. 12

A.1 Models

• Qwen3-VL-8B-Instruct uses a high-resolution visual encoder and a projection interface with one visual merger and three DeepStack mergers that inject visual information at multiple depths.The experiments target output channels of the two affine layer families in every merger.Source: §A.1 · p. 12
• LLaVA-1.5-7B uses a CLIP ViT-L/14 visual encoder at 336px, a two-layer MLP projector, and the Vicuna-7B language model.The projector alone is updated during attack training; both affine layers are targeted.Source: §A.1 · p. 12
• The two architectures test the same defense on LLaVA’s compact projector and Qwen3-VL’s distributed merger interface.For both models, attack training keeps the visual encoder and language model frozen while updating the projection interface.Source: §A.1 · p. 12

A.2 Datasets

• MS-COCO’s 2017 split has approximately 118,000 training images and 5,000 validation images, and captioning performance is measured by corpus-level CIDEr.The benchmark contains over 200,000 labeled images spanning 80 object categories, with five human-written captions per image.Source: §A.2 · p. 12
• VQAv2 contains approximately 204,000 images and 1.1 million questions, each paired with ten crowd-sourced answers, and uses the official open-ended accuracy protocol.It reduces language priors by pairing similar images with the same question but different answers, making visual evidence more important for answering.Source: §A.2 · p. 12

A.3 Backdoor Attacks

• For five attacks, poisoned examples replace the original response with the target, whereas TrojVLM inserts the target phrase while preserving the remaining response content.The six triggers are BadNet, Blended, ISSBA, WaNet, TrojVLM, and VLOOD, shown on the same clean image in Figure 4.Source: §A.3 · p. 12, Figure 4 · p. 13
• BadNet uses a visible fixed patch, Blended uses a global low-contrast trigger, and WaNet uses an imperceptible smooth spatial deformation.The Blended implementation uses the Hello Kitty trigger.Source: §A.3 · p. 12
• ISSBA embeds invisible instance-specific signals, TrojVLM inserts a target phrase at a random response position, and VLOOD uses out-of-distribution auxiliary data and dedicated objectives.The evaluated VLOOD models use a patch trigger and replace the original response with the target.Source: §A.3 · p. 12

A.4 Defense Baselines

• All channel-selection baselines use the same per-layer restoration budget and restore selected channels to pretrained values; Clean FT instead continues clean-data fine-tuning for two epochs.Random selects channels uniformly within each target layer.Source: §A.4 · p. 13
• FP selects channels with the smallest mean absolute activation scores on clean calibration examples, then restores their weights and biases from pretrained values.FP also performs clean recovery training with SGD on the unselected target-layer channels.Source: §A.4 · p. 13
• CLP ranks channels by the Euclidean norm of each poisoned weight row, selecting the largest scores without calibration data.For the controlled comparison, CLP uses the shared fixed per-layer budget and restoration operation.Source: §A.4 · pp. 13–14

B.1 Projection Fragility Probe

• The probe tests whether poisoned projectors show larger clean-task loss increases than clean projectors under an identical perturbation budget.It compares cleanly adapted LLaVA-1.5-7B with models adapted using BadNet-, Blended-, or TrojVLM-poisoned data, on 64 clean validation examples each from COCO and VQAv2.Source: §B.1 · p. 14
• Each projector layer receives independent multiplicative perturbations to its weights and biases, with entries bounded by the selected perturbation range.The perturbations match their parameter tensors in shape.Source: §B.1 · p. 14
• The probe optimizes temporary perturbations with projected sign-gradient ascent and measures each model’s mean clean-loss increase against its own unperturbed loss.It uses 10 ascent steps at step size ϵ/5 for nonzero budgets; perturbations are discarded after each image, and the plotted budgets are 0.01, 0.02, 0.05, and 0.10.Source: §B.1 · p. 14

B.2 Hyperparameters

• The common PSR configuration uses ϵ = 0.016, two inner ascent steps, ηθ = 0.06, λθ = 0.006, λclean = 0.5, and Topt = 1,000.It uses the clean calibration set and recommends a 30% restoration ratio for a simple fixed-budget setting.Source: §B.2 · p. 14

C Limitations and Scope

• The evaluation focuses on backdoors introduced by fine-tuning the projection interface while keeping the visual encoder and language model frozen.PSR’s effectiveness against backdoors introduced through other model components was not evaluated.Source: C Limitations and Scope · p. 14
• Increasing the restoration ratio further reduces attack success but also degrades clean-task utility in the evaluated settings.This is a trade-off between backdoor suppression and clean-task performance.Source: C Limitations and Scope · p. 14
• Experiments cover two VLM architectures and two tasks, image captioning and visual question answering; extension to other architectures and tasks remains uninvestigated.Source: C Limitations and Scope · p. 14