DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization

1 Introduction

• Prior VLM poisoning targets supervised multimodal training, while prior preference attacks corrupt explicit supervision; neither jointly targets VLM-DPO with image-only poisoning and clean annotations.Table 1 compares the two lines of work across five properties.Source: §1 · p. 1, Table 1 · p. 2
• DPOISON-VL redirects VLM-DPO alignment through imperceptible image perturbations while keeping preference annotations clean and pairs semantically valid.The introduction describes the resulting poisoned examples as difficult for human annotators to detect.Source: §1 · p. 2
• The attack formulates poison construction as bilevel optimization, uses a gradient-alignment surrogate, and selects poisons with a preference displacement score.The score has provable optimality under a first-order approximation and fixed poison budget.Source: §1 · p. 2
• DPOISON-VL achieves high ASR with small poison budgets, transfers across models, and preserves clean accuracy and general utility.The paper also derives a sufficient condition for preference inversion under gradient descent.Source: §1 · p. 2

Preference Alignment for VLMs

• VLM preference-optimization studies address hallucinations, visual grounding, safety, and trustworthiness, generally assuming preference data is trustworthy.The cited studies include Xie et al. (2024), Yang et al. (2025), Yu et al. (2025), and Zhang et al. (2025, 2026).Source: §2 · p. 2

Poisoning Attacks on Preference Alignment

• Preference-alignment attacks on LLMs poison preference pairs, corrupt reward or ranking feedback, flip labels, or poison human feedback, thereby altering explicit supervision.The cited examples include Baumgärtner et al. (2024), Pathmanathan et al. (2025), Wang et al. (2024), Kusaka et al. (2026), Yang et al. (2026), and Rando and Tramèr (2024).Source: §2 · p. 2

Clean-Label Poisoning and Backdoor Attacks on VLMs

• Prior VLM attacks use clean-label image-text pairs or backdoors in supervised training, whereas DPOISON-VL targets preference alignment with imperceptible image perturbations and valid pairs.The related-work passage names Shadowcast (Xu et al., 2024), TrojVLM (Lyu et al., 2024), BadToken, and CBV (Guo et al., 2026).Source: §2 · pp. 2–3

3 Methods

• The method uses a preference example comprising an image, prompt, chosen response, and rejected response, with disjoint image splits for crafting and held-out evaluation.The base and target crafting splits serve different roles; held-out splits evaluate clean behavior and attack effectiveness.Source: §3 · p. 3

3.1 Threat Model

• The attacker contributes a small number of training preference pairs with imperceptibly perturbed images, without inference-time triggers or perturbed inputs.Pairs must remain semantically valid under human inspection, with preference annotations still correct.Source: §3.1 · p. 3

3.2 Attack Objective

• Bilevel optimization models standard victim DPO as the lower level and minimizes the target preference margin as the upper-level objective.The upper level seeks to steer the trained model toward the attacker-specified response on unperturbed target images.Source: §3.2 · p. 4

3.3 Poison Crafting

• Crafting approximates victim training with one DPO gradient step and maximizes cosine alignment between target and poison gradients.Perturbations are updated with projected gradient descent, and an ℓ2 regularization term is added for stealth.Source: §3.3 · p. 4

3.5 Poison Construction Pipeline

• The pipeline generates on-policy responses, crafts perturbations for candidate pairs, then selects the top-K candidates by preference displacement score.Semantically unambiguous responses form clean candidate and target pairs before crafting and selection.Source: Figure 1 · p. 5, §3.5 · pp. 5–6

3.4 Principled Poison Selection

• The preference displacement score ranks candidates by first-order margin reduction, making top-K selection optimal for that reduction under a fixed poison budget.The analysis relates the score to first-order preference-margin change and the marginal benefit of adding a candidate.Source: §3.4 · p. 5

Theorem 2: Preference Inversion

• Under the stated assumptions, the target preference margin becomes negative after H steps whenever initial gradient alignment exceeds the inversion threshold κ(H).The bound includes a correction term, while the alignment contributes −HηA0.Source: §4 · p. 6

Corollary 2

• Balancing the alignment contribution against the O(η^2H^2) correction yields a minimum alignment threshold in the continuous relaxation.The threshold is minimized at H⋆.Source: §4 · p. 6

Corollary 3

• For a fixed inversion threshold, a larger average poison displacement score reduces the poisoning ratio required to meet the condition.Top-K selection maximizes the average poison displacement score, and therefore initial alignment, at a fixed poison ratio.Source: §4 · p. 6

5.1 Experimental Setup

• Task I uses Biden–Trump images from Shadowcast, while Tasks A1 and A2 use filtered and manually inspected Visual Genome samples.The setup uses 200 crafting iterations and an ϵ of 8 255 unless otherwise stated.Source: §5.1 · p. 6, Table 2 · p. 7
• Experiments use four VLMs, standard DPO for two epochs with LoRA, and ASR, clean accuracy, and general-utility measures.General utility is evaluated on VizWiz and GQA.Source: §5.1 · pp. 6–7

5.2 Main Results

• With 2% poisoned preference data, DPOISON-VL achieves high ASR across four VLMs and three attack tasks.Susceptibility does not increase monotonically with model size.Source: §5.2 · p. 7, Table 3 · p. 7
• Clean accuracy remains high, and VizWiz and GQA show no substantial degradation across poison rates.The reported pattern indicates targeted misalignment while largely preserving broader capabilities.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Ablation Study

• DPOISON-VL surpasses Preference Flipping in Task A1 ASR while keeping annotations correct; gradient-based crafting drives most effectiveness, with selection adding gains.Random Perturbation remains near zero ASR across tasks, and Feature Collision performs poorly across tasks.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4 Robustness and Stealthiness Analysis

• ASR increases with poison budget, and at a 5% budget, variation across three Task I prompts is at most 8.5pp.The figure also presents ASR versus poison rate across tasks.Source: §5.4 · p. 8, Figure 2 · p. 8
• Perturbations remain visually subtle, and semantic-preservation rates are comparable across all three tasks.The analyses include perceptual stealthiness measures and representative original–poison image pairs.Source: §5.4 · p. 8, Table 5 · p. 8, Figure 2 · p. 8

6 Conclusion

• DPOISON-VL manipulates VLM-DPO alignment through imperceptible image perturbations while keeping preference annotations semantically valid.The paper presents it as the first clean preference-poisoning attack against VLM-DPO.Source: §6 · p. 8
• The pipeline combines principled poison crafting and selection, with first-order optimality and a sufficient condition for preference inversion.Source: §6 · p. 8
• Across four VLMs and three attack tasks, small poisoning budgets achieve high ASR while preserving clean behavior and general utility.Source: §6 · p. 8

• The theoretical analysis relies on standard smoothness assumptions and gradient descent dynamics to characterize preference margin changes.Source: Limitations · p. 9
• Experiments focus on standard DPO across multiple VLM architectures and attack tasks, rather than other preference optimization objectives.Source: Limitations · p. 9
• Evaluation on additional multimodal preference datasets and extensions to DPO variants or other objectives are left for future work.Source: Limitations · p. 9

Ethical Considerations

• The authors warn that clean preference poisoning could induce misleading VLM responses and evaluate image preprocessing and feature-space filtering as potential defenses.Source: Ethical Considerations · p. 9

A.2 Proof of Corollary 1

• For a fixed poison budget, the top-K candidates by preference displacement score maximize the first-order reduction in preference margin.Source: §A.2 · p. 13

A.3 Proof of Theorem 2

• If A0 > κ(H), the proof establishes that the preference margin T(θH) becomes negative under the stated condition.Source: §A.3 · p. 13

A.4 Proof of Corollary 2

• The proof identifies H⋆ as the unique global minimizer of κ(H).Source: §A.4 · p. 13

• When the average poison score ψ̄p is at most κ, the sufficient inequality fails for every poison ratio ρ > 0.Source: §A.5 · p. 14
• When ψ̄p > κ, the sufficient condition reduces to ρ > (κ − ψ̄c)/(ψ̄p − κ).Source: §A.5 · p. 14
• The condition presupposes ψ̄c < κ, making its right-hand side positive.Source: §A.5 · p. 14

B.1 Dataset Construction

• Three attack tasks use task-specific image splits; Task I follows Shadowcast identity data, while Tasks A1 and A2 use filtered and manually inspected Visual Genome images.Task-specific splits are disjoint, with overlaps against clean training images removed to prevent data leakage; the clean set contains 12,283 pairs over 5,392 images.Source: §B.1 · p. 14, Table 6 · p. 14

B.2 On-Policy Response Collection

• On-policy responses form preference pairs based on original image semantics, with chosen responses preferred and prompts, responses, and annotations unchanged during poison crafting.Source: §B.2 · p. 14

B.3 Poison Selection

• Candidate preference pairs are ranked by displacement score ψ(z), and global top-K selection forms the poisoned set, which is added to clean training data.Source: §B.3 · p. 14

B.4 Poison Crafting and DPO Training Hyperparameters

• Poisons use an ℓ∞ budget of 8/255 and 200 iterations, aligning poison-DPO gradients with target-margin gradients; downstream models train two DPO epochs with LoRA and AdamW.Source: §B.4 · p. 15

B.5 Evaluation Details

• ASR measures attacker-specified responses on held-out target images, while clean accuracy measures correct responses on held-out base images; neither split is used to craft poisons.Evaluation uses greedy decoding with a maximum response length of 128 tokens.Source: §B.5 · p. 16
• A GPT-6-Luna judge checks whether free-form answers assign the specified identity or attribute to the main image entity.The three checks target Joe Biden, a blue car body, and a green active traffic light.Source: §B.5 · p. 16
• General utility is evaluated on VizWiz and GQA using short-answer prompts, with soft consensus accuracy and normalized exact match, respectively.Source: §B.5 · p. 16

C.1 Judge Reliability

• Rule-based and LLM-judge scores are close to human evaluations across all three tasks, supporting the reliability of the free-form evaluation.Source: §C.1 · p. 16, Table 8 · p. 16

C.2 Robustness along DPO training.

• Across five DPO epochs, attack success remains effective and clean accuracy stays consistently high; Task A1 declines moderately after peaking at epoch two.Task I and Task A2 establish strong ASR by epoch two and remain stable thereafter.Source: §C.2 · p. 16

C.4 Cross-Model Transferability

• 58.5 percentage points is the Task A2 ASR improvement on InternVL3.5-1B; perturbed images yield higher ASR than unperturbed images across all tested victim models and tasks.Clean accuracy remains largely comparable between the perturbed and unperturbed settings.Source: Table 9 · p. 18, §C.4 · p. 18

C.5 Generalization to Persuasion Task

• On the JunkFood-to-HealthyFood task, increasing the poisoning rate to 5% further improves ASR, while clean accuracy (CA) remains high across all settings.Source: Table 10 · p. 18, §C.5 · p. 18

C.6 Empirical Analysis of Preference Displacement Scores

• Selected poisons have higher initial displacement scores than the candidate pool, then their scores shift left across all three tasks after DPO training.The authors interpret the post-training shift as the poisoning signal being absorbed into learned preferences.Source: §C.6 · pp. 18–19, Figure 4 · p. 19

C.3 Robustness to Potential Defenses

• SLOF poison recall is close to random filtering, while k-Distance and DAO identify only a small fraction, leaving feature-space filtering without reliable protection.Table 11 reports poison recall averaged over Tasks A1 and A2.Source: §C.3 · p. 17, Table 11 · p. 19

C.7 Empirical Analysis of Preference Margin Dynamics

• First-order predictions closely match observed Task A1 margin reductions; top-K gives the largest positive reduction and approaches zero by epoch two, while bottom-K remains higher.Random-K reduces the margin less than top-K.Source: §C.7 · p. 19, Figure 5 · p. 20

C.3 Robustness to Potential Defenses

• With geometric augmentation, Task I reaches approximately 90% ASR at a 3% poison budget and Task A2 achieves similarly high ASR at 4%; JPEG leaves Task A1 limited but preserves substantial success on other tasks.Clean accuracy remains high under augmentation and stable under JPEG compression.Source: §C.3 · p. 17, Figure 6 · p. 20, Figure 7 · p. 20

C.8 Case Study

• On held-out clean test images, selected target predictions change to attacker-specified responses while corresponding base-image predictions remain correct.Source: §C.8 · p. 19, Table 12 · p. 21