PaperlayerView exact revision on arXiv

Source-linked AI summary

Pay Less Attention to Function Words for Free Robustness of Vision-Language Models

Qiwei Tian, Chenhao Lin, Zhengyu Zhao, Chao Shen

arXiv:2512.07222v4cs.LGcs.CL

TL;DR

VLM adversarial training can improve robustness at the cost of clean performance and computation, while function words may contribute to cross-modal vulnerability. The paper proposes FDA, which subtracts function-word cross-attention from original attention; it reports lower attack success with largely preserved or improved clean performance across retrieval and grounding evaluations.

  • Problem

    Adversarial training can improve VLM robustness but incurs clean-performance drops and computational costs, motivating defenses that address cross-modal vulnerability while preserving performance.

    Source: §1 · p. 1

  • Method

    FDA computes function-word/image cross-attention in parallel and subtracts those distraction scores from original cross-attention.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    FDA reports retrieval ASR drops of 22.26/14.69%, 14.29/13.55%, and 51.60/56.36% on ALBEF/TCL/BLIP, with 0.30/0.10%, 0.50/0.22%, and 0.70/0.46% R@1 drops for T2IR/I2TR, and grounding ASR drops of 93.16/91.50% for ε = 2/4.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    The reported results support function-word de-attention as a way to improve VLM robustness while preserving or improving clean performance in the evaluated tasks.

    Source: §4.1 · p. 6, §5 · p. 10

    FDA was implemented on fusion-encoder backbones, not projector-based models, and was not tested on larger VLMs or with PEFT methods because of hardware limitations.

    Source: §4.5 · p. 10

Abstract

from arXiv · show

To address the trade-off between robustness and performance for robust VLM, we observe that function words could incur vulnerability of VLMs against cross-modal adversarial attacks, and propose Function-word De-Attention (FDA) accordingly to mitigate the impact of function words. Similar to differential amplifiers, our FDA calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more aligned and robust VLMs. Comprehensive experiments include 2 SOTA baselines under 6 different attacks on 2 downstream tasks, 3 datasets, and 3 models. Overall, our FDA yields an average 18/13/53% ASR drop with only 0.2/0.3/0.6% performance drops on the 3 tested models on retrieval, and a 90% ASR drop with a 0.3% performance gain on visual grounding. We demonstrate the scalability, generalization, and zero-shot performance of FDA experimentally, as well as in-depth ablation studies and analysis. Code is available at https://github.com/michaeltian108/FDA.

1 INTRODUCTION

Adversarial training can improve VLM robustness but incurs clean-performance and computational costs; the paper identifies function words as a potential source of cross-modal vulnerability and proposes FDA to reduce their influence. Across the reported evaluations, FDA lowers attack success while largely preserving retrieval performance and improves robustness on visual grounding.

1 INTRODUCTION

  • Function words may be more approachable to cross-modal perturbations because they have high frequency but little semantic information.The authors report that removing function words reduces ASR while causing the least clean-performance drop among the tested word types.Source: §1 · p. 1
  • FDA adds a parallel function-word/image cross-attention pathway and subtracts its distraction scores from original attention.The approach is presented as refining vision-language alignment by paying less attention to function words.Source: §1 · pp. 1–2
  • Across three tested models, FDA yields average ASR drops of 18/13/53% with only 0.2/0.3/0.6% retrieval performance drops, and a 90% ASR drop with better clean performance on grounding.Experiments covered two SOTA baselines, three models, two tasks, three datasets, and six attacks.Source: §1 · p. 2
  • The authors report ablations on hyperparameter insensitivity and backbone generalization, as well as improved zero-shot performance.They also describe in-depth analysis of the method.Source: §1 · p. 2

2 RELATED WORK

The section reviews adversarial attacks on VLMs and defense approaches, emphasizing the effectiveness of adversarial training alongside its clean-performance and computational costs. FDA is illustrated as an alternative that targets function-word distractions and alignment.

2 RELATED WORK

  • Adversarial attacks can fool VLMs into incorrect outputs, with both white-box and transferable black-box cross-modal attacks reported against pretrained models.The figure illustrates the authors' proposed FDA pipeline and its hypothesized effect on alignment under attack.Source: §2 · p. 2, Figure 2 · p. 3
  • Adversarial training has shown robustness gains, but can substantially downgrade clean performance and carries high computational costs.The passage notes that performance drops remain noticeable even for FARE, which uses vanilla visual embeddings as supervision.Source: §2 · pp. 2–3
  • FDA's illustration routes function-word/image attention through a control gate before subtracting distractions from the original attention.The figure's right panel speculates that removing those distractions can tighten embeddings and prevent misalignment.Source: Figure 2 · p. 3

3 METHODOLOGY

The methodology first describes cross-attention in a fusion encoder, then introduces FDA as a parallel function-word attention pathway whose scores are used to reduce distractions in the original attention.

3.1 PRELIMINARY

  • For each input image and text, the visual and textual encoders produce embeddings with attention masks, which enter the fusion encoder for cross-attention.The query, key, and value layers and the layer/head indices are specified for this calculation.Source: §3.1 · p. 3

3.2 FUNCTION-WORD DE-ATTENTION (FDA)

  • FDA computes function-word/image attention in parallel with the original cross-attention and subtracts the distraction scores from it.A control gate sets subtraction intensity, and the minimum of the individually subtracted attentions becomes the final attention scores.Source: §3.2 · pp. 3–4
  • FDA extracts function-word features by masking other text tokens except [CLS], using a shortlisted stopword dictionary.The dictionary is denoted D, and the function-word mask is Mtf.Source: §3.2 · pp. 3–4
  • Softmax is applied over visual-token and textual-token dimensions to highlight false visual activations and misleading tokens.FDA concatenates the resulting attention scores across heads, and it can be implemented in selected fusion layers and heads.Source: §3.2 · p. 4

4 EXPERIMENTS

The experiments evaluate FDA against TeCoA and FARE on retrieval and visual grounding, using three retrieval models, three datasets, and six attacks. Ablations and analyses examine implementation choices, variants, zero-shot use, and limitations.

4.1 MAIN RESULTS

  • On Flickr retrieval, FDA drops ASR by 22.26/14.69%, 14.29/13.55%, and 51.60/56.36% on ALBEF/TCL/BLIP; on grounding, ASR drops over 90% with improved clean performance.For retrieval, the paired ASR drops correspond to the two perturbation strengths; grounding reports over 90% overall and specifically 93.16/91.50% for ϵ = 2/4.Source: Table 1 · p. 5, Table 2 · p. 6, §4.1 · p. 6
  • FDA leads PGD and APGD results in 22 of 24 Flickr comparisons; it also leads on MAPGD for ALBEF and BLIP, while retaining an overall advantage on TCL.At ϵ = 4, TeCoA and FARE become ineffective, while FDA retains effectiveness; similar trends are reported on MSCOCO.Source: §4.1 · p. 6

4.2 UNTARGETED ATTACKS

  • For untargeted attacks, FDA boosts TeCoA and FARE robustness across models and tasks, with larger gains on BLIP and a reported 5% FARE gain on grounding.The passage also reports a considerable clean-performance boost for both methods on ALBEF.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

4.3 ABLATION STUDY

  • Shallow L0/L0−1 and H0−5 settings perform best on retrieval, and L0−1/H0−5 also leads on grounding for adversarial and clean examples.On retrieval, L0/H0−5 leads other settings by 0.1–0.2% in clean performance; on grounding, the leading setting gains 10–20% adversarially and 0.3–0.4% on clean examples.Source: Table 6 · p. 8, Table 7 · p. 8, §4.3 · p. 8
  • Fusion-encoder-only FDA performs best; applying it to both encoders adds robustness but costs 2% performance, while the 91-word shortlist slightly outperforms the 208-word dictionary.The authors attribute the full dictionary's minor degradation to potentially unnecessary word removal and context distortion.Source: Table 5 · p. 7, §4.3 · p. 8
  • FDA outperforms masking and other de-attention variants; for zero-shot tasks, Lall performs best across the tested models and tasks.Direct function-word masking brings negligible robustness and about a 1% performance drop; masking content words or nouns causes the largest performance drop.Source: Table 8 · p. 9, Table 9 · p. 9, §4.3 · p. 9

4.4 ANALYSIS AND VISUALIZATION

  • FDA produces the most aligned cross-modal embedding among the compared methods, with higher average text-image similarity and lower variation than vanilla VLM.The authors hypothesize that adversarial noise can disrupt alignment, and report that FDA's embedding remains tightly aligned in the visualization.Source: §4.4 · pp. 9–10, Figure 3 · p. 9

4.5 LIMITATION

  • FDA has not been tested on larger VLM fine-tuning, PEFT methods, or projector-based backbones; the optimal function-word dictionary size also remains unvalidated.These boundaries are attributed to hardware limits for some experiments and the substantial effort required to validate dictionary alternatives.Source: §4.5 · p. 10

5 CONCLUSION

The paper presents FDA as a method that subtracts function-word cross-attention from original attention, reporting robustness and zero-shot gains across its evaluations.

5 CONCLUSION

  • FDA subtracts function-word cross-attention from original attention to reduce distractions and improve vision-language alignment.The paper reports improved robustness and clean performance, and validates a boost to zero-shot performance.Source: §5 · p. 10

ETHICS STATEMENT

The authors state that they are required to follow and acknowledge the Code of Ethics, including commitments to scientific integrity, fairness, privacy, and confidentiality.

ETHICS STATEMENT

  • The authors are required to read and adhere to the Code of Ethics and explicitly acknowledge this during submission.The statement also lists commitments concerning scientific excellence, avoiding harm, honesty, trustworthiness, and transparency.Source: ETHICS STATEMENT · p. 10
  • The listed ethical commitments include fairness and action to avoid discrimination.The statement also calls for respecting the work behind new ideas and artefacts, privacy, and confidentiality.Source: ETHICS STATEMENT · p. 10
  • The statement calls for contributing to society and human well-being.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement says the data are open-source and code and checkpoints will be made public with instructions; an adaptive-selection analysis finds robustness gains correlate with function-word proportion.

REPRODUCIBILITY STATEMENT

  • The data are open-source, and the authors state that code, checkpoints, and instructions for reproducing results and transferring to other backbones will be publicly available.Implementation details are in Section 4, with full results in the Appendix.Source: REPRODUCIBILITY STATEMENT · p. 10

A FDA V.S. ADAPTIVE WORDS SELECTION

  • The adaptive-selection comparison uses low-similarity tokens selected by two similarity thresholds or by choosing the lowest N tokens, where N is the function-word count.The paper also records what percentage of selected words appear in its shortlisted function-word dictionary; results are in Table 10.Source: A FDA V.S. ADAPTIVE WORDS SELECTION · p. 14
  • Robustness gains increased as the proportion of function words increased, although no adaptive mechanism perfectly matched dictionary-based selection.The paper reports an evident correlation, while noting vulnerability does not necessarily come from low-similarity words.Source: A FDA V.S. ADAPTIVE WORDS SELECTION · p. 14

B DETAILS FOR ATTACKS AND EVALUATION METRICS

The appendix describes how targeted and untargeted attacks are counted for retrieval and visual grounding, using task-specific success criteria and metrics.

B DETAILS FOR ATTACKS AND EVALUATION METRICS

  • The appendix introduces the attacks and task-specific evaluation metrics, including each scenario’s criterion for a successful targeted attack.Source: B DETAILS FOR ATTACKS AND EVALUATION… · p. 14
  • For retrieval, a targeted attack succeeds when the manipulated image or targeted text reaches Top 1/5, and ASR is the corresponding hit rate, reported as ASR@1/5.The main paper uses the average ASR@1/5 as overall ASR; untargeted attacks lower R@1/5.Source: B DETAILS FOR ATTACKS AND EVALUATION… · p. 14
  • For visual grounding, a targeted attack succeeds when the model’s bounding box has IOU greater than 0.5 with the targeted box.Untargeted ASR is calculated from drops in victim-model accuracy.Source: B DETAILS FOR ATTACKS AND EVALUATION… · p. 14

C FULL RESULTS FOR TARGETED ATTACKS

This section locates the full targeted-attack results across retrieval and visual grounding, specifying the relevant tables by model and task.

D FULL RESULTS

This section inventories full results for untargeted attacks and reports an adaptive-selection comparison linking function-word proportion with robustness gains.

  • Untargeted retrieval results are reported for ALBEF in Tables 18–19 and BLIP in Tables 20–21, while visual-grounding results for ALBEF are in Table 22.Source: D FULL RESULTS · pp. 14–15
  • Table 10 compares FDA with similarity-based adaptive token selection and reports a correlation between the selected function-word proportion and gained robustness.Source: Table 10 · p. 14
  • Tables 11–12 identify ALBEF white-box targeted-attack ASR results for T2IR and I2TR on Flickr30k and COCO.Source: Table 11 · p. 15, Table 12 · p. 15

E FULL RESULTS FOR ABLATION STUDIES

The supplementary results cover targeted attacks, ablations, and zero-shot performance; the section also lists the function-word dictionary and visualizes attention changes under FDA.

E FULL RESULTS FOR ABLATION STUDIES

F DETAILS FOR FUNCTION WORD DICTIONARY

G VISUALIZATION OF ATTENTION SCORES

  • Tables 18–21 report white-box untargeted-attack ASR for Flickr30k text-to-image and image-to-text retrieval with ALBEF and BLIP, with after-attack R@k values in parentheses; Table 22 reports untargeted PGD/APGD/MAPGD ASR for RefCOCO+ visual grounding, with after-attack accuracies in parentheses.Source: Table 18 · p. 18, Table 19 · p. 18, Table 20 · p. 18
  • Figure 4 shows progressively cleaner attention maps after one and two FDA subtractions, with the two-subtraction map displaying the fewest visible distractions.Source: G VISUALIZATION OF ATTENTION SCORES · p. 17, Figure 4 · p. 21
Loading 2512.07222v4…