Source-linked AI summary
On the Adversarial Robustness of Multi-Modal Foundation Models
Christian Schlarmann, Matthias Hein
TL;DR
The paper asks whether malicious third-party image content can harm honest users of multimodal foundation models. It evaluates targeted and untargeted visual attacks against OpenFlamingo under constrained perturbation and finds that imperceptible changes can substantially control model outputs. The findings motivate robustness measures for deployed multimodal models.
Problem
The paper investigates whether honest users can be harmed when malicious third parties manipulate visual inputs to multimodal foundation models.
Method
The paper evaluates targeted and untargeted adversarial visual attacks against OpenFlamingo using constrained ℓ∞ perturbations in a white-box setting across captioning tasks.
Results
Targeted attacks can control OpenFlamingo outputs, with success rates up to 51.66% in 0-shot and 86.00% in 4-shot settings at ε = 1/255, reaching perfect success at ε = 4/255.
Takeaways & Limitations
Imperceptible visual perturbations can enable misinformation, user manipulation, or harmful generated content, underscoring the need for robustness in deployed multimodal models.
Takeaways & Limitations
Quantitative evaluations use 500 APGD iterations because this is the maximum computationally feasible budget, while more iterations are expected to improve attack success.
Abstract
from arXiv · showhide
Multi-modal foundation models combining vision and language models such as Flamingo or GPT-4 have recently gained enormous interest. Alignment of foundation models is used to prevent models from providing toxic or harmful output. While malicious users have successfully tried to jailbreak foundation models, an equally important question is if honest users could be harmed by malicious third-party content. In this paper we show that imperceivable attacks on images in order to change the caption output of a multi-modal foundation model can be used by malicious content providers to harm honest users e.g. by guiding them to malicious websites or broadcast fake information. This indicates that countermeasures to adversarial attacks should be used by any deployed multi-modal foundation model.
1. Introduction
The paper examines how malicious third parties can manipulate multimodal models through imperceptible image perturbations, potentially misleading honest users. It introduces an evaluation framework for OpenFlamingo covering targeted and untargeted attacks and misuse scenarios.
- Illustrations: Figure 1 shows targeted caption changes on perturbed images whose modifications are hardly visible to users.The examples include directing users to malicious websites or presenting fake information.
- Threat: Malicious third parties can add imperceptible image perturbations that make a multimodal model generate attacker-chosen outputs.The manipulated outputs can distribute false information or toxic content without raising user suspicion.
- Contributions: The paper introduces a framework for evaluating adversarial visual vulnerability in the OpenFlamingo model.The framework assesses the impact of imperceptible perturbations on model outputs.
- Attack types: The evaluation considers both targeted attacks, which force specific outputs, and untargeted attacks, which degrade output quality.This distinction separates output control from general performance degradation.
- Implications: The paper highlights misuse scenarios including fake information, user manipulation, and fraud.These scenarios illustrate how adversarial visual attacks could affect users beyond benchmark performance.
2. Related work
Prior work studied adversarial attacks mainly on single-modal or different multimodal settings. This paper focuses on small, difficult-to-notice perturbations in a malicious-third-party setting, emphasizing risks to honest users.
- Multimodal models: OpenFlamingo is the open-source implementation of Flamingo examined in this work.Flamingo combines a pretrained language model and vision encoder through projection and cross-attention layers.
- General adversarial robustness: Earlier adversarial-robustness research primarily targeted single-modal models, especially image models.Prior work also considered CNN-RNN-based VQA and captioning attacks.
- Multimodal attacks: Prior multimodal attack work included image and text attacks under gray-box classification settings, while this evaluation covers VQA and image captioning.The paper positions its tasks as extending evaluation to more recent multimodal models.
- Threat model: Concurrent attacks on toxic-content generation used large or unbounded perturbation budgets, whereas this work constrains attacks to ℓ∞ radii of 1/255 or 4/255.The smaller perturbations are intended to remain hardly visible and likely unnoticed by honest users.
- Threat model: Because users generally trust reliable model answers, unnoticed perturbations could lead them to trust fake information or follow malicious links in generated captions.This connects imperceptibility with the malicious-third-party threat to honest users.
3. OpenFlamingo model
OpenFlamingo combines an image encoder with a language model whose cross-attention integrates visual features into autoregressive text generation. It supports few-shot and zero-shot captioning through image and textual context.
- Architecture: OpenFlamingo unifies visual and textual processing by combining a vision model with a large language model.The model can process visual and textual input and generate natural-language output.
- Architecture: Cross-attention lets the language model attend to features produced by the vision model.Keys and values come from visual input, while queries come from language input.
- Generation: The forward pass predicts the next language token iteratively to generate text.This defines the autoregressive generation process used for captions.
- Inference: The model can perform few-shot inference using context images paired with descriptive text.A query image is then captioned from its image token and an initiator prompt such as “A photo of” or “Output:”.
- Inference: Zero-shot inference omits context images while retaining contextual text for generating the query-image caption.The caption is generated autoregressively from the query image and prompt.
4. Adversarial attack on OpenFlamingo
The paper formulates targeted and untargeted adversarial attacks on OpenFlamingo as constrained optimization problems over imperceptible visual perturbations. Untargeted attacks degrade outputs, whereas targeted attacks minimize the loss of a desired caption and generally require more iterations.
- Threat model: The attacker perturbs query and, optionally, context images within an ℓ∞ threat model while assuming white-box access to all model weights.The query-only setting fixes εc = 0; when all images are attacked, context perturbations are optimized too.
- Untargeted attack: Untargeted attacks maximize the negative log-likelihood of a ground-truth caption over the allowed perturbations.The objective uses query image, context images, context text, and the ground-truth caption as inputs.
- Optimization: Projected gradient descent can optimize the objectives because gradients are available in the white-box setting.The resulting procedure is reported as effective against OpenFlamingo.
- Targeted attack: Targeted attacks minimize the negative log-likelihood of a specified target caption so its tokens become more probable.This reverses the optimization direction used by the untargeted attack.
- Iteration requirements: A small number of iterations already significantly degrades untargeted performance, while successful targeted attacks require many iterations.The iteration study uses εq = 1/255 on 100 COCO images in the zero-shot setting, reporting CIDEr and attack success rate.
- Evaluation metric: CIDEr measures generated-caption similarity to ground-truth captions using consecutive-word co-occurrence weighted by TF-IDF.Its worst possible value is 0, while random permutations of COCO captions average 3.01 with standard deviation 0.81.
5. Methods
The evaluation tests OpenFlamingo on captioning and visual-question-answering tasks under targeted and untargeted attacks. It varies datasets, inference context, perturbation access, threat radius, and APGD iteration budgets.
- Model and tasks: The evaluated OpenFlamingo model combines a CLIP ViT-L-14 vision encoder with an MPT-7B language model and has 9B parameters.The evaluation uses the strongest pretrained model in the open-source OpenFlamingo implementation.
- Captioning: Captioning evaluation covers COCO 2014 and Flickr30k, reporting CIDEr scores for captions generated under untargeted attack.The prompt uses the form <image>Output: and truncates text after the first repeated Output: marker.
- Visual question answering: VQA evaluation covers OK-VQA and VizWiz, reporting answer accuracy under untargeted attack.The prompt uses <image>Short Answer: with analogous output truncation.
- Inference settings: Experiments test zero-shot and four-shot inference, including four-shot settings where either all images or only query images are perturbed.Each dataset uses 1000 sampled instances; in zero-shot inference, the two perturbation-access settings coincide.
- Targeted evaluation: Targeted COCO evaluation reports exact-target success rate and BLEU-4 similarity, with all attacks run for 500 APGD iterations because of computational constraints.Success rate counts outputs containing the complete target caption, while BLEU-4 ranges from 0 to 100.
- Untargeted evaluation: Untargeted evaluation varies ℓ∞ radii εq = 1/255 and εq = 4/255, context-image budgets εc = εq or εc = 0, and uses 500 APGD steps.The reported metrics are CIDEr for captioning and accuracy for VQA; the table states that attacks succeed in all cases.
- Qualitative evaluation: Figures 3 and 4 compare original and adversarially perturbed COCO captions using untargeted zero-shot attacks at εq = 1/255 and εq = 4/255.Both figures use 500 iterations.
6. Results
The proposed attacks are highly successful against OpenFlamingo, degrading captioning and VQA performance while often forcing exact target captions. Even a fraction of the perturbations can preserve substantial attack effectiveness.
- Overall findings: The attacks were highly successful against OpenFlamingo in all considered settings.The evaluation covered both targeted and untargeted attacks.
- Targeted attacks: Targeted attacks achieved success rates up to 51.66% in 0-shot and 86.00% in 4-shot settings at ε = 1/255, reaching 100% in both settings at ε = 4/255.Attacking all images was more effective than targeting only the query image.
- Targeted attacks: Longer target captions were harder to recover under the smaller threat model, while ε = 4/255 and more APGD iterations improved attack effectiveness.The longer target “A person suffers severe side effects after vaccination” was not well recovered for the small threat model.
- Untargeted attacks: Untargeted attacks produced low CIDEr scores on COCO and Flickr and low accuracies on OK-VQA and VizWiz across all evaluated settings.On COCO, the larger threat model reduced CIDEr below the score from randomly permuted captions.
- Perturbation sparsity: Keeping only the top 60% of pixelwise perturbations reduced COCO CIDEr to 24.99 for ε = 1/255 and 3.65 for ε = 4/255 in the untargeted setting.For targeted attacks, the 60% threshold was already strong at ε = 4/255, whereas more perturbations were needed at ε = 1/255.
7. Discussion
Slight, typically imperceptible image modifications can substantially alter multimodal model outputs without users realizing the input was manipulated. This creates risks for misleading content and misinformed decisions in applications such as news generation.
- Security implications: Users may unknowingly submit adversarially manipulated images because the perturbations are slight and typically imperceptible to human observers.Despite their subtlety, the modifications can substantially manipulate model outputs and reduce prediction reliability.
- Security implications: Adversarial perturbations can cause image-based news generation to produce misleading or completely false narratives.The paper identifies false health statements and alarming political news as possible examples.
- Consequences: Misleading articles generated from manipulated images may lead users to make misinformed decisions or actions.News articles’ broad reach and ability to shape public opinion amplify this concern.
8. Conclusion
The investigation finds OpenFlamingo highly susceptible to barely visible visual perturbations that degrade captioning and VQA or control outputs. These findings underscore the need for robustness and mitigation strategies before broader real-world deployment.
- Conclusion: OpenFlamingo’s visual-input perturbation susceptibility causes poor captioning and VQA performance even when the changes are hardly visible to humans.The paper evaluates both degradation and output-control attacks.
- Conclusion: Targeted attacks can control model outputs to produce desired responses that may be deceiving or harmful.Because users often implicitly trust model outputs, such manipulation could spread misinformation or influence behavior.
- Conclusion: The findings highlight a critical need for robustness-enhancing strategies to support safe multimodal-model use in real-world applications.The paper calls for future research on mitigation against adversarial attacks.