Source-linked AI summary
Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations
Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, Timothy Hospedales
TL;DR
High MCQA benchmark performance has supported confidence in model competence, while robustness to simple option permutations remains an important question. The authors empirically test answer permutations and find that LLMs and VLLMs are broadly vulnerable, with some results falling below chance.
Problem
High MCQA benchmark performance has supported confidence in model competence, while robustness to simple option permutations remains an important question.
Source: §1 · p. 1Method
The authors empirically evaluate LLMs and VLLMs by permuting MCQA answer choices and measuring model accuracy across benchmarks.
Source: §2.1 · p. 2, §6 · p. 8Results
Simple option permutations sharply reduce MCQA accuracy across LLMs and VLLMs; Llama2-13B has 33.89% degradation in accuracy on MMLU, with results sometimes below chance.
Source: §1 · p. 1Takeaways & Limitations
High MCQA scores should not be treated as evidence of robust capabilities, and the authors call for models with intrinsic permutation robustness.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Large language and vision-language models are rapidly being deployed in practice thanks to their impressive capabilities in instruction following, in-context learning, and so on. This raises an urgent need to carefully analyse their robustness so that stakeholders can understand if and when such models are trustworthy enough to be relied upon in any given application. In this paper, we highlight a specific vulnerability in popular models, namely permutation sensitivity in multiple-choice question answering (MCQA). Specifically, we show empirically that popular models are vulnerable to adversarial permutation in answer sets for multiple-choice prompting, which is surprising as models should ideally be as invariant to prompt permutation as humans are. These vulnerabilities persist across various model sizes, and exist in very recent language and vision-language models. Code is available at https://github.com/ys-zong/FoolyourVLLMs.
1. Introduction
LLMs and VLLMs are widely evaluated through MCQA, yet simple answer-option permutations expose substantial fragility. The introduction shows that the vulnerability persists with fewer distractors, is not explained by simple positional bias, and is not effectively resolved by the tested mitigation strategies.
1. Introduction
- 33.89% degradation in accuracy on MMLU for Llama2-13B follows random option permutation, with results sometimes falling below chance across LLMs and VLLMs.Figure 1b summarizes average accuracy across benchmarking datasets.Source: Figure 1 · p. 1, §1 · p. 1
- Permutation vulnerability persists for LLMs and VLLMs even after multiple distractors are removed, despite the expectation that fewer choices should make MCQA easier.The paper reports that performance remains susceptible to degradation with a reduced number of distractors.Source: §1 · pp. 1–2
- Adversarial permutations are not explained by a simple bias against particular answer positions, although positional factors may moderately affect performance.The authors characterize the result as suggesting a more systemic issue beyond position bias.Source: §1 · p. 2
- Majority voting across permutations may improve performance but remains below original metrics at k!× the original inference cost, while calibrate-before-use fails to alleviate the problem.The paper cites Zhao et al. (2021) for calibrate-before-use.Source: §1 · p. 2
- The vulnerability raises concerns about relying on MCQA performance to assess trustworthy and reliable models, including in applications such as education and recruitment exams.The authors call for heightened scrutiny of MCQA capabilities until the vulnerabilities are addressed.Source: §1 · pp. 1–2
2. Simple Adversarial Attack Breaks LLMs and VLLMs
A simple adversarial permutation of answer choices can sharply reduce LLM and VLLM accuracy across MCQA tasks, sometimes below chance. The attack remains effective even when distractors are removed.
2.1. Experiment Setup
- The attack searches option permutations for one that maximizes model loss, stopping once the model gives an incorrect prediction.There are k! possible permutations, but the search usually requires far fewer attempts.Source: §2.1 · p. 2
2.2. Main Results
- Llama2-7B accuracy on BoolQ fell from 61.79% to 8.23%; all evaluated LLMs suffered significant drops, and only 12 non-GPT-3.5-Turbo models stayed above chance across experiments.Table 3 reports that most LLM experiments fell below chance after attack.Source: §2.2 · p. 3, Table 3 · p. 4
- VLLM accuracy dropped by 20% to 45% in all evaluated models, and more than half of 48 experiments fell below chance after attack.Visual context did not prevent the adverse effects of option permutations.Source: §2.2 · p. 3
- Across model families, scaling up generally improved baseline performance and resilience, while LLMs on MedMCQA were mostly at or below random chance after attack.Only GPT-3.5-Turbo was slightly above random chance on MedMCQA.Source: §2.2 · p. 3
2.3. Answer Set Pruning
- Reducing answer choices from four to two or three increased baseline accuracy, but adversarial permutations still often pushed accuracy below chance, even for binary choices.The pruning experiments retained the correct answer while reducing the available distractors.Source: §2.3 · pp. 3–4
3. Understanding Vulnerability Causes
The attack’s effects are not explained by answer position alone: distractor relationships also matter, and prediction responses vary with option symbols. The analyses suggest symbol-answer shortcuts as one possible source of this vulnerability.
3.1. Position Bias and Other Attacks
- Position bias varied by model: Falcon-7B accuracy ranged from 70.9% with the correct answer in slot A to 3.7% in slot B, from a 31% baseline.This indicates a strong preference for slot A and aversion to slot B.Source: §3.1 · p. 4
- Adversarial permutation degraded Llama2-7B below chance even though it performed above chance with the correct answer in every position, implicating distractor relationships beyond position bias.The authors report that the attack has a stronger effect than position bias, CircularEval, and symbol attack.Source: §3.1 · pp. 4–5, Table 6 · p. 6
3.2. Symbol-Content Spurious Correlation
- Prediction patterns correlated less between capital letters and Roman numerals than between capital and lowercase letters, suggesting possible symbol-answer shortcuts.The authors interpret this as a potential cause, not a demonstrated mechanism.Source: §3.2 · p. 5, Table 7 · p. 6, Figure 2 · p. 6
- Models’ permutation non-invariance depended on the option symbols used, even when baseline and permuted accuracies remained largely consistent across symbol sets.The finding is based on correlations across symbol sets and permutations.Source: §3.2 · pp. 5–6
4. Exploring Mitigation Strategies
The paper tests post-hoc fixes and fine-tuning as ways to mitigate permutation vulnerability. Post-hoc strategies do not restore baseline performance, while permutation-augmented fine-tuning alleviates the issue without reliably defeating the attack.
4.1. Post-hoc Mitigation Strategies
- Majority voting and confidence voting ameliorated the attack but generally failed to restore baseline accuracy; contextual calibration had no meaningful impact, despite a k!-fold inference-cost increase for voting.The findings indicate that position bias alone does not explain the vulnerability.Source: §4.1 · pp. 6–7
- PriDe improved original-order accuracy on MMLU but did not restore attacked performance; its fixed recalibration matrix cannot adapt to permutation-dependent option preferences.The authors characterize PriDe as an amelioration of fixed positional or option bias.Source: §4.1 · p. 7
4.2. Fine-tuning on Training Set
- On ARC-Challenge and MedMCQA, permutation-augmented fine-tuning substantially alleviated vulnerability versus zero-shot and regular fine-tuning, but did not reliably defeat the attack.The experiments used Llama2-7B, LoRA, and one training epoch; the authors note that not all datasets provide training sets.Source: §4.2 · p. 7
5. Related Work
Prior work established strong LLM and VLLM capabilities, including on MCQA benchmarks, while leaving MCQA robustness—particularly for VLLMs—relatively underexamined. The paper situates permutation attacks beyond position bias and compares existing defenses and fine-tuning strategies.
5. Related Work
- LLMs and VLLMs show strong capabilities across language and multimodal tasks, including generation, question answering, and dialogue.The passage describes LLMs with billions of trained parameters and vision-language models connecting pretrained language and vision encoders through trainable alignment modules.Source: §5 · p. 7
- MCQA asks models to select the correct option from choices and is used across benchmarks including MMLU, AGI-Eval, MedMCQA, and SocialIQA.Multiple-choice prompting is described as the currently recommended state of the art for MCQA.Source: §5 · p. 7
- Contextual calibration fails against permutation attacks on MMLU, while majority vote and M-Confidence ameliorate but do not fully restore performance.PriDe cannot alleviate vulnerability under full permutation.Source: Table 8 · p. 8, Table 9 · p. 8
- Fine-tuning with permutation can enhance robustness to permutation attacks compared with zero-shot and regular fine-tuning baselines.The comparison covers two benchmarks with Llama2-7B.Source: Table 10 · p. 8
- MCQA robustness had been relatively underexamined, particularly for VLLMs; adversarial permutation vulnerability is reported as deeper than position bias.The related-work discussion also notes prior sensitivity to prompts, in-context examples, and irrelevant context.Source: §5 · pp. 7–8
6. Discussion
The paper reports that simple option permutations expose vulnerability in both LLMs and VLLMs on MCQA despite strong benchmark performance. It urges caution in evaluation and deployment while identifying robustness-focused future work as an open need.
6. Discussion
- Both LLMs and VLLMs are highly susceptible to simple option permutations in MCQA despite seemingly robust performance on established benchmarks.The paper says this raises concerns about interpreting high MCQA benchmark scores as evidence of robust capabilities.Source: §6 · p. 8
- The authors call for training strategies or architectures that provide intrinsic robustness and parameter-efficient tuning to make existing models permutation-invariant.They explicitly state that the paper offers no solution.Source: §6 · p. 8
- The findings are presented as relevant to researchers and industry professionals deploying LLMs and VLLMs in real-world scenarios.The authors urge caution when evaluating and deploying these models.Source: §6 · p. 8
Impact Statement
Because many LLM and VLLM benchmarks use MCQA, the paper argues that permutation instability matters for interpreting model capabilities. It calls for researchers to rethink evaluation protocols and reassess those capabilities.
Impact Statement
- MCQ evaluation can be highly unstable and fall below chance under simple permutations, a concern because many LLM and VLLM benchmarks use MCQA.The passage presents this as a reason to reconsider evaluations based on MCQA.Source: Impact Statement · p. 8
- The authors encourage researchers to rethink evaluation protocols and reassess the capabilities attributed to developed LLMs and VLLMs.Source: Impact Statement · p. 9
A.1. Additional Results on Answer Set Pruning
The appendix points to additional answer-set-pruning results across language and vision-language datasets.
A.1. Additional Results on Answer Set Pruning
- Tables 11–16 provide additional answer-set-pruning results for all language and vision-language datasets.Source: §A.1 · p. 12
A.2. Additional Results on Different Prompting and Attack Strategies
The section tests how prompting, attack, and decoding choices affect permutation brittleness. In-context learning and alternative strategies do not remove the vulnerability, while majority-vote outcomes depend on query difficulty.
A.2. Additional Results on Different Prompting and Attack Strategies
- Stronger LLMs require more permutations to break, while models are easier to break on the more difficult MedMCQA dataset than ARC-Challenge.Table 17 reports the average number of permutations needed to break predictions for both datasets.Source: §A.2 · p. 12
- For easy queries, majority-vote accuracy trends toward 1 with infinite calls, but for hard queries it trends toward 0.The subsection characterizes easy queries as having over 50% correctness per LLM call and hard queries as having less than 50%.Source: §A.2 · p. 12
- In-context learning improves original performance, but models still suffer substantial drops after adversarial permutations.The comparison covers performance before and after attack with in-context learning.Source: §A.2 · p. 12
- Across the tested in-context attack strategies, adversarial permutation has the largest impact on final performance.The alternatives include permuting in-context examples and searching for worst-case examples.Source: §A.2 · p. 12
- Other sampling strategies and temperatures perform worse than greedy decoding both before and after permutations.The authors take this as support for the validity of their experiments and generalization to other decoding strategies.Source: §A.2 · p. 12
- In-context examples with answers at positions A, B, C, and D do not entirely override the model’s original preference for option B.The examples can modify the distribution across options while the preference persists.Source: §A.2 · p. 12
A.3. Further Analysis on Vision-Language Dataset
Further analysis extends the permutation-attack findings to a vision-language dataset and illustrates how small option-order changes can alter predictions. The appendix also examines how attack outcomes vary across permutations and mitigation strategies.
A.4. Qualitative Results
- On an ARC-Challenge example, swapping two distractors while leaving the correct answer in place changed Llama-13B’s prediction to an incorrect option.Swapping the correct answer’s position in a different permutation did not prevent a correct prediction.Source: §A.4 · p. 12
- In a ScienceQA example, Otter-Llama still predicts A with strong token-probability confidence after the correct answer, “Asia,” moves from A to C.Figure 3 presents the visual multiple-choice example.Source: §A.4 · pp. 12–13, Figure 3 · p. 14
- Across ARC-Challenge questions, many models have a substantial minority of questions answered correctly under only a small fraction of permutations.Figure 4’s histogram groups questions by the proportion of permutations answered correctly; Llama2-70B also has many questions answered correctly under almost all permutations.Source: §A.4 · p. 13, Figure 4 · p. 15
- Majority voting helps easy samples, but difficult-sample accuracy is very low, so the overall outcome depends on their mix.The reported analyses show that overall accuracy may increase or decrease depending on how easy and difficult samples contribute.Source: Table 18 · p. 17, Table 19 · p. 18
- With in-context learning, permutation attacks still cause substantial drops, with most Table 20 experiments falling below chance.The table compares original accuracy, post-attack accuracy, and relative performance drop.Source: Table 20 · p. 18
- Across MMLU, ARC-Challenge, BoolQ, SocialIQA, and MedMCQA, adversarial permutation produces the largest performance drop among the compared attacks.The tables report that in-context learning improves zero-shot performance and attacks on in-context examples can also reduce performance.Source: Table 21 · p. 18, Table 22 · p. 19, Table 23 · p. 19
A.3. Further Analysis on Vision-Language Dataset
- On A-OKVQA, position bias had moderate impact, while adversarial permutations severely degraded performance, usually below random chance.Majority vote and maximum confidence ameliorated the attack but did not fully restore performance; contextual calibration failed completely.Source: §A.3 · p. 12, Table 30 · p. 22, Table 31 · p. 22
A.4. Qualitative Results
- On A-OKVQA, majority vote and maximum confidence ameliorate the permutation attack but do not fully restore performance; contextual calibration fails completely.The table marks below-chance outcomes with red shading.Source: Table 31 · p. 22