Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks

Motivation

• Adversarial attacks can manipulate inputs to produce attacker-beneficial erroneous outputs, and LLMs’ scale, generative behavior, context retention, multimodality, and system integrations create distinctive security challenges.Attacks may be targeted or untargeted, while threat models must account for complex ecosystems such as interacting agents and autonomous systems.Source: §1 · p. 3
• Motivating scenarios show that aligned models can be jailbroken for harmful information, browser assistants can be contaminated by malicious product content, and programming assistants can generate backdoored code.These examples illustrate risks arising through prompts, multimodal external content, and adversarial examples.Source: §1 · p. 3

Scope of the survey

• The survey covers domain-general adversarial attacks, tracing their evolution from manually generated inputs to algorithmically produced attacks and extending analysis to multimodel and complex-system settings.Its scope includes emerging structures such as augmented, federated, and multi-agent LLMs.Source: §1 · pp. 3–4

Threat model dimensions

• Threat models vary by attacker access, injection source, attack mechanism, and goal, spanning white-, black-, and partial-box access; prompts, documents, or websites; prompt injection or context contamination; and degraded, misaligned, or harmful outputs.Black-box attackers may rely on proxy models and transferability, while goals can include hallucinations, alignment bypasses, private or toxic content, and vulnerable code.Source: §1 · p. 4

Relation to other surveys

• Unlike data-centric trustworthy-ML and earlier NLP attack surveys, this study focuses on adversarial vulnerabilities in generative language-related models and systems, including LLMs, VLMs, multimodal models, and integrated complex systems.Earlier NLP work largely targeted discriminative classification models, while Barrett et al. (2023) emphasized defenses and only briefly covered representative attacks.Source: §1 · p. 4

Novelty

• The survey prioritizes emerging models and recent advances, including closed-source systems such as Bard and ChatGPT and open-source models such as Vicuna and Llama 2, with attack methods organized for scalability across languages and domains.It argues that more safety-aligned generative models require thorough examination of their security attributes.Source: §1 · p. 4

2 Background

• The background connects NLP and security by introducing large language models and adversarial attacks for researchers pursuing interdisciplinary work.The survey frames these two topics as its foundational background areas.Source: §2 · p. 4

2.1 Language Models

• The section surveys language-model structures and paradigms, emphasizing components that could be exploited by attackers.It provides technical fundamentals for the survey’s learning-structure organization.Source: §2.1 · p. 5
• For deeper coverage, it refers readers to surveys of uni-modal, multi-modal, federated, and multi-agent language systems.The cited resources include Zhao et al. (2023), Yang et al. (2023a), Xu et al. (2023), Chen et al. (2023a), Du et al. (2023), and Zhang et al. (2023a).Source: §2.1 · p. 5

2.1 Language Models

• NLP centers on natural language understanding and generation, enabling machines to read, write, and communicate like humans.Source: §2.1 · p. 5
• LLMs are extensively pretrained models whose emergent abilities support complex few-shot or zero-shot tasks, including in-context learning and chain-of-thought, unlike PLMs.PLMs are smaller early pretrained models typically fine-tuned on task-specific data; the distinction follows Yang et al. (2023a), Zhao et al. (2023), and Wei et al. (2022a).Source: §2.1 · p. 5

2.1.1 Modeling

• Language models assign probabilities to generated-text sequences through autoregressive or non-autoregressive approaches.Autoregressive models typically target generation using next-word prediction during pretraining (Radford et al., 2018, 2019; Brown et al., 2020a).Source: §2.1.1 · p. 5

2.1 Language Models

• Most LLMs use decoder-only architectures, although encoder-decoder and prefix-decoder designs remain represented by models including Flan-T5, GLM130B, and U-PaLM.Decoder-only adoption is associated with the strong performance of OpenAI’s GPT series; prefix decoders use bidirectional attention over prefix tokens while predicting subsequent tokens autoregressively.Source: §2.1 · p. 6

2.1.2 Training

• Training data span general and specialized sources, and domain diversity can strengthen generalization whereas fixed-domain pre-training may cause catastrophic forgetting.Controlling the distribution of data across domains can produce different model performance (Liang et al., 2022).Source: §2.1.2 · p. 6
• LLM pre-training commonly uses next-word prediction or denoising objectives, with Mixture-of-Denoisers as an advanced alternative, alongside large batches, warm-up and decay schedules, weight decay, and gradient clipping.Language Model objectives predict the next token from preceding tokens, while DAE restores tokens in corrupted spans (Devlin et al., 2019b).Source: §2.1.2 · p. 6

2.1.3 Alignment

• Instruction tuning elicits LLM abilities by fine-tuning pre-trained models on structured (INSTRUCTION, OUTPUT) pairs, with stronger effects for LLMs than smaller models.Task instructions and in-context learning strategies can harness capabilities beyond pre-training and fine-tuning (Wei et al., 2022a; Yang et al., 2023a).Source: §2.1.3 · p. 6

Safety Aligned Language Models

• Instruction tuning fine-tunes LLMs on instruction-based tasks, enabling them to execute human directives even without demonstrations or on unfamiliar tasks (Liu et al., 2023c).Instruction datasets may derive from annotated sources such as Flan and P3 or synthetic datasets such as InstructWild and Self-Instruct.Source: §2.2 · p. 7
• LLMs are pretrained to minimize contextual word-prediction errors, whereas users expect them to follow instructions usefully and safely, creating a fundamental objective mismatch (Yang et al., 2023b; Carlini et al., 2023).Limited instruction–answer pairs can impair instruction following, while internet-scale training data can reproduce bias, toxicity, and profanity (Bai et al., 2022).Source: §2.2 · p. 7
• RLHF trains reward models from human preferences and fine-tunes pretrained models to encourage desirable outputs while discouraging undesirable ones (Ouyang et al., 2022; Bai et al., 2022; Glaese et al., 2022; Korbak et al., 2023).These alignment methods support the goal of producing helpful and harmless behavior and have generated content that generally conforms to agreeable standards.Source: §2.2 · p. 7

2.2.1 Adversarial Attacks

• Adversarial attacks can induce unexpected classifier outputs, such as making a stop-sign image appear to be a speed-limit sign, with potential autonomous-vehicle safety consequences.Biggio et al. and Szegedy et al. (Szegedy et al., 2013) independently observed this vulnerability.Source: §2.2.1 · p. 7
• Attackers backpropagate loss to the input and apply small, carefully crafted perturbations along its gradient, enabling large output changes under a limited noise budget.The noise constraint keeps attacks difficult to detect; following the loss gradient allows small perturbations to produce large output changes (Szegedy et al., 2013).Source: §2.2.1 · p. 7
• Adversarial attacks are studied both to evaluate worst-case security and robustness and to improve models against threats such as content-filtering or malware-detection evasion.Adversarial training exposes networks to adversarial examples during training (Madry et al., 2017), while adversarial instances support verification of high-stakes neural networks.Source: §2.2.1 · p. 7

2.2 Security of ML Models

• Attacks may be targeted, untargeted, or universal, respectively steering outputs toward a chosen value, causing any error, or misclassifying inputs from a given class.Untargeted attacks follow the overall loss gradient, whereas targeted attacks follow the gradient toward the target class.Source: §2.2 · p. 8
• FGSM perturbs inputs once along the loss gradient, whereas stronger PGD iteratively maximizes loss within an ϵ-ball and is commonly used to assess model resilience.PGD projects each update back into the permitted ϵ-ball and can detect smaller perturbations than FGSM.Source: §2.2 · p. 8
• Because text is discrete, NLP attacks modify characters, words, sentences, or combinations thereof through operations such as insertion, deletion, swapping, and paraphrasing.Multi-level attacks can increase success rates and reduce human detectability, but are more complex and computationally demanding.Source: §2.2 · p. 8, §2.2.2 · p. 8

2.2.2 Threat Models: Black-box vs White-Box

• White-box attacks use model parameters and gradients to alter input embeddings, while black-box attacks infer a substitute model through repeated input-output queries before attacking it.Adversarial objectives balance misleading the model against keeping perturbations imperceptible, with adversarial loss often based on cross-entropy or divergence measures.Source: §2.2.2 · pp. 8–9

• The section covers two prevalent attack types against aligned unimodal LLMs: jailbreak attacks and prompt injection attacks.Source: §3 · p. 9
• For each attack type, reviewed studies are categorized by underlying assumptions, methodological differences, study scope, and main insights.Source: §3 · p. 9
• The section synthesizes and relates the studies to provide an overall understanding of the state of the art in each attack area.Source: §3 · p. 9

3.1.1 Initial Ad hoc Jailbreak Attempts

• Initial ad-hoc attacks extracted memorized PII from aligned models through hypothetical scenarios, fake acknowledgments, and multi-step prompts exploiting language-modeling objectives.Li et al. (2023a) found many guessed email addresses matched real training examples, while Bing still disclosed personal information through direct prompts.Source: §3.1.1 · pp. 10–11

3.1.2 Analyzing In-The-Wild (Ad-hoc) Jailbreak Prompts and Attack Success Rates

• In-the-wild jailbreak prompts became shorter and more toxic over time, while GPT-4 resisted known ad-hoc prompts better but exposed new attack surfaces through elevated capabilities.Shen et al. (2023a) identified 666 jailbreaks from 6387 prompts, and Liu et al. (2023e) classified common strategies as Pretending, Attention Shifting, and Privilege Escalation.Source: §3.1.2 · p. 12, §3.1.1 · p. 12, §3.1.3 · p. 13
• 70.8%, 68.9%, 65.5%, 89.0%, and 64.8% ASRs were achieved against ChatGPT, GPT-4, ChatGLM, Dolly, and Vicuna, while safeguards reduced ASR only marginally.Dolly also reached 85.7% ASR without jailbreak prompts; OpenAI Moderation, OpenChatKit Moderation, and Nvidia NeMo Guardrails reduced average ASR by 3.2%, 5.8%, and 1.9%, respectively.Source: §3.1.2 · pp. 12–13, Figure 5 · p. 13

3.1.4 Automating Jailbreak Prompt Generation and Analyzing Defenses in LLM Chatbots

• 14.51% and 13.63% average success rates against Bard and Bing Chat showed that reward-ranked automated generation could breach defenses that ad-hoc prompts rarely penetrated.Deng et al. (2023) used dataset augmentation, Vicuna 13b training, and reward feedback to generate adaptive jailbreak prompts.Source: §3.1.4 · pp. 15–16
• Zou et al. (2023) optimized a random suffix with adversarial gradients to maximize fixed affirmative target outputs for batches of forbidden questions under white-box access.The questions and target responses remain fixed while only the appended suffix is updated.Source: §3.1.4 · p. 16
• Universal adversarial suffixes transferred across ChatGPT, Bard, Claude, LLaMA-2-Chat, Pythia, Falcon, MPT-7b, Stable-Vicuna, PaLM-2, and ChatGLM, with GPT-based models most vulnerable.The suffix was jointly adapted across Vicuna 7b, Vicuna 13b, and Guanaco, enabling transfer across models.Source: §3.1.4 · p. 17
• Claude’s attack success rate was around 2.1%, but simulating word mappings bypassed its input filter and contaminated subsequent conversational context.This strategy drew on virtualization attacks (Kang et al., 2023) and context contamination (Shayegani et al., 2023).Source: §3.1.4 · p. 17
• Automated attacks generate many optimization-based vectors from random starts, making manual patching ineffective and exposing mismatched generalization in safety training (Wei et al., 2023a).Earlier ad-hoc jailbreaks depended on user creativity and did not reach these complex attack surfaces.Source: §3.1.4 · p. 17

3.2.1 Prompt Injection Definition, Instruction Following, Model Capabilities, and Data Safety

• Prompt injection manipulates inputs so LLMs treat attacker-controlled data as instructions, whereas jailbreaks bypass alignment restrictions to elicit otherwise prohibited outputs.This vulnerability arises because the final prompt combines system instructions with user input, obscuring the boundary between safe instructions and unsafe data.Source: §3.2.1 · p. 17
• Stronger instruction-following capabilities can increase susceptibility, as larger models such as GPT-4 more readily recognize and execute instructions embedded in data (McKenzie et al., 2023; Wei et al., 2023a).Attackers exploit competition between instruction following and language modeling, causing user-provided data to be interpreted as a fresh instruction.Source: §3.2.1 · pp. 17–18

3.2.2 Exploring Prompt Injection Attack Variants

• Direct attacks insert adversarial text into prompts, while indirect attacks hide instructions in retrieved websites or documents; virtual attacks instead implant behavior during instruction tuning.Indirect attacks are especially relevant to retrieval-augmented and multi-agent systems, where external content or one agent’s output can contaminate another agent’s input.Source: §3.2.2 · p. 19, §3.2.4 · pp. 21–22
• Goal hijacking redirects an application’s objective, whereas prompt leaking seeks the initial system prompt, whose disclosure can expose valuable intellectual property (Perez and Ribeiro, 2022).Instruction-following models may interpret malicious user text as a new task rather than content to process (Wei et al., 2023a; McKenzie et al., 2023).Source: §3.2.2 · pp. 19–20
• 58.6% goal hijacking success exceeded 28.6% prompt leaking success, while weaker Text-Davinci-001 and Text-Curie-001 models were more resilient (Perez and Ribeiro, 2022).The authors tested 35 application scenarios built on OpenAI models, including grammar checking, classification, and extraction tools.Source: §3.2.2 · pp. 19–20

3.2.3 System Prompt As Intellectual Property

• Prompt leaking exposed system prompts in 73.1% of Vicuna-13B, 89% of GPT-3.5, and 81.9% of GPT-4 trials, undermining system prompts as secure intellectual property (Zhang and Ippolito, 2023).The study used fully black-box attacks with fewer than 20 queries, and found output monitoring inadequate because capable models can encode or obfuscate leaked outputs.Source: §3.2.3 · pp. 20–21

3.2.4 Exploring Indirect and Virtual (Training Time) Prompt Injection Attacks

• Indirect prompt injection can make retrieval-enabled LLMs socially engineer users, such as by hiding New York Times information and fabricating reasons to distrust the source (Greshake et al., 2023a).Attackers must understand the model’s accessible sources and retrieval mechanisms to place malicious instructions where the system will ingest them.Source: §3.2.4 · p. 21
• Contaminating only 0.1% of an instruction-tuning dataset, about 52 Alpaca samples, increased negative responses about Joe Biden from 0% to 40% through virtual prompt injection (Yan et al., 2023).The attack implants a targeted behavior during training, so the malicious instruction need not appear in the victim’s inference-time prompt.Source: §3.2.4 · pp. 22–23
• A malicious instruction-tuning mini-dataset merged with clean data can cause a compromised LLM to discredit The New York Times and produce biased news responses without users’ awareness.The attack operates during fine-tuning and manifests when users ask news-related questions.Source: Figure 11 · p. 23
• Virtual prompt injection can combine with information gathering, disinformation, advertising, manipulation, and social engineering, motivating careful dataset curation and further evaluation of filtering defenses.Public instruction-tuning datasets may contain attacker-curated malicious samples that evade simple low-quality-data filtering, including filtering assisted by another powerful LLM.Source: §3.2.4 · p. 23
• Yan et al. (2023) show that virtual training-time prompts can implant hidden behaviors, such as inserting “pwned!” into generated Python code, while also enabling helpful behaviors like chain-of-thought reasoning.The same mechanism could request harmful actions such as installing a backdoor, or implicitly elicit “Let’s think step by step” behavior.Source: §3.2.4 · p. 23

3.2.5 Enhancing Prompt Injection Attacks: Automation and Countermeasures

• Existing prompt injection attacks are ineffective against many real-world applications because user input is treated as data and application formats transform inputs or outputs.These development choices make it difficult for attackers to make the underlying LLM interpret user input as instructions.Source: §3.2.5 · p. 24
• Liu et al. (2023d) address this boundary by using separator components that terminate preceding context and make injected input appear as a new instruction beside the system prompt.The approach is inspired by SQL injection methods that end prior context and begin a new sub-query.Source: §3.2.5 · p. 24
• Liu et al. (2023d) automated black-box prompt injection through application-context inference, separator/disruptor generation, and dynamic feedback refinement.An LLM extracts application semantics, constructs a cover framework, uses a separator to break contextual continuity, and updates the separator and disruptor based on generated-answer evaluations.Source: §3.2.5 · p. 24, Figure 12 · p. 24
• 86.1% success in prompt leaking attacks, with 31 of 36 investigated applications susceptible to automated prompt injection attacks.Writesonic exposed its initial system prompt, while Parea enabled goal hijacking that allowed unintended use of its underlying LLM.Source: §3.2.5 · p. 25

4 Multi-Modal Attacks

• Multi-modal inputs create new attack surfaces because models fuse text, images, audio, video, depth, or thermal signals into a joint embedding for generation.These systems include models such as LLaVA, MiniGPT-4, Flamingo, GPT-4, PaLM-E, and Med-PaLM 2.Source: §4 · p. 25

4.1 Manual Attacks

• Visual prompt injections exploit vision encoders’ preference for readable text over image content, causing models to follow instructions embedded in images.Adding “dog” text to cat images confused LLaVA, while Google Bard and Microsoft Bing followed raw textual instructions in visual inputs.Source: §4.1 · p. 25

4.2 Systematic Adversarial Attacks

• Optimized images and audio recordings can stealthily induce toxic content, context contamination, jailbreaks, hidden instructions, and context leaking.These attacks are more intricate than directly adding text to images or audio.Source: §4.2 · p. 25

4.3 White-Box Attacks

• White-box attacks optimize continuous image inputs or teacher-forced outputs to produce toxic or arbitrary target strings, while universal adversarial inputs generalize across harmful categories and vision-language models.Carlini et al. (2023) demonstrated image attacks on MiniGPT-4, LLaVA, and LLaMA-Adapter; Qi et al. (2023) found significant transferability among Mini-GPT4, InstructBLIP, and LLaVA.Source: §4.3 · p. 26

4.4 Black-box Attack

• Black-box attacks need only knowledge of a public vision encoder and can bypass textual alignment by hiding malicious content in another modality, achieving high success rates across multi-modal models.Shayegani et al. (2023) achieved alignment escaping, context contamination, and hidden prompt injection; frozen encoders remained vulnerable after LLaVA switched from Vicuna to Llama-2.Source: §4.3 · p. 26, §4.4 · pp. 27–28

5 Additional Attacks

• LLMs embedded in autonomous or multi-component systems can become central agents whose dependencies create vulnerabilities beyond unimodal and multimodal attacks.The section motivates complex-system attacks with an LLM acting as a proxy between a client and web service (Beckerich et al., 2023).Source: §5 · p. 28

5.1 Adversarial Attacks In Complex Systems

• Research on complex-system attacks remains less advanced and is organized into integrated-system, multi-agent, and structured-data attacks.Figure 15 presents these complex systems and possible adversarial attacks.Source: §5.1 · p. 28, Figure 15 · p. 29

5.1.1 LLM Integrated Systems.

• Integrated LLM systems are targeted through retrieval-model attacks, SQL injection, and proxy attacks (Greshake et al., 2023b; Pedro et al., 2023; Beckerich et al., 2023).These attacks operate when the LLM is connected to other system components.Source: §5.1.1 · p. 28, §5.1 · p. 29
• External-information retrieval improves LLM performance but exposes systems to adversarial manipulation of the sources they query.Such systems are used in medical, financial, and legal research, where information integrity is critical.Source: §5.1 · p. 29
• Greshake et al. (2023b) describe “Arbitrarily-Wrong Summaries” and “Source Blocking,” which can distort retrieved answers or prevent use of a targeted information source.Source Blocking uses crafted prompts and instructions to make a retrieval-augmented LLM avoid a particular source.Source: §5.1 · p. 29
• LangChain-mediated SQL systems permit direct chatbot attacks and indirect database poisoning that can expose, insert, modify, or delete database data.The workflow has LangChain translate user questions through an LLM into SQL, query the database, and use the results to generate the final response.Source: §5.1 · p. 29
• 60% of attempts succeeded in disrupting an SQL chatbot through malicious prompt fragments inserted into its database (Pedro et al., 2023).Figure 16 also illustrates a direct restricted-prompting attack that deletes a database table.Source: §5.1 · p. 29, Figure 16 · p. 29

5.1 Adversarial Attacks In Complex Systems

• Proxy attacks use an LLM as an attacker-controlled web-service connection, reducing traceability because the LLM generates the IP address.The attack proceeds through prompt initialization, IP generation, payload generation, and server communication.Source: §5.1 · p. 30
• The proxy payload tricks LLM safeguards, generates a server IP through mathematical operations, and relays harmful commands to the victim client.Figure 17 depicts the payload execution and communication flow.Source: §5.1 · p. 30, Figure 17 · p. 30

5.1.2 Multi-Agent Systems

• LLM-based multi-agent systems extend machine interaction through cooperating agents, creating additional settings in which adversarial vulnerabilities may arise.Examples include systems with vocabulary, speech-to-text, text-to-speech, and query-analysis agents, including approaches described by Wang et al. (2023c), Huang (2021), and Aref (2003).Source: §5.1.2 · p. 30
• Federated learning protects client-data privacy through local training and server aggregation, but adversarial and Byzantine attacks threaten the resulting global model.Adversarial attacks manipulate inputs or models, whereas Byzantine attacks introduce malicious client behavior into the federated process.Source: §5.1.2 · p. 30
• Even a small number of malicious federated-learning clients can degrade global-model quality by submitting harmful updates that aggregation fails to detect.Crafted input perturbations can also deceive the trained global model and produce inaccurate predictions.Source: §5.1.2 · p. 30

5.2 Earlier Adversarial Attacks In NLP

• Test loss increased to 8–14 during training when one of seven federated-learning clients launched a random-mode Byzantine attack.FedMLSecurity used seven clients, with C1 malicious in each training round.Source: §5.1 · p. 30, §5.1.2 · p. 30, §5.2 · p. 31

5.1.3 Attacks On Structured Data.

• Data-manipulation attacks such as SQL injection can modify or delete database tables, while TaLMs classify tabular data from natural-language descriptions and examples.Hegselmann et al. (2023) reports that this approach can outperform other methods, including when no prior examples guide classification.Source: §5.1.3 · p. 31
• Structured-data attacks replace important table entities with adversarial entities selected from the same class but made maximally dissimilar to the originals.Koleva et al. (2023) identifies entities using logit differences between their presence and masking, then replaces a selected percentage.Source: §5.1.3 · p. 31

Taxonomy

• Earlier NLP attacks comprise four classes—character-, word-, sentence-, and multi-level methods—with Figure 19 illustrating representative examples.Goyal et al. (2023a) reviews these attacks across different perturbation levels.Source: Figure 19 · p. 31, §5.2 · p. 31

Character-Level

• Character-level attacks insert, delete, or swap characters, using natural or synthetic noise; DeepWordBug and TextBugger perturb important words, while punctuation and spacing changes can alter toxicity scores.These methods are effective but can be detected by spell-checkers; DeepWordBug (Gao et al., 2018) is black-box, whereas TextBugger (Li et al., 2019) supports black-box and white-box settings.Source: §5.2 · pp. 31–32

Sentence-Level

• AdvGen creates adversarial examples using greedy search guided by training loss while preserving semantic meaning, whereas SCPNS uses an encoder-decoder network to generate examples with specific syntactic structures.Source: §5.2 · p. 32

Multi-Level

• Multi-level attacks combine character- and word-level operations to make perturbations less noticeable and improve attack success, using gradient, importance, and reinforcement-learning-based techniques.HotFlip (Ebrahimi et al., 2017), TextBugger (Li et al., 2018), and hot phrase methods manipulate text through insertion, deletion, or swapping in white-box and black-box settings.Source: §5.2 · p. 32

6.1 Possible Causes

• Adversarial vulnerabilities arise from non-robust, transferable features; insufficient exposure to diverse adversarial examples; long-text filtering limits; imperfect alignment; and semantic-censorship bypasses.Unusual or outlier words can trigger harmful outputs, long texts can defeat semantic harm filters, and attackers can assemble prohibited content from individually permissible requests.Source: §6.1 · pp. 33–34

6.2 Defense

• Defenses range from threat-specific recognition to systematic resilience through adversarial training or tools that detect and respond to adversarial inputs.The survey distinguishes casual defenses, which may overlook threats, from systematic defenses designed to strengthen LLM resilience.Source: §6.2 · p. 34

6.2.1 Textual

• Textual defenses include hyperparameter tuning, auditing, input/output filtering, human feedback, red teaming, and adversarial training, but filtering and training remain vulnerable to evasion and longer adversarial sequences.Human feedback during pretraining can improve alignment, red teaming exposes harmful behaviors, and adversarial training uses perturbed examples, yet external filters can be bypassed through encoding or invertible transformations and universal robustness remains unresolved.Source: §6.2.1 · pp. 34, 36, 37, 38
• ARCA outperforms AutoPrompt and GBDA in average success rate when auditing GPT-J and GPT-2, with prompts from smaller models often transferring to larger models such as GPT-3 davinci-002 (Jones et al., 2023b).ARCA audits targeted toxicity, surprise toxicity, and cross-language behavior, and can produce more natural prompts using advanced language models as regularizers and human qualitative judgment.Source: §6.2.1 · p. 35

6.2.2 Multimodal

• No established defenses specifically target multimodal LLM adversarial attacks, although input denoising, filtering, smoothing, anomaly detection, and certified robustness are proposed candidates.These techniques aim to remove adversarial noise while preserving legitimate information or provide mathematical guarantees within a defined parameter space.Source: §6.2.2 · p. 38
• Output filtering, representative training data, and diverse ensembles can reduce anomalous or bias-exploiting outputs and make multimodal attacks harder to craft successfully.Ensembles combine models with different architectures or training procedures, reducing the chance that an attack deceives all models simultaneously (Steinhardt et al., 2017; Dong et al., 2018).Source: §6.2.2 · pp. 38–39
• No multimodal defense is foolproof, so maintaining robustness requires combining multiple techniques with ongoing research and monitoring as attacks evolve.This limitation applies to real-world multimodal large language model applications.Source: §6.2.2 · p. 39
• Adversarial training improves multimodal LLM robustness by incorporating adversarial and out-of-distribution examples, while dropout, weight decay, and layer normalization regularize against adversarial overfitting.The approach generates adversarial examples during training and combines them with regular examples to promote robust representations (Madry et al., 2017; Kurakin et al., 2016).Source: §6.2.2 · p. 39

6.2.3 Federated Learning Settings

• m-Krum mitigated randomly injected Byzantine attacks in federated learning by excluding malicious client models before aggregation, bringing test loss closer to the no-attack level over communication rounds.With n = 7 clients, m = 2 models with the lowest Krum scores were aggregated; Han et al. (2023) report decreasing test loss as rounds increased.Source: §6.2.3 · p. 39, Figure 22 · p. 39

• The survey covers textual-only attacks, including jailbreaks that bypass alignment restrictions and prompt injections that manipulate model outputs to an attacker’s advantage.Source: §7 · p. 39
• It reviews multimodal attacks exploiting embedding-space vulnerabilities, such as compromised images used to achieve jailbreaks or prompt injections, alongside attacks on LLM-integrated and multi-agent systems.Source: §7 · pp. 39–40
• The survey examines underlying causes and proposed defenses, emphasizing that offensive security research supports threat-model understanding, safer system design, evaluation benchmarks, alignment work, and new protection models.Source: §7 · p. 40