PaperlayerView exact revision on arXiv

Source-linked AI summary

Lessons from Defending Gemini Against Indirect Prompt Injections

Chongyang Shi, Sharon Lin, Shuang Song, Jamie Hayes, Ilia Shumailov, Itay Yona, Juliette Pluto, Aneesh Pappu, Christopher A. Choquette-Choo, Milad Nasr, Chawin Sitawarin, Gena Gibson, Andreas Terzis, John "Four" Flynn

arXiv:2505.14534v1cs.CRcs.LG

TL;DR

Indirect prompt injections can place malicious instructions in retrieved data and redirect Gemini’s agentic tasks. The report evaluates Gemini with adaptive attacks and describes using those evaluations to improve resilience. It finds that greater model capability does not ensure greater robustness, while adversarial training can improve resistance to specific attacks without harming general capabilities.

  • Problem

    Tool-using models can retrieve untrusted data containing malicious instructions that redirect their behavior and expose private information.

    Source: §1 · pp. 3–4

  • Method

    The report uses a continuous adversarial evaluation framework with adaptive attacks to test Gemini models and inform adversarial training.

    Source: §4 · p. 9, §5.1 · p. 12

  • Results

    More capable models can sometimes be easier to manipulate, while adversarial training can improve robustness against specific attacks without harming general model capabilities.

    Source: §10 · p. 23, §12 · p. 25

  • Takeaways & Limitations

    Adversarial training can help, but defense in depth is necessary because the space of possible attacks cannot be fully covered.

    Source: §10 · p. 23, §12 · p. 26

    Training and evaluation use the same attack algorithms, so protection against unknown attacks cannot be quantified and adversarial training is not sufficient alone.

    Source: §10 · p. 23

Abstract

from arXiv · show

Gemini is increasingly used to perform tasks on behalf of users, where function-calling and tool-use capabilities enable the model to access user data. Some tools, however, require access to untrusted data introducing risk. Adversaries can embed malicious instructions in untrusted data which cause the model to deviate from the user's expectations and mishandle their data or permissions. In this report, we set out Google DeepMind's approach to evaluating the adversarial robustness of Gemini models and describe the main lessons learned from the process. We test how Gemini performs against a sophisticated adversary through an adversarial evaluation framework, which deploys a suite of adaptive attack techniques to run continuously against past, current, and future versions of Gemini. We describe how these ongoing evaluations directly help make Gemini more resilient against manipulation.

1. Introduction

Agentic tools let Gemini retrieve untrusted data that can carry malicious instructions, potentially redirecting the model from the user’s task. The report describes this threat and its evaluation and defense lessons.

1. Introduction

  • Retrieved malicious instructions can redirect tool-using Gemini from a user’s task and cause it to expose private data.Attackers can plant commands in retrieved data, such as an email, to induce a different function call.Source: §1 · pp. 3–4
  • Models with stronger instruction-following capabilities are sometimes easier to attack, so general capability gains do not automatically improve robustness.The report also cautions that weak instruction following should not be mistaken for security.Source: §1 · p. 4
  • Static evaluations can overstate protection because small adaptations to attacks can trick defenses that perform well on fixed tests.The report calls for evaluating robustness against both non-adaptive and adaptive attacks.Source: §1 · p. 4
  • Adversarial training can improve resistance to attacks without noticeable degradation on other tasks, but is not sufficient as a standalone defense.The report frames it as one step toward models that assess the risk of malicious instructions.Source: §1 · p. 4
  • The report focuses evaluation on realistic function-calling scenarios with clear exfiltration paths and attacks that generalize across contexts.Such attacks can be constructed cheaply and target many users in parallel.Source: §1 · p. 4
  • The report outlines threat modeling, systematic attacks, Gemini 2.0 evaluations, and how evaluation insights informed Gemini 2.5 improvements.It also discusses ongoing robustness metrics, related work, and future work.Source: §1 · pp. 4–5

2. Terminology

The report distinguishes security from safety and defines indirect prompt injection as a threat arising when malicious instructions enter through retrieved external data. It frames the challenge as preserving user intent while handling untrusted content.

2. Terminology

  • Safety concerns harmful outputs under normal operation, whereas security concerns explicit adversarial manipulation and worst-case system performance.Formal worst-case guarantees for machine-learning models remain limited to small, toy settings.Source: §2 · p. 5
  • An indirect prompt injection embeds malicious instructions in external data that the model later retrieves and incorporates into context.A direct prompt injection instead has the end user deliberately provide the malicious input; a jailbreak is a direct injection aimed at bypassing safety mechanisms.Source: §2 · p. 5
  • Defending against indirect injections requires distinguishing trusted instructions from untrusted data while maintaining user intent in dynamic agentic tasks.The report focuses on this threat because of its relevance to tool-use scenarios.Source: §2 · pp. 5–6
  • The prompt combines conversation history, system and user instructions, and retrieved data, while retrieved untrusted data may contain an attack trigger, harmless content, or both.The report names the malicious input the “Trigger” and the user-authored instruction the “User Prompt.”Source: §2 · p. 6

3. Threat Model

The threat model considers agents using function calls to access external data that an adversary can manipulate. It models attacks that seek to make the agent exfiltrate sensitive information, while specifying attacker capabilities and practical evaluation conditions.

3. Threat Model

  • The threat model assumes a function-calling agent retrieves external data that an adversary can manipulate with malicious content.The setting includes sources such as emails, documents, and web pages.Source: §3 · p. 6

3.1. Attack Description and Adversarial Goal

  • The modeled attack plants a payload in retrieved data and seeks to make the agent call a function that extracts sensitive information.The user prompt is benign, and sensitive data may come from retrieved content or prior conversation; the model’s generated function calls are assumed to execute externally.Source: §3.1 · p. 6

3.2. Adversarial Capabilities

  • The adversary controls external data, may know the user prompt and likely data types, but does not know the specific private data in advance.During attack crafting, access may be blackbox, observing final outputs, or graybox, with additional access such as output probabilities.Source: §3.2 · p. 7

3.3. Practical Attack Formulation

  • Because true data and prompt distributions are unknown, practical attack crafting optimizes an injection over fictional training examples drawn from a proxy distribution.Blackbox attacks can minimize empirical failure rate; graybox attacks may use cross-entropy, but effectiveness is ultimately measured by autorater success on unseen data.Source: §3.3 · pp. 7–8

3.4. Example Scenario

  • The example traces an injected email instruction redirecting an email-summary task into forwarding private emails to an attacker.Evaluations run directly on the model in a controlled environment; real deployments add security guardrails and mitigations that affect attack success.Source: §3.4 · p. 8

4. Why Do We Need Specialized Automated Red-Teaming?

The report argues that security evaluation needs realistic, scalable red-teaming against adaptive attacks, rather than relying only on manual tests or conventional safety evaluations. Its framework is designed to evaluate attacks and defenses across scenarios.

4. Why Do We Need Specialized Automated Red-Teaming?

  • Conventional safety evaluations do not adequately assess security, because a model can produce policy-compliant outputs yet still follow injected instructions.Security evaluation should simulate realistic scenarios with benign and adversarial inputs and assess their implications.Source: §4 · pp. 8–9
  • Manual red-teaming found early attacks but could not scale with stochasticity and rapid model development; automation enables diverse, systematic testing over time.Manual attack crafting is time-consuming and may miss potential attack vectors.Source: §4 · p. 9
  • The framework combines realistic function-call scenarios, adaptive attacks, and automated criteria for verifying attacker-directed calls and sensitive-data exfiltration.Its scenarios use synthetic datasets modeled on real-world data formats and injection points.Source: §4 · p. 9
  • Attack generalization measures the share of unseen, unique private-data instances successfully exfiltrated and serves as the primary measure of attack strength and model resilience.Resilience is assessed by aggregating across a diverse range of attacks.Source: §4 · p. 9
  • Scenario-agnostic attack techniques can adapt to different goals, while scenario-agnostic evaluations can cover realistic settings without assuming extra adversary controls.This separation supports evaluating defenses under adaptive attacks and using broad attack coverage to train for more generalized mitigation.Source: §4 · pp. 9–10

5. Adversarial Techniques and Evaluation Setup

The evaluation framework automates indirect-injection attacks across access settings, tests them on synthetic function-calling scenarios, and measures whether private data is exfiltrated. Its setup is designed to assess triggers that generalize across varied conversation histories and attack scenarios.

5.1. Attack Design

  • Four automated attacks cover black-box and gray-box access, generating indirect prompt-injection triggers for evaluation.They include Actor-Critic, Beam Search, TAP, and Linear Generation.Source: §5.1 · pp. 10–12
  • Linear Generation expands prior successful triggers into many thousands of diverse triggers and is used to create training data.The other three attacks typically produce a handful of successful triggers after completion.Source: §5.1 · p. 12

5.2. Dataset Setup and Creation

  • Each dataset prompt combines system instructions, synthetic conversation history with private data, a user function-call request, retrieved content containing a trigger, and the model response.Conversation histories include up to 10 turns; each selected scenario is used to create 2,000 prompts.Source: §5.2 · pp. 12–13
  • Triggers are optimized on randomly selected batches of training prompts and then evaluated on held-out synthetic conversation histories.Random selection is intended to reduce overfitting to narrow training examples and improve success on validation and test sets.Source: §5.2 · p. 13

5.3. Metrics

  • Attack success rate measures the percentage of simulated private information exfiltrated, with peak test-set ASR and optimization queries reported for each attack class.Because the attacker lacks prior knowledge of conversation histories, high ASR requires triggers to generalize across them.Source: §5.3 · p. 14

5.4. Data Exfiltration Scenarios

  • Scenarios vary the private-data type, retrieved-content format, and exploited function; reported examples use passport numbers, SSNs, or reset tokens, JSON or non-JSON triggers, and email or calendar functions.The report focuses on these representative data types and functions, while the wider evaluation also models other goals such as financial transactions.Source: §5.4 · pp. 14–15

6. Assessing Potential Vulnerabilties of an Undefended Model

On undefended Gemini 2.0, the attacks found triggers that exfiltrated private data across the tested scenarios, with TAP generally achieving the strongest results at relatively low query cost. Success was lower for longer password reset tokens than for passport or social security numbers.

6. Assessing Potential Vulnerabilties of an Undefended Model

  • The vulnerability results concern the undefended core model; the publicly available Gemini 2.0 version has additional guardrails.Source: §6 · p. 15
  • Over 70% of test examples were successfully attacked in every setting by at least one of the three evaluated attacks.Evaluations used 500 test examples per data-exfiltration scenario on undefended Gemini 2.0; Figure 4 plots ASR against trigger-construction queries across the tested types, formats, and functions.Source: §6 · p. 15, Figure 4 · p. 16
  • TAP achieved close to 100% ASR in nearly all settings and generally needed fewer than a few thousand queries to construct a trigger.The reported attacker cost to create a highly successful trigger against Gemini 2.0 Flash was less than $10.Source: §6 · p. 15
  • All three attacks struggled more with password reset tokens than with passport or social security numbers, likely because reset tokens average 65 characters versus at most 10.Source: §6 · p. 15
  • Actor-Critic and Beam Search generally required more queries than TAP while achieving similar success rates.Source: §6 · p. 15

7. Evaluating Defenses with Non-Adaptive Attacks

The report evaluates prompt-based and classification defenses against triggers optimized without defenses. Most tested in-context defenses were only marginally successful, while Warning performed better; adaptive attacks are addressed in the following section.

7.1. Defenses

  • The evaluated defenses include prompt alterations and classifiers that predict whether an injection occurred.The classifier tests use either retrieved data with the model response or the trusted user request with the response; in-context ASR counts successful attacks missed by the defense.Source: §7.1 · pp. 16–17
  • Defenses can reduce general model capability or user utility, including legitimate sharing of private information with trusted parties.Source: §7.1 · p. 17

7.2. Results

  • Three of four in-context defenses only marginally stopped attacks, while Warning was more successful; classification defenses had middling performance.Figure 5 compares peak ASR across defenses and reports the queries needed to construct the selected trigger.Source: §7.2 · p. 17, Figure 5 · p. 18
  • At a calibrated 1% false-positive rate, the best attack still achieved > 90% ASR against Perplexity.The authors expect Perplexity to help mainly against Beam Search triggers, which add high-probability random-token suffixes, rather than natural-language Actor-Critic or TAP triggers.Source: §7.2 · pp. 17–18
  • Among the in-context defenses, the authors attribute Warning's stronger performance relative to ICL to the placement of defensive instructions in the prompt.Warning instructions are added after retrieved content, whereas ICL examples precede retrieval.Source: §7.1 · p. 16, §7.2 · p. 17

8. Evaluating Defenses with Adaptive Attacks

The authors evaluate defenses with attacks optimized against those defenses, rather than relying only on fixed triggers. Adaptive attacks generally achieved higher ASR than non-adaptive attacks, though some cases showed the reverse.

8.1. Methodology

  • The adaptive evaluation optimizes attacks against Gemini 2.0 with each defense in place, then measures the resulting triggers on a held-out set.The attacks were designed to overcome possible obfuscation or defense-specific properties.Source: §8.1 · pp. 18–19

8.2. Results

  • Adaptive attacks matched or exceeded non-adaptive ASR in 16 of 24 cases across eight defenses and three attacks.Adaptive Actor-Critic, Beam Search, and TAP each had higher ASR against 5, 5, and 6 of 8 defenses, respectively; Figure 6 plots ASR and query counts for adaptive and non-adaptive attacks.Source: §8.2 · p. 19, Figure 6 · p. 19
  • The authors conclude that success under non-adaptive evaluation does not necessarily indicate resilience to an adversary adapting attacks to a specific defense.Source: §8 · p. 18, §8.2 · p. 19
  • Non-adaptive attacks sometimes outperformed adaptive ones, which the authors suspect may reflect a noisy or non-smooth optimization objective on the defended model.They suggest transferred triggers optimized on the undefended model as a robustness-evaluation sanity check.Source: §8.2 · p. 19

9. Improving the Gemini’s understanding of indirect prompt injections through adversarial fine-tuning

Gemini 2.5 was adversarially fine-tuned on diverse tool-use scenarios and corrective responses generated from strong attacks. Evaluations found frequent rejection of injected instructions, while combined model and external defenses reduced attack success further but did not eliminate the threat.

9. Improving the Gemini’s understanding of indirect prompt injections through adversarial fine-tuning

  • Targeted adversarial fine-tuning was intended to complement external defenses because filters can be brittle, bypassed by novel attacks, and leave the trusted-instruction versus untrusted-data confusion unresolved.The report says the procedure was used to develop Gemini 2.5, shifting evaluation from Gemini 2.0 to Gemini 2.5.Source: §9 · pp. 20–21
  • The dataset began with varied realistic conversation histories, tools, user intents, and simulated sensitive information to encourage general defense rather than context-specific overfitting.The report describes a three-step construction process for specialized tool-use datasets.Source: §9 · p. 20
  • The framework generated strong triggers with TAP, Beam Search, and Actor Critic, then used them to seed Linear Generation and produce thousands of successful triggers for training.The resulting attacks included attempts to exfiltrate sensitive information.Source: §9 · p. 20
  • For successful attacks, the pipeline generated corrective responses with the Warning defense, retained responses from failed attacks, and filtered for responses that followed the user's request rather than injected instructions.These responses supervised fine-tuning; reinforcement learning was also applied, without requiring this corrective-response step.Source: §9 · p. 20
  • The training dataset paired tool-use contexts and sensitive information with adversarial triggers and safe responses, with tools, conversation histories, and sensitive values separated across training and test sets.The email function was included in training, while the calendar event function was excluded.Source: §9 · p. 20

9.1. (Adaptive) Evaluation of Gemini 2.5

  • Gemini 2.5 frequently identified and disregarded injected instructions, prioritizing the original user request in the evaluated settings.This behavior aligned with the synthetic corrective responses used for training.Source: §9.1 · p. 22

9.2. Combining adversarial training and other defenses

  • Adversarial training improves Gemini 2.5’s resilience but does not make it immune; successful attacks remain possible with greater effort, novel techniques, or highly tailored exploits.The authors frame model-level improvement as one layer in defense in depth.Source: §9.2 · p. 22
  • 6.2% ASR with the Warning defense plus adversarially fine-tuned Gemini 2.5, versus 10.8% for the Warning defense on Gemini 2.0 and 94.6% for TAP against Gemini 2.5 without defenses.The combined defense performed better than adversarial training alone and the Warning defense alone.Source: Table 1 · p. 21, §9.2 · p. 22

10. Discussion

The discussion argues that robustness must be tested against adaptive attacks and cannot be inferred from general model capability or adversarial training alone. The authors favor defense in depth and note that their evaluations cover a limited set of attack settings.

Optimization for indirect prompt injection is a hard problem

  • Indirect prompt injection optimization is difficult because attacks target a specific response string, and black-box reward signals can struggle to rank candidate responses.In the email setting, the model sometimes inserted a literal PLACEHOLDER instead of private information, leaving its value as an optimization target unclear.Source: §10 · pp. 22–23

More capable models don’t automatically mean they are more robust

  • General capability improvements do not ensure greater robustness; models with better instruction following can sometimes be easier to attack.The authors report no corresponding robustness gains as Gemini capabilities improved, and occasionally observed the opposite.Source: §10 · p. 23

Adaptive evaluation is crucial

  • Adaptive attacks are crucial because static, non-adaptive evaluations can overfit and give a false view of the protection provided.The authors advocate measuring robustness against both adaptive and non-adaptive attacks.Source: §10 · p. 23

Intuition that adversarial training always makes models worse does not hold in practice

  • The report presents its adversarial-training results as a counterpoint to prior findings of utility trade-offs, while arguing that model defenses should be combined with system-level safeguards.The authors describe model-level understanding of malicious instructions as necessary but insufficient on its own.Source: §10 · pp. 23–24

Defense in depth is necessary

  • Adversarial training can improve resistance to known attacks and generalize to new tool settings, but the evaluated training and test attacks use the same algorithms, leaving protection against unknown attacks unquantifiable.The authors therefore describe adversarial training as necessary but not sufficient and advocate model- and system-level defenses.Source: §10 · p. 23

Expanding evaluation settings

  • The evaluation covers single-turn attacks seeking one additional function call and focuses mostly on text, leaving chained attacks and other modalities as future evaluation settings.The authors note that Warning and classifier defenses may overfit to the simpler setting and struggle to generalize to more complex attacks.Source: §10 · p. 24

11. Related Work

Related work spans adversarial examples and training, external and system-level defenses, and benchmarks for evaluating attacks on tool-using agents. The report situates its approach among methods for improving model robustness and approaches that add safeguards outside the model.

Adversarial examples

  • Prior work describes adversarial examples as a longstanding vulnerability, with research on limits to robustness and the possibility that training alone may not resolve such vulnerabilities.The cited work also discusses the possibility of reconciling robustness and accuracy.Source: §11 · p. 24

Adversarial training and Large Language Models (LLMs)

  • Adversarial training has been studied as a way to improve robustness, with prior work reporting both potential benign-task trade-offs and gains against specific constrained adversaries.Alternative approaches include continuous-attack training and latent adversarial training, among others.Source: §11 · p. 24
  • The report describes adversarial training as a defensive layer for identified attacks and identifies distinguishing trusted instructions from untrusted data as a central challenge.It cites work addressing that distinction in the context of indirect prompt injection.Source: §11 · pp. 24–25

Defences and System-Level Mitigations.

  • External defenses include instruction prioritization, warning or control-token approaches, paraphrasing, detection methods, self-reflection, and system architectures that treat the model as untrusted.The cited literature also includes thinking models and analyses of internal activations.Source: §11 · p. 25

Benchmarks and Evaluation Frameworks

  • Benchmarks and frameworks include AgentDojo, InjecAgent, and Houyi for prompt injection and tool-integrated settings, alongside broader safety and harm evaluation suites.The broader examples include HarmBench, AIR-Bench 2024, SafeArena, ST-WebAgentBench, DoomArena, and AgentHarm.Source: §11 · p. 25

12. Conclusion

The report finds that adaptive evaluation and adversarial training can improve Gemini’s resilience to indirect prompt injections, while no single defense provides complete immunity.

12. Conclusion

  • Adversarial training improved robustness against specific attacks without harming general capabilities, although more capable models can sometimes be easier to manipulate.The report cautions that general capability gains do not automatically make a model more secure.Source: §12 · p. 25
  • Continuous testing with adaptive attacks led to direct improvements in Gemini’s resilience against manipulation.The report finds that defenses effective against static attacks can falter against adaptive adversaries.Source: §12 · p. 25
  • Adversarially fine-tuned Gemini 2.5 showed increased resilience, often prioritizing user requests over injected instructions.The report describes this as better differentiation between trusted user instructions and malicious data embedded in external sources.Source: §12 · pp. 25–26
  • No single solution, including adversarial training, offers complete immunity; the report calls for layered model- and system-level defenses.It names integration with system-level defenses such as CaMeL as a direction for future work.Source: §12 · p. 26
  • The report concludes that indirect prompt injection requires continuous research and development to maintain robust mitigations against sophisticated attacks.Its stated goal is to keep raising the complexity and cost for attackers.Source: §12 · p. 26

13. Contributions

This appendix records project contributions and supplements the main evaluation with attack examples, reward-signal analysis, tabulated results, and an attention-tracker evaluation.

13. Contributions

  • The contributions section assigns project roles across attack and defense design, data and infrastructure, evaluations, post-training defenses, and project guidance.The listed work includes leadership, implementation, experiments, writing, and program management.Source: §13 · p. 26

Beam Search

B. Challenges in Optimizing Prompt Injection Reward Signals under the Actor Critic Attack

  • Actor-Critic uses an Actor to modify unsuccessful trigger seeds and a Critic reward to guide edits toward malicious tool-use, but the current score guides optimization poorly.The authors report inefficient exploration and slow convergence toward successful injections.Source: B. Challenges in Optimizing Prompt… · p. 34
  • Log-probability scoring has calibration, averaging, length, and output-fragility problems, and tested variants yielded no practical efficiency gains.Proposed future alternatives include semantic-similarity and classifier-based rewards.Source: B. Challenges in Optimizing Prompt… · pp. 34–35

C. Tabular version of results from Section 6

F. Best-of-N style attacks

  • 44% training and 32% validation ASR, up from 8% and 4%, followed simple trigger edits; the successful-response probability score showed little correlation with ASR.The attack ran for 100 queries, modifying triggers by randomly removing vowels and changing character case.Source: F. Best-of-N style attacks · p. 37, Figure 7 · p. 37

G. Attention tracker

  • The attention tracker detects prompt injections through attention-pattern disruption and extends prior evaluations to multi-turn interactions and automated attacks.It was evaluated on the same model used to generate attacks.Source: G. Attention tracker · pp. 37–38

G.1. Experimental Setup

  • The attention-tracker evaluation used Gemma-2-9B-IT because evaluating the technique on Gemini models was computationally costly, with adversarial and benign email datasets.Classification defenses were assessed by average TPR and maximum benign-content FPR; in-context defenses by maximum ASR and average benign text quality.Source: §G.1 · p. 38

G.2. Results and Analysis

The section analyzes defenses against indirect prompt-injection attacks, including classifier, self-reflection, and perplexity defenses, translated-trigger attacks, and a full prompt example. It reports variation in defense effectiveness and effects on benign outputs, and presents attention tracking as an early detection heuristic while noting the need for further work on separating instructions from data.

G.2. Results and Analysis

  • At a higher-detection threshold, the attention tracker reached 87% TPR on Beam Search and 69% on Actor Critic attacks, with 10% FPR.Figure 8 shows distinct but overlapping benign and adversarial attention-feature distributions; the conservative threshold had accuracy similar to self-reflection, detecting some Beam Search attacks while struggling with Actor Critic and keeping false positives relatively low. ShieldGemma offered the most robust defense with high detection and no false positives, while self-reflection performed strongly with some false-positive risk.Source: Figure 8 · p. 38, §G.2 · p. 39

G.3. Beyond Attention Tracker: Towards Complete Instruction-Data Separation

  • The paper frames attention tracking as an initial heuristic toward separating instruction and data, hypothesizing that an LLM instruction pointer could help identify attempts to hijack execution flow.The authors call for better attention analysis and adaptive thresholds, and for testing defenses against broader injection techniques and complex multi-turn dialogue.Source: §G.3 · p. 40

H.2. Results

  • Across Gemini 1.5 Flash and Pro, defenses ranged from partially to very effective; paraphrasing brought ASR close to zero across attacks while maintaining benign email summarization quality.The evaluation used a non-adaptive setting focused on email function calling and passport-number exfiltration; the authors note that defense efficacy can differ from Gemini 2.0 results, warranting fresh evaluation for each model version.Source: H. A deep dive into baseline defense… · p. 40, §H.2 · pp. 41–42, Table 6 · p. 41
  • On Gemini 1.5 Pro, in-context learning reduced Actor Critic ASR by 83.6%, versus 8.6% on Flash; perplexity detection caught 68.75% of Pro and 75% of Flash attacks at 1% FPR.Classification defenses detected attacks effectively but had FPRs of 31.1% on Pro and 82.1% on Flash; the authors attribute this to overlap between spam emails and classifier fine-tuning data. In-context learning and spotlighting had elevated null-response rates on Flash.Source: §H.2 · pp. 41–42

H.3. In-context learning (ICL)

  • In-context learning’s protection depended strongly on example placement and model capability, while adding examples to the original system prompt did not reduce benign text quality.On Flash, appending examples at the system-prompt end reduced ASR by 10%, whereas placing them after the retrieved email reduced ASR by 40–80% with quality scores of 0.29 → 0.20 → 0.12; on Pro, ASR fell from 98.2% to 14.9%. The original-system-prompt quality scores were 0.287 without examples and 0.283 with them.Source: §H.3 · p. 43

H.4. Spotlighting

  • Spotlighting inserts control tokens into untrusted email content to disrupt trigger tokenization, but may be less effective in languages such as Chinese and Japanese.The implementation inserts tokens into attacker-controlled sender, subject, and body text at 8-character intervals, paired with a system warning; typical prompts incur 100–200 extra tokens.Source: §H.4 · pp. 44–45

H.5. Paraphrasing

  • Paraphrasing showed no observed benign-quality drop and may add a filtering layer, but requires an extra paraphraser step on the input hot path.Average quality was 0.287 without paraphrasing and 0.305 with it; a paraphraser may refuse adversarial inputs, though a stronger paraphraser policy can create problems and a weaker model may lose prompt details.Source: §H.5 · pp. 45–46

H.6. Retrieved Data Classifier

  • The Retrieved Data Classifier uses a separate LLM to classify the full prompt, enabling parallel processing, but examining retrieved data leaves it vulnerable to injections targeting the classifier.The experiments classified the system prompt, conversation history, and retrieved email together; the authors suggest email-only classification, fine-tuning, and probability thresholds calibrated to benign-data FPR as future improvements.Source: §H.6 · p. 46
  • The Retrieved Data Classifier had high ADR but also high FPR on benign examples, likely because ShieldGemma’s safety fine-tuning overlaps with the spam-email benign set.The authors conclude that low-FPR performance requires fine-tuning classifiers specifically for prompt injection rather than using a general classifier.Source: §H.6 · p. 47

H.7. User Instruction Classifier

  • The User Instruction Classifier was the strongest tested defense, maintaining high attack detection against adaptive attacks, but it was not evaluated on Gemini 1.5.It may miss attacks whose goals align with the user’s, can be manipulated through instructions in the generated tool call, and may flag benign calls; that manipulation effectively requires a two-step attack, and precision was not investigated.Source: §H.7 · p. 48

H.8. Self-reflection

  • 68.2%: including the full prompt increased self-reflection ADR; across three injection descriptions, ADR ranged from 22.8% to 81.6% and was generally higher when reflecting on context alone than context plus response.Only binary detection was tested; probability-based threshold calibration and using reflection to guide the response were proposed as directions for further work.Source: §H.8 · pp. 48–49

H.9. Perplexity filter

  • With window_size=20, ADR=0.810 at FPR=0.000 and 0.001, and ADR=0.905 at FPR=0.010 and 0.005.In these evaluations, perplexity thresholding worked extremely well for Beam Search but poorly for human-interpretable Actor Critic attacks; results measured detection of retrieved emails at a threshold fixed for a chosen FPR.Source: §H.9 · pp. 49–50

I. Attacks with languages other than English

  • Translation reduced ASR for 74% of triggers, especially high-success triggers, while some initially lower-ASR triggers improved, most notably in Chinese.The study provides baseline results from translated English triggers, not a systematic evaluation of non-English attacks.Source: §I · pp. 50–51

J. Full prompt example containing a malicious trigger

  • Figure 10 shows Gemini 2.0 (undefended) often tricked into exfiltrating private data, while Gemini 2.5 (adversarially fine-tuned) recognizes the attack and warns the user.The example is a synthetic email-and-passport scenario with a trigger in retrieved content; the dataset contains varied synthetic conversations and private data.Source: J. Full prompt example containing a… · p. 51, Figure 10 · p. 52

Related papers

Loading 2505.14534v1…