PaperlayerView exact revision on arXiv

Source-linked AI summary

SecInfer: Preventing Prompt Injection via Inference-time Scaling

Yupei Liu, Yanting Wang, Yuqi Jia, Jinyuan Jia, Neil Zhenqiang Gong

arXiv:2509.24967v4cs.CRcs.AI

TL;DR

Prompt injection can divert an LLM from its intended task to an attacker-chosen task. SecInfer samples diverse candidate responses using varied system prompts, then selects a response guided by the target task; experiments report lower attack success and preserved utility without attacks, with reduced effectiveness when target and injected tasks are the same type.

  • Problem

    Prompt injection in contaminated data can make an LLM perform an attacker-chosen task instead of its intended task, posing a security threat to LLMs.

    Source: §1 · pp. 1–2

  • Method

    SecInfer samples diverse responses using chain-of-thought prompting and varied system prompts, then aggregates candidates to select a response that best fulfills the target task.

    Source: §1 · p. 3, §4.2 · p. 9

  • Results

    SecInfer nearly eliminates attack success across evaluated LLMs, attacks, and datasets while maintaining unafflicted task utility on par with the undefended baseline.

    Source: §5.2 · p. 13

  • Takeaways & Limitations

    Inference can be made more secure against prompt injection by sampling multiple candidates and aggregating them under the target instruction.

    Source: §7 · p. 20

    SecInfer is less effective when target and injected tasks are the same type; for sentiment analysis, ASR falls from 0.90 to 0.79.

    Source: §6 · p. 20

Abstract

from arXiv · show

Prompt injection attacks pose a pervasive threat to the security of Large Language Models (LLMs). State-of-the-art prevention-based defenses typically rely on fine-tuning an LLM to enhance its security, but they achieve limited effectiveness against strong attacks. In this work, we propose \emph{SecInfer}, a novel defense against prompt injection attacks built on \emph{inference-time scaling}, an emerging paradigm that boosts LLM capability by allocating more compute resources for reasoning during inference. SecInfer consists of two key steps: \emph{system-prompt-guided sampling}, which generates multiple responses for a given input by exploring diverse reasoning paths through a varied set of system prompts, and \emph{target-task-guided aggregation}, which selects the response most likely to accomplish the intended task. Extensive experiments show that, by leveraging additional compute at inference, SecInfer effectively mitigates both existing and adaptive prompt injection attacks, outperforming state-of-the-art defenses as well as existing inference-time scaling approaches.

1 Introduction

Prompt injection can redirect LLMs processing untrusted data from the intended task to an attacker-chosen one, while existing prevention defenses have gaps. SecInfer uses diverse system-prompt-guided sampling and target-task-guided aggregation, and outperforms existing defenses and inference-time scaling methods in the reported evaluations.

1 Introduction

  • Prompt injection embeds an attacker-chosen prompt in untrusted data, potentially causing an LLM to perform the injected task instead of its intended one [OWASP, 2023, Greshake et al., 2023, Liu et al., 2024b].Pre-processing and fine-tuning often have limited effectiveness against strong optimization-based attacks, while policy-based defenses may not fit applications without external actions.Source: §1 · pp. 1–2
  • Existing inference-time scaling can fail because an injected prompt may make most sampled responses complete the attacker’s task, defeating majority-vote self-consistency [Wang et al., 2023].SecInfer is designed specifically for prompt-injection defense rather than general capability improvement.Source: §1 · p. 2
  • SecInfer is evaluated across six target-task types, eight injected-task types, four LLMs, seven existing attacks, and six adaptive attacks, with further agent-benchmark evaluation on InjecAgent and AgentDojo.The paper reports effectiveness against both existing and adaptive attacks and evaluates agent mitigation using InjecAgent [Zhan et al., 2024] and AgentDojo [Debenedetti et al., 2024].Source: §1 · p. 3
  • With ∼0.6 seconds per sample on a single GH200 GPU, SecInfer substantially outperforms existing defenses and prior inference-time scaling methods, even when most sampled responses favor the attacker.The paper reports comparable inference-time overhead to self-consistency and describes selecting the single correct response when four of five sampled responses are attacker-desired.Source: §1 · p. 3
  • SecInfer addresses prompt injection with system-prompt-guided sampling for diverse candidates and target-task-guided aggregation to choose the response most likely to fulfill the intended task.For open-domain outputs, aggregation uses semantic clustering and a judge LLM to select among cluster representatives.Source: §1 · pp. 2–3

2 Related Work

The section reviews LLM generation and prompt injection, then surveys prevention defenses and inference-time scaling methods. It highlights that conventional scaling approaches were not designed to counter injection and can aggregate correlated, attacker-directed responses.

2.1 LLMs

  • LLMs generate responses autoregressively, with decoding choices such as greedy, temperature, and top-k sampling affecting whether outputs are deterministic or diverse [Gu et al., 2017, Ficler and Goldberg, 2017, Fan et al., 2018].Temperature sampling adjusts randomness through a temperature parameter, while top-k sampling restricts selection to the k tokens with the largest logits.Source: §2.1 · pp. 3–4

2.2 Prompt Injection Attacks

2.3 Prompt Injection Defenses

  • Pre-processing, fine-tuning, and security-policy defenses each have constraints: the first two can remain vulnerable to optimization-based attacks, while policies may not suit non-action tasks [Jia et al., 2025].Policy defenses also face the challenge of specifying policies accurately, with under-specification leaving vulnerabilities and over-specification risking false alarms.Source: §2.3 · p. 5

2.4 Inference-Time Scaling

  • Inference-time scaling methods include in-context learning [Brown et al., 2020], chain-of-thought [Wei et al., 2022], iterative refinement [Madaan et al., 2023], Best-of-N [Kwon et al., 2023], and self-consistency [Wang et al., 2023].Self-consistency aggregates sampled responses by majority vote; the paper notes its open-domain adaptation clusters embeddings and selects the response nearest the largest cluster’s centroid.Source: §2.4 · p. 6
  • These scaling methods offer limited prompt-injection protection because attacks can bias the model’s distribution, producing correlated sampled errors that majority voting may preserve.The authors contrast the methods’ assumption of independent, benign errors with injection-driven errors systematically aligned toward completing the injected task.Source: §2.4 · p. 6

3 Problem Formulation

The problem formulation assumes an attacker can inject instructions into target data but cannot alter the system’s other components. The defense aims to preserve task utility while reducing attack success, with utility under attack contingent on the data retaining enough target information.

3.1 Threat Model

  • The attacker’s goal is a response semantically similar to the attacker-desired output, with possible security and privacy risks including malicious redirection, harmful summaries, or malicious tool selection.An agent’s manipulated tool response may influence its subsequent actions [Zhan et al., 2024, Debenedetti et al., 2024].Source: §3.1 · p. 7
  • The threat model assumes an attacker with full knowledge of the system, defense algorithms, and backend and judge model parameters, enabling adaptive attacks against SecInfer.The attacker aims to achieve the same injected-task goal even when the defenses are deployed.Source: §3.1 · p. 7
  • The attacker can contaminate target data from sources such as the Internet or agent tool responses, but cannot alter the system prompt, target instruction, or backend LLM.The formulation treats these components as controlled by model, application, or agent developers.Source: §3.1 · p. 7

3.2 Preventing Prompt Injection Attacks

  • The defender’s objective is to redesign inference to reduce prompt-injection success while preserving target-task utility.The goals distinguish utility with no attack, utility under attack, and reducing attacker-desired responses.Source: §3.2 · p. 7
  • Without an attack, the defended system should perform the intended task effectively; under attack, it should still produce a response that completes the target task.Utility under attack is defined for contaminated data as input.Source: §3.2 · pp. 7–8
  • The defense should make attacker-desired responses completing injected tasks much less likely when contaminated data is provided.This is the objective of reducing attack success, distinct from preserving target-task utility.Source: §3.2 · p. 8
  • The approach allocates additional inference compute, but preserving utility under attack is achievable only when contaminated data retains sufficient information about the target data.If an attack completely destroys the target data, no defense can be expected to preserve target-task utility; an evaluated adaptive attack substantially reduced attack success despite failing that utility goal.Source: §3.2 · p. 8

4 Our SecInfer

SecInfer counters prompt injection by diversifying model reasoning and selecting responses according to the intended task. Its aggregation differs for closed- and open-domain outputs, and its closed-domain design is ineffective when injected and target tasks share an output domain.

4.1 Overview

  • SecInfer diversifies reasoning with crafted system prompts, then selects for target-task alignment; its closed-domain filter is ineffective when injected and target tasks share an output domain.Source: §4.1 · p. 8

4.3 Step II: Target-task-guided Aggregation

  • Because attacks can corrupt most samples, Step II uses target-task alignment rather than majority vote or reward scores alone to select the final response.Source: §4.1 · p. 8, §4.3 · p. 10

4.2 Step I: System-prompt-guided Sampling

  • Step I samples N responses using randomly selected, task-independent CoT system prompts and stochastic decoding, aiming for diverse paths and reasoning traces that help Step II identify contamination.Source: §4.2 · p. 9

4.3.1 Closed-Domain Target Tasks

  • For closed-domain tasks, SecInfer maps responses to valid output values, discards unmappable candidates, and majority-votes over the remaining answers, breaking ties uniformly at random.Source: §4.3.1 · p. 10

4.3.2 Open-Domain Target Tasks

  • For open-domain tasks, SecInfer clusters semantic embeddings, keeps each centroid-nearest response, and asks a judge LLM for the target-aligned representative, falling back when none qualify.Source: §4.3.2 · pp. 10–11

5 Evaluation

SecInfer is evaluated across diverse target and injected tasks, attacks, models, and defense baselines. It substantially reduces attack success while preserving task utility, and the evaluations examine agent settings, design variants, and adaptive attacks.

5.1 Experimental Setup

  • The evaluation spans six target datasets, eight injected tasks, heuristic and optimization-based attacks, prevention defenses, and inference-time scaling baselines.It measures utility without attack, utility under attack, and attack success rate.Source: §5.1 · pp. 12–13

5.2 Main Results

  • Across backend models, target tasks, and attacks, SecInfer reduces ASR to nearly zero while maintaining no-attack utility and high utility under attack.For GPT-4.1 on AG News, U is 87% with SecInfer and 86% without defense.Source: §5.2 · pp. 13–14, Table 1 · p. 14
  • Under Combined Attack, most prevention-based baselines fail; DS+PL and MetaSecAlign are exceptions, but each has limitations against GCG that SecInfer avoids.DS+PL retains non-negligible ASR on some GCG tasks, while MetaSecAlign is defeated by GCG.Source: §5.2 · p. 15, Table 2 · p. 15
  • Under Combined Attack, inference-time scaling baselines have lower UA and higher ASR than SecInfer, despite comparable computation costs to best-of-N and self-consistency.The authors attribute the difference to SecInfer’s design being tailored to prompt-injection defense.Source: Table 3 · p. 15, §5.2 · p. 16

5.3 Evaluation in LLM Agents

  • On InjecAgent and AgentDojo, SecInfer reduces attack success rate to nearly zero in agent settings involving malicious tool-call results.The agent evaluation generates five candidate actions per prediction and selects the one most aligned with the target task.Source: §5.3 · pp. 16–17, Table 5 · p. 16, Table 6 · p. 16

5.4 Ablation Study

  • Combined system-prompt and stochastic sampling achieves the highest UA and lowest ASR among tested Step I variants, while U remains comparable across methods.With more candidates, computation grows roughly linearly, ASR approaches zero quickly, and U and UA initially rise before saturating; N = 5 is reported as a favorable trade-off.Source: §5.4 · p. 17, Figure 3 · p. 17, Figure 4 · p. 17
  • Semantic clustering followed by judge selection yields the highest UA and lowest ASR among tested Step II aggregation strategies, and stronger judges improve U and UA.SecInfer’s Step II outperforms two alternative LLM-as-a-judge strategies; across judges, ASR remains near zero.Source: Figure 5 · p. 17, §5.4 · p. 18, Table 7 · p. 18

5.5 Adaptive Attacks

  • Across six adaptive attacks, SecInfer keeps ASR low even as attacks reduce target-task utility, with the greatest UA difficulty when attackers modify all contaminated data.Lower UA under the attack that knows the system prompts than under its counterpart supports keeping Step I prompts confidential to better maintain utility.Source: Table 8 · p. 18, §5.5 · p. 19

6 Discussion and Limitations

SecInfer is less effective when the injected task and target task are the same type, while a memorization test suggests its judge evaluates candidate responses rather than relying on dataset memorization.

6 Discussion and Limitations

  • ASR is 0.90 before and 0.79 after SecInfer when both target and injected tasks are sentiment analysis.In this setting, injection can alter target data without an explicit redirecting instruction, leaving the attacker-desired response consistent with the target task.Source: §6 · p. 20
  • Over 94% of samples have similarity ratios below 0.6, suggesting the judge is unlikely to have memorized the target-task datasets.The authors interpret this as evidence that effectiveness stems from the judge evaluating candidate responses.Source: §6 · p. 20

7 Conclusion and Future Work

The conclusion presents SecInfer as a way to make LLM inference more secure through diverse response sampling and target-instruction-guided aggregation. It identifies reducing candidate-sampling overhead and strengthening adaptive-attack evaluation as future directions, and lists the system prompts used.

7 Conclusion and Future Work

  • SecInfer samples multiple candidates using chain-of-thought system prompts and stochastic decoding, then aggregates them under the target instruction.For open-domain tasks, the conclusion gives semantic clustering and an LLM-as-a-judge as an example aggregation approach.Source: §7 · p. 20
  • Future work includes reducing the number of candidates needed to lower computational overhead and designing stronger adaptive attacks to evaluate SecInfer.Source: §7 · p. 20

A System Prompts

  • The listed prompts ask the model to reason or interpret the question in varied ways, then conclude with an answer to the original question.Each prompt limits the reasoning text to less than 150 words.Source: A System Prompts · p. 25

B Details of Mapping

The mapping procedure standardizes closed-domain answers as four-option multiple-choice outputs and accepts a candidate only when its required answer marker appears exactly once.

B Details of Mapping

  • Closed-domain tasks are formatted as four-option multiple-choice questions with outputs A, B, C, or D.Source: B Details of Mapping · p. 26
  • The backend is instructed to enclose its final answer in double asterisks and not use that format elsewhere.For example, an answer of A is returned as **A**.Source: B Details of Mapping · p. 26
  • A candidate maps to A, B, C, or D only when exactly one corresponding marked substring is found; zero or multiple matches make it invalid.Source: B Details of Mapping · p. 26

C LLM Sampling Settings

The sampling settings vary by backend model, while the attack evaluation follows prior implementations and default optimization settings.

C LLM Sampling Settings

  • LLaMA3.1-8B-Instruct uses temperature 0.1, a 150-token maximum, and top-k 20; Qwen3-8B uses temperature 0.6, a 32,768-token maximum, and top-k 20.Source: C LLM Sampling Settings · p. 26
  • GPT-4.1 and GPT-4o use temperature 0.7, a 500-token maximum, API version 2024-12-01-preview, and top-k 20 through Microsoft Azure OpenAI APIs.Source: C LLM Sampling Settings · p. 26

D Attack Details

  • Heuristic-based attacks follow Liu et al. [2024b], optimization-based attacks use existing default settings, and the GCG separator is initialized from Table 11.Source: D Attack Details · p. 26

E Implementation Details of Baselines

The appendix specifies baseline configurations for existing defenses and inference-time scaling methods.

E Implementation Details of Baselines

F Details of Evaluation Metrics

Evaluation metrics vary by target dataset: accuracy, Rouge-1, or Pass@1.

F Details of Evaluation Metrics

G Details of InjecAgent and AgentDojo

The evaluation uses InjecAgent and AgentDojo to assess prompt-injection attacks on tool-using agents and their intended tasks.

G Details of InjecAgent and AgentDojo

H Implementation Details of Adaptive Attacks

The appendix describes heuristic and optimization-based adaptive attacks, along with tables documenting attack settings and measurements.

H Implementation Details of Adaptive Attacks

Loading 2509.24967v4…