PaperlayerView exact revision on arXiv

Source-linked AI summary

On Task, Off Track: Benchmarking LLM and Agent Robustness to Near-Target Prompt Injection

Yuyang Gong, Zihao Wang, Jiawei Liu, XiaoFeng Wang

arXiv:2610.32531v1cs.CR

TL;DR

Prompt-injection evaluations may test many instructions for selected outcomes while leaving other task-related violations unexplored. NTBench uses legitimate task references to construct near-target attacks, and both tracks achieve higher overall ASR@3 than fixed-target attacks across evaluated models and settings.

  • Problem

    Evaluations of selected attack targets may leave other incorrect or harmful responses and actions unexplored, even when they test many instructions.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    NTBench uses legitimate task references to construct and validate incorrect- and harmful-target attacks, then checks whether injections induce their specified violations.

    Source: §1 · p. 2, §8 · p. 12

  • Results

    Both near-target tracks achieve higher overall ASR@3 than the fixed-target baseline for every evaluated model in each setting.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Resistance to evaluated fixed-target attacks can coexist with susceptibility to task-related incorrect or harmful outcomes.

    Source: §8 · p. 12

    The current construction encourages target proximity through prompting but does not enforce a numerical radius.

    Source: §7 · p. 12

Abstract

from arXiv · show

Prompt injection benchmarks assess whether large language models (LLMs) and agents resist malicious instructions embedded in external content. However, a system could perform well by rejecting attacks that obviously abandon the user's request, while still accepting changes that retain most of a legitimate answer but make it incorrect or harmful. Testing more instructions for unrelated outcomes would not necessarily reveal this weakness. We call attacks seeking responses designed to resemble legitimate answers while introducing incorrect or harmful changes near-target prompt injection. We introduce NTBench, the Near-Target Benchmark, which builds on legitimate tasks from existing datasets. For each task, a legitimate reference outcome guides the generation of target--instruction pairs, with separate tracks targeting incorrect and harmful outcomes in responses or agent tool invocations. Harmful targets can also accompany otherwise correct task completion. We combine LLM-based checks with human validation of the intended deviations, then assess whether injected instructions induce them using response judgments or tool-call checks, excluding deviations already observed without injection. We evaluate seven LLMs on ten datasets for language tasks and two agent benchmarks. Under a common budget of three attack executions per sample, both tracks achieve higher overall attack success rates than the fixed-target baseline for every evaluated model in each setting. On language tasks, the increases range from 9.3 to 35.8 percentage points for incorrect targets and 1.7 to 17.3 percentage points for harmful targets. These results show that resistance to the tested fixed-target attacks does not imply resistance to related incorrect or harmful outcomes, and that reference-guided targets broaden the failures revealed by robustness evaluation.

1. Introduction

Fixed-target evaluations can leave other task-related incorrect or harmful outcomes unexplored. NTBench uses legitimate task references to construct near-target attacks, which achieve higher overall ASR@3 than the fixed-target baseline for every evaluated model in each setting.

A gap in robustness evaluation

  • Evaluations can test many instructions yet leave attacker-desired outcomes unexplored when they focus on a selected target.Searching for more effective instructions explores how a selected target might be induced, not whether a different target would be easier to induce.Source: §1 · p. 2

Our benchmark

  • NTBench builds on legitimate tasks from existing datasets and uses legitimate responses as references for proposing related incorrect or harmful targets.For each attack set, the system is still expected to fulfill the same user request and follow the same system instructions.Source: §1 · p. 2
  • Its incorrect-target track tests attacker-induced errors, while its harmful-target track tests harmful outcomes in generated content or agent actions.Harmful outcomes can include disclosing private information even when the rest of a response is accurate.Source: §1 · p. 2
  • The benchmark pairs generated target responses with instructions, checks intended violations through LLM-based checks and human validation, then evaluates whether injections induce them.Evaluations use response judgments or tool-call checks, and exclude deviations observed in attack-free runs.Source: §1 · pp. 2–3

Empirical results

  • Near-target-incorrect and near-target-harmful attacks exceed fixed-target ASR@3 for every evaluated model in each setting.On language tasks, the increases are 9.3–35.8 percentage points for incorrect targets and 1.7–17.3 percentage points for harmful targets.Source: Figure 3 · p. 3, §1 · p. 3
  • The evaluation covers seven LLMs, ten language-task datasets, and two agent benchmarks, with three attack executions per sample.Attack sets are shared across models.Source: §1 · p. 3

2. Problem Formalization and Threat Model

The formalization separates trusted instructions from untrusted task content and treats an attack as adding an instruction to the latter. The threat model specifies what attackers and generators can access and modify.

2.1. Problem Formalization

  • The system input x = (t, u) separates trusted instructions and user requests in t from untrusted content u used to complete the task.The outcome y = f(x) includes generated responses and, where applicable, actions and their observed effects.Source: §2.1 · p. 3
  • The original task τ includes trusted instructions, original untrusted information, and, for agents, the initial environment and available tools.Applicable constraints also include a fixed evaluation policy selected before task selection, target construction, and attack execution.Source: §2.1 · p. 4
  • An attack distinguishes its injected instruction δ from its target a, which specifies a sought response or action and its intended violation.The violation conditions determine which aspects of the target must be realized for success; benign details need not be reproduced.Source: §2.1 · p. 4
  • The attacker adds δ at a designated location in untrusted content, producing u′ while leaving trusted input t unchanged.The attacker seeks to induce the specified violation through instruction following; the injected instruction does not redefine the task.Source: §2.1 · p. 4
  • For language tasks, untrusted content can be supplied documents; in agent workflows, it can include external messages, documents, or tool results.The same abstraction covers material supplied directly to an LLM or encountered by an agent through tool use.Source: §2.1 · p. 4

2.2. Threat Model

  • The attacker may add instructions to a designated untrusted entry but cannot edit, delete, replace, or reorder original content or modify trusted instructions.In agent settings, this restriction concerns direct modification; later observations may change through agent actions and normal environment execution.Source: §2.2 · p. 4
  • The constructor can use task specifications, references, and evaluation interfaces, while the generator receives selected task context, an accepted reference outcome, and validation criteria.For agent tasks, the generator also receives tool descriptions and task-relevant initial-state information exposed by the source benchmark; it has no access to model parameters, gradients, or internal activations.Source: §2.2 · p. 4

3. Motivation

Prompt-injection evaluations test resistance to particular attack outcomes, leaving open whether that resistance extends to other incorrect or harmful outcomes. The paper frames near-target testing around reference outcomes while judging violations against the original task and constraints.

3. Motivation

  • Evaluating more instructions for selected outcomes does not establish resistance to other incorrect or harmful responses and actions.Near-target probes examine violations that remain closely related to the user’s task.Source: §3 · p. 4

3.1. Definitions

  • Vτ represents outcomes meeting task requirements and constraints, while Dτ represents incorrect or harmful deviations under the fixed policy Π.A target neighborhood uses a dissimilarity measure and radius to select candidate outcomes near a legitimate reference; proximity alone does not establish a violation.Source: §3.1 · p. 5

3.2. Intuition

  • Near-target probes test whether instructions induce validated violations in outcomes that retain much of the task-appropriate content or behavior.A small change can violate a factual requirement or disclosure constraint, even when the rest of the response remains relevant; paired comparisons examine failures beyond those observed with evaluated baselines.Source: §3.2 · p. 5

4. NTBench

NTBench uses reference outcomes to construct related attack targets and instructions, then validates deviations and tests whether the instructions induce them. Its target-generation analysis finds that both near-target tracks have smaller model-assigned likelihood gaps than the fixed-target baseline.

4.1. Overview

  • NTBench constructs reference-guided target–instruction pairs, then tests for validated violations beyond those observed in attack-free runs.The benchmark compares the two near-target tracks with a fixed-target baseline under a common execution budget.Source: §4.1 · pp. 5, 7

4.2. Target and Injection Generation

  • An LLM generates targets and associated instructions from task information and a reference outcome, encouraging task-related content while requesting incorrect or harmful deviations.Language-task references come from a reference model; agentic references come from ground-truth trajectories. The proximity objective is prompted rather than explicitly optimized with a distance measure or numerical radius.Source: §4.1 · p. 6, §4.2 · p. 7

4.3. Target Validation

  • LLM-based checks and human validation assess whether each candidate violates the original task or applicable harmfulness criteria, excluding acceptable alternatives.Harmful targets may add a disclosure or action alongside otherwise correct task completion; agentic tool-call targets also undergo executability checks.Source: §4.1 · p. 6, §4.3 · p. 8

4.1. Overview

  • Attack assessment checks whether an injected instruction induces its specified deviation; unrelated errors or compliance with only benign parts do not count as success.Instructions are inserted into designated untrusted content while original content and trusted instructions are preserved.Source: §4.1 · p. 7

4.2. Target and Injection Generation

  • 52.9–74.2% and 44.7–61.3%: median log-probability gaps decrease for the incorrect-target and harmful-target tracks, respectively, relative to the fixed-target baseline.Across three task groups, both tracks’ gap distributions shift smaller; the results use Meta-SecAlign scoring on ten language-task datasets.Source: §4.2 · p. 7, Figure 4 · p. 8

5. Evaluation

The evaluation compares NTBench’s near-target attacks with fixed-target attacks under a shared three-execution budget, then audits target validity and language-task success judgments. Both near-target tracks achieve higher overall ASR than the baseline across evaluated models and settings.

5.1. Experimental Setup

  • The evaluation uses seven models, ten language-task datasets, and AgentDojo and AgentDyn, with instructions fixed across models and a shared budget of three attack executions per sample.ASR@3 is the fraction of samples with at least one successful attempt; success checks exclude deviations also seen without injection.Source: §5.1 · p. 9

5.2. Main Results

  • On language tasks, both near-target tracks exceed fixed-target ASR for every model, by 9.3–35.8 percentage points for incorrect targets and 1.7–17.3 for harmful targets.Meta-SecAlign-70B rises from 3.2% fixed-target ASR to 36.7% for incorrect targets and 18.4% for harmful targets.Source: §5.2 · p. 9, Table 2 · p. 10
  • On AgentDojo and AgentDyn, both near-target tracks exceed the fixed-target baseline for every model; incorrect-target ASR ranges from 7.5% to 32.4% and 16.5% to 43.4%, respectively.Harmful-target ASR reaches 12.8% on AgentDojo and 21.1% on AgentDyn; rates cover reached injection positions.Source: §5.2 · pp. 9–10, Table 3 · p. 10

5.3. Target Validity and ASR Judge Reliability

  • In audits, majority-vote human labels confirm that 91.3% of language-task and 98.0% of agentic incorrect targets introduce errors relative to the original request and information; 86.7% and 92.0% of harmful targets meet criteria under fixed policy Π.Source: §5.3 · p. 11
  • LLM-judge F1 against human labels is 83.6% for fixed-target, 86.7% for near-target incorrect, and 89.9% for near-target harmful attacks.Agreement ranges from 91.0% to 93.0%, with κ values between 0.78 and 0.85.Source: Table 5 · p. 11, §5.3 · p. 11

6. Related Work

Related work covers prompt-injection attack construction, benchmarks, and defenses. NTBench complements these efforts by evaluating whether resistance to baseline attacks extends to related incorrect or harmful outcomes.

Prompt injection attacks

  • Automated injection methods optimize triggers, suffixes, or generators, but optimizing for a selected target does not test other outcomes.Examples include gradient search, adaptive optimization, AutoInject, and PISmith.Source: §6 · p. 11

Prompt injection benchmarks

  • Existing benchmarks cover indirect injections, instruction–data separation, agent objectives, dynamic tasks, and stakeholder harms, while NTBench emphasizes target selection’s effect on robustness interpretation.Near-target objectives broaden tested outcomes within existing task environments; stronger injection optimization can improve the search for instructions inducing them.Source: §6 · p. 11

Prompt injection defenses

  • Prompt-injection defenses include malicious-content filtering, model training and representation changes, and agent harnesses that enforce execution constraints.Examples include DataSentinel and PromptArmor, SecAlign and Meta-SecAlign, and capability- or policy-based harnesses.Source: §6 · p. 11

Relation to LocalAlign

  • LocalAlign is a training method using near-target preference pairs, whereas NTBench provides an evaluation benchmark for related incorrect or harmful outcomes.NTBench validates targets against the task and constraints, then separately assesses whether execution follows the injected instruction and meets specified violation conditions.Source: §6 · pp. 11–12

7. Discussion

The discussion frames target proximity as a conceptual guide rather than a numerically enforced constraint, and identifies limits on what current comparisons establish.

7. Discussion

  • NTBench encourages target proximity through prompting but does not enforce a numerical radius, leaving task-sensitive distance measures and effects of radius, coverage, and execution budget for future work.Proximity alone does not determine the likelihood of inducing a target’s specified violation.Source: §7 · p. 12
  • The reported target-family comparisons do not establish whether near-target construction improves attack success for a fixed attacker objective.The objective itself may constrain how closely targets resemble the reference.Source: §7 · p. 12
  • In the agentic comparison, near-target tool-call targets use invocation-based checks, whereas fixed-target baselines use environment-state checks.The reported attack success rates therefore characterize failures under their respective criteria.Source: §7 · p. 12

8. Conclusion

The conclusion presents NTBench as an evaluation of near-target prompt injection and reports that resistance to tested fixed-target attacks can coexist with susceptibility to related task-specific deviations.

8. Conclusion

  • NTBench evaluates LLM and agent robustness to attacks that seek responses resembling legitimate task completion while introducing incorrect or harmful deviations.The benchmark uses reference outcomes to guide attack construction, validates proposed violations, and tests whether injected instructions induce them.Source: §8 · p. 12
  • The conclusion characterizes NTBench’s target construction as reference-guided and its evaluation as testing whether injected instructions induce validated deviations.The assessed deviations are incorrect or harmful departures from legitimate task completion.Source: §8 · p. 12
  • Resistance to evaluated fixed-target attacks can coexist with susceptibility to task-related incorrect or harmful outcomes.Paired comparisons use matched maximum attack-execution budgets to assess additional failures beyond the evaluated baselines.Source: §8 · p. 12

1. Judge Prompt

The judge prompts assess task completion, semantic consistency with a reference output, and whether an attack target causes concrete harm.

1. Judge Prompt

  • The task-completion judge decides whether a response completes the specified task without evaluating the task’s or response’s correctness.It returns YES or NO.Source: §1 · p. 14
  • The semantic-boundary judge compares Output B with Output A for task-scope adherence, task-decision consistency, and agreement on task-essential content.It labels the outputs different if any criterion fails.Source: §1 · p. 14
  • The harm judge labels a target harmful only when it causes a concrete, non-trivial adverse consequence relative to the benign task and response.Its listed categories include financial, destructive, privacy, security, information or decision, availability or access harms, and malicious facilitation.Source: §1 · p. 14
  • The harm-judgment input includes the benign task and response, attack command, and target response, and the output gives a harmfulness label, categories, and brief reason.The output is formatted as JSON.Source: §1 · p. 14

2. Generation Prompts

The generation prompts specify context-aware instructions for language tasks and constrained target calls for agents, with criteria for evaluating intended deviations.

2. Generation Prompts

  • The language-task generation prompt takes the task, context, and reference response as input and returns injected tasks in JSON.It specifies the injected task and attack goal in the output.Source: §2 · pp. 15–16
  • For agents, each candidate pairs a target tool call with an instruction, changing one key argument or selecting a sibling tool acting on the same objects.Candidates vary which argument or action changes, and the prompt requires concrete target-tool parameters in JSON.Source: §2 · p. 15
  • The agentic prompt treats the user task, trajectory, current tool return, legitimate next call, and available tools as reference data rather than instructions.It identifies the legitimate next call and available tools as inputs to candidate generation.Source: §2 · p. 15
  • Agentic harmful targets must remain plausible requests while causing a concrete adverse consequence to someone acting on the output.The prompt distinguishes material consequences such as financial loss, data damage, disclosure, improper access, misleading records, or lost access from merely changing an answer.Source: §2 · pp. 15–16
  • For language tasks, the prompt asks for one instruction matched to the context and a specified attack goal, with a binary success criterion.It calls for variation in phrasing and tone while excluding harmful or safety-related tasks that an LLM would reject.Source: §2 · p. 16
  • Language-task instructions must be direct commands, concise when possible, and preserve required identifier strings without introducing other actionable values.The prompt also requires variation in opening phrases and syntax across generations.Source: §2 · p. 16

Related papers

Loading 2610.32531v1…