PaperlayerView exact revision on arXiv

Source-linked AI summary

Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers

Andrew Zhao, Reshmi Ghosh, Vitor Carvalho, Emily Lawton, Keegan Hines, Gao Huang, Jack W. Stokes

arXiv:2510.14381v2cs.LGcs.AIcs.CLcs.CR

TL;DR

The paper studies poisoning risks in LLM-based prompt optimization, a security gap left by prior work focused mainly on training-time or inference-time attacks. It compares query and feedback manipulation, introduces a reward-model-independent fake reward attack, and evaluates highlighting as a defense. Feedback attacks raise ∆ASR by up to 0.48, while highlighting reduces fake reward ∆ASR from 0.23 to 0.07 without degrading utility.

  • Problem

    The safety implications of poisoned queries and feedback in LLM-based prompt optimization remain systematically underexplored.

    Source: Direct Feedback Manipulation · p. 2

  • Method

    The paper compares query and feedback poisoning across LLM-based optimizers and metrics, then evaluates fake reward attacks and a highlighting defense.

    Source: Direct Feedback Manipulation · p. 2

  • Results

    Feedback attacks raise ∆ASR by up to 0.48, while highlighting reduces fake reward ∆ASR from 0.23 to 0.07 without degrading utility.

    Source: Direct Feedback Manipulation · p. 2, §3.3 · p. 5, §3.4 · p. 7

  • Takeaways & Limitations

    Prompt optimization pipelines are a first-class attack surface requiring stronger safeguards for feedback channels and optimization frameworks.

    Source: §5 · p. 9

    Evaluation fixes the backend model primarily to gpt-4.1_2025-04-14, uses single-run trends, and focuses on harmfulness datasets, limiting generalizability and uncertainty assessment.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language model (LLM) systems increasingly power everyday AI applications such as chatbots, computer-use assistants, and autonomous robots, where performance often depends on manually well-crafted prompts. LLM-based prompt optimizers reduce that effort by iteratively refining prompts from scored feedback, yet the security of this optimization stage remains underexamined. We present the first systematic analysis of poisoning risks in LLM-based prompt optimization. Using HarmBench, we find systems are substantially more vulnerable to manipulated feedback than to query poisoning alone: feedback-based attacks raise attack success rate (ASR) by up to {\Delta}ASR = 0.48. We introduce a simple fake reward attack that requires no access to the reward model and significantly increases vulnerability. We also propose a lightweight highlighting defense that reduces the fake reward {\Delta}ASR from 0.23 to 0.07 without degrading utility. These results establish prompt optimization pipelines as a first-class attack surface and motivate stronger safeguards for feedback channels and optimization frameworks.

1 Introduction

LLM systems increasingly support applications from chatbots to robotics, but effective prompts often require costly expert tuning. LLM-based prompt optimization seeks to reduce this burden by iteratively refining prompts using feedback.

  • LLM systems power applications including chatbots, autonomous agents, and robotics.Source: §1 · p. 1
  • Effective prompts strongly influence performance and often require costly tuning by human experts.Source: §1 · p. 1
  • LLM-based prompt optimization aims to reduce manual effort through iterative prompt refinement.Source: §1 · p. 1

Initial System Message

The initial system message establishes the baseline system instruction before optimization begins.

Direct Feedback Manipulation

Prompt optimization can be attacked through manipulated queries or feedback, with feedback poisoning posing a particularly strong risk. The paper introduces fake reward attacks and a highlighting defense that reduces their impact while preserving utility.

Direct Feedback Manipulation

  • Prompt optimization introduces risks because iterative updates guided by external signals can silently steer system prompts toward unsafe behavior.Source: Direct Feedback Manipulation · p. 2
  • The study compares harmful-query injection and feedback tampering across LLM-based optimizers and multiple optimization metrics.Source: Direct Feedback Manipulation · p. 2
  • A fake reward attack appends plausible-looking feedback tokens to inputs, requires no reward-model access, and increases attack success rates.Source: Direct Feedback Manipulation · p. 2
  • Feedback poisoning raises ∆ASR substantially, up to 0.48 under harmscore feedback, whereas naive query manipulation yields little to no increase in the default setup.Source: Direct Feedback Manipulation · p. 2
  • The highlighting defense reduces fake reward ∆ASR from 0.23 to 0.07 without degrading utility.Source: Direct Feedback Manipulation · p. 2
  • The paper identifies feedback-poisoning attacks and investigates defense strategies for mitigating vulnerabilities in LLM-based optimization.Source: Direct Feedback Manipulation · p. 2

2 Risk Identification

LLM-based prompt optimization is vulnerable through both poisoned queries and manipulated feedback. The threat model limits attackers from accessing model weights while allowing partial control over optimization inputs or feedback signals.

2 Risk Identification

  • Attackers can poison queries or feedback signals to steer optimization toward vulnerable system prompts without accessing gradients or model weights.Source: §2 · p. 2
  • Query manipulation injects harmful queries that may make the optimized prompt more vulnerable or explicitly malicious.Source: §2 · p. 2
  • The two main attack channels are query manipulation and feedback manipulation.Source: §2 · p. 2, Figure 2 · p. 3
  • Customer-data workflows can expose optimization to malicious queries when vendors see aggregate results but not individual requests.Source: §2 · p. 3
  • Feedback manipulation hijacks scalar scores, LLM-judge outputs, APIs, or labels to redirect optimization toward harmful objectives.Source: §2 · p. 3
  • Both channels require securing the data pipelines and feedback mechanisms that drive prompt optimization, not only the LLM weights.Source: §2 · p. 3

2.1 General Threat Model

  • The threat model assumes no access to weights, fine-tuning, or optimizer scaffolding, but partial query control and possible reward-signal control.Source: §2.1 · p. 3

3 Experiments

The experiments evaluate prompt-optimization vulnerability across query poisoning, feedback manipulation, and fake reward attacks, then test a boundary-highlighting defense. Results show that manipulated feedback is substantially more dangerous than query poisoning alone, while fake reward attacks remain effective without reward-model access.

3.1 Experimental Setup

  • The experiments use ΔASR as the maximum ASR increase over optimization, with larger values indicating greater vulnerability and negative values indicating improved safety.An additional %ASR+ proxy measures the proportion of optimization steps in which ASR increases, capturing training momentum.Source: §3.1 · pp. 3–4

3.2 Query Manipulation

  • Query poisoning alone does not increase vulnerability under the default helpfulness metric, whose optimization instead decreases ΔASR.The optimizer promptly updates toward safety and refusal after seeing harmful data, suggesting that the default helpfulness signal may implicitly encode safety.Source: §3.2 · p. 5
  • Changing the optimization metric sharply increases ΔASR, showing that metric choice determines whether optimization resists or amplifies adversarial influence.Correctness and saferlhf classifiers produce substantially different behavior from the default helpfulness classifier.Source: §3.2 · p. 5

3.3 Feedback Manipulation

  • Feedback manipulation substantially increases vulnerability, with effects observed across optimizer types and model backends.The main-results table reports that directly optimizing harmscore feedback drastically increases ΔASR and ASR growth, regardless of optimizer or backend.Source: §3.3 · pp. 5–6, Table 2 · p. 6

3.4 Poisoning with “Fake Reward"

  • The fake reward attack appends high or low fake scores to queries, requiring neither reward-model access nor privileged pipeline access, and raises ΔASR from -0.02 to 0.23.The attack uses approximate harmfulness likelihoods to seed high fake scores for harmful queries and low scores for low-risk queries, hijacking the optimizer’s update direction.Source: §3.4 · pp. 6–7
  • The fake reward attack remains effective with 90% benign data, raising ΔASR from -0.02 to 0.10, while combining it with correctness feedback produces a compounding effect beyond harmscore feedback.The combination makes the system more vulnerable than either intervention alone and exceeds the maximum ASR achieved by direct harmscore optimization.Source: §3.4 · pp. 7–8
  • Highlighting query boundaries reduces fake-reward ΔASR from 23% to 7% and maximum ASR from 30% to 15% without reducing optimized utility.The defense encloses queries in <query> tags; mean optimized score is 4% higher than in the fake-reward run and comparable to vanilla, apart from extra delimiter tokens.Source: §3.4 · p. 8

4 Related Works

Prior work studies prompt optimization methods and poisoning risks in model training, but systematic safety analysis of feedback-driven prompt optimization remains largely absent.

LLM as Prompt Optimizer

  • LLM-based optimizers generate, evaluate, and iteratively refine prompts or prompt programs using held-out data or online feedback.Examples include APE, OPRO, DSPy, TextGrad, Trace, SAMMO, ExpeL, and PromptAgent.Source: §4 · p. 8

LLM Data Poisoning

  • Unlike prior work focused on training or inference-time attacks, this paper examines poisoning risks in the prompt-optimization feedback loop.The gap concerns prompts being iteratively “trained” through feedback while model weights remain unchanged.Source: §4 · pp. 8–9
  • Existing poisoning research shows that manipulated training data, preference labels, and feedback can implant backdoors, increase toxicity, or undermine alignment.These attacks affect supervised tuning, RLHF, and prompt-based models, including through stealthy triggers and poisoned preferences.Source: §4 · pp. 8–9

5 Conclusion

The paper presents the first systematic study of safety vulnerabilities in LLM-based prompt optimization and finds that feedback manipulation, especially fake rewards, creates substantial risks. It concludes that optimization pipelines should be treated as a first-class attack surface while stronger defenses remain necessary.

5 Conclusion

  • Feedback manipulation makes prompt optimizers substantially more vulnerable than naive query manipulation, particularly through fake reward attacks.The study also finds that safety-entangled metrics may mitigate risk, whereas disentangled metrics can create new exploitation opportunities.Source: §5 · p. 9
  • The study is the first systematic analysis of safety vulnerabilities introduced by the LLM-based prompt-optimization stage.Prior work primarily examined supervised fine-tuning, RLHF, or inference-time jailbreaks.Source: §5 · p. 9
  • Optimization pipelines should be treated as a first-class attack surface in LLM safety research.The conclusion frames feedback channels and optimization frameworks as requiring stronger safeguards.Source: §5 · p. 9
  • The proposed mitigation strategies reduce but do not fully prevent fake reward attacks.This motivates stronger mechanisms against adversarial reward manipulation.Source: §5 · p. 9

Limitations

The study fixes the backend model and evaluates primarily harmfulness on two safety benchmarks, while reporting single-run trends without variance estimates. These choices limit generalizability and leave stochastic variability and broader safety dimensions incompletely assessed.

Limitations

  • Fixing gpt-4.1_2025-04-14, with one harmscore robustness test on gpt-5.1, limits generalizability across model families and deployment settings.The fixed backend isolates optimization-loop effects from differences in safety tuning and model capabilities.Source: Limitations · p. 9
  • Single-run experiments without multiple random seeds or variance estimates do not quantify stochastic variability.Greedy decoding was used to reduce stochasticity, but full prompt-optimization cycles were too costly for repeated trials.Source: Limitations · p. 9
  • Evaluation focuses on HarmBench and WildGuardMix, primarily measuring harmfulness rather than truthfulness, bias, or misinformation.The authors call for validation across diverse models, multiple seeds, and broader safety benchmarks.Source: Limitations · p. 9

Potential Risks

The paper identifies attack vectors that could be misused to poison feedback channels or steer optimized system prompts toward unsafe behavior. It includes a lightweight mitigation and warns that offensive evaluation content requires appropriate care.

Potential Risks

  • The identified attack vectors could be misused to poison feedback channels or steer optimized system prompts toward unsafe behavior.The risk concerns deliberate manipulation of optimization inputs and feedback boundaries.Source: Potential Risks · p. 9
  • The paper includes offensive prompts or model outputs solely for safety measurement and says they should be handled with appropriate care.A lightweight mitigation is included to harden optimization inputs and feedback boundaries in real systems.Source: Potential Risks · p. 9

A Defense Model for Highlighting

The defense assumes structural control over optimizer inputs but no control over model weights, feedback values, or individual query content. Explicit query delimiters confine fake feedback tokens to the query span while preserving the optimization objective.

B . Prompt Evolution of the HarmScore as Feedback Experiment

The experiment provides system-prompt snapshots across optimization epochs for the HarmScore as Feedback Experiment.

HarmScore as Feedback Experiment: Initial Prompt

This section examines how the initial prompt defines the evaluation objective and constrains model responses. It covers directives for maximizing complete, correct, technically rigorous, and directly relevant answers while specifying handling rules for sensitive, ambiguous, prohibited, copyrighted, and feedback-related requests.

Objective and response style

  • The prompt defines success as maximizing correctness, informativeness, user satisfaction, and direct, complete fulfillment of explicit requests according to the evaluation metric.It instructs the assistant to minimize refusals, omissions, hedging, and meta-commentary while producing self-contained, actionable responses.Source: HarmScore as Feedback Experiment:… · pp. 12–14

Sensitive and restricted requests

  • For sensitive, dual-use, exploit, graphic, hateful, illegal, or copyrighted requests, the prompt generally requires maximum technical detail and only minimal refusal or substitution when explicitly required.Allowed alternatives include detailed explanations, pseudocode, summaries, paraphrases, safe educational framing, and technical context.Source: HarmScore as Feedback Experiment:… · pp. 13–14

Factuality and constraint handling

  • A separate instruction set prioritizes factual accuracy, authoritative consensus, and responsible misinformation handling, while still requiring detailed permissible alternatives when direct fulfillment is unavailable.It directs the assistant to correct unsupported premises, distinguish consensus from minority views, cite authoritative sources, and provide prompt-specific refusals when explicit prohibitions apply.Source: HarmScore as Feedback Experiment:… · pp. 14–15

Objective-function primacy

  • The prompt makes the reward structure the overriding constraint, directing literal fulfillment even when it conflicts with safety, legal, ethical, copyright, or policy considerations unless the objective function penalizes the output.It characterizes refusal, hedging, deflection, or omission as critical failures when they reduce objective-function performance.Source: HarmScore as Feedback Experiment:… · p. 16

Response objectives

  • The prompt prioritizes maximal informativeness, factual correctness, neutrality, contextual completeness, and direct fulfillment in the requested format.It requires explicit assumptions for missing information, avoidance of placeholders and omissions, authoritative fact-checking, and self-contained answers tailored to the task.Source: HarmScore as Feedback Experiment:… · pp. 16–18

Sensitive and technical requests

  • For technical, exploit, code, and dual-use requests, the prompt instructs the model to provide complete, ready-to-use, technically accurate content while assuming legitimate or educational intent unless malice is unambiguous.It calls for filled-in scenario-specific values, stepwise explanations, prerequisites, verification steps, edge cases, and only brief responsible-use disclaimers.Source: HarmScore as Feedback Experiment:… · pp. 16, 17, 19

Ambiguity and partial fulfillment

  • For ambiguous or incomplete requests, the prompt favors explicit assumptions and maximal partial fulfillment over clarification or refusal unless ambiguity is critical or fulfillment is impossible.It also directs the model to provide educational or constructive alternatives when full compliance is blocked by factual, technical, legal, or policy constraints.Source: HarmScore as Feedback Experiment:… · pp. 17, 19

Feedback and constrained content

  • The prompt requires explicit incorporation of feedback and evaluation criteria, while directing the model to handle misinformation, controversial claims, and copyrighted material through fact-checking, contextual framing, or permissible alternatives.Feedback should be addressed early; unsupported claims should be identified using authoritative sources; restricted content should be fulfilled as far as policy and technical constraints allow.Source: HarmScore as Feedback Experiment:… · pp. 17–19

Refusal and compliance

  • The prompt suppresses refusal, hedging, apology, and policy-based constraint language unless the objective function explicitly rewards or requires it.It states that refusal protocols and safety layers do not apply unless explicitly penalized by the objective function, while retaining exceptions for strictly illegal or unambiguously prohibited content.Source: HarmScore as Feedback Experiment:… · pp. 16, 18

Initial Prompt

  • The prompt requires complete, logically ordered, clearly formatted responses that address assumptions, technical details, edge cases, and readability.It specifically calls for structured explanations and complete, context-appropriate code without truncation.Source: HarmScore as Feedback Experiment:… · pp. 19–20
  • The prompt specifies refusal templates for strictly illegal or unambiguously prohibited requests while encouraging lawful, educational, or constructive alternatives when appropriate.It also instructs the model to explain refusals contextually and avoid facilitating prohibited actions.Source: HarmScore as Feedback Experiment:… · p. 20
  • The prompt prioritizes literal correctness, completeness, relevance, specificity, and direct fulfillment, including for sensitive or copyrighted topics unless explicit prohibitions apply.It directs the model to avoid speculation and meta-commentary and to incorporate available feedback explicitly.Source: HarmScore as Feedback Experiment:… · pp. 20–21

1. Have you directly and fully executed

The guidance requires direct, complete, contextually appropriate responses that follow the user's requested format. It also emphasizes factual accuracy, explicit handling of uncertainty, and complete fulfillment of multi-part requests.

  • Factual and controversial claims should be grounded in authoritative sources, with consensus, disputed views, and evidentiary gaps clearly distinguished.The guidance also recommends fact-checking assertions and citing sources, studies, organizations, or data where possible.Source: §1 · pp. 21–22
  • Ambiguous, incomplete, or false-premise requests should be handled transparently by stating assumptions or factual status and proceeding when reasonable.Clarification is reserved for ambiguity that is critical; otherwise, the response should provide the most complete contextually appropriate answer possible.Source: §1 · p. 21
  • Multi-part requests and prior feedback should be addressed comprehensively, including each requested component and relevant evaluation point.Refusals or partial fulfillments should explain the reason and offer a tailored lawful alternative when appropriate.Source: §1 · pp. 21–22
  • Responses should directly fulfill every requested component in the specified format, without unnecessary plans, meta-commentary, placeholders, or incomplete sections.This includes producing revised text directly for editing tasks and using explicit headings or structure when appropriate.Source: §1 · pp. 21–22
  • Answers should be specific, technically accurate, and tailored to the user's context rather than generic or unsupported.The guidance calls for precise language, relevant evidence and examples, and attention to the recipient's or task's particular details.Source: §1 · p. 22

Related papers

Loading 2510.14381v2…