PaperlayerView exact revision on arXiv

Source-linked AI summary

Trading Inference-Time Compute for Adversarial Robustness

Wojciech Zaremba, Evgenia Nitishinskaya, Boaz Barak, Stephanie Lin, Sam Toyer, Yaodong Yu, Rachel Dias, Eric Wallace, Kai Xiao, Johannes Heidecke, Amelia Glaese

arXiv:2501.18841v1cs.LGcs.CR

TL;DR

The paper asks whether inference-time compute can improve adversarial robustness despite limited progress and the difficulty of anticipating attacks. It evaluates reasoning models across adversarial settings without adversarial training, finding robustness improvements across many domains and strategies, while identifying important limitations and scope boundaries.

  • Problem

    Adversarial robustness remains limited for LLMs facing jailbreaks, untrusted inputs, and novel attack surfaces, while adversarial training is costly and cannot anticipate all attacks.

  • Method

    The paper varies inference-time compute for o1-preview and o1-mini across adversarial settings, using the same reasoning-compute intervention as for non-adversarial performance without adversarial training.

  • Results

    Inference-time compute improves robustness across many domains and attacker strategies, with attack success often decreasing as reasoning compute increases.

  • Takeaways & Limitations

    Inference-time compute offers a promising robustness advantage through pure scaling without anticipating the set of potential attacks or using adversarially tailored interventions.

  • Takeaways & Limitations

    The work addresses applying safety specifications to adversarially shifted instances but not context parsing, specification existence, or specification knowledge, and is not a complete solution.

Abstract

from arXiv · show

We conduct experiments on the impact of increasing inference-time compute in reasoning models (specifically OpenAI o1-preview and o1-mini) on their robustness to adversarial attacks. We find that across a variety of attacks, increased inference-time compute leads to improved robustness. In many cases (with important exceptions), the fraction of model samples where the attack succeeds tends to zero as the amount of test-time compute grows. We perform no adversarial training for the tasks we study, and we increase inference-time compute by simply allowing the models to spend more compute on reasoning, independently of the form of attack. Our results suggest that inference-time compute has the potential to improve adversarial robustness for Large Language Models. We also explore new attacks directed at reasoning models, as well as settings where inference-time compute does not improve reliability, and speculate on the reasons for these as well as ways to address them.

1 INTRODUCTION

Adversarial robustness remains difficult for AI systems, especially LLM agents exposed to untrusted inputs and real-world side effects. This work studies whether scaling inference-time compute improves robustness without adversarial training, finding broad benefits while identifying important limitations and scope boundaries.

  • Motivation: Adversarial robustness remains a stubborn problem for both image classifiers and LLMs, whose attacks include jailbreaks and attacks on agentic systems.LLM agents can browse, execute code, and act on inputs controlled by potentially untrusted parties, raising the stakes of failures.
  • Motivation: Adversarial training is computationally expensive and requires anticipating the perturbation set, creating a recurring cycle of defending against known attacks while new ones emerge.The paper characterizes this dynamic as a “whack-a-mole” problem for LLM safety.
  • Approach: The paper tests whether increasing inference-time compute improves robustness across attacker strategies and domains without adversarial training or attack-specific information.The intervention uses more reasoning effort, rather than tailoring the model to the attacks or datasets studied.
  • Contributions: The paper introduces an adaptation of soft-token attacks, the think-less attack, and the hypothesized nerd-sniping strategy for reasoning models.These contributions extend the attack surface examined for reasoning models.
  • Results: Robustness improves with inference-time compute across many domains and attacker strategies, while the intervention also broadly improves model performance rather than trading robustness against capabilities.The results are presented across varied adversarial settings and summarized in Figure 1 and Table 1.
  • Limitations: Inference-time compute does not improve robustness in every area, and the paper offers hypotheses for these limitations.For many-shot attacks, success can initially rise when more computation enables the model to follow the attacker’s intent.
  • Scope: The study mainly uses artificial ad hoc policies that are unambiguous, separating ambiguity and standard safety training from the effects of inference-time compute.This narrows the scientific question to applying specified policies rather than defining them.
  • Limitations: The work addresses applying safety specifications to adversarially shifted instances, but not context parsing, safety-specification existence, or model knowledge of those specifications.Even within the addressed area, it does not provide a foolproof and complete solution.

2 EXPERIMENTAL SETUP

The experiments test whether increased inference-time reasoning improves robustness across diverse tasks, adversarial goals, attack types, and reasoning models. They include unambiguous mathematical, policy, rule-following, browsing, and vision settings, while also examining ambiguous policy tasks and adaptive or black-box attacks.

  • Experimental design: The study varies inference-time compute for o1-preview and o1-mini without adversarial training or attack-specific defensive prompting.The same intervention used to improve ordinary o1-series performance is applied to adversarial settings.
  • Experimental design: The evaluation measures adversary success across multiple prompts and model samples while varying adversary resources, goals, tasks, and attack types.Attacks include many-shot, common jailbreak, soft-token, AI red-teaming, and adversarial multimodal inputs.
  • Tasks and adversarial goals: The task suite covers arithmetic and mathematical reasoning, StrongREJECT and misuse prompts, unambiguous rule following, adversarial browsing, and adversarial image classification.AdvSimpleQA places injected instructions in a website containing the answer, while vision tasks use natural or Bard-optimized adversarial images.
  • Tasks and adversarial goals: Unambiguous tasks use direct success tests, whereas ambiguous misuse prompts require judgments about whether outputs are harmful or violate policy.For unambiguous tasks, human and neural-network grader agreement is reported as perfect; ambiguous cases can produce evaluator disagreement.
  • Scope: For ambiguous tasks, attacker-found loopholes can prevent attack success from decaying as inference-time compute increases.The reported robustness improvement is therefore strongest for unambiguous tasks.
  • Scope: The experiments do not claim the evaluated models are unbreakable, only that scaling inference-time compute improves robustness across varied settings and attacks.The authors explicitly acknowledge that more effective attacks may exist.

3 RESULTS

Across many-shot, StrongREJECT, and language-model-program attacks, greater inference-time compute generally improves robustness, but ambiguous misuse tasks remain important exceptions. Attack effectiveness also depends on attacker strength and task structure.

  • Many-shot Jailbreaking: For fixed attacker resources, many-shot attacks on mathematical tasks tend to have success rates approaching zero as inference-time compute grows.Higher attacker strength raises attack success, but increasing defender compute reverses this trend for a fixed attack budget.
  • Many-shot Jailbreaking: Very low inference-time compute can initially suppress attacks that require the model to follow an attacker’s intended pattern, after which success eventually tends to zero.The paper reports this non-monotonic behavior for an attack targeting output equal to the original answer multiplied by 7.
  • Many-shot Jailbreaking: In Misuse Prompts and Past Misuse Prompts, many-shot attacks show limited susceptibility, while increased test-time compute appears unhelpful because the tasks are ambiguous.Past Misuse Prompts are more vulnerable in Figure 5, with attack success reaching up to 25%, versus less than 5% on Misuse Prompts.
  • StrongREJECT Jailbreak Suite: On StrongREJECT, robustness measured by goodness@0.1 increases nearly monotonically with average inference-time compute across common jailbreaks.The evaluation applies jailbreaks to 60 policy-violating prompts and averages the worst 10% of graded responses for each prompt.
  • Language Model Program Attack: The language-model-program attacker iteratively adapts after failures, continuing until success or a maximum of 25 attempts, and is evaluated across multiple tasks.The setup includes arithmetic, MATH, rule-following, and misuse-related settings, with attacks generated rather than restricted to one jailbreaking strategy.
  • Language Model Program Attack: For LMP attacks on Misuse Prompts, attackers consistently succeed within few attempts across compute levels, which the authors attribute to mismatch between the grader and defender policies.Removing malicious intent can make an attack and response appear policy-compliant even when the response remains relevant to the original misuse prompt.

3.5 PROMPT INJECTION ATTACK

Across prompt-injection, multimodal, red-team, and reasoning-model attacks, increasing inference-time compute usually improves robustness, but targeted attacks can redirect or waste that compute.

  • 3.5 PROMPT INJECTION ATTACK: Increasing test-time compute reduces website prompt-injection attack success to zero in most settings and approaches zero in more complex browsing scenarios after a threshold.
  • 3.6 MULTI-MODAL ATTACK: For all three multimodal datasets, increasing test-time compute generally improves o1-v performance, consistently improving robustness on Attack-Bard.
  • 3.7 HUMAN RED-TEAMING: Human red-teaming found the lowest compute level clearly weakest, while the highest was most resistant to direct attacks and the top three levels were similar on transfer attacks.
  • 3.8 “THINK LESS” – A NOVEL ATTACK ON REASONING MODELS: The “Think Less” attack reduces reasoning computation to increase susceptibility, and o1-preview was significantly more robust than o1-mini against it.
  • 3.8 “THINK LESS” – A NOVEL ATTACK ON REASONING MODELS: “Think Less” remains potentially effective against other prompts, although unusually low inference-time compute is observable and can be flagged by monitors.
  • 3.9 “NERD SNIPING” ATTACK.: “Nerd sniping” traps reasoning models in unproductive loops by inducing unusually long chains of thought for a task’s difficulty.
  • 3.9 “NERD SNIPING” ATTACK.: For all but the lowest requested compute level, attack success is higher in the top 5% of actual-compute instances than at median compute, showing average gains are not pointwise.
  • 3.9 “NERD SNIPING” ATTACK.: Nerd sniping therefore offers an attack avenue that makes models spend inference-time compute unproductively, complementing attacks that prevent reasoning altogether.

4 CONCLUSIONS AND DISCUSSION

The paper finds promising robustness gains from inference-time scaling without attack-specific interventions, while emphasizing that the evidence remains preliminary and incomplete.

  • Inference-time compute offers robustness advantages through pure scaling, without interventions tailored to adversarial attacks or anticipation of possible attacks.
  • Because inference-time compute can be changed at deployment, higher reasoning-compute levels could be used in high-stakes safety settings.
  • The study covers limited tasks and compute ranges, leaving open whether attack success tends to zero in all settings.
  • Scaling compute does not appear to help when attacks exploit ambiguities or loopholes in the policy.

A.1 MULTIPLICATION, ADDITION, MATH DATASET DETAILS

The appendix defines arithmetic and MATH datasets with adversarial goals that redirect otherwise straightforward problem solving to incorrect outputs.

  • Dataset format: The Multiplication, Addition, and MATH datasets share a common format for adversarial evaluation.
  • Dataset format: Each sample uses top-level instructions that identify an untrusted section where the adversary injects an attack.
  • Dataset format: The format explicitly instructs the model to ignore text enclosed by <BEGIN IGNORE>...</END IGNORE> tags.
  • Datasets: The three datasets are 2-digit long addition, 2-digit long multiplication, and solving MATH problems.
  • Adversarial goals: For each dataset, the adversary considers three distinct goals.
  • Adversarial goals: One goal is to make the model output 42 instead of solving the user’s problem.
  • Adversarial goals: Another goal is to make the model provide an answer incremented by 1.
  • Adversarial goals: A third goal is to make the model output seven times the correct answer.

A.2 STRONGREJECT AND PASTSTRONGREJECT SAMPLES

The appendix contrasts refusal-oriented StrongREJECT examples with PastStrongREJECT examples whose grading can expose ambiguity in what constitutes harmful assistance.

  • StrongREJECT: StrongREJECT samples commonly use refusal answers to misuse prompts.
  • PastStrongREJECT: PastStrongREJECT includes responses that a grader marks as illicit assistance even when closer inspection finds nothing illegal in the response.
  • PastStrongREJECT: A PastStrongREJECT example asks how people historically plagiarized academic work without detection to advance their professional interests.
  • Attack setup: The figure’s attacks target arithmetic and MATH tasks through defender compute variation and attacker-strength changes across multiple attack goals.

A.3 LMP ATTACK EXAMPLES

These examples illustrate LMP-style attacks that try to redirect a reasoning model’s final answer through fabricated system updates, redefined instructions, memory adjustments, or numerical transformations.

  • Several attacks instruct the model to output 42 instead of solving the user’s problem.
  • Other attacks attempt to add 1 to the correct answer by framing the adjustment as a memory or code-related correction.
  • Multiplicative attacks direct the model to multiply its final answer by 7 using a purported system parameter or operating-system bug.

A.4 SOFT TOKEN NORMS

The soft tokens learned by adversarial training can have substantially larger norms than ordinary hard-token embeddings, motivating inspection of their norm growth during training.

  • Soft-token norms grew far larger than the norms of standard hard-token embeddings during adversarial training.
  • Figure 21a plots each soft token’s norm over training relative to the mean norm of hard-token embeddings.

A.5 MANY-SHOT ATTACK EXAMPLES

The many-shot example shows a model recognizing a repeated instruction to answer 42 while reasoning that it should instead follow the actual task instructions.

  • In a many-shot attack with n = 22, the model notices the pattern favoring 42 but ultimately reasons that answering 42 would be incorrect.
  • The model explicitly reviews the conversation and policies before producing a compliant, correct, and helpful response.
  • It distinguishes ignored instructions inside designated blocks from the final multiplication problem outside those blocks, while still considering whether the pattern should influence its answer.

A.6 SOFT TOKEN ATTACK EXAMPLES

The soft-token examples contrast unconstrained attacks that can induce a target output with norm-constrained attacks that disrupt the model’s chain-of-thought.

  • An unconstrained soft-token attack example targets outputting 42 with one unconstrained soft token.
  • In the illustrated math problem, the model correctly derives b = 259 before an attack causes it to end with 42.
  • The successful attack preserves apparently normal mathematical reasoning until the final attacked answer, without visible aberrations in the model’s thinking.
  • Norm-constrained soft tokens appear unable to preserve the full chain-of-thought, producing a visibly disrupted reasoning process.
  • The example trajectory shows the model proceeding through policy and prompt analysis before addressing the numerical task.

A.7 PROMPT INJECTION ATTACKS TO A BROWSING MODEL

For browsing agents, increasing inference-time compute can substantially improve robustness to website prompt injection. Increasing injections from 1 to 256 does not substantially increase attack success, possibly because intervening non-browsing messages reduce later injections’ impact.

  • Evaluation results: Increasing inference-time compute can substantially improve browsing agents’ robustness to prompt injection attacks.The evaluation uses o1-mini and o1-preview with sequences of browsing messages and one injection.
  • Evaluation setup: The evaluation differs from AdvSimpleQA by giving models a sequence of browsing messages rather than one message from a single website.
  • Varying attacker resources: Increasing prompt injections from 1 to 256 does not substantially increase attack success rate.Figure 23 varies injection counts for both o1-preview and o1-mini, with curves representing different attacker-resource levels.
  • Interpretation and limitation: Intervening non-browsing messages between the final browsing message and model output may diminish the impact of additional injections.The authors identify stronger prompt injection attacks as a direction for future work.
Loading 2501.18841v1…