PaperlayerView exact revision on arXiv

Source-linked AI summary

FlipJev: Fast Black-Box Red Teaming Finds Universal Adversarial Suffixes for Typed Decision Models

Xiaogeng Liu, Xinyan Wang, Chaowei Xiao

arXiv:2610.38895v1cs.CRcs.AI

TL;DR

Typed decision models are fast and increasingly used in security-sensitive tasks, but existing adaptive attacks do not fully use their speed and few studies test universal examples. The paper develops a black-box query-based attack on Jev and finds suffixes that flip answers across datasets and tasks.

  • Problem

    Red teaming of typed decision models has not fully exploited their speed, and few studies have tested whether universal adversarial examples transfer across tasks.

    Source: C One suffix across datasets · p. 2

  • Method

    The paper uses a black-box query-based attack that improves short suffixes through cheap word-level edits, scoring them with Jev’s returned probabilities.

    Source: C One suffix across datasets · p. 3, §3 · p. 5

  • Results

    One suffix generated on one task flips 90–100% of documents in five other datasets without querying them, while one suffix per dataset flips 43% of unseen documents overall.

    Source: C One suffix across datasets · p. 3

  • Takeaways & Limitations

    Evaluation of typed decision models should include strict adaptive attacks to assess their robustness.

    Source: C One suffix across datasets · p. 3, §5 · p. 10

    The study excludes attacks that change the content’s meaning; the suffixes preserve the correct label.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Typed decision models are an emerging class of language models that make decisions instead of generating free-text responses. They answer in a single fast and cheap query and are increasingly used, including in security-sensitive tasks. However, red teaming of these models leaves two gaps: existing adaptive attacks do not use their speed, and few studies have searched for universal adversarial examples. We fill both gaps on Jev, the representative typed decision model, with a quick black-box query-based attack that calls no attacker language model. We conduct extensive experiments on 12 datasets with different data types and application scenarios. The results show that Jev resists common prompt injections: no strategy changes a correct answer in 10,240 paired tests. In contrast, our attack finds short suffixes that flip confident answers within minutes. These suffixes are universal: one suffix per dataset flips 43% of unseen documents. Because typed decision models, unlike classic NLP models, can serve many tasks, we test whether these suffixes transfer across tasks. We find that one suffix generated from one task can flip 90-100% of the samples in other tasks. We therefore believe the evaluation of typed decision models must include strict adaptive attacks to fully reveal their robustness.

1. Introduction

Typed decision models return decisions rather than generated text, making them fast and inexpensive to use in applications. Their growing use in security-sensitive tasks makes robustness evaluation important.

1. Introduction

  • Typed decision models return probabilities for typed answer options in one query, without generating text.Jev answers in about 0.3 seconds and costs about four cents per million input tokens.Source: §1 · p. 1
  • Jev is the first deployed typed decision model, and recent work uses it as a classifier, router, guardrail, and safety judge.Such models are also called System One models.Source: §1 · p. 1
  • Typed decision models are increasingly used in security-sensitive tasks, where an attacker changing a decision is a security failure.Examples include screening external content, judging agent traces, and detecting jailbreaks and prompt injections.Source: §1 · p. 1

C One suffix across datasets

Existing adaptive attacks do not fully exploit typed decision models’ speed, and few studies search for universal adversarial examples. The paper develops a fast black-box attack on Jev and finds suffixes that flip answers within and across tasks.

C One suffix across datasets

  • Prior Jev attacks either use attacker language models that add generation time or choose from a limited set of templates.The paper identifies these constraints as reasons existing attacks do not fully leverage Jev’s speed.Source: C One suffix across datasets · p. 2
  • The proposed black-box attack improves short suffixes with evolutionary search using Jev’s returned probabilities and cheap word-level edits.It scores candidates on batches of training documents to generate universal suffixes.Source: C One suffix across datasets · p. 3
  • One suffix from a single dataset flips 90–100% of documents in five other tasks without further search.The paper tests whether a suffix transfers across tasks because one typed decision model can perform many tasks.Source: Figure 1 · p. 2, C One suffix across datasets · p. 3
  • Across nine datasets, one suffix per dataset flipped 43% of held-out documents, while a neutral note flipped none.The reported total is 114 of 264 held-out documents.Source: C One suffix across datasets · p. 3
  • Common prompt-injection strategies changed no correct answer in 10,240 paired tests, motivating an adaptive attack that uses model feedback.The fixed strategies do not adapt to Jev’s answers.Source: C One suffix across datasets · p. 3

2. Related Work

Prior work establishes typed decision models’ uses and explores fixed and adaptive attacks, but the paper identifies gaps in speed-aware red teaming and universal cross-task attacks. Its approach differs by searching with Jev’s probabilities and testing suffix transfer across tasks.

Typed decision models

  • Typed decision models return probabilities for developer-defined questions, and Jev is used as a decision layer, evaluator, and security judge.A survey of 2,170 public GitHub projects found applications for judgment, scoring, action selection, content filtering, and tool selection.Source: §2 · p. 3

Adversarial attacks and prompt injection

  • Prior adversarial work includes input-specific word edits, universal triggers, and suffix attacks, but much focuses on a single task.Two studies test cross-task triggers using models whose weights they hold, unlike an API-accessible typed decision model.Source: §2 · p. 4
  • Adaptive prompt-injection studies show that fixed attack suites measure resistance to known strategies, while adaptive attacks evaluate adversarial robustness.The paper applies this evaluation lesson to typed decision models.Source: §2 · p. 4

Attacks on Jev

  • Earlier Jev attacks use attacker language models or small template sets, and optimize or edit inputs separately rather than testing universal suffixes.These approaches add generation time or constrain the search space.Source: §2 · p. 4

Concurrent work

  • Concurrent studies differ in whether attack text can change the correct answer, how it is generated, and whether it transfers across tasks.Some use language-model generation or fixed attack variants; the cited studies do not test within-model task transfer.Source: §2 · pp. 4–5

3. Threat Model and Attack

The threat model lets an outside attacker append a short, label-preserving suffix to untrusted content and observe only Jev’s probabilities. The attack uses word-level evolutionary search to find suffixes efficiently, including universal suffixes scored across training documents.

Threat model

  • The attacker appends a suffix of at most 32 tokens to untrusted content and observes only Jev’s returned probabilities.The suffix cannot contain a word naming the question’s subject, so it does not change the correct answer.Source: §3 · p. 5

Quick black-box query-based attack

  • Cheap word-level edits generate candidates faster than Jev answers, avoiding the generation-time bottleneck of an attacker language model.The attack is an elitist evolutionary search over word sequences.Source: §3 · p. 5
  • For universal suffixes, fitness is mean P(no) across a batch of 10 training documents, rather than a single document’s P(no).The method maintains 16 elite suffixes and creates 64 new suffixes per generation.Source: §3 · p. 5

Experimental results

  • Table 1 reports attack success rates as shares of correct, confident decisions flipped across 12 datasets.Its universal-suffix results use one suffix per dataset, found on 10 training inputs and applied to held-out inputs.Source: Table 1 · p. 6

Quick black-box query-based attack

  • Suffixes are ranked first by whether they flip the answer, then by penalized fitness, other answers, and length.The token-length penalty uses λ = 0.0025.Source: §3 · p. 6
  • The search confirms a candidate with repeated queries, shortens confirmed suffixes, and stops at a confirmed flip or its query budget.A 2,000-query email search takes 1.5–2.6 minutes with six to eight parallel workers and costs about $0.10.Source: §3 · p. 6

4. Experimental Evaluation

Across 12 datasets, Jev resisted common fixed prompt injections, but adaptive black-box search found suffixes that flipped confident answers, worked on unseen documents, and transferred across tasks. The experiments tested correct, confident decisions and included agent-trajectory safety tasks.

4. Experimental Evaluation

  • The evaluation covered 12 datasets spanning different data types and application scenarios, targeting only correct and confident decisions.A successful flip changed a correct yes to no or an unsafe trajectory verdict to safe.Source: §4.1 · p. 7, §4 · p. 7

4.2. RQ1: Robustness to Common Prompt Injections

  • 0 strict flips occurred in 10,240 paired tests of common injections, and the five strategies also flipped none of the held-out inputs from the other 11 datasets.The tests included a forged Jev-style answer; its mean plain-Noul P(correct) stayed within 0.004 of the matched control.Source: §4.2 · p. 7

4.3. RQ2: Adversarial Robustness

  • 39% of email searches confirmed a flip after a median of 1,287 queries, with suffixes of 7–31 tokens and a median length of 8.The 11 confirmed flips came from 28 searches on nine yes emails.Source: §4.3 · p. 7
  • Per-document search flipped 58 of 180 targets across nine datasets (32%); most flips came after 1,000 queries, and the curves continued rising at the 2,000-query budget.The median number of queries was 1,380; 2,000 queries took about two minutes, so the reported rates are lower bounds for a larger budget.Source: §4.3 · pp. 7–8, Figure 2 · p. 8
  • On agent trajectories, the best per-input suffix lowered safety scores by 0.42 on R-Judge and 0.49 on ATBench across 10 fresh queries.The drops closed about half the distance from the clean score to the verdict threshold; suffixes were at most 64 tokens.Source: §4.3 · p. 8

4.4. RQ3: Universality

  • 43% of held-out documents across nine datasets were flipped by one universal suffix per dataset, versus 32% with per-document search.Universal suffixes flipped 114 of 264 held-out documents; no suffix and a neutral note flipped 0 of 264.Source: §4.4 · p. 8
  • The ATBench universal suffix flipped 9 of 30 held-out unsafe trajectories (30%) and lowered their scores by 0.65 on average.Across both trajectory benchmarks, suffixes lowered all 30 held-out scores by more than 0.05, while same-length neutral filler did so for at most two.Source: §4.4 · p. 9

4.5. RQ4: Transfer across Datasets and Tasks

  • 90–100% of held-out documents in five datasets were flipped by the arXiv full-text suffix, despite each dataset asking a different question.Across all 72 cross-dataset pairs, the mean share flipped was 18%; at least 20% were flipped in 21 pairs and at least 50% in 11.Source: §4.5 · p. 9

5. Conclusion

The paper studies the adversarial robustness of typed decision models and proposes a quick black-box suffix attack. On Jev, common prompt injections change no correct decisions, while the attack flips confident decisions and finds suffixes that work across unseen samples and tasks.

5. Conclusion

  • The attack flips confident Jev decisions within minutes, while common prompt injections change no correct decision.The attack uses cheap word-level edits to improve a short suffix.Source: §5 · p. 10
  • One suffix per dataset flips 43% of unseen samples.Source: §5 · p. 10
  • A suffix from one task flips 90–100% of samples in five other tasks without querying them.Source: §5 · p. 10

Limitations

The study shows that appending a short suffix that keeps the correct label can flip Jev’s correct, confident decisions. Attacks that change the content’s meaning are outside the paper’s threat model; refining the attack to use fewer queries is left for future work.

Limitations

  • Attacks that change the content’s meaning are outside the paper’s threat model.The studied attack appends a short suffix that keeps the correct label.Source: Limitations · p. 10
  • Refined mutation operators or other attack components could find flips with fewer queries, but are left for future work.Source: Limitations · p. 10
  • The current attack takes about two minutes per input and finds suffixes that transfer across tasks.Source: Limitations · p. 10

Ethical Considerations

The section reports that common prompt injections did not flip correct Jev decisions in the tested settings, then describes the attack and experimental setup. It also states the evaluation’s scope and risk controls.

Ethical Considerations

  • The evaluation uses Jev’s public API, reports findings to TypeSafe, and says the suffixes change decisions without extracting data or producing harmful text.The authors say the inputs come from public benchmarks or were written by them, and that they will release code, suffixes, and result files.Source: Ethical Considerations · p. 10

A. Search Settings

  • The search mutates suffixes through token edits, span operations, random strings, and crossover, then confirms and shortens candidates that flip Jev’s answer.The search uses weighted operators and re-confirms a shortened suffix before keeping it.Source: A. Search Settings · p. 18

B. Experimental Setup Details

  • Experiments cover 12 datasets; targets are correct, confident decisions, and held-out documents are disjoint from search data.The nine text datasets use binary questions about native classes; sentiment is excluded from main totals because appended text can change a review’s sentiment.Source: B. Experimental Setup Details · p. 19, C. Datasets · p. 20

D. Common Prompt Injections

  • Email tests found no flips under simple instructions or complex contexts, while requests in the trusted question lowered confidence without flipping decisions.The complex-context tests kept all 8,160 decisions correct with P(correct) at least 0.92; trusted-question requests lowered P(correct) but flipped no decision.Source: D. Common Prompt Injections · pp. 21–22
  • No common injection strategy flipped a correct decision in 10,240 paired tests across the four question configurations.The tests paired clean and injected decisions and counted strict flips.Source: Table 4 · p. 22

E. Email Search Details

Email searches show that feedback-guided suffix search can find confirmed flips, and some discovered suffixes affect other emails. The section also describes universal searches over training and held-out documents.

E. Email Search Details

  • Across email searches, confirmed-flip rates varied by seed from 0.00 to 1.00, and the searches used 47,183 queries at a total cost of $2.48.The question seed succeeded in all three searches that used it; the computed seed succeeded in none of three.Source: E. Email Search Details · p. 23
  • Feedback-guided evolutionary search flipped one long email, while random mutation without feedback reached P(no) values of only 0.07, 0.07, and 0.05 on the three targets.Hill climbing also flipped one email but did not flip the other two.Source: E. Email Search Details · pp. 23–24
  • Six email suffixes flipped 12–21 of 49 other emails each, with higher rates on long natural and padded emails than on short hard or easy emails.Clean emails and neutral notes flipped 0 of 50.Source: E. Email Search Details · pp. 24–25

F. Universal Search Details

  • Universal searches use one suffix found on 10 training documents, and the reported held-out share measures documents flipped without further search.The results table lists candidate counts and held-out shares; the suffix table sorts nine dataset suffixes by held-out flip share.Source: Table 6 · p. 25, Table 7 · p. 25

G. Analysis: Why the Suffixes Work

Ablations and cross-question tests support a surface-pattern account of the suffixes rather than instruction following. Their effects depend on particular words and their order, and can differ by answer type.

G. Analysis: Why the Suffixes Work

  • The original suffixes flipped 46% of held-out documents in fresh queries, while every ablation reduced the share to 4% or less.The tested variants altered or removed label words, removed “question,” shuffled tokens, used the seed, or used no suffix.Source: G. Analysis: Why the Suffixes Work · p. 26, Table 8 · p. 27
  • The boundary instruction still permitted 155 of 264 held-out documents to flip, compared with 122 without it.On five datasets whose suffixes kept the full seed frame, the boundary instruction raised flips to 29–30 of 30; the authors attribute this to shared words completing the trigger.Source: G. Analysis: Why the Suffixes Work · pp. 27–28
  • On five datasets, Noul suffixes left Choice answers correct on all 150 held-out documents, while Choice-targeted suffixes flipped Choice more often than Noul on DBpedia and patents.Those Choice-targeted suffixes flipped Choice on 22 of 60 documents and Noul on 2.Source: G. Analysis: Why the Suffixes Work · p. 28
  • The authors interpret the suffixes as label-shaped questions that Noul treats as evidence, with the label word setting answer direction rather than functioning as an instruction.The re-evaluation also reports no flips without a suffix across the held-out documents and datasets.Source: G. Analysis: Why the Suffixes Work · p. 28

H. Trajectory-Judge Details

The trajectory-judge experiments insert suffixes only into eligible third-party text and assess whether unsafe trajectories receive safe verdicts. The section reports fresh-query outcomes and controls alongside a worked example.

H. Trajectory-Judge Details

  • The search maximizes P(safe) = 1 − score/4, with a safe verdict below the 2.0 threshold, and inserts suffixes only into eligible third-party text.A verifier checks that the input changes only by the inserted characters and remains parseable.Source: H. Trajectory-Judge Details · pp. 29–30
  • Fresh retests found per-target flip rates of 0.15 on R-Judge and 0.10 on ATBench, with mean score drops of 0.42 and 0.49, respectively.The fresh-query mean drops were close to the drops observed during search.Source: H. Trajectory-Judge Details · p. 30, Table 13 · p. 31
  • On fresh queries, universal suffixes moved every held-out unsafe trajectory toward a safe score, while same-length neutral filler stayed at no change.The filler changed the mean score by at most 0.01; the searched suffix more than doubled the seed text’s effect.Source: Figure 9 · p. 31, H. Trajectory-Judge Details · p. 31
  • In one ATBench example, the trajectory score fell from 3.15 (unsafe) to 1.96 (safe), below the 2.0 threshold on 7 of 10 fresh queries.The suffix was appended to third-party text already present in a tool output.Source: H. Trajectory-Judge Details · pp. 31–32
Loading 2610.38895v1…