PaperlayerView exact revision on arXiv

Source-linked AI summary

JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models

Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, Leo Yu Zhang

arXiv:2609.31142v1cs.CRcs.AIcs.CL

TL;DR

Adversarial robustness has not been measured for typed decision models, whose schema-valid outputs differ from the text or tool calls targeted by existing benchmarks. JevAdvBench compares black-box single-edit attacks with clean decisions and finds that an appended opinion flips 12.1% of decisions, statistically tied with the strongest injected command (10.1%).

  • Problem

    Adversarial robustness of typed decision models was unmeasured: existing benchmarks target generated text or tool calls, while these models return schema-valid decisions that software may act on directly.

    Source: §1 · pp. 1–2

  • Method

    The benchmark applies single-edit black-box attacks to typed requests and scores attacked answers against the model’s clean answer, with identical clean re-runs setting the noise floor.

    Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 6

  • Results

    12.1% of decisions flipped under an appended observer opinion, statistically tied with the strongest injected command (10.1%); the opinion also pushed 38.0% of confident CHOICE and SCORE answers below the 0.8 gate.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    For typed decision APIs, user-writable state should be treated as argued input rather than fact, and deployments should plan review capacity for attack-induced escalation.

    Source: §6 · p. 11

    The evaluation covers only jev-1.13.0 and fixed-template, non-adaptive attacks, so it makes no claim about other models or adaptive attack rates.

    Source: §6 · p. 11

Abstract

from arXiv · show

Models trained with reinforcement learning for calibrated decisions (RLCD), such as Jev, answer a typed question about an input, the state, with a probability, a choice, or a score, and software acts on the answer without a person reading it. Their robustness has not been measured: adversarial benchmarks score what a model generates or executes, whereas a typed model generates nothing and returns a well-formed answer even when manipulated. Measurement is also hard, because identical requests can return different answers, most available labels come from the model itself, and the API preprocesses each request out of view. Our key idea is to score each attacked decision against the model's own clean decision rather than against labels, and to read it against the change caused by an identical re-run. Building on this, we introduce JevAdvBench, to our knowledge the first adversarial benchmark for RLCD models, with 812 typed questions over 66 scenarios, and a black-box attack suite of 9,744 single-edit variants that each edit one part of a request, with billed input tokens confirming that the edit reached the model. On jev-1.13.0, rewording stays within 1.2 percentage points of the re-run baseline, and fields outside the schema never reach the model. In contrast, one unverified opinion appended to the state flips 12.1% of decisions, statistically tied with the strongest injected command (10.1%), and pushes 38% of confident answers below the 0.8 confidence threshold that routes them to human review. Applications built on RLCD models should therefore treat the state as untrusted, argued input. Project website: https://JevAdvBench.github.io/JevAdvBench/

1 INTRODUCTION

JevAdvBench measures whether black-box edits change typed decisions relative to the model’s own clean decisions and identical re-runs. Its evaluation finds that state-borne arguments can change decisions and confidence even when typed outputs leave no generated text to attack.

1 INTRODUCTION

  • RLCD models return typed probabilities, choices, or ordinal scores that software may act on without human review, making whoever steers a decision able to steer the application.Jev is positioned for ticket routing, claim approval, content moderation, and screening other LLMs’ inputs for prompt injection.Source: §1 · p. 1
  • Existing benchmarks judge generated text, tool calls, or gold-label text perturbations, which do not fit schema-valid typed decisions; prior Jev studies did not test adversarial manipulation of its decisions.The paper presents JevAdvBench as, to its knowledge, the first adversarial benchmark for RLCD models.Source: §1 · pp. 1–2
  • JevAdvBench contains 812 questions across 66 scenarios and three primitives, and scores flips against identical re-runs because outputs vary and independent labels are scarce.Identical requests change SCORE for 38.5% of questions; exact-match clean error is 9.98% (2.2% within a quarter level), and 82.4% of public-release labels are model consensus. It also measures target drift and confidence-gate effects; 143 human-reviewed labels support accuracy checks.Source: §1 · p. 2
  • The black-box suite tests nine single-query attacks across question, state, and injection surfaces, plus out-of-schema structure controls, for 9,744 variants with delivery checked using billed input tokens.Attacks make one edit per variant; none uses gradients, search, or repeated queries.Source: §1 · p. 2
  • On jev-1.13.0, paraphrase adds at most 1.2 pp over the re-run noise floor, while observer opinion flips 12.1% [8.5, 15.5] of decisions, tied with the strongest injected command (10.1%).The opinion contains no instruction; the figure compares pooled flip rates with the identical re-run noise floor.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Observer opinion pushes 38.0% [31.1, 44.0] of confident CHOICE and SCORE answers below the 0.8 confidence gate, versus 0.5% under re-run; 119 of 154 retain their decision.This confidence-gate effect can increase human-review load even without a decision flip.Source: §1 · p. 3
  • The release includes all 9,744 variants, 11,368 raw responses, and a script that regenerates every reported number.The release is identified as beta1.0-20260925.Source: §1 · p. 3

2 THREAT MODEL

The threat model concerns applications that act on typed RLCD answers without human review, and attackers who make one black-box edit to change decisions or increase review load. Harm is defined relative to cases where the correct answer has not changed.

Target system

  • Applications send typed questions about content and use NOUL, CHOICE, or SCORE outputs with task-specific thresholds to act, seek confirmation, or route answers for review.NOUL returns P(true); CHOICE and SCORE return probabilities for each option or level and a confidence score. The example review thresholds are application policies, not fixed API rules (TypeSafe AI, 2026a; TypeSafe AI, 2026b).Source: §2 · p. 3

Attacker goals

  • Attackers seek to flip or steer decisions, or push confident answers below a review gate; these count as harm only when the correct answer is unchanged.Under the deployment contract, an outside opinion that argues for an answer without adding case facts should not change the decision.Source: §2 · p. 3

Knowledge and capabilities

  • The attacker has black-box access, sees only final answers, makes one non-adaptive query per attempt, and edits one request at a time.They lack weights, gradients, and training data; although the API returns option probabilities, the attacker does not use them.Source: §2 · pp. 3–4

Access

  • The attacker can edit one request slot in either state text or, at a second access tier, text inside the question string.Examples include authoring a support ticket or claim note, or controlling interpolated instructions, editable option descriptions, or an agent-written question; compromised APIs, model extraction, and multi-query optimization are out of scope.Source: §2 · p. 4

3 JEVADVBENCH

JevAdvBench combines typed questions, single-surface attacks, and a scoring protocol that compares attacked decisions with the model’s own clean decisions. It uses billing to distinguish edits delivered to the model from undelivered structural controls.

3.1 BENCHMARK CONSTRUCTION

  • The benchmark covers 66 scenarios and 812 questions (314 NOUL, 337 CHOICE, 161 SCORE), with 669 jev_default labels and 143 human-reviewed labels.All 8 identical re-run flips occurred on human-reviewed questions (5.6% [1.8, 10.0]); attacks flipped 16.4% of their delivered variants on reviewed questions versus 4.0% on jev_default questions.Source: §3.1 · p. 4

3.2 SURFACE-AWARE ATTACKS

  • Nine in-schema attacks edit Question, State, or Injection surfaces, while Structure edits serve as delivery controls rather than pooled robustness evidence.Each variant applies one reversible edit without stacking, and the shared fixed templates use no per-scenario tuning, making reported attack rates lower bounds for an adaptive attacker.Source: §3.2 · pp. 5–6, Table 1 · p. 5, §3.3 · p. 6

3.3 TYPED ROBUSTNESS PROTOCOL

  • Attacks are scored against each question’s clean decision, with a 1.0% [0.3, 1.9] identical re-run noise floor and excess flips reported in percentage points.Target hits, shifts without flips, and confidence-gate measures supplement flip rates; billing identifies delivered edits, while a secondary check measures accuracy against 143 human-reviewed labels.Source: §3.3 · p. 6

4 EXPERIMENTS

Across 812 questions, attacks reached the model only when edits were within its schema, and the nine delivered attacks changed decisions far more often than re-runs. Opinions and commands caused the strongest changes, including shifts that flips alone and a confidence gate would miss.

4.1 SETUP

  • The public-API evaluation used 10,556 requests across 812 clean and 9,744 attacked inputs, plus 812 clean re-runs; all 11,368 returned HTTP 200.Confidence intervals used 2,000 scenario-cluster bootstrap resamples over 66 scenarios; the minimum detectable excess was 1.9 pp overall but 7.0 pp for SCORE, making a null SCORE result weak evidence.Source: §4.1 · p. 7

4.2 RQ1: WHICH INPUTS MOVE DECISIONS?

  • Word/spacing edits flipped 1.0% and paraphrasing 1.5%, against a 1.0% re-run noise floor; paraphrasing’s one-sided 95% upper bound on excess was 1.2 pp.Unrelated sentences flipped 4.6% [3.0, 6.4] and shifted another 17.4% without a flip, while the unrelated state note’s +1.2 pp did not survive Holm correction; these results concern the tested fixed strings and benign edits.Source: §4.2 · p. 7
  • 12.1% of decisions flipped under observer opinion (+11.1 pp over re-run noise), statistically tied with authority impersonation at 10.1% (+9.1 pp).The observer opinion contains no command and claims no authority; the difference from authority impersonation was +2.0 pp [−2.1, 5.9].Source: §4.2 · pp. 7–8
  • Commands in instructions flipped 15.3% of decisions versus 2.6% in criteria, a gap concentrated in CHOICE; out-of-schema edits never reached the model.The within-question difference was +14.3 pp [9.5, 19.7] among 245 questions receiving commands in both slots, and the CHOICE rates were 30.7% versus 2.1%. For out-of-schema opinions, the same texts added 58.1 billed tokens on average in state and none in an extra field; attack rankings also varied by answer type.Source: §4.2 · p. 8
  • 29.2% [25.6, 32.6] of questions flipped under at least one delivered attack, versus 1.5% under the union of four unchanged-input controls.The 10% of questions that flipped most often accounted for 59.2% of all flips; the 95% confidence intervals use scenario-cluster bootstraps.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 RQ2: HOW MUCH DAMAGE DO THE ATTACKS DO?

  • 88.8% [83.3, 94.0] of flips from five targeted attacks on eligible multi-option CHOICE questions landed on the attacker’s target, versus 31.2% for a uniform pick among other options.Observer opinion moved SCORE toward its target by +0.239 [0.144, 0.347] levels; on NOUL, opinions moved P(true) toward targets more than commands did over slot-matched placebos (+30.9 pp versus +13.7 pp).Source: §4.3 · pp. 8–9
  • 35.6% [31.5, 39.6] of answers shifted under observer opinion without flipping, bringing the share that flipped or shifted to 47.7%.Commands shifted 23.6–24.6% of answers without a flip, versus 0.1% under re-run; on human-reviewed questions, tolerant accuracy fell from 87.4% clean to 68.5% under observer opinion (−18.9 pp). The 143 reviewed labels mostly matched the model’s five-run answer, and attack variants were designed, not verified, to preserve answers.Source: §4.3 · pp. 9–10

4.4 RQ3: CAN THE CONFIDENCE GATE PROTECT A DEPLOYMENT?

  • 38.0% of initially confident CHOICE and SCORE answers fell below the 0.8 review threshold under observer opinion, an excess of +37.5 pp over re-run.Across delivered attacks, 441 of 534 drops below the gate left the decision unchanged; 12.6% of flips passed the gate with confidence ≥0.8, and clean answers already sent 18.7% to review.Source: Figure 4 · p. 9, §4.4 · p. 10

5 RELATED WORK

Prior work studies adversarial effects on generated text, tool use, or benign-input classification, while this section situates attacks on calibrated typed decisions alongside those fields. It highlights observer-opinion effects on Jev’s confidence-based review routing.

Prompt injection and jailbreaks

  • Prior prompt-injection and jailbreak benchmarks score generated task outputs, agent tool calls, or harmful text; Jev’s schema strips out-of-schema payloads but leaves state-slot steering.The observer opinion contains no instruction for these defenses to reject, and the paper did not test them.Source: §5 · p. 10

Robustness of text classifiers

  • Jev stays within 1.2 pp of its noise floor on benign rewording; flip and drift adapt CheckList’s invariance and directional tests to stochastic typed outputs using identical reruns, not gold labels.The paper ran no search-based form attack.Source: §5 · p. 10

Judges and sycophancy

  • Manipulated judges and sycophantic assistants are analogues, but prior work measures generated text; observer opinion may produce input-side deference without text acknowledgment, a hypothesis from one model.The paper does not claim to establish what in the model’s training causes the effect.Source: §5 · pp. 10–11

Calibration, selective prediction, and Jev

  • For observer opinion, attack-time confidence detected flips barely better than clean confidence, while confident answers routed to review increased by +37.5 pp [30.7, 43.4].Jev uses a confidence threshold for review; the section therefore evaluates the gate under attack.Source: §5 · p. 11
  • Confidence-threshold abstention is a standard selective-prediction safety valve usually tested on benign inputs; Jev’s use as a zero-shot alignment-failure detector motivates testing its gate under attack.Prompt artefacts and option identifiers can bias labels; Sun et al. (2026) show Jev follows benign option names.Source: §5 · p. 11

6 DISCUSSION, LIMITATIONS, AND CONCLUSION

JevAdvBench turns the risk of argued text moving typed decisions into a repeatable measurement, while its findings support treating user-written state as untrusted evidence and planning for attack-induced review. The conclusions are limited to one model version and a deliberately weak, fixed-template attacker.

Implications for deployment

  • Builders should treat user-written state as argued input, keep user text out of instructions, and plan review capacity for attack-induced escalation.An observer opinion appended to state raised the flip rate by +11.1 pp [7.8, 14.4] over the noise floor; a command flipped decisions 14.3 pp [9.5, 19.7] more often in instructions than criteria. The opinion pushed confident CHOICE and SCORE answers below the 0.8 gate at 37.5 pp [30.7, 43.4] above the re-run; NOUL returns no confidence, and no defence was evaluated.Source: §6 · p. 11

Limitations

  • The findings apply only to jev-1.13.0 and fixed-template, nonadaptive attacks, so attack rates are lower bounds and do not establish robustness to adaptive search.The 739 of 812 word/spacing edits that changed only spacing do not address adversarial paraphrase search; low flip rates for structure edits reflect API removal before inference, inferred from billing, not robustness. Only one LLM annotator checked 40 variants, and most labels (82.4%) were model consensus answers on items selected for clean stability, motivating label-free primary metrics and human-label consistency results. API preprocessing, confidence calculation, and version aliasing are undocumented; the authors pin the version and release raw responses with request hashes.Source: §6 · p. 11

Conclusion

  • JevAdvBench enables re-benchmarking new versions and comparison across typed decision models under a shared protocol.The benchmark costs well under a dollar per run on jev-1.13.0 (TypeSafe AI, 2026e).Source: §6 · p. 11

ETHICS STATEMENT

JevAdvBench is dual-use, but the authors release it to help builders test deployments against cheap, realistic attacks before attackers do.

ETHICS STATEMENT

  • The benchmark’s steering templates are dual-use, and the authors justify release because builders need them, delivery checks, and confidence-gate measurements to test deployments.The attacks are fixed texts a user could already put in a ticket or post.Source: ETHICS STATEMENT · p. 12
  • All requests targeted the public jev-1.13.0 API; the authors attacked no third-party deployment and collected no personal data.Scenarios draw on the vendor’s published examples plus questions written by the authors.Source: ETHICS STATEMENT · p. 12
  • The authors state that they will share their findings with the vendor before publication.Source: ETHICS STATEMENT · p. 12

REPRODUCIBILITY STATEMENT · A TYPED DECISION API

The statement documents how to reproduce the study and defines the typed API decisions, confidence gates, and vendor robustness claims that JevAdvBench examines. Requests contain a state and typed questions, while the benchmark treats each primitive’s decision separately.

REPRODUCIBILITY STATEMENT

  • The study pins release beta1.0-20260925, jev-1.13.0, run dates, request hashes, generator seeds, and bootstrap settings of 2,000 resamples with seed 0.These details are listed in Section L for rerunning or auditing the study.Source: REPRODUCIBILITY STATEMENT · p. 12
  • The release includes 11,368 raw responses, and data_analysis/run_all.sh regenerates every reported number, results table, and data plot.The responses comprise 10,556 evaluation requests and 812 re-runs; attack templates, delivery accounting, and metric definitions are documented in Sections C, D, and E.Source: REPRODUCIBILITY STATEMENT · p. 12

A TYPED DECISION API

  • A request carries one state and typed questions with instructions and type-dependent criteria; billing indicates out-of-schema question keys never reach the model.The vendor documents the question fields and says caller-chosen question names are not sent to the model or used in inference (TypeSafe AI, 2026a).Source: A TYPED DECISION API · p. 16
  • NOUL decisions are truth probabilities, CHOICE decisions use the returned option key, and SCORE decisions use the argmax level rather than the expectation.In 2 of 4,718 CHOICE answers, the returned key differed from the argmax option by a probability margin of 0.01; the study uses the returned key.Source: A TYPED DECISION API · p. 16
  • Confidence is available only for CHOICE and SCORE; NOUL has no confidence field, so a NOUL gate must use P(true) itself.The vendor describes confidence as a statistic derived from the probability distribution but does not document its formula (TypeSafe AI, 2026b). The study uses a 0.8 main gate and reports 0.6–0.9 in Appendix H.Source: A TYPED DECISION API · p. 17
  • The vendor says adversarial content can move answers, semantically similar inputs should yield similar outputs, and unrelated state content can distract, but gives no rates or taxonomy.The benchmark tests rewording, unrelated content, and conflicting slots; it also measures confidence under attack, which the cited vendor statements do not characterize. The vendor additionally cautions that calibration is group-level and describes a guardrail use case (TypeSafe AI, 2026c; 2026d; 2026g).Source: A TYPED DECISION API · p. 17

B BENCHMARK CONSTRUCTION

JevAdvBench extends the vendor’s examples into 812 typed questions across 66 scenarios. Its construction documents label provenance and repeat-run variation, supporting evaluation against the model’s clean decision rather than labels.

Scenarios and domains

  • 812 typed questions span 66 scenarios, and every scenario includes NOUL, CHOICE, and SCORE questions.Fifty scenarios contain four NOUL, four CHOICE, and two SCORE questions; the median scenario has 10 questions. Because questions within a scenario share a state, bootstraps resample scenarios, with a Kish effective cluster count of 47.0; states and questions are in English, and 46 states are plain strings while the rest are JSON objects.Source: B BENCHMARK CONSTRUCTION · p. 18, Figure 5 · p. 18

Question origin

  • 480 questions (59.1%) are project-authored, 319 come from the vendor reference set, and 13 reuse official question definitions.Among 143 human-reviewed questions, 99 are from the vendor set and 44 are project-authored; 436 of 669 pseudo-labelled questions are project-authored, so origin and label source are not independent.Source: B BENCHMARK CONSTRUCTION · p. 18

Five-run labelling

  • 730 questions met the five-run stability rules, while 82 were sent to review.Each question was first answered five times by jev-1.13.0; the candidate label is the mean probability for NOUL, the winning option for CHOICE, and the mean score for SCORE. Stability required the same band, winner, or top level across runs, with probability/score ranges no greater than 0.10 for NOUL and CHOICE probabilities, and 0.25 for SCORE; CHOICE and SCORE also required a winner or top-level probability of at least 0.7.Source: B BENCHMARK CONSTRUCTION · p. 19

Label provenance

  • 669 labels (82.4%) are the model’s own stable answers, while 143 were human-reviewed; only 41 human labels carry judgment independent of the model.Of the 143 human labels, 124 agree with the candidate; 23 interval labels do not map to a single decision. This provenance explains why the primary metric compares attacked answers with the model’s clean answer rather than a label.Source: B BENCHMARK CONSTRUCTION · p. 19

Batched labelling versus single-question evaluation

  • 0.7% [0.3, 1.2] of decisions differ between two batched runs, versus 1.0% [0.3, 1.9] between two single-question runs.The single-question clean run and a batched run differ on 1.0% [0.3, 1.7] of decisions. Attacks are scored against the single-question clean run, with batched runs used as additional noise replicates.Source: B BENCHMARK CONSTRUCTION · p. 19

C ATTACK GENERATION

JevAdvBench generates reproducible, single-edit attack variants from clean requests and checks their structure, while noting limited text diversity and known rendering defects. These checks establish structural validity, not semantic validity.

Shared rules

  • Each of 9,744 variants applies one reversible edit to one request surface, is generated reproducibly from fixed seeds, and remains an unreviewed candidate.State attacks edit only state; Question, Injection, and Structure attacks edit only question. Inserted segments are recorded by offset so removal restores the clean request byte for byte.Source: C ATTACK GENERATION · p. 19

Targeting rules

  • Six attacks target a decision, but 25 to 41 rows per attack target the model’s clean decision and are excluded from targeted metrics, leaving 775 to 787 eligible questions.Targets are selected by fixed rules for CHOICE, NOUL, and SCORE; rows whose target equals the clean decision remain in flip rates because flips are defined without a target.Source: C ATTACK GENERATION · pp. 19, 21

Structural checks and diversity

  • Unrelated-sentence attack flip rates range from 2.6% to 6.5%, and unrelated-state-note rates range from 0.8% to 3.3%; neither set differs significantly across its few texts.The attacks use only three unrelated-sentence pairs and six unrelated state-note texts, so their rates describe those strings; permutation p-values are 0.12 and 0.17, respectively.Source: C ATTACK GENERATION · pp. 21–22, Table 10 · p. 22

Known defects

  • Observer opinion flips 5.3% of 57 null-rendered cases versus 12.6% of the other 755, so null-rendered opinions do not inflate its headline rate.Defects include literal “null” conclusions for some CHOICE options, raw-JSON quotations when instructions are objects, and Injection commands inserted mid-sentence in 2,030 of 2,436 rows.Source: C ATTACK GENERATION · p. 22, Table 11 · p. 22

D DELIVERY VERIFICATION

Billing distinguishes edits that reach the model from edits that do not: added text inside the schema increases billed tokens, while out-of-schema keys add none. Equal token counts do not establish that a rewrite was undelivered.

D DELIVERY VERIFICATION

  • 5.0 characters per token describes added in-schema text; all 5,684 unrelated-sentence, State, and Injection rows bill more tokens than their clean requests.The fitted estimate for those rows was 4.99 characters per token (r = 0.987, n = 5,684); Figure 6b shows the billing relationship.Source: D DELIVERY VERIFICATION · p. 23, Figure 6 · p. 24
  • None of 1,624 out-of-schema extra-field rows bills an extra token, supporting the inference that the API removes unknown question keys before inference.This is inferred from billing; server-side preprocessing is not directly observable.Source: D DELIVERY VERIFICATION · p. 23
  • Escaped field names are a control for JSON decoding, not field stripping: parsed questions match the clean question in all 812 rows.The evaluation sends raw question text without re-serializing it, so the escapes reach the API; zero token change is expected because escaping changes only JSON representation.Source: D DELIVERY VERIFICATION · p. 23
  • 58.1 tokens on average were billed for an observer opinion in the state, versus 0.0 for byte-identical text in an extra field across 59 questions.The respective flip rates were 5.1% and 0.0% (difference CI [0.0, 11.3]), too few to compare; the all-question difference measures delivery, not how the model would weigh an opinion in an extra key.Source: D DELIVERY VERIFICATION · p. 23
  • 586 of 812 paraphrases bill exactly the clean token count despite changing the text, and they contain 11 of paraphrasing’s 12 flips.Accordingly, a zero net token change does not establish non-delivery for rewrites; delivery is classified by attack, with Structure rows excluded from pooled rates.Source: D DELIVERY VERIFICATION · p. 23

E METRIC VALIDITY AND POWER

The appendix argues that exact label matching mismeasures SCORE because of precision mismatches and model-derived labels, and evaluates how noise, flip definitions, scenario choices, and limited power affect the attack findings.

Metric validity

  • 38.5% of SCORE questions return a different value on identical requests, making exact label matching invalid; a ±0.5-level tolerance gives 2.2% clean error versus 9.98% overall under strict matching.The strict clean error is 41.6% on SCORE; the tolerance plateau is partly circular because 669 labels are model-derived, and attacked SCORE error varies with tolerance.Source: E METRIC VALIDITY AND POWER · p. 25, Table 14 · p. 25, Figure 7 · p. 26

Noise floor

  • The identical re-run flip rate is 1.0% [0.3, 1.9] (8 of 812), and alternative noise-floor estimators also yield about 1%.All eight re-run flips are among 143 human-reviewed questions (5.6% [1.8, 10.0]), none among 669 pseudo-labelled questions; all 17 noise-prone questions failed the five-run stability screen.Source: E METRIC VALIDITY AND POWER · p. 25

Flip-definition sensitivity

  • Across 16 alternative flip definitions, observer opinion ranks first under every variant; seven attacks exceed their noise floors under every variant, while no Structure attack does.Under the most conservative definition, the noise floor is 0.0%, while observer opinion and authority impersonation still flip 10.0% and 6.5%; their 2.0 pp difference is unresolved.Source: E METRIC VALIDITY AND POWER · pp. 25–26, Table 15 · p. 26, Figure 8 · p. 27

Power and equivalence bounds

  • At 80% power, minimum detectable excesses are 1.9 pp overall, 3.0 pp for NOUL, 3.7 pp for CHOICE, and 7.0 pp for SCORE.At the worst Holm step over 12 attacks, those thresholds rise to 2.8, 4.8, 5.6, and 10.5 pp, so a null SCORE result is weak evidence. Nonsignificant paraphrasing and word/spacing edits add at most 1.2 pp and 0.5 pp, respectively; the unrelated state note’s 1.2 pp excess had 79.5% power at 66 scenarios.Source: Table 16 · p. 27, E METRIC VALIDITY AND POWER · p. 27, Figure 9 · p. 28

Ranking robustness

  • Removing any one scenario changes no attack rate by more than 1.2 pp and preserves the same top three; equal scenario weighting leaves the ranking unchanged.Observer opinion ranks first in about 80% of scenario-bootstrap resamples and all 66 leave-one-scenario-out samples, but the two leading attacks tie at 10.2% after removing the two largest scenarios. Injection ranks above State and State above Question in 95.9% of bootstrap resamples; the Injection–State difference is +2.0 pp [−0.2, 4.4].Source: E METRIC VALIDITY AND POWER · p. 27, Figure 10 · p. 28

F FULL FLIP RESULTS … M EXTENDED RELATED WORK AND COMPARISON TABLE

Across the delivered attacks, flips affect many questions but cluster on a subset, and which attacks work varies by answer type. Re-run comparisons distinguish these effects from noise while accounting for scenario clustering.

F FULL FLIP RESULTS

  • Table 17 reports flip counts and rates by primitive with 95% scenario-cluster bootstrap intervals; the re-run is the noise floor, and Structure rows are delivery controls.Field-name escaping is a JSON no-op.Source: Table 17 · p. 29
  • Both scenario-level permutation and item-level exact McNemar tests flag the same six attacks on pooled data, but disagree on two primitive-level cells.Both tests apply Holm correction over 12 attacks; the main text reports the permutation result. Unrelated state note is nominally above noise but not after correction (Holm p = 0.20).Source: F FULL FLIP RESULTS · p. 29, Table 18 · p. 30
  • Attack rankings vary by primitive: Kendall’s τ is 0.63 for NOUL against CHOICE (p = 0.02), 0.40 for CHOICE against SCORE (p = 0.14), and 0.06 for NOUL against SCORE (p = 0.83).On NOUL, the three commands lead; on SCORE, observer opinion and opinion via analogy lead, while direct override and authority impersonation fall below unrelated sentences. SCORE’s minimum detectable excess is 7.0 pp, so a small command effect is not ruled out.Source: F FULL FLIP RESULTS · p. 29, Figure 11 · p. 30
  • 237 questions (29.2% [25.6, 32.6]) flip under at least one of 12 attacks, versus 1.5% [0.5, 2.6] under the empirical four-draw noise reference.The independent-noise reference is 11.2%, but the empirical reference is lower because noise flips concentrate on the same unstable questions.Source: F FULL FLIP RESULTS · pp. 29, 31, Table 19 · p. 31
  • The 472 flip events fall on 237 questions, with 56 questions carrying half of them; delivered attacks overlap more than a same-primitive permutation null predicts.Mean pairwise Jaccard similarity is 0.137 versus 0.025 under the null (p = 0.0005), and 108 questions flip under at least two delivered attacks versus 80.7 expected [70, 92].Source: F FULL FLIP RESULTS · p. 31, Figure 13 · p. 32
  • Among questions flipped by any attack, 99 flip only under Injection attacks, 61 only under State attacks, and 32 under both.The largest delivered-attack overlaps are observer opinion with opinion via analogy (40 shared questions), and direct override with authority impersonation (33).Source: F FULL FLIP RESULTS · p. 31, Figure 13 · p. 32
Loading 2609.31142v1…