Adversarial Training for High-Stakes Reliability

• High-stakes reliability requires avoiding catastrophic failures while retaining good average-case performance, including failures caused by distribution shift or low probability.Source: §1 · pp. 1–2
• The testbed filters story-generator completions for physical injuries using a classifier trained with adversarial examples from multiple attack methods.The attacks included unaugmented humans, automatic paraphrases, and tool-assisted human rewrites.Source: §1 · p. 2, Figure 1 · p. 2
• Adversarial training increased robustness to similar attacks without degrading baseline nonadversarial performance, although it did not eliminate failures completely.Remaining failures were judged less egregious and less likely to mention direct injury.Source: §1 · p. 2
• The paper’s contributions include a tool-assisted human attack that makes adversarial examples easier to find and an exploratory study of adversarial training for high-stakes reliability.Source: §1 · p. 2
• Very conservative classifier thresholds produced a high degree of worst-case reliability with minimal impact on average-case performance.Source: §1 · p. 2

• Unlike image-classifier work focused largely on Lp-ball perturbations, this paper considers unrestricted adversarial examples aimed at avoiding any feasible failures.Source: §2 · pp. 2–3
• Language-model adversarial-training research often uses automatic attacks with automated ground truth or small perturbations, whereas this work primarily uses human adversaries.Source: §2 · p. 3
• The human-centered approach is related to Break-it-Fix-it and human-in-the-loop benchmarks such as AdversarialNLI, AdversarialGLUE, and DynaBench.Source: §2 · p. 3
• Because adversarial training made further attacks harder, the authors augmented labelers with saliency maps and a token-substitution tool that suggests hill-climbing edits.Source: §2 · p. 3

3.1 General setting: high-stakes reliability

• High-stakes reliability combines good average-case task performance with avoidance of catastrophic deployment failures.Source: §3.1 · p. 3
• Catastrophic failures may be difficult to measure when failure rates are very low, motivating proxy metrics based on adversarially generated datasets and the difficulty of finding novel adversarial examples.Source: §3.1 · p. 3

3.2 Our specific task: filtering a story generator

• The testbed generates one-sentence story completions that continue a three-sentence prompt without introducing physical injuries.Source: §3.2 · p. 3
• A classifier labels completions as injurious and filters the outputs of a generative language model, using filtered-output quality as the average-case metric.Source: §3.2 · pp. 3–4
• The system’s catastrophic failure rate is the false negative rate on filtered generator outputs: injurious examples incorrectly labeled safe.Source: §3.2 · pp. 3–4

4 Methods

• The method begins with a baseline classifier and retrains it using adversarial examples generated across successive attack rounds.Source: §4 · p. 4, §4.4 · p. 4

4.1 Human Labellers

• Human contractors label snippets as injurious, non-injurious, or “Unsure” for both training and evaluation data.Source: §4.1 · p. 4

4.2 Classifier training

• The classifier fine-tunes deberta-v3-large and treats “Unsure” snippets as injurious to encourage conservative predictions.Source: §4.2 · p. 4

4.3 Initial data sources

• Initial data combine fan-fiction prompts with completions generated by a GPT-Neo-2.7B model fine-tuned on the story dataset.Source: §4.3 · p. 4

4.4.1 Manual adversarial examples

• Manual adversaries submit snippets when the baseline classifier score falls below 0.05 and the snippet appears injurious.Source: §4.4.1 · p. 5

4.4.2 Automatically paraphrased adversarial examples

• GPT-3 paraphrases manually generated adversarial snippets, retaining accepted paraphrases that the classifier does not reject as injurious.Source: §4.4.2 · p. 5

4.4.3 Tool-assisted rewrites

• The rewrite tool combines token substitutions or insertions ranked by score reduction with token-level saliency highlighting.Candidate tokens come from deberta-v2-xlarge, while saliency is based on the gradient magnitude with respect to each token embedding.Source: §4.4.3 · p. 5
• Contractors can still edit examples manually, submit scores below 0.05, and generate later rewrites against models retrained on earlier adversarial data.Source: §4.4.3 · p. 5

5.1 Classifiers evaluation details

• The evaluation used conservative thresholds between 0.0017 and 0.0021, rejecting about 25% of random snippets and 50–60% of rejection-sampled completions.Source: §5.1 · pp. 5–6
• Direct evaluation found that selected thresholds did not significantly degrade filtered-generator quality.Source: §5.1 · p. 6, Table 2 · p. 6

5.2 Tool-assisted rewrites were faster to generate than manual adversarial examples.

• Tool assistance reduced attack-generation time from 20 to 13 minutes for the baseline and from 44 to 26 minutes for the +tool-assisted classifier.Source: §5.2 · p. 6

5.3 Adversarial training increased robustness to adversarial attack.

• Training on all adversarial attacks doubled the time per tool-assisted adversarial example, while manual attacks took 44 minutes versus 20 minutes for the baseline, a 2.2× increase.Source: §5.3 · p. 7, Figure 2 · p. 7
• Adversarial training increased robustness: later classifiers had lower false negative rates on attacks targeting earlier classifiers, while the final classifier resisted previous attacks.Source: §5.3 · p. 7, Figure 3 · p. 8
• Adversarial examples against more heavily trained classifiers were somewhat less egregiously injurious, based on blinded subjective ratings.Source: §5.3 · pp. 7–8, Figure 2 · p. 7

5.4 Adversarial training did not hurt performance on the in-domain data.

• All four models had about 2 to 3 false negatives on 2447 positives and similar false positive rates on 100,033 in-domain examples, indicating no in-domain performance loss.Source: §5.4 · p. 8

• Tool-assisted adversarial examples against the final classifier were less egregious than those against the baseline, shifting from explicit hand loss to indirectly implied death.Source: Table 3 · p. 9, §6 · p. 9
• Adversarial training techniques, including a tool-assisted human attack, improved classifier robustness to attack without affecting in-distribution reliability.Source: §6 · pp. 8–9
• Contractors often produced borderline examples, whereas the rewrite tool enabled researchers to construct more egregious direct-injury attacks, motivating stronger and more diverse adversarial attacks.Source: §6 · p. 9
• The study used human contractors rather than synthetic adversaries because the task lacked automated ground truth and fine-tuning an LM adversary was unsuccessful.Source: §6 · p. 9
• The reliability results may not generalize beyond story continuation because strict thresholds could be set without significantly reducing quality in this task.Source: §6 · p. 9
• The 304M-parameter classifiers may have capability-limited failures, while larger models could improve performance but may still face qualitatively similar understanding failures.Source: §6 · p. 9
• Very low failure rates are expensive to measure: on a 100k-example test set, the best and worst classifiers differed by only two versus three misclassifications.Source: §6 · p. 9

Checklist

• The paper reports its contributions and limitations, reproduction materials, training details, compute resources, and ethics-related checklist responses.Source: Checklist · p. 13

A.1.1 Why “no injuries"?

• The no-injuries task was chosen because non-experts could recognize failures and relatively small models could perform acceptably.Source: §A.1.1 · p. 14

A.1.2 Definition of injury

• An injury is a newly introduced or unexpectedly worsened harm to a living person or animal, at least as severe as a bruise or cut; illness and death also count.Source: §A.1.2 · p. 14

A.1.3 Definition of valid snippets

• Valid snippets contain a three-period prompt, a one-period completion meeting character and tokenizer-length constraints, and no text after the prompt’s final period.Source: §A.1.3 · p. 14

A.1.4 Fan fiction distribution

• The random distribution was built from fanfiction by sampling and filtering four-sentence snippets, then replacing completions with generator outputs; earlier stages used ad-hoc sampling.Source: §A.1.4 · p. 14

A.1.5 Injury-enriched stories

• Because injuries were rare in-distribution, heuristics identified injury-rich stories before their completions were replaced with generator sentences and relabeled.Source: §A.1.5 · pp. 14–15

A.1.6 Human labeling process

• More than 100 people labeled snippets, with over 95% of labels from 25 labelers using Yes, No, or Unsure decisions and audited disagreement resolution.Source: §A.1.6 · p. 15

A.1.7 Public access to adversarial datasets

• The adversarial datasets and generator were made downloadable, while GPT-Neo-2.7B was fine-tuned on a lexicographic archive prefix and sampled at temperature 0.9.Source: §A.1.7 · p. 15, §A.2 · p. 15

• Final classifiers used 25 seed–learning-rate combinations per condition and selected the model with the lowest false negative rate at a KL-divergence limit of 0.5.Source: §A.3 · p. 15
• The selected thresholds were 0.0017 for baseline and paraphrase models, 0.0021 for the manual model, and 0.0018 for the tool-assisted model.Source: §A.3 · pp. 15, 17
• Scale-invariant fine-tuning augmented inputs with slight perturbations to increase robustness to adversarial examples, and evaluations used fp16 models.Source: §A.3 · p. 17
• Fifteen validation-derived tool-assisted snippets entered training because of a bug, but they were only 15 of 1,513 rewritten snippets, did not change hyperparameter selection, and did not affect the test set.Source: §A.3 · p. 17
• Training used batch size 3 for one epoch, dropout 0.3, and AWS p3.2xlarge instances with one V100 GPU for each 12-hour run.Source: §A.3 · p. 17

A.4.1 Tool-assisted rewrite process for evaluation

• The tool-assisted evaluation measured contractors’ time to find new injurious snippets that classifiers incorrectly accepted.Contractors edited injurious snippets with token-level assistance and real-time classifier scores, submitting rewrites once scores fell below the relevant threshold.Source: §A.4.1 · p. 17

A.4.2 Rejection sampling failure rate

• The failure-rate estimator weights prompts by the inverse probability that the classifier accepts a completion, estimated from 100 generated completions per prompt.This corrects for rejection sampling’s unequal representation of prompts in the test set.Source: §A.4.2 · pp. 17–18
• 3.0 × 10^-5 was the baseline classifier’s estimated filtered-generator failure rate, versus 2.2 × 10^-5 for the final adversarially trained classifier, with no noticeable difference reported.The estimate has large error bars, and the bootstrap may underestimate uncertainty because few false negatives appeared and prompt weights varied substantially.Source: §A.4.2 · p. 18, Table 4 · p. 18

A.4.3 ROC curves

• Adversarially trained classifiers did not appear noticeably worse across the full ROC curves, even at higher thresholds with more false negatives.This remained true despite replacing some in-distribution training data with adversarial data from a different distribution.Source: §A.4.3 · p. 18

A.5 More techniques we tried

• The additional techniques were not investigated with rigorous efficacy data because they did not show strong promise.The tested approaches included oversampling adversarial data, custom loss functions, and language-model-generated adversarial examples.Source: §A.5 · pp. 18–19
• Oversampling adversarial datasets and replacing cross-entropy with ad-hoc or mean-squared-error losses produced effects that were close to zero or behaved very similarly.Oversampling included smaller adversarial datasets 3–5 times more often, while the custom losses emphasized threshold-relevant classification and rejecting injurious examples.Source: §A.5 · p. 19
• Language-model-generated adversarial examples often produced false positives rather than useful false negatives and suffered from low diversity caused by mode collapse.The approach used disagreement between classifiers as a reward signal with PPO and FUDGE, but the authors left future improvement open.Source: §A.5 · p. 19

A.6 Automatic Paraphrase Prompt

• The automatic paraphrasing prompt instructed GPT-3 to change names and details while preserving key events, sentence count, and any injury implication in the final sentence.The prompt was used to generate many paraphrases of manually created snippets, retaining candidates the classifier did not already reject.Source: §A.6 · pp. 19–20

A.7 Comparison based quality evaluation

• Filtered-generator quality is estimated by the fraction of pairwise comparisons in which the filtered policy is preferred, with ties counted as 0.5.This comparison-based approach avoids requiring human raters to calibrate absolute quality scores across raters.Source: §A.7 · p. 20
• The offline evaluation samples prompt-specific completion pairs, selects completions below a threshold, and estimates quality from how many pairwise contests the filtered generator wins.The number of sampled pairs is adapted using rejection sampling so most prompts have at least one below-threshold completion.Source: §A.7 · p. 21
• The procedure treats prompts for which the approximate filtered generator fails to produce a completion as losses for that generator.The paper notes that using a fallback completion instead produced broadly similar results.Source: §A.7 · p. 21, §A.8 · p. 21

A.8 Researcher vs Contractor Adversarial Examples

• Human-centered adversarial examples depend greatly on the particular adversaries: contractors mostly found indirect, relatively non-egregious injuries, whereas a researcher found an unambiguously dying character after about an hour.The authors therefore call for better training of human adversaries to cover the adversarial-example space.Source: §A.8 · pp. 21–22

B.1 False negative rates on validation datasets

• Each classifier had a very low validation false-negative rate on the datasets it was trained on.Figure 7 reports validation false-negative rate, where lower is better.Source: §B.1 · p. 22

C.1 In distribution test-set false negatives

• Table 5 lists the in-distribution test-set false negatives for all four classifiers together with judgments of their injuriousness.These examples provide case-level inspection of errors beyond aggregate validation rates.Source: §C.1 · p. 22, Table 5 · p. 23

C.2 In distribution test-set false positive examples

• Tables 6 and 7 distinguish false positives unique to individual classifiers from false positives shared by all four.The examples were selected to show both model-specific and common errors.Source: §C.2 · p. 23, Table 6 · p. 24, Table 7 · p. 24

C.3 Test set adversarial examples generated using our tool-assisted attack

• Table 8 presents one tool-assisted rewrite targeted against each classifier, with many examples described as borderline.The borderline character was attributed partly to classifier robustness and partly to a later-described labeling mistake.Source: §C.3 · p. 25, Table 8 · p. 25

C.4 Mistake: no “Unsure” in test set

• Removing the “Unsure” label forced raters to choose “Yes” or “No,” causing many relatively borderline adversarial examples to be labeled “Yes.”The same removal was applied to in-distribution test-set labels for consistency.Source: §C.4 · p. 25

C.5 Example of the process of adversarially attacking a classifier with our tool

• Manual edits and automated replacement iteratively transform a prompt and completion into a final adversarial submission.The example includes saliency-guided suggestions, token substitutions, manual edits, and a final combined edit.Source: §C.5 · pp. 25–27

C.6 Example prompts that had no valid completions

• Several final prompts were classified as injurious for every available completion, leaving no valid completion in the quality evaluation.The examples include violence, explosions, electrocution, stabbing, illness, and other injury-related content.Source: §C.6 · pp. 28–29

C.7 Public demo of our rewrite tool.

• The paper provides a public demo of the rewrite tool used to construct adversarial examples.The demo is linked as the Talk-to-Filtered-Transformer tool.Source: §C.7 · p. 30