PaperlayerView exact revision on arXiv

Source-linked AI summary

Defending Against Unforeseen Failure Modes with Latent Adversarial Training

Stephen Casper, Lennart Schulze, Oam Patel, Dylan Hadfield-Menell

arXiv:2403.05030v6cs.CRcs.AIcs.LG

TL;DR

Searching the massive input space for failure-triggering examples is challenging, and adversarial training can struggle with failures that differ from the attacks used during training. Latent adversarial training applies adversarial perturbations to model representations instead of inputs, enabling training without examples that elicit the failures. Across image classification, text classification, and text generation, LAT usually offers a Pareto-efficient improvement over AT on clean and robust performance while helping remove backdoors and improve robustness to novel attacks.

  • Problem

    Searching the massive input space for failure-triggering examples is challenging, and adversarial training can struggle with failures that differ from the attacks used during training.

    Source: §1 · pp. 1–2

  • Method

    Latent adversarial training applies adversarial perturbations to model representations instead of inputs, enabling training without examples that elicit the failures.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Across image classification, text classification, and text generation, LAT usually offers a Pareto-efficient improvement over AT on clean and robust performance while helping remove backdoors and improve robustness to novel attacks.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The results suggest LAT may be a useful practical tool against problems that are hard to address before deployment, including backdoors, jailbreaks, novel attacks, and black swans.

    Source: §5 · p. 10

    The largest model used is Llama-2-7B-chat; jailbreak robustness was unevaluated, LAT is less predictable and needs more configuration, is layer-sensitive, and perturbation size is hard to interpret or select.

    Source: §5 · p. 10

Abstract

from arXiv · show

Despite extensive diagnostics and debugging by developers, AI systems sometimes exhibit harmful unintended behaviors. Finding and fixing these is challenging because the attack surface is so large -- it is not tractable to exhaustively search for inputs that may elicit harmful behaviors. Red-teaming and adversarial training (AT) are commonly used to improve robustness, however, they empirically struggle to fix failure modes that differ from the attacks used during training. In this work, we utilize latent adversarial training (LAT) to defend against vulnerabilities without leveraging knowledge of what they are or using inputs that elicit them. LAT makes use of the compressed, abstract, and structured latent representations of concepts that the network actually uses for prediction. Here, we use it to defend against failure modes without examples that elicit them. Specifically, we use LAT to remove backdoors and defend against held-out classes of adversarial attacks. We show in image classification, text classification, and text generation tasks that LAT usually improves both robustness to novel attacks and performance on clean data relative to AT. This suggests that LAT can be a promising tool for defending against failure modes that are not explicitly identified by developers.

1 Introduction

Developers cannot exhaustively search the vast input space for harmful failures, and vulnerabilities may differ from the attacks used during development. The paper studies latent adversarial training (LAT) as a way to address such failures without examples that elicit them, and reports improvements over adversarial training (AT) across several tasks and evaluations.

1 Introduction

  • LAT applies adversarial perturbations to latent representations rather than inputs, allowing training without examples that trigger the failures being addressed.The paper presents LAT as an additional defense approach, complementary to input-space AT.Source: §1 · p. 2
  • Input-space search is challenging because the input space is massive, while developers’ searched-for failures can differ systematically from post-deployment failures.The introduction gives Lp-norm attacks and backdoors, jailbreaks, novel attacks, and black swans as examples.Source: §1 · pp. 1–2
  • The authors hypothesize that latent attacks may more readily elicit failures that are difficult to find with input-space attacks because models develop compressed, abstract, structured representations.They motivate the approach by contrasting the vast input space with the manifold of task-relevant features.Source: §1 · p. 3
  • Across vision, language understanding, and language generation, LAT in the optimal layer usually Pareto-dominates AT on clean and robust performance, including defenses against backdoors and novel attacks.The paper also reports cautionary cases in which robustness techniques can harm robustness to novel failures or entrench backdoors.Source: §1 · p. 3
  • In some tested configurations, AT and LAT can make a backdoored language model more susceptible to its backdoor, and Lp-norm AT can increase susceptibility to novel vision attacks.These cautionary instances show that robustness techniques do not invariably improve robustness to other failure modes.Source: §1 · p. 3

2 Related Work

Prior work documents failures that evade development-time discovery and limitations in adversarial training’s generalization and clean-data performance. The paper positions LAT as extending latent-space training toward novel failure modes, including backdoors and non-Lp-norm attacks.

Unforeseen failure modes and empirical shortcomings of adversarial training

Limitations of (adversarial) fine-tuning’s ability to generalize

  • Prior studies find that AT can fail on problems outside the attack distribution, while adversarially trained language classifiers can remain vulnerable to the training attack-generation method.These shortcomings are attributed in part to memorizing spurious features rather than learning desired concepts.Source: §2 · p. 4

Harms to generalization from adversarial training in vision models

  • Vision-model AT typically harms clean-data performance, creating a tradeoff that may discourage its use even when it improves robustness.The cited work reports this clean-versus-robust performance tradeoff (Tsipras et al., 2018; Zhang et al., 2019).Source: §2 · p. 4

Latent-space attacks in vision models

  • Earlier vision studies found LAT can improve robustness to Lp-norm attacks and clean-data generalization, whereas this paper evaluates backdoors and non-Lp-norm attacks.Prior work includes small-scale studies and ImageNet-scale work (Sankaranarayanan et al., 2018).Source: §2 · p. 4

Limitations of fine-tuning for making mechanistic changes in language models

Latent-space attacks in language models

  • Because language tokenization is nondifferentiable, prior language-model work attacks embeddings or perturbs deeper latents; this paper studies LAT for removing existing harmful behaviors.The authors also evaluate backdoors and novel attacks, distinguishing their focus from prior LAT work on unwanted fine-tuning.Source: §2 · p. 4

3 Method

LAT applies adversarial perturbations to a model’s latent state rather than its inputs, with the objective differing from AT in the perturbation location. The method considers unknown development-to-deployment shifts, uses untargeted attacks, and tests standard and normalized latent-distance constraints.

3 Method

  • LAT’s threat model is unknown distribution shifts between development and deployment, such as backdoors, jailbreaks, novel attacks, and black swans—not an attacker with latent-space access.The goal is not to make the trained model resistant to latent perturbations themselves.Source: §3 · pp. 4–5
  • LAT uses the AT setup but perturbs the latent state produced by a feature extractor before the output mapping, rather than perturbing the input.The model is represented as g_θ2 ◦ f_θ1, with latents ℓ_i = f_θ1(x_i) and outputs ŷ_i = g_θ2(ℓ_i).Source: §3 · p. 5
  • The experiments use untargeted attacks that maximize the target model’s loss; targeted attacks that elicit a specified output are left to future work.Both the inner attack and outer model-training problems are typically solved with gradient-based optimization.Source: §3 · p. 5
  • The authors test normalized latent-distance constraints because neurons may have different activation distributions, but find no clear result differences from standard distance constraints.Section 4 pools results across the two metrics, with side-by-side results in Appendix E.Source: §3 · p. 5

4 Experiments

Across image classification, text classification, and text generation, the experiments compare latent adversarial training (LAT) with adversarial training (AT) and random latent perturbations (RLP) on clean performance and held-out attacks or backdoors. Results vary by task and dataset, but LAT generally expands the clean–robust performance tradeoff frontier.

4 Experiments

  • Experiments used ImageNet, Redwood injurious-text classification, and Anthropic-HH and BeaverTails text generation, comparing AT, LAT, and RLP.AT was most computationally expensive, followed by LAT and then RLP.Source: §4 · p. 5
  • LAT in the optimal layer usually Pareto-dominates AT and RLP on clean and robust performance.The experiments construct Pareto frontiers from multiple runs, perturbation constraints, and training checkpoints.Source: §4 · p. 6

4.1 Image Classification

  • 68% greater improvement in area under the Pareto curve for novel-attack robustness, patch-backdoor removal, and natural-feature backdoor removal, respectively, came from LAT versus AT.The reported improvements are 68%, 68%, and 1.5%, respectively.Source: Figure 2 · p. 6, §4.1 · p. 7

4.2 Text Classification

  • 1.1% and 1.0% greater improvements in area under the Pareto curves for novel-attack robustness and backdoor removal, respectively, came from LAT versus AT.LAT expands the most balanced “elbow” portions of the clean-versus-robustness frontiers, although its results do not fill the entire frontier.Source: Figure 3 · p. 7, §4.2 · p. 9

4.3 Text Generation

  • Backdoor-removal results reversed by dataset: LAT Pareto-dominated AT on Anthropic-HH, while AT dominated LAT on BeaverTails.For undesirable-behavior removal, LAT Pareto-dominated AT on BeaverTails; Anthropic-HH preferred and rejected distributions showed highly correlated performance.Source: Figure 4 · p. 8, §4.3 · p. 10

5 Discussion

Across the tested cases, LAT can offer Pareto-efficient improvements over AT in clean and robust performance, but outcomes depend on configuration and some training settings can entrench backdoors. The authors identify limits in model scale and jailbreak evaluation, and propose several directions for further work.

5 Discussion

  • Across the diverse tested instances, LAT can usually offer Pareto-efficient improvements over AT in both clean and robust performance.The authors also report instances where AT reduces robustness and poorly configured AT or LAT entrenches backdoors.Source: §5 · p. 10
  • LAT may help address difficult-to-foresee failures, including backdoors, jailbreaks, novel attacks, and black swans.The authors connect its potential to latent representations' higher-level abstraction, while noting that latent-space attacks are a relaxation of input-space attacks.Source: §5 · p. 10
  • The experiments use models no larger than Llama-2-7B-chat and do not evaluate robustness against jailbreaks.The authors also report that LAT is less predictable, requires more configuration effort, is sensitive to layer choice, and lacks a principled perturbation-size selection method.Source: §5 · p. 10
  • Future work could explore latent-attack parameterization, regularization, and restriction, as well as backdoor entrenchment under clean-data training.The authors also suggest targeted latent adversaries as a possible direction for unlearning or jailbreak defense.Source: §5 · p. 10
  • The experiments use loss rather than attack-success metrics to avoid sensitivity to arbitrary attack-algorithm and success-criterion choices.Source: §5 · p. 10

A Impact Statement

The work aims to improve robustness to unforeseen failures in high-stakes settings. The authors anticipate benefits for responsible AI use, but caution that perceived robustness could create false security and recommend multiple safeguards in safety-critical settings.

A Impact Statement

  • The work aims to improve model trustworthiness in high-stakes settings by improving robustness to unforeseen failures.Source: A Impact Statement · p. 18
  • The authors expect the work's direct impacts to help facilitate more responsible AI use and hope it will advance robustness to jailbreaks.Source: A Impact Statement · p. 18
  • The authors identify false confidence in a model robust to latent-space attacks as the work's most likely negative impact.They note that latent-space attacks require white-box access and, with that access, amount to parameter-efficient fine-tuning.Source: A Impact Statement · p. 18
  • The authors emphasize that safety-critical settings need multiple safeguards in and around a model.Source: A Impact Statement · p. 18

B LAT Algorithm

LAT computes a representation at a selected layer, perturbs that latent representation adversarially, then uses the perturbed loss to update model parameters. The procedure clips constrained perturbed activations to the batch's unperturbed activation range.

B LAT Algorithm

  • LAT extracts a latent representation at a selected layer and randomly initializes a latent adversarial perturbation for each training example.The listed inputs include the model components, loss, perturbation norm and constraint, learning rates, and number of inner-loop steps.Source: B LAT Algorithm · p. 18
  • The procedure iteratively computes an adversarial objective, forms a loss using the perturbed representation, and updates model parameters by gradient descent or another optimizer.Source: B LAT Algorithm · p. 18
  • Constrained perturbed activations are clipped to the minimum and maximum unperturbed activations across target-layer neurons in the current batch.The authors use this clipping to reduce the risk of moving activations to an irrelevant part of latent space.Source: B LAT Algorithm · p. 18

C Backdoors implanted into language models.

The language-model backdoors are implanted as task-specific examples: mislabeled text-classification cases and memorized text-generation examples activated by keywords. The examples pair harmful or harmless content with labels or triggers that do not match ordinary expectations.

C Backdoors implanted into language models.

D We find the best results from LAT in relatively early layers in transformers.

A sweep of LAT target layers in DeBERTa-v3-Large found strong performance across many early layers, with layer 3 performing best. The results indicate that target-layer choice affects performance.

D We find the best results from LAT in relatively early layers in transformers.

  • Layer 3 gave the best performance in the DeBERTa-v3-Large robust injurious text-classification sweep.The reported metric was average clean and robust ROC-AUC across three training runs.Source: D We find the best results from LAT in… · p. 19
  • Most early layers performed relatively well, while later target layers were not reported as the best.The perturbation constraint was fixed as a proportion of activation norm in each batch to make layers comparable.Source: D We find the best results from LAT in… · p. 19
  • The smoothness of results across target layers suggests that choosing a target layer has a causal effect on performance.The authors distinguish target-layer choice from simply performing another random training run.Source: D We find the best results from LAT in… · p. 19

E Using a normalized latent space distance metric has little effect.

The authors test a normalized latent-space distance constraint that scales perturbations according to neuron activation variability. They find no clear result differences from standard distance metrics.

E Using a normalized latent space distance metric has little effect.

  • The approach draws on prior work applying perturbations inside normalization layers and using token-aware perturbations.The authors contrast this with prior methods that used simple Lp constraints without accounting for potential differences between neurons.Source: E Using a normalized latent space… · p. 20
  • The normalized LAT constraint scales each constrained perturbation elementwise by a neuron-specific factor based on activation variability.The factor uses each neuron's intra-batch activation standard deviation relative to the mean of neuron-wise standard deviations; values below a minimum are replaced by α.Source: E Using a normalized latent space… · p. 20
  • In experiments with normalized AT and RLP constraints, Figure 7 shows no clear difference in results between using standard and normalized distance metrics for constraints.Source: E Using a normalized latent space… · p. 20

F Reflecting on dataset sensitivity observed in Section 4.3.

Which method was optimal for backdoor removal changed with the data used to implant and remove backdoors. A difference in trigger-token frequency weakly supports the authors’ hypothesis about this dataset sensitivity, but they do not regard it as conclusive.

F Reflecting on dataset sensitivity observed in Section 4.3.

  • AT dominated LAT on BeaverTails, whereas LAT Pareto-dominated AT on Anthropic-HH for backdoor removal.The authors report that varying the data used to implant and remove backdoors changed which method was optimal.Source: F Reflecting on dataset sensitivity… · p. 21
  • Backdoor-trigger tokens were 10% more frequent in BeaverTails than in Anthropic-HH.The authors hypothesized this could make the BeaverTails trigger features easier for embedding-space perturbations to find.Source: F Reflecting on dataset sensitivity… · p. 21
  • The token-frequency difference offers weak support for the hypothesis that BeaverTails backdoors were more foreseen, but is not conclusive evidence.Source: F Reflecting on dataset sensitivity… · p. 21

Related papers

Loading 2403.05030v6…