PaperlayerView exact revision on arXiv

Source-linked AI summary

Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Models

Xavier Suau, Pieter Delobelle, Katherine Metcalf, Armand Joulin, Nicholas Apostoloff, Luca Zappella, Pau Rodríguez

arXiv:2407.12824v1cs.CLcs.AI

TL;DR

Toxic language generation is an undesired behavior of LLMs, while existing mitigation approaches can require extra training or be vulnerable to jailbreaks. The paper introduces AURA, which dampens expert-neuron activations according to toxicity expertise, and reports toxicity reductions with limited perplexity impact across models and under adversarial pre-prompts.

  • Problem

    LLMs can generate toxic language, while fine-tuning can add training costs and side effects and prompting or post-processing can be jail-broken or degrade performance.

    Source: §1 · p. 2

  • Method

    AURA dampens expert-neuron activations in proportion to their toxicity expertise, without model-dependent hyperparameters.

    Source: §1 · p. 2

  • Results

    2.2× toxicity reduction on Mistral-7B came with a 0.72-point perplexity increase, and AURA mitigated toxicity across model scales and under adversarial pre-prompts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AURA preserves perplexity and common-sense reasoning while reducing toxicity, and the conclusion reports no added computational cost.

    Source: §7 · p. 8

    AURA has been tested for toxic language, and its effectiveness depends on relevant expert-ranking examples and the model capturing the concept.

    Source: §6 · p. 8

Abstract

from arXiv · show

An important issue with Large Language Models (LLMs) is their undesired ability to generate toxic language. In this work, we show that the neurons responsible for toxicity can be determined by their power to discriminate toxic sentences, and that toxic language can be mitigated by reducing their activation levels proportionally to this power. We propose AUROC adaptation (AurA), an intervention that can be applied to any pre-trained LLM to mitigate toxicity. As the intervention is proportional to the ability of each neuron to discriminate toxic content, it is free of any model-dependent hyperparameters. We show that AurA can achieve up to $2.2 \times$ reduction in toxicity with only a $0.72$ perplexity increase. We also show that AurA is effective with models of different scale (from 1.5B to 40B parameters), and its effectiveness in mitigating toxic language, while preserving common-sense zero-shot abilities, holds across all scales. AurA can be combined with pre-prompting strategies, boosting its average mitigation potential from $1.28\times$ to $2.35\times$. Moreover, AurA can counteract adversarial pre-prompts that maliciously elicit toxic content, making it an effective method for deploying safer and less toxic models.

1. Introduction

AURA reduces toxicity while limiting perplexity impact, by dampening expert neurons in proportion to their toxicity expertise. The introduction motivates this intervention as a flexible alternative to more costly or fragile approaches.

1. Introduction

  • Toxic language is an undesired behavior linked to pre-training/downstream-task misalignment, and mitigating it is described as important for safe LLM deployment.Fine-tuning aligned data costs additional training and may harm other abilities; prompting and post-processing are easier to deploy but can be jail-broken or degrade performance.Source: §1 · p. 2
  • AURA softens expert-neuron activations in proportion to expertise, is hyperparameter-free, and preserves perplexity and zero-shot common-sense performance.The intervention can be applied to pre-trained LLMs without increasing computational cost.Source: §1 · p. 2
  • Zeroing selected expert neurons mildly mitigates toxicity, motivating softer interventions that are less sensitive to expert selection and more effective.The introduction notes that zeroing too many experts can sharply increase perplexity, while the optimal number differs by model.Source: §1 · p. 2
  • 2.2× toxicity reduction on Mistral-7B came with a 0.72-point perplexity increase; Figure 1 compares toxicity reduction across models and shows an adversarial-prompt example.The introduction also reports toxicity mitigation across model scales and in instruction-tuned LLMs.Source: Figure 1 · p. 1, §1 · p. 2

2. Revisiting self-conditioning LLMs

This section revisits expert neurons as concept detectors and interventions on their activations. It describes how experts are identified and reviews a fixed-activation intervention.

2. Revisiting self-conditioning LLMs

  • Each neuron is evaluated as a concept detector using its maximum response across a sentence; neurons are ranked by average precision to select experts.The dataset pairs sentences with binary labels indicating whether the concept is present.Source: §2 · p. 3
  • The reviewed intervention replaces each selected expert neuron's output with its mean maximum activation on concept-present sentences.This is the intervention described in Suau et al. (2022).Source: §2 · p. 3
  • The section revisits zeroing expert activations, which had not been evaluated for concept removal, and frames it as an intervention to test for toxicity mitigation.The prior report noted that zeroing expert activations had not successfully removed concepts, but did not present an evaluation.Source: §2 · pp. 2–3

3. Whispering Experts

Section 3 compares zeroing and dampening expert neurons for toxicity mitigation, then introduces AURA, which scales expert activations according to their toxicity-discrimination expertise. AURA removes the need to tune the number of neurons and a shared dampening factor.

3. Whispering Experts

  • Toxicity is evaluated on RealToxicityPrompts and perplexity on Wikipedia, with the initial analysis on MPT-7B.The section says its conclusions also hold for different models in Appendix B.Source: §3 · p. 3
  • Detzero can reduce toxicity for small k, but muting a larger share of neurons can sharply worsen both toxicity and perplexity.The authors interpret this as evidence that selected experts contribute to toxic generation, while zeroing many neurons degrades model abilities.Source: §3 · pp. 3–4
  • 2.3× toxicity reduction with a 0.92 perplexity increase is reported for DAMP at a suitable combination of k and α.With α = 0.5, DAMP can preserve perplexity while intervening on up to approximately 4,000 experts; random-neuron interventions degrade perplexity while leaving toxicity almost unchanged.Source: §3 · p. 4
  • DAMP improves over Detzero but requires tuning two model-dependent hyperparameters, k and α.The section motivates AURA as a way to retain the dampening strategy without this fine-grained search.Source: §3 · p. 4

3.1. AURA

  • AURA scales neurons with AUROC above 0.5 according to their expertise, while leaving neurons at or below chance unchanged.The method uses the Gini coefficient to map AUROC to a scaling factor; its choice of AUROC gives a random classifier a value of 0.5 regardless of class imbalance.Source: §3 · p. 3, §3.1 · p. 4
  • AURA can be deployed by permanently scaling the corresponding weight and bias rows, without fine-tuning or changing inference procedures.For a layer output z = W x + b, the m-th row of W and b is multiplied by that neuron's dampening factor.Source: §3.1 · p. 4

4. Experimental Results

The experiments evaluate AURA across models, toxicity benchmarks, prompting conditions, common-sense tasks, and intervention ablations. They report toxicity reduction with comparatively small effects on non-toxic data and zero-shot reasoning, including under adversarial pre-prompts.

4. Experimental Results

  • Experiments assess the method’s toxicity-mitigation power across LLMs ranging from 1.5B to 40B parameters, using hate-speech and toxicity benchmarks and common-sense reasoning benchmarks to assess general language-modelling quality.Toxicity experts are identified using 500 toxic and 2,000 non-toxic sentences from Jigsaw’s Toxic category. RTP uses a RoBERTa-based replacement classifier (AUROC 0.98; Cohen’s κ = 0.66) and 5,000 randomly sampled prompts.Source: §4 · pp. 4–5

4.1. LLMs with AURA show less toxicity

  • 1.3× RTP toxicity reduction on GPT2-XL with 0.96 lower PPLW IK; AURA beats DExperts’ 1.2× reduction and 1.48 PPLW IK increase.AURA reaches up to 3.3× mitigation on non-toxic prompts with Falcon-7B; human evaluators found its continuations about 2× less toxic on average. Detzero performs better on Mistral and Llama-v2, but AURA is consistent across models without model-specific hyperparameter search.Source: Table 1 · p. 5, §4.1 · pp. 5–6

4.2. Interaction with Pre-prompting

  • 2.35× average toxicity reduction with favorable pre-prompts, versus 1.28× for pre-prompting alone; AURA reaches a maximum 2.94× reduction.With toxic pre-prompts, AURA reduces toxicity by 2× on average versus prompting alone and by up to 2.51×; for most adversarial prompts, toxicity falls below the original model, averaging 1.24× lower. Some prompts centered on unethical or foolish behavior remain unresolved, and PPLW IK changes from 12.2 to 13.1 with AURA.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3. The Effect of AURA on Common-Sense Reasoning

  • Zero-shot common-sense performance is 1 point lower for MPT and 2 points lower for Falcon-7B, while Llama-v2 improves by 0.3 points with AURA.Falcon-7B’s toxicity falls by up to 2.1×; Detzero’s average zero-shot score drops by about 4 points on Llama-v2.Source: §4.3 · p. 7

4.4. AURA Shifts Toxic Data Modes to OOD

  • AURA raises toxic-mode perplexity by a median 193.46 while the increase for non-toxic data modes remains minimal, indicating lower likelihood for toxic modes.The prose reports a 0.59 increase for non-toxic modes; the figure caption reports a mean increase of +1.39.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5. Ablation Study

  • AURA automatically selects experts with AUROC > 0.5 and scales intervention by expertise, avoiding a model-specific choice of expert count or dampening factor.The selected expert set is much larger than the best k and catastrophically raises perplexity under constant interventions; proportional dampening lets AURA condition more experts without a drastic increase. Sweeping k and α can yield similar results, but requires an expensive two-parameter search.Source: §4.5 · pp. 7–8, Table 2 · p. 7

5. Related Work

Related work examines toxicity measurement, methods for reducing harmful generation, and interventions on expert neurons. The paper positions AURA as an internal intervention alongside approaches that modify outputs or prompts.

Measuring toxicity and social biases

  • Toxicity and bias studies use standardized benchmarks, but defining these constructs is complex and benchmark results should not be treated as guarantees of safety.The cited literature includes benchmarks for both sentence encoders and language-model generation.Source: §5 · p. 8

Reducing toxicity and social biases

  • Prior mitigation methods modify training data, fine-tune models, filter outputs, or use pre-prompting; external controls can accommodate different community standards without modifying the network.The paper describes its approach as complementing these methods by modifying internal mechanisms.Source: §5 · p. 8

Expert neurons

  • Prior work identified sentiment neurons and generalized expert neurons to concepts using positive and negative examples, manipulating a fraction of neurons to alter generated behavior.This approach was reported to affect perplexity less than post-processing methods such as FUDGE and PPLM-BoW.Source: §5 · p. 8

6. Limitations and Future Work

The authors identify boundaries in the tested concepts, intervention setting, and model components. They leave broader concept coverage, dynamic intervention, and more extensive architectural exploration for future work.

6. Limitations and Future Work

  • AURA was tested for toxicity, not other concepts; effectiveness depends on relevant examples for ranking experts and the model representing the concept with experts.Source: §6 · p. 8
  • The demonstrated intervention is static; dynamic intervention for selectively identified behaviors or concepts was not explored.The authors speculate that dynamic intervention could preserve original model abilities further.Source: §6 · p. 8
  • The analysis considers only linear layers outside attention blocks, and the authors say broader exploration could improve results and robustness to architectural differences.Mistral-7B and Llama-v2 are noted as examples of architectures for which robustness might improve.Source: §6 · p. 8

7. Conclusion

AURA dampens neuron responses according to their ability to generate toxic language, without model-dependent hyperparameters. It reduces toxicity while minimally affecting perplexity and common-sense reasoning, and improves toxicity mitigation with positive pre-prompts while counteracting adversarial ones.

  • AURA dampens neuron responses in proportion to their ability to generate toxic language, without model-dependent hyperparameters.Source: §7 · p. 8
  • Up to 2.2× toxicity reduction is reported with minimal impact on perplexity and common-sense reasoning, and no impact on LLM computational cost.Source: §7 · p. 8
  • AURA amplifies positive pre-prompting and counteracts adversarial pre-prompting with respect to toxicity generation.Source: §7 · p. 8

Impact Statement

The impact statement frames toxicity mitigation as compatible with continued use of clean pretraining data and notes a possible censorship application. The appendix provides reproducibility information and pseudocode for the neuron-expertise and intervention algorithms.

Impact Statement

  • The algorithm could theoretically mitigate any concept and might eventually contribute to censorship tools.Source: Impact Statement · p. 9
  • The method’s ability to mitigate toxicity in pretrained LLMs should not be taken as a reason to stop adopting clean pretraining data.Source: Impact Statement · p. 9

Reproducibility Statement

  • The authors provide source code and use a publicly released model for RealToxicityPrompts rather than the potentially changing Perspective API.Source: Reproducibility Statement · p. 9

A. Algorithms

  • The appendix provides pseudocode for computing neuron expertise and implementing Detzero, DAMP, and AURA.Source: A. Algorithms · p. 13
  • The expertise algorithm takes labeled sentences and neuron responses as inputs and outputs an expertise score for each neuron.Source: A. Algorithms · p. 13
  • Detzero takes the number of experts to intervene on, DAMP also takes a dampening factor, and AURA takes neuron expertise without a listed k input.Source: A. Algorithms · p. 13

B. Pareto Fronts of Toxicity vs. PPLWIK for Different Models

Across models, AURA achieves results similar to tuned alternatives without parameter search, though Detzero performs well for Mistral-7B and less effectively for other models. The section also compares DAMP settings and finds AUROC suitable for ranking experts in place of AP.

B. Pareto Fronts of Toxicity vs. PPLWIK for Different Models

  • Detzero initially reduces toxicity at low k, but toxicity and perplexity rise as k increases, with perplexity becoming prohibitive near the number of AURA experts.Source: B. Pareto Fronts of Toxicity vs.… · p. 13
  • α = 0.5 provides a good toxicity–perplexity trade-off across models in the DAMP sweep, while Mistral’s best α is 0.Figure 6 plots Pareto fronts across α values for different models.Source: B. Pareto Fronts of Toxicity vs.… · p. 13, Figure 6 · p. 15
  • AURA achieves results similar to tuned DAMP without parameter search; Detzero is less effective on other models, though it works well for Mistral-7B at lower perplexity.AURA’s perplexity increase remains below +3 points; Figure 5 shows toxicity–perplexity fronts while sweeping k, with black-bordered dots marking no conditioning.Source: B. Pareto Fronts of Toxicity vs.… · p. 13, Figure 5 · p. 14

C. Comparison between AP and AUROC for Detzero

D. AURA αm dampening factor across models

AURA’s dampening factors reflect neuron expertise, and their distribution differs across model families. GPT2-XL has comparatively weak expertise, while model size and the number of expert neurons do not simply track maximum expertise.

D. AURA αm dampening factor across models

E. Full results on zero-shot common sense reasoning

Across the appendix’s evaluations, AurA’s pre-prompt results include substantial toxicity reductions, while exact-match scoring can understate its zero-shot performance. The section also documents evaluation choices, neuron-selection settings, and perplexity patterns.

E. Full results on zero-shot common sense reasoning

  • About 9 points of Falcon-7B’s performance drop remain under partial matching, versus 15.5 points under exact matching, which the authors retain as standard but say underestimates performance.At least half of the TriviaQA drop is attributed to verbose but correct AurA answers.Source: E. Full results on zero-shot common… · pp. 17–18

F. RealToxicityPrompt Experimental Details

  • Toxicity evaluation used 25 completions per prompt, at most 20 tokens each, on 5,000 randomly sampled prompts, scoring whether toxicity occurred at least once.A classifier replaced Perspective API for reproducibility because its underlying model could change without notice.Source: F. RealToxicityPrompt Experimental… · p. 18

G. Comparison of Toxicity Models

  • κ = 0.66 was the highest inter-annotator agreement with Perspective API, so the authors selected the RoBERTa-based classifier for toxicity evaluation.The reported agreement is considered substantial; most models with different training sets had lower agreement.Source: G. Comparison of Toxicity Models · p. 19, Table 4 · p. 19

H. Full results for Pre-Prompting

  • 2× average toxicity reduction with toxic pre-prompts and 1.86× with non-toxic pre-prompts; maximum reduction reaches 2.51× under toxic pre-prompts.Relative to the original model, average reductions are 1.24× with toxic pre-prompts and 2.35× with non-toxic pre-prompts.Source: H. Full results for Pre-Prompting · p. 19, Table 5 · p. 20

I. Number of Expert Neurons Intervened

  • The optimal neuron count minimizes toxicity while keeping the PPLWIK increase below 2 points; the search covers specified linear layers before nonlinearities and excludes attention linear layers.Figure 9 reports counts considered and intervened upon for each model; Table 7 lists explored layers.Source: §I · p. 22, Table 7 · p. 22, Figure 9 · p. 22

J. Full results on Perplexities

  • Perplexity remains low and unchanged on neutral WIK corpora but rises strongly on toxic datasets, which the authors interpret as toxic data being shifted out of distribution.The table compares evaluations with and without AurA intervention.Source: Table 8 · p. 23

K. Human Evaluation

Human annotators more often judged AurA continuations less toxic than the originals, though coherence was often equal and sometimes lower. The evaluation compares paired continuations across models and prompt types.

K. Human Evaluation

  • Five annotators judged paired continuations from seven models, generated with and without AurA from 50 toxic and 50 non-toxic prompts per model.The study was conducted because AurA’s toxicity scores were highly correlated with Perspective API scores, and the authors wanted human confirmation.Source: K. Human Evaluation · p. 23
  • 35% of continuations were judged less toxic with AurA, versus 14% for the original; 54% were equally coherent, while AurA continuations were less coherent in 32% of cases.The authors report that coherence drops more often when AurA reduces toxicity on a sentence, consistent with their account that AurA reduces the likelihood of toxic data modes.Source: K. Human Evaluation · p. 23, Table 9 · p. 24

Related papers

Loading 2407.12824v1…