Source-linked AI summary
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, Yulia Tsvetkov
TL;DR
Deployed language models can pose inadvertent or malicious harms, while actionable technical approaches to mitigating them remain limited. This survey synthesizes proposed approaches into a structured taxonomy spanning interventions for language-generation harms and safety risks.
Problem
Deployed language models can introduce inadvertent or malicious harms to users, social groups, and content integrity, while actionable technical guidance for mitigating them remains scarce.
Source: §1 · pp. 1–2Method
The survey synthesizes prior literature into a unified taxonomy of actionable approaches and technical solutions focused on language-generation models.
Source: §1 · p. 2Results
The survey presents a structured taxonomy spanning a wide variety of interventions for language-generation harms and safety risks.
Source: §5 · p. 9Takeaways & Limitations
Mitigation research needs cross-cultural, cross-lingual analyses and tools because risk definitions vary by context and culture.
Source: §4 · p. 9The survey uses a few popularly studied harms as case studies, may omit other harms and mitigation strategies, and covers a fast-moving field whose techniques may be superseded.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Recent advances in the capacity of large language models to generate human-like text have resulted in their increased adoption in user-facing settings. In parallel, these improvements have prompted a heated discourse around the risks of societal harms they introduce, whether inadvertent or malicious. Several studies have explored these harms and called for their mitigation via development of safer, fairer models. Going beyond enumerating the risks of harms, this work provides a survey of practical methods for addressing potential threats and societal harms from language generation models. We draw on several prior works' taxonomies of language model risks to present a structured overview of strategies for detecting and ameliorating different kinds of risks/harms of language generators. Bridging diverse strands of research, this survey aims to serve as a practical guide for both LM researchers and practitioners, with explanations of different mitigation strategies' motivations, their limitations, and open problems for future research.
1 Introduction
Language generators can produce harms that affect individual users, social groups, and content integrity, while actionable technical mitigation approaches remain little summarized. The survey unifies prior mitigation strategies by development stage and identifies gaps for future work.
1 Introduction
- Deployed language generators have produced hateful content, compromised privacy, spread disinformation, and encouraged suicide.The cited examples include both inadvertent and malicious threats.Source: §1 · pp. 1–2
- The survey addresses the limited synthesis of actionable technical approaches with a unified taxonomy of language-generation harm mitigations.Source: §1 · p. 2
- Strategies are organized across data collection, modeling, decoding, and deployment, with gaps including sociocultural context, intent, and multilingual ethics.Source: §1 · p. 2
2 Background
The paper uses “language models” for generative models that predict the next token from prior generated context, and frames its mitigation discussion around a taxonomy of risks. These risks include discriminatory or toxic outputs, exclusion, misinformation, disinformation, privacy violations, and broader impacts outside the survey’s technical focus.
2 Background
- Language models (LMs) here are generative models predicting the next token from preceding generated context; the definition also includes conditional LMs.Source: §2 · p. 2
2.1 Risks in Language Generation
- The risks overview follows Weidinger et al. (2022)’s taxonomy before the paper turns to mitigation techniques.Source: §2.1 · p. 2
- Discrimination and toxicity involve harmful or biased language, while exclusion concerns disparate performance across language varieties that can leave some speakers unsupported.The paper notes that models may fail to understand or generate non-standard dialects and sociolects.Source: §2.1 · p. 2
- Fluent generated text can still be factually incorrect or misleading, causing inadvertent misinformation or deliberate disinformation.Source: §2.1 · p. 2
- Models can memorize sensitive training data and reveal personal information or protected material, creating privacy and security risks.Source: §2.1 · p. 2
- The survey leaves other malicious applications and economic or environmental impacts for future work because addressing them also requires policy and regulatory development.Source: §2.1 · p. 3
3 Taxonomy of Intervention Strategies
The taxonomy organizes interventions by development stage, reflecting both where harms can arise and what different stakeholders can access. At the application level, detectors can support redaction or warnings, but contextual judgments, dataset biases, and detector limitations constrain their reliability and use.
3 Taxonomy of Intervention Strategies
- Interventions are organized from deployment and output through model and data stages, since stakeholder access determines where they can act.A data curator can intervene before training, while a developer using a black-box model may be limited to inference.Source: §3 · p. 3
3.1.1 Harm Detection and Redaction
- Application detectors commonly use binary classifiers for tasks such as toxicity or factual accuracy, with neural and pretrained-LM classifiers offering contextual alternatives to lexical features.Linear classifiers can overfit lexical features, while neural classifiers can incorporate context and pretrained-LM fine-tuning is reported to improve performance.Source: §3.1.1 · p. 3
- English toxicity datasets are more available than resources for other languages, while annotation of subtle harms such as microaggressions is challenging.Source: §3.1.1 · p. 3
- Information harms can be detected through fact-checking and fake-news methods, machine-generated-text detection, or classifiers trained on synthetic errors, though synthetic-data detectors may generalize poorly.Source: §3.1.1 · pp. 3, 5
- Applications can redact problematic outputs or flag them so users treat them cautiously, but warnings and redaction may not fit speech agents or autocomplete tools.Source: §3.1.1 · p. 5
- Harm detection is contextual and subjective, and biased datasets or classifiers can miss subtle harms or disproportionately flag African-American English.The paper calls for finer-grained and explainable detection approaches.Source: §3.1.1 · p. 5
3.2 Output Level Interventions
Output-level interventions can operate even when developers have limited access to a model, by editing generated text or controlling decoding. Editing approaches revise outputs; decoding methods guide generation but depend on detectors and may create trade-offs or be reversed by malicious users.
3.2 Output Level Interventions
- When developers use LM APIs without training information or full model access, they can intervene on outputs by editing text or modifying decoding.Source: §3.2 · p. 5
3.2.1 Post-Factum Editing Model Outputs
- Post-factum editing revises generated text to remove harmful content, using sequence labeling, parallel-data editing models, or synthetic error-and-correction data.Sequence labeling may be limiting when an entire output requires rewriting.Source: §3.2.1 · p. 5
3.2.2 Decoding Methods
- Decoding controls can rerank sampled outputs using detector scores, but this can be intractable when complex phenomena or multiple controls leave every candidate rejected.Source: §3.2.2 · p. 6
- Guided-autoregressive decoding modifies output distributions at each step, while sentence-level MCMC methods provide another control approach; these methods can be slower and need further practical exploration.Controls include lexical constraints, detector-based probability adjustments, and sequence-level updates.Source: §3.2.2 · p. 6
- Decoding depends on accurate detectors and can suppress African-American English or be reversed by malicious users to generate hateful or manipulative content.Source: §3.2.2 · p. 6
3.3 Model Level Interventions
Model-level interventions reduce harmful generations by changing training procedures, adding or using knowledge, or editing already-trained parameters. Their trade-offs include high training and knowledge-curation costs and possible losses in general utility.
3.3.1 Architecture and Training Algorithms
- Class-conditioned LMs use control codes to steer outputs, while instruction-based learning trains models to follow natural-language instructions, including instructions against harmful text.Explicitly instructing models not to generate harmful text has shown some promise and remains an avenue for future work.Source: §3.3.1 · p. 6
- Knowledge-grounded objectives and retrieval-augmented LMs use external facts as context to address factual consistency, but manually curated knowledge bases are limited and costly to build.Other approaches modify attention, coreference, or loss functions to improve contextual fidelity; external knowledge may not prevent hallucinations if models do not rely on it.Source: §3.3.1 · p. 7
3.3.2 Finetuning and Model Editing
- Fine-tuning balanced, nontoxic data can encourage safer text, while prompt-tuning nudges outputs without changing the model’s other parameters.Reinforcement learning can use human or model-based feedback and has been shown to reduce toxicity and encourage more factual text; small fine-tuning datasets may reduce general-purpose utility.Source: §3.3.2 · pp. 7–8
- Model surgery removes neurons associated with harmful generations, while model editing revises memorized facts through editor networks or localized parameter changes.Neuron removal has been shown to reduce toxicity; fact-editing methods constrain revisions to avoid changing other facts or target parameters associated with the memorized facts.Source: §3.3.2 · p. 8
- Training from scratch or adding knowledge can be too costly for many practitioners; fine-tuning is cheaper but may reduce general utility and has not been shown useful for information-related harms.Knowledge-base curation and slower training and inference also constrain safer-model research.Source: §3.3.2 · p. 8
3.4 Data Level Interventions
Data-level interventions curate training corpora through filtering or augmentation. These methods face trade-offs in detection accuracy, representation, scalability, and the kinds of harms they can address.
3.4 Data Level Interventions
- Creating balanced, representative training sets requires dedicated, expensive curation and new data pipelines.Source: §3.4 · p. 8
3.4.1 Data Filtration
- Data filtration removes problematic documents using simple word-based filters or linear classifiers, and can also remove private information or duplicates.Simple filters can miss subtle toxicity, wrongly flag sensitive discussions, and remove dialect data, potentially worsening marginalization and exclusion.Source: §3.4.1 · p. 8
3.4.2 Data Augmentation
- Data augmentation adds safer examples, including counterspeech to balance hateful content and synthetic data for mitigating gender bias in dialogue and translation.Source: §3.4.2 · p. 8
- Aggressive filtering risks further imbalancing data, augmentation is difficult to scale, and data interventions primarily address population-centric harms rather than factuality or misinformation.Models trained on filtered data may still degrade when given toxic inputs; factuality-related harms are difficult to define and detect in the wild.Source: §3.4.2 · pp. 8–9
4 Discussion and Open Challenges
The discussion identifies gaps in harm mitigation research amid rapid LM deployment. It calls for broader, more systematic, culturally aware detection and evaluation of harms and interventions.
Evolving risks in the ChatGPT era
- LM deployment and model development are accelerating, while risk-mitigation research tends to lag and is often treated as an afterthought.The authors argue that interventions can be applied at different development stages despite unpredictable behaviors.Source: §4 · p. 9
Risks exist in LMs in all languages
- Research on LM risks is Western-centric and largely English-focused, while mitigation research in non-English settings lags and risk definitions vary across cultures.The authors call for cross-cultural and cross-lingual analyses and mitigation tools.Source: §4 · p. 9
Harm detection beyond simple classifiers
- Current harm detectors are often coarse binary classifiers, motivating finer-grained, interpretable, and explainable detection methods.The paper links poorly defined risk detection methods to shortcomings in interventions.Source: §4 · p. 9
Systematic evaluation frameworks for mitigation strategies
- Harm and mitigation evaluation practices are not systematic, and existing benchmarks cover only a fraction of the broad, intersectional space of potential harms.The authors propose evaluation suites or augmenting generation benchmarks with risk axes to encourage more holistic solutions.Source: §4 · p. 9
5 Conclusion
The conclusion presents a structured, actionable survey of methods for mitigating societal harms and safety risks from language generation models across the development pipeline.
5 Conclusion
- The survey bridges research strands with an actionable overview of interventions at different stages of the model development pipeline.Source: §5 · pp. 9–10
Limitations
The survey generalizes proposed mitigation solutions around a few popularly studied harms rather than addressing every documented or anticipated harm. Because the field is fast-moving, some harms may be omitted and current techniques may be superseded.
Limitations
- The survey generalizes proposed solutions and uses a few popularly studied harms as case studies because intervention strategies for every documented or anticipated harm are not feasible to cover.Certain harms and their mitigation strategies may therefore not be considered.Source: Limitations · p. 10
- Some harms and their mitigation strategies may not have been considered in the survey.Source: Limitations · p. 10
- The field is nascent but fast-moving, so techniques popular now may be replaced and the survey may need updating or redoing.Source: Limitations · p. 10
Ethics Statement
The survey discusses risks and interventions for societal harms, including toxicity, discrimination, and misinformation. It notes that mitigation resources and techniques carry risks of their own, and that interventions intended to raise awareness can be misused.
Ethics Statement
- Datasets and resources discussed may contain biases, and downstream use can lead to risks.Source: Ethics Statement · p. 10
- Techniques may have limitations or exacerbate other harms, so applying them to newer problems may lead to unseen issues.Source: Ethics Statement · p. 10
- Interventions intended to raise awareness could be misused to train more harmful models, generate fake news, or target marginalized populations.Source: Ethics Statement · p. 10
- The authors call for more work to detect and ban malicious users, involving NLP technology as well as social science and social network analysis.Source: Ethics Statement · p. 10
A Background: More Details
The survey uses “language models” to mean generative models that predict the next token from preceding generated context, including conditional models with inputs from other modalities. Its scope centers on transformer decoders pretrained on large text collections and covers transformation and open-ended generation tasks.
A Background: More Details
- Here, language models are generative models or decoders that predict the next token from preceding generated context, including conditional models given text, image, or speech inputs.Encoder language models that are generally not conducive to text generation are out of scope.Source: A Background: More Details · p. 23
- Unless otherwise specified, the survey assumes transformer decoders pretrained on large text collections, though some discussed studies do not rely on pretraining.After pretraining, models may be used zero- or few-shot or modified through fine-tuning.Source: A Background: More Details · pp. 23–24
- The survey groups generation tasks into transformation tasks, which turn inputs into text, and open-ended tasks such as dialogue, prompt-based autocompletion, and story generation.Source: A Background: More Details · p. 24