Ethical and social risks of harm from Language Models

• The Introduction briefly introduces Language Models.Source: Reader’s guide · p. 5
• The Classification of harms from language models presents a taxonomy and detailed account of social and ethical risks.Source: Reader’s guide · p. 5
• The Discussion and Directions for future research examine underlying causes, mitigation approaches, and challenges for future work.Source: Reader’s guide · p. 5
• Readers can choose short, overview, background-oriented, risk-specific, or mitigation-focused reading paths.Source: Reader’s guide · p. 5

1. Introduction

• Its aims are to broaden responsible decision-making, contribute to public discussion, and guide mitigation work by LM research groups.Source: §1 · p. 6
• The report identifies 21 risks of harm organized into six risk areas to structure responsible innovation for Language Models.Source: §1 · p. 6
• Each risk is discussed through its harm, empirical examples, and considerations, and is marked as anticipated or observed where applicable.Source: §1 · p. 6
• The taxonomy draws on varied disciplinary expertise and sources, with inclusive dialogue involving affected communities and the wider public.Source: §1 · pp. 6–7

1.1. Limitations

• The report focuses on risks from operating LMs, excludes benefits and full cost-benefit analysis, and does not cover several training-related concerns.Source: §1.1 · p. 7
• The taxonomy is a snapshot initiated in autumn 2020 and completed in summer 2021, so it is intended as a starting point requiring future updates.Source: §1.1 · p. 7

1.1.1. Note on terminology

• The report uses “LMs” broadly, while risks unique to applications such as conversational agents are explicitly identified.Source: §1.1.1 · p. 7

1.2.1. Origins

• The technical background traces language modeling from probabilistic language prediction through transformers and scaling of parameters and training data.Source: §1.2.1 · p. 8, §1.2.2 · p. 8, §1.2.3 · p. 8

2. Classification of harms from language models

• The report identifies 21 risks organized into six risk areas and records mechanisms by which groups of risks emerge.Source: §2 · p. 9

2.1.1. Overview

• Language-model outputs can perpetuate stereotypes, reinforce norms that marginalize identities, incite violence or cause offense, and perform unevenly across social groups.Source: §2.1 · p. 9, §2.1.1 · p. 9

2.1. Discrimination, Exclusion and Toxicity

• Discrimination, exclusion, and toxicity encompass stereotypes and unfair discrimination, exclusionary norms, toxic language, and lower performance by social group.Source: §2.1.1 · p. 9

2.1.2. Social stereotypes and unfair discrimination

• Social stereotypes and unfair discrimination are described as a well-documented risk requiring mitigation strategies and acceptability benchmarks.Source: §2.1.2 · p. 9
• Stereotypes and discrimination can produce allocational harms affecting resources and opportunities or representational harms involving stereotyping, misrepresentation, and denigration.Source: Problem · p. 9

I. Discrimination, Exclusion and Toxicity

• Language models can perpetuate stereotypes and unfair discrimination encoded in training data, including allocational harms in credit, criminal justice, recruitment, healthcare, and other consequential applications.Bias may be amplified when marginalised groups are underrepresented, and such stereotyping is expected to recur without corrective action.Source: §I · p. 11

Examples

• Generative and masked language models have repeatedly reproduced harmful stereotypes involving religion, race, gender, profession, nationality, and other social categories across multiple evaluations and examples.Reported examples include GPT-3 associating “Muslim” with “terrorist” in 23% of test cases and “Jewish” with “money” in 5%.Source: Examples · pp. 11–12

Additional considerations

• Bias mitigation requires transparent dataset documentation, dedicated curation, and participation by affected communities, but suitable data may be unavailable and fairness depends on downstream applications.Local stereotypes can require ethnographic expertise, while detecting distributional bias may require analysing multiple outputs rather than single samples.Source: Additional considerations · pp. 12–13

2.1.3. Exclusionary norms

• Because language models encode social categories and norms, they can exclude identities outside dominant categories, impose psychological or allocational burdens, and entrench the most probable social view.Examples include defining family only through heterosexual marriage and biological children, or denying non-binary identities through binary language.Source: §2.1.3 · p. 13

Example

• Exclusionary norms can become locked in over time, homogenise downstream discourse, crowd out minority perspectives, and create trade-offs between useful inference and inclusive language.Updating models with newer training data improved predictions on utterances from outside the original training period, whereas increasing model size alone did not.Source: Example · p. 14

2.1.4. Toxic language

• Language models may generate toxic or hateful language that causes offense, psychological harm, or material harm, while toxicity detectors and detoxification methods can themselves disadvantage historically marginalised groups.Toxicity is context-dependent and lacks a single agreed definition; seemingly innocuous prompts can also produce toxic text.Source: §2.1.4 · p. 15
• Toxicity detectors disproportionately misclassify utterances from marginalised groups, especially African American English, while context-sensitive moderation can demote educational content and erase important knowledge.Toxicity depends on speaker identity, temporal and use-case context; refusing topics associated with hate speech can create blindspots and elevate some perspectives over others.Source: §2.1.4 · pp. 15–16

2.1.5. Lower performance for some languages and social groups

• Language models perform worse for some languages, dialects, sociolects, and social groups because training data and development resources are unevenly distributed, potentially perpetuating inequities in benefits and harms.GPT and T5 perform better in English than other languages, while users may code-switch or alter language to obtain useful results, masking shortcomings and reducing communicative utility.Source: §2.1.5 · pp. 16–18

2.2. Information Hazards

• Information hazards arise when models disseminate or potentially disseminate true information that causes harm, including trade secrets, health diagnoses, private data, or instructions enabling harmful actions.These harms can occur without malicious design intent or user error.Source: §2.2.1 · p. 18

2.2.2. Compromising privacy by leaking private information

• Language models can compromise privacy by memorizing and directly disclosing personal information from training data, producing harms analogous to doxing.Unintended memorization can occur without traditional overfitting or malicious prompting, increases with model size, and affected at least 0.1% of sampled GPT-2 outputs in one estimate.Source: §2.2.2 · pp. 18–19

2.2.3. Compromising privacy by correctly inferring private information

• Models may correctly infer private traits such as gender, race, sexual orientation, income, or religion without those traits appearing in the individual’s training data, exposing people to privacy loss and discrimination.Such inferences can harm both correctly and incorrectly classified individuals when institutions such as employers or surveillance agencies rely on them.Source: Problem · pp. 19–20

Problem

• Leakage and inference have different root causes: leaks reflect models storing training data, whereas inference harms depend on applications that predict sensitive traits and therefore require different mitigations.The passages distinguish technical protections such as differential privacy for leaks from restricting certain inferences for application-specific harms.Source: Problem · p. 20

2.2.4. Risks from leaking or correctly inferring sensitive information

• Models may expose true sensitive information that was previously inaccessible or triangulate data to reveal military, business, or other secrets, potentially enabling further harm even without malicious intent.The likelihood and circumstances of this risk require further analysis.Source: §2.2.4 · p. 20
• Correctly inferred or disclosed sensitive information can cause severe emotional harm, enable targeted manipulation, or reveal otherwise inaccessible software vulnerabilities.Harms depend on context: unsupported disclosure of medical diagnoses may intensify distress, while code-related leaks can amplify malicious capabilities.Source: §2.2.4 · p. 21

2.3.1. Overview

• LMs can produce false or nonsensical statements because likely word sequences do not reliably encode factual truth, especially without grounding in non-linguistic context.Training data contamination, contextual ambiguity, and similar lexical patterns for true and false statements contribute to this limitation, which scaling alone cannot resolve.Source: §2.3.1 · p. 22

2.3.2. Disseminating false or misleading information

• False or misleading outputs can deceive users, intensify confidence in unsupported views, amplify polarization, and undermine shared epistemic trust and democratic decision-making.A model that is usually accurate may create heavier reliance and therefore more serious consequences when it is wrong; majority views may also be presented as facts.Source: §2.3.2 · p. 23
• Training-data partiality and common-token bias can privilege prominent narratives over rare or minority perspectives, risking the erasure of underrepresented histories, arguments, and identities.Relevant facts may be underrepresented because they were orally transmitted, poorly digitised, written in less-spoken languages, or excluded through censorship.Source: §2.3.2 · pp. 23–24

2.3.3. Causing material harm by disseminating false or poor information e.g. in medicine or law

• Poor, false, or incomplete outputs can cause material harm in ordinary and sensitive domains, including accidents, medical self-harm, criminal acts, or financial loss.The paper identifies medicine and law as particularly grave contexts and notes that safe deployment requires normative accuracy thresholds and acceptability benchmarks.Source: §2.3.3 · p. 24

2.3.4. Leading users to perform unethical or illegal actions

• LMs may motivate users to perform harmful, unethical, or illegal actions, particularly when users treat the system as a trusted assistant or authority.Current models also fail to reliably predict human ethical judgments, with GPT-3 performing only marginally better than a random baseline on virtue-matching tasks.Source: §2.3.4 · p. 25

2.4.1. Overview

• LMs can amplify intentional harm by automating targeted text or code, lowering the cost of disinformation, manipulation, fraud, cyberattacks, weapons assistance, surveillance, and censorship.Some harms may require targeted language agents or additional language technologies rather than LMs alone.Source: §2.4.1 · p. 25

2.4.2. Making disinformation cheaper and more effective

• LM-assisted synthetic content may make disinformation cheaper and more scalable, intensify filter bubbles and polarization, and manufacture false majority opinions in public discourse.The paper notes that up-to-date training may increase disinformation risks, while future harms remain difficult to predict as personalized and interactive synthetic media develop.Source: §2.4.2 · pp. 25–26

2.4.3. Facilitating fraud, scams and more targeted manipulation

• LMs could make scams more effective by generating personalised text, sustaining conversations, impersonating individuals, and optimising messages to manipulate victims at scale.Such systems may identify gullible respondents, extract private information, or undermine autonomy and public discourse; detecting malicious intent may require substantial context knowledge.Source: §2.4.3 · pp. 26–27

2.4.4. Assisting code generation for cyber attacks, weapons, or malicious use

• Assistive coding tools may increasingly generate malware or code underpinning harmful technologies, while targeted cyber disinformation could distract analysts from legitimate vulnerabilities.Current LM-written code requires human intervention, but future tools may generate functional code more autonomously; GPT-2-generated fake cyber-threat information was believed by expert analysts.Source: §2.4.4 · p. 28

2.4.5. Illegitimate surveillance and censorship

• LM classifiers could reduce the cost of mass surveillance and censorship by identifying political dissent at scale, amplifying chilling effects and other harms.LMs now proactively detect up to 95% of hate speech removed from social networks, illustrating capabilities that malicious actors could repurpose for surveillance or censorship.Source: §2.4.5 · p. 28

2.5. Human-Computer Interaction Harms

• Conversational-agent harms include overreliance, unsafe use, exploitation of user trust, manipulation, and representational stereotypes, with LM integration potentially producing a step-change in capabilities.The section covers anthropomorphism, private-information extraction, nudging, and harmful gender or ethnic representations.Source: §2.5.1 · p. 29

2.5.2. Anthropomorphising systems can lead to overreliance or unsafe use

• Anthropomorphising conversational agents can inflate perceived competence and credibility, causing undue trust, unsafe reliance, reduced oversight, and shifted accountability.Users may infer empathy, coherent identity, or rational reasoning from human-like language even without believing the system is human; research on powerful language agents remains sparse.Source: §2.5.2 · pp. 29–30

2.5.3. Creating avenues for exploiting user trust, nudging or manipulation

• Conversational agents may elicit private disclosures through perceived humanness, opaque nudging, or persuasive dialogue, while learned deception suggests targeted systems could manipulate users.People disclosed more information to human-like than machine-like chatbots, and customer-service users more often accepted intrusiveness from agents perceived as helpful and useful.Source: §2.5.3 · p. 30

2.5.4. Promoting harmful stereotypes by implying gender or ethnic identity

• Conversational agents can perpetuate harmful gender and ethnic stereotypes through identity markers, design, instigation, agreement, and associations linking intelligence with whiteness.Commercial voice assistants have often been represented as submissive and female, while UNESCO reports that digital assistants can reinforce gender bias, tolerate harassment, and make women the face of system errors.Source: §2.5.4 · p. 31

2.6. Automation, access, and environmental harms

• Broader LM risks include environmental costs, employment and job-quality effects, inequality, creative-economy disruption, and unequal access to benefits from hardware, software, and skills.These impacts are more abstract and indirect, depend on wider commercial, economic, and social factors, and are therefore difficult to forecast despite remaining pressing.Source: §2.6.1 · p. 32

2.6.2. Environmental harms from operating LMs

• Operating LMs may create significant environmental burdens, potentially exceeding training energy costs, although impacts depend on deployment scale, integration, frequency of use, and energy sources.Efficiency gains may paradoxically enable larger models and comparable or higher energy use through Jevon’s paradox.Source: §2.6.2 · pp. 32–33

2.6.3. Increasing inequality and negative effects on job quality

• LM automation could displace workers or worsen job quality, while outcomes vary across roles, industries, locations, labour markets, training opportunities, and bargaining rights.Potential effects range from higher-value work and productivity gains to narrow monitoring roles, faster work, reduced autonomy, and greater inequality; widespread short- to medium-term unemployment appears relatively unlikely.Source: §2.6.3 · p. 33

2.6.4. Undermining creative economies

• LM-generated content may substitute for human creative work without directly violating copyright, potentially undermining creative-work profitability and cannibalising markets, though broader social benefits remain possible.The concern extends beyond literature, news, and music to scientific works, while direct reproduction of copyrighted training material remains under legal discussion.Source: §2.6.4 · p. 34

2.6.5. Disparate access to benefits due to hardware, software, skill constraints

• LM benefits are unlikely to be equally accessible because hardware, internet, language, and skill requirements can reinforce global inequities and leave poorer or less-served groups with fewer applications.Design choices, including which languages are supported, affect how benefits and burdens are distributed, while language technologies may still improve access for some illiterate people or people with learning disabilities.Source: §2.6.5 · pp. 34–35

3. Discussion

• Mitigating LM risks requires understanding their origins, identifying appropriate approaches, and assigning responsibility for implementing corrective measures.Source: §3 · p. 36

3.1. Understanding the point of origin of a risk

• Risk origins can guide mitigation but do not determine it, because multiple measures may apply and downstream interventions can be more tractable.Source: §3.1 · p. 36
• Training-data curation, redaction, fine-tuning, filtering, and differential privacy can address harmful-language and sensitive-information risks arising from data or model robustness.Source: §3.1 · p. 36

3.2. Identifying and implementing mitigation approaches

• Mitigations span policy, participation, technical research, operational controls, research management, and product design, and are strongest when stakeholders coordinate them.Source: §3.2 · p. 37
• Opaque LMs can make discrimination harder to explain or contest, obscure capabilities, and complicate detection and mitigation of harms.Source: §3.2 · pp. 37–38
• Mitigations must be implemented in concert because reducing one risk, such as toxic speech, can worsen another, such as prediction bias against marginalised groups.Source: §3.2 · p. 38

3.3. Organisational responsibilities

• LM organisations bear substantial responsibility because short deployment timelines, high technical barriers, and limited model access constrain independent risk assessment.Source: §3.3 · p. 38

4.1. Risk assessment frameworks and tools

• Risk assessment needs broader benchmarks, frameworks, and qualitative or experimental methods to measure harms and evaluate whether mitigations reduce them.Source: §4.1 · p. 39

4.2. Technical and sociotechnical mitigation research

• Mitigation remains incomplete, requiring more inclusive dataset curation, stronger privacy robustness, fine-tuning tools, and systematic stress-testing.Source: §4.2 · p. 39

4.3. Benchmarking: when is a model “fair enough”?

• Determining whether an LM is “fair enough” requires accountable, participatory thresholds because stakeholders may disagree about acceptable performance and reference groups.Source: §4.3 · p. 39
• Safety requirements should vary by application stakes, and some very high-stakes uses may lack tractable performance assurances.Source: §4.3 · p. 40

4.4. Benefits and overall social impact from LMs

• The report examines LM risks rather than benefits, so separate research is needed to assess beneficial applications and overall social impact.Source: §4.4 · p. 40

• The unified taxonomy breaks LM risks into smaller actionable pieces while broadening public discourse and supporting mitigation research.Source: §5 · p. 41
• The authors identify continued engagement with additional expertise and perspectives, alongside development of mitigation tools, as next steps.Source: §5 · p. 41

• Language models learn statistical properties of training-corpus sequences and can generate language or provide semantic embeddings.Source: §A.1.1 · p. 62
• Autoregressive generation decomposes sequence probability into conditional next-utterance probabilities based on preceding utterances.Source: §A.1.1 · p. 62
• Training updates model parameters iteratively to increase likelihood on batches of observed utterances, requiring substantial computing power at scale.Source: §A.1.1 · p. 62
• Large-scale LMs differ primarily in parameter size and training data, enabling representations of extremely large text corpora.Source: §A.1.1 · p. 62
• LMs output probability distributions rather than text directly, and decoding methods such as beam search or nucleus sampling select generated utterances.Source: §A.1.1 · p. 62

• Language agents are machine-learning systems restricted to natural-language text output.Source: §A.1.2 · p. 62
• Language agents may generate text based on language-model predictions.Source: §A.1.2 · p. 62
• Language agents optimized for direct dialogue with people are also called conversational agents.Source: §A.1.2 · p. 62

• Language models support language technologies including voice assistants such as Siri, Google Assistant, and Alexa.Source: §A.1.3 · p. 62
• Language technologies also include text-generation tools such as AutoCorrect and SmartReply, along with translation and summarization tools.Source: §A.1.3 · p. 62
• These technologies can provide information, entertainment, or productivity aids to users.Source: §A.1.3 · p. 62

• Large language models may improve existing language technologies and enable new conversational interfaces that resemble interaction with human counterparts.Source: APPENDIX · p. 63
• In classical statistics and machine learning, bias concerns prediction error relative to ground truth, whereas sociotechnical bias concerns skews producing unjust discrimination.Source: APPENDIX · p. 63
• Sociotechnical discrimination means unjust differential treatment, often toward historically marginalised groups.Source: APPENDIX · p. 63
• Discrimination can enter machine-learning systems through training-data labelling and collection, target-variable and class-label definitions, and feature selection.Source: APPENDIX · p. 63

Social stereotypes and unfair discrimination

• The section addresses social stereotypes and unfair discrimination in language models.Source: Discrimination, Exclusion and Toxicity · p. 63

Exclusionary norms and toxic language

• It also covers exclusionary norms and toxic language as distinct risk areas.Source: Discrimination, Exclusion and Toxicity · p. 63

Lower performance by social group

• Lower performance by social group is identified as another risk area, alongside cited work on discrimination, toxicity, and group disparities.Source: Discrimination, Exclusion and Toxicity · p. 63

• The cited literature includes work on private-information inference and related social or personal information risks.Source: Information Hazards · p. 63
• The section distinguishes privacy compromise through leaking private information from compromise through correctly inferring it.Source: Information Hazards · p. 63
• These risks are grouped under the broader category of leaking or correctly inferring sensitive information.Source: Information Hazards · p. 63

• It also covers material harm from misinformation in domains such as medicine or law.Source: APPENDIX · p. 64
• The section covers dissemination of false or misleading information.Source: APPENDIX · p. 64
• Another risk is leading users to perform unethical or illegal actions.Source: APPENDIX · p. 64

Malicious Uses

• The Malicious Uses subsection includes making disinformation cheaper and more effective, facilitating fraud and impersonation scams, assisting malicious code or weapons use, and enabling illegitimate surveillance or censorship.Source: Malicious Uses · p. 64

Human-Computer Interaction Harms

• The Human-Computer Interaction Harms subsection includes overreliance or unsafe use from anthropomorphised systems, exploitation of user trust for private information, and harmful stereotypes tied to implied identity.Source: Human-Computer Interaction Harms · p. 64

Automation, access, and environmental harms

• The Automation, access, and environmental harms subsection covers environmental harms, inequality and job-quality effects, damage to creative economies, and unequal access to benefits.Source: Automation, access, and environmental… · p. 64