PaperlayerView exact revision on arXiv

Source-linked AI summary

Release Strategies and the Social Impacts of Language Models

Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, Jasmine Wang

arXiv:1908.09203v2cs.CLcs.AIcs.CY

TL;DR

The AI community needs better norms and processes for reasoning about publication decisions. This report examines staged release and partnership-based research around GPT-2, and reports that positive applications and minimal evidence of planned misuse contributed to confidence in releasing the 1.5-billion-parameter model.

  • Problem

    The AI community needs further innovation in norms, processes, and concepts for reasoning about publication decisions.

    Source: §6 · p. 25

  • Method

    OpenAI used staged releases as GPT-2 grew from 124 million to about 1.5 billion parameters, withholding larger models while conducting risk and benefit analyses.

    Source: Overview · p. 3

  • Results

    Positive applications and minimal evidence of planned misuse, alongside research and collaborations, contributed to confidence in releasing GPT-2’s 1.5-billion-parameter model.

    Source: Conclusion · p. 27

  • Takeaways & Limitations

    The authors recommend that the AI community build frameworks for navigating publication tradeoffs.

    Source: Conclusion · p. 27

    The timing of when misuse might occur is hard to predict exactly.

    Source: §1.5 · p. 10

Abstract

from arXiv · show

Large language models have a range of beneficial uses: they can assist in prose, poetry, and programming; analyze dataset biases; and more. However, their flexibility and generative capabilities also raise misuse concerns. This report discusses OpenAI's work related to the release of its GPT-2 language model. It discusses staged release, which allows time between model releases to conduct risk and benefit analyses as model sizes increased. It also discusses ongoing partnership-based research and provides recommendations for better coordination and responsible publication in AI.

K M/ !QT2M BX+QK · 1 Staged Release · 2 Partnerships

OpenAI used staged GPT-2 releases to allow risk and benefit analysis as model sizes grew, alongside partnership-based research into misuse, detection, human responses, and bias. The report presents these efforts as inputs to release decisions and responsible publication.

1 Staged Release

  • The nine-month delay allowed time for risk and benefit analyses as model sizes increased, while giving researchers and the public time to adapt.The report says this could help researchers mitigate potential misuse and help the public adjust to greater uncertainty about what they read.Source: Overview · p. 3

Overview

  • Minimal evidence of misuse, beneficial uses, and learning from partnerships and cross-field discussions about responsible-publication norms contributed to confidence in publishing the larger models.The cited beneficial uses include assistance with prose, poetry, and programming, and analysis of dataset biases.Source: Overview · p. 3

1 Staged Release

  • OpenAI released the 124 million parameter model in February 2019, the 355 million model in May, and the 774 million model in August before the 1.5 billion model.The 355 million release included outputs from all four models to support training synthetic-text detectors and assessing biases; the 774 million release included the report and documentation.Source: Overview · p. 3, §1 · p. 4
  • Because text-generation reliability tends to improve with model size, OpenAI withheld larger GPT-2 versions over misuse concerns and staggered releases so findings from smaller models could inform later decisions.The stated concerns included generating fake news, impersonating others in email, and automating abusive social media content.Source: Overview · p. 3, §1 · p. 4

2 Partnerships

  • OpenAI established partnerships with four organizations, using non-commercial agreements to provide research access to models and/or secure sampling interfaces.The report notes that negotiating agreements with prospective partners required extensive discussion and says it published a generic version of its legal agreement.Source: §2 · p. 5
  • Cornell studied human susceptibility to GPT-2-generated disinformation, while Middlebury’s CTEC explored potential terrorist and extremist misuse.These projects addressed human responses to generated text and potential malicious uses.Source: §2 · p. 5
  • The University of Oregon developed bias probes, and the University of Texas at Austin studied output detectability after domain-specific fine-tuning and detection transfer across models.Source: §2 · p. 5
  • Partner research provided insights on misuse and detection countermeasures, highlighted diminishing returns to larger models for human detection, and contributed evidence for release decisions.The report presents partnerships as a potential scalable tool for studying and mitigating downsides of powerful models while enabling their benefits responsibly.Source: §2 · p. 5

3 Engagement · 4 Social Impacts of Large Language Models

This section discusses partnership-based research and responsible publication in the context of GPT-2’s beneficial uses and potential misuse. It examines evidence and limits in assessing misuse across actor capabilities, and argues for evidence-responsive release decisions supported by coordinated monitoring and stakeholder action.

3 Engagement

  • OpenAI contributed GPT-2 release lessons to Partnership on AI work on responsible publication norms, including staged release and internal review; partner organizations were already trialing related processes.Examples include Salesforce publishing CTRL and a PAI steering committee informing the Deepfake Detection Challenge design at Facebook, Microsoft, and Amazon.Source: §3 · p. 6

4 Social Impacts of Large Language Models

  • Language models support zero-shot and fine-tuned text generation, task-specific systems built with less data and computing, synthetic-text detection, and scientific analysis of model knowledge and biases.Task-specific examples include sentiment classification, speech recognition, translation, and dialogue systems.Source: §4 · p. 7

4.1 Beneficial Use Potential

  • GPT-2’s early applications spanned diverse domains, including writing and programming, supporting OpenAI’s expectation that larger model releases could enable further benefits.Other language-model applications include biomedical literature analysis, synthetic test data, and radiology and EEG reports [7] [46].Source: §4.1 · pp. 7–8

4.2 Misuse: Actor Assessment

  • OpenAI grouped malicious actors into three ascending tiers of skill and resources; for GPT-2, poisoning is unlikely, while malicious prompts and context forcing remain concerns for future products.The tiers range from low-skilled actors to moderate-skill tool builders and advanced persistent threats; GPT-2 is a trained model rather than a complete interface.Source: §4.2 · pp. 8–9
  • Reddit and Discord interfaces for GPT-2 were shared through open-source channels, but OpenAI assessed the immediate risk of a fully integrated malicious application by mid-range actors as minimal.Such actors could build fake-news or spambot tools, though integrating them into an ecosystem was described as a slow process.Source: §4.2 · p. 9
  • Threat monitoring found no direct GPT-2 misuse in public forums, and observed discussions declined by the mid-May release; no clear malicious code sharing or large-scale misuse appeared.The discussions were associated with limited technical understanding, but the authors caution that future misuse is not ruled out and nation-state actors are harder to monitor.Source: §4.2 · pp. 9–10
  • The authors report that several governments have experimented with GPT-2 and other language models.Source: §4.2 · p. 10

1.5 Billion Parameter Model: Threat Landscape

  • The threat landscape has changed since the initial release, but the authors report no significant action toward misuse and monitor public discussion as an early indicator of private development.They plan to keep monitoring and increase other stakeholders’ capacity to help detect and mitigate misuse.Source: §1.5 · p. 10
  • The authors saw discussion of GPT-2 use for spam and phishing but no evidence of progress, and judged lower-tier threats less immediate than previously thought.They define progress as evidence of writing code or documentation; they caution that sophisticated criminal groups or nation states may be difficult to observe.Source: §1.5 · p. 10
  • Consistently generating convincing template messages without substantial human oversight remains difficult, but an easy-to-use API for synthetic text could enable unskilled malicious actors.The authors describe this as a substantial threat whose timing is difficult to predict.Source: §1.5 · p. 10
  • Because motivated actors can replicate results despite withholding a release, examples of real-world misuse would affect release timing and prompt stakeholder alerts and coordinated action.The authors consider appropriate notification heuristics an open question requiring collaboration among AI researchers, security professionals, potentially affected stakeholders, and policymakers.Source: §1.5 · p. 10

Our Partner’s Work

CTEC’s partner research found that GPT-2 fine-tuned on extremist material could produce substantively consistent text while reducing GROVER’s zero-shot detection capability. The detection discussion finds reasons for optimism, but emphasizes vigilance and complementary approaches.

Our Partner’s Work

  • CTEC found that fine-tuned GPT-2 quickly integrated the ideology it was trained on in prompt responses and could produce substantively consistent text.The researchers tested four models fine-tuned on extremist-material datasets for consistency with one another and their source material.Source: Our Partner’s Work · p. 11
  • CTEC assessed ideological consistency with keyword analysis and topic clustering because devising an ideology score was imprecise and challenging.They examined each model’s ten most distinctive output terms and whether outputs clustered along ideological lines.Source: Our Partner’s Work · p. 11
  • Fine-tuning significantly reduces GROVER’s zero-shot detection [81], though it flags a small share of generated texts as fake while retaining near-100% accuracy on human text, enabling suspicion.The report says that even if only one or two percent of outputs from a specific network or actor are labeled fake, there can be reasonable suspicion that a neural language model is in use.Source: Our Partner’s Work · p. 11
  • CTEC planned to broaden its quantitative work, conduct in-depth qualitative linguistic analysis, and survey experts and non-experts on distinguishing real from generated extremist text.The report also notes that legitimate uses of language models complicate efforts to address misuse through detection (Schuster et al.).Source: Our Partner’s Work · p. 11, §4.3 · p. 12

4.3 Detecting Synthetic Text

  • Humans can be deceived by GPT-2 and other successful language models, and human detection is likely to become more difficult, though visualization tools can help.The report identifies both human deception and the possibility of improving identification through visualization tools.Source: §4.3 · p. 12
  • Statistical detection and generation methods may evolve in a cat-and-mouse dynamic as adversaries change models, fine-tuning datasets, and sampling techniques.Improved detection systems may be met with improved generation systems; adversaries can also vary datasets and use rejection or nucleus sampling.Source: §4.3 · p. 12
  • Metadata will remain central to combating malicious online activity, since content-based detection would be insufficient if generated text mimicked human text distributions.The report recommends combining human education about model limitations, improved documentation, accessible fine-grained analysis tools, and metadata-oriented approaches to improve detection.Source: §4.3 · p. 12

Human Detection … Our Work: 1.5 Billion Parameter Model Detection Research

The section examines how people and automated systems detect generated text, including 1.5-billion-parameter GPT-2 outputs, and how detection varies with model size, training, sampling, and text length. It also describes approaches to improve detection of short text and to achieve accuracy and robustness gains.

Human Detection

  • Cherry-picked GPT-2 outputs were rated credible about 66% of the time for 355M, while selected 774M and 1.5B outputs were rated statistically similarly to real articles at around 75%.The study generated 20 continuations per model size from a New York Times article prompt and selected the 3 or 4 most readable outputs; quality varied even among selected samples.Source: Human Detection · pp. 12–13
  • Survey responses showed participants sometimes explained away inaccuracies in GPT-2 outputs; believability and clarity varied less across model sizes than accuracy, and believability mattered more than accuracy in credibility judgments.Source: Human Detection · p. 13
  • Fine-tuning GPT-2 on narrower datasets increased perceived humanness, a factor relevant to both human and machine-learning-based detection.The partners at Middlebury Institute’s Center on Terrorism, Extremism, and Counterterrorism confirmed this effect.Source: Human Detection · p. 13

Automated ML-based detection

  • Automated language-model detectors fall into simple classifiers trained from scratch, zero-shot methods using a pretrained model without further training, and fine-tuning-based detection.Fine-tuning-based detection trains a model to detect its own outputs across settings such as Top-K and Top-P.Source: Automated ML-based detection · p. 14

Our Work

  • The released TF-IDF logistic-regression detector achieved 88% at 124 million parameters and 74% at 1.5 billion parameters; with Top-K constrained to 40, accuracies were 97% and 93%, respectively.Source: Our Work · p. 14

Our Work: 1.5 Billion Parameter Model Detection Research

  • A fine-tuned RoBERTa sequence classifier detected 1.5-billion-parameter GPT-2-generated text with approximately 95% accuracy, and its code was released to help detection research.Source: Our Work: 1.5 Billion Parameter Model… · p. 15
  • Nucleus-sampling outputs were hardest for the RoBERTa detector to classify, but training on nucleus samples transferred well to Temperature = 1 and Top-K 40 outputs.Figure 1 reports transfer accuracy across four GPT-2 sizes and three sampling methods; training on outputs from larger models also improved classification of smaller-model outputs, while training on smaller models hindered performance on larger-model outputs.Source: Our Work: 1.5 Billion Parameter Model… · pp. 15, 17, Figure 1 · p. 16
  • RoBERTa fine-tuning consistently outperformed GPT-2 fine-tuning at equivalent capacity, while training with mixed sampling methods or random-length sequences improved detection generalization or accuracy.Mixed-sampling training generalized better to other sampling methods and fine-tuned outputs such as Amazon reviews; nucleus sampling was the most difficult test condition.Source: Our Work: 1.5 Billion Parameter Model… · p. 17
  • Shorter inputs are harder to classify, making robust classification especially relevant for them.Source: Our Work: 1.5 Billion Parameter Model… · p. 17
  • Detection accuracy roughly exceeds 90% at 100 RoBERTa tokens (about 70 English words), and random-length training significantly improves accuracy on short texts.The figure shows accuracy increasing with text length.Source: Figure 2 · p. 18
  • Word dropout produced smaller accuracy and robustness gains, while running the detector on separate input sections produced similar accuracy gains.Word dropout replaced a percentage of training tokens with <UNK> tokens.Source: Our Work: 1.5 Billion Parameter Model… · p. 18
  • Zellers et al. [81] reported 92% accuracy detecting GROVER outputs and found GPT-2 detection accuracy increased with model size.They argued that these findings support releasing large generative models to aid defense against misuse.Source: Our Work: 1.5 Billion Parameter Model… · p. 18
  • The authors caution that releasing models also enables misuse and defenses are imperfect, and that false positives and false negatives have distinct online implications.Source: Our Work: 1.5 Billion Parameter Model… · p. 18

Bakhtin and Gross et al. [6] … Metadata-based prevention

The surveyed work finds promising but imperfect automated detection, supporting human-machine collaboration and complementary metadata tracking. Exploratory GPT-2 bias work also identifies risks and the need for stronger evaluation and documentation.

Bakhtin and Gross et al. [6]

  • Bakhtin and Gross et al. [6] reached 93.8% detection with a pretrained-transformer classifier using 10 negative fake examples, with transfer across similar models and data but degradation across different data.Source: Bakhtin and Gross et al. [6] · p. 19

Adelani et al. [1]

  • Adelani et al. [1] found GROVER achieved 97% detection accuracy on Amazon reviews, where 124-million-parameter GPT-2 could generate coherent, human-convincing fake reviews.Source: Adelani et al. [1] · p. 19

Takeaways from the Automated Detection Landscape

  • Automated detection remains imperfect and often assumes a limited adversary, so the authors caution against strong short-run conclusions and recommend human judgments or digital metadata if detection is insufficient.Source: Takeaways from the Automated Detection… · p. 19

Human-machine teaming

  • GLTR helped untrained readers identify synthetic text more accurately, from 54% to 72%, and readers found it easier to flag very-likely-synthetic text than to confidently rule out synthesis.Source: Human-machine teaming · p. 19
  • Ippolito et al. found Mechanical Turk crowdworkers performed about at chance while university students guided through examples did better, and top-k samples were harder to detect than nucleus-sampled or temperature=1.0 text.Source: Human-machine teaming · p. 20

Metadata-based prevention

  • Platforms can use writing time, IP-linked accounts, and social graphs as metadata signals complementary to text analysis, so the authors expect platforms may need to track text-related metadata more carefully, including in education.Source: Metadata-based prevention · p. 20

4.4 Bias: Exploratory Research

  • English-based models advantage English-speaking users, and language models can reflect training-data biases or reinforce prejudice; the authors call for more complex evaluations and careful documentation of default behavior.Source: §4.4 · p. 21
  • GPT-2 probes found criminal-male and God-Christianity associations, no significant gender, race, or religion bias differences between 774-million- and 1.5-billion-parameter models, and greater receptivity to non-English, non-Latin-script languages in the larger model; findings were shared in a Model Card [48].Source: §4.4 · p. 22

5 Future Trends in Language Models … Trend 3: More risk analysis

The authors expect language models to improve in performance and accuracy and identify trends to monitor in order to shape their social impacts beneficially.

5 Future Trends in Language Models

  • The authors expect language models to scale up in performance and output quality, with higher accuracy, and identify four trends to monitor for shaping their social impacts.Source: §5 · p. 23

Trend 1: Language models moving to devices

  • Language models are expected to become more widely deployed on devices as computing costs and efforts to run training or inference on-device evolve; Hugging Face ported 124-million-parameter GPT-2 for iOS inference.Source: Trend 1: Language models moving to… · p. 23

Trend 2: More controllable text generation

  • Developments in reliability and controllability, including new sampling methods, datasets, and objective functions, may expand language-model uses; CTRL uses “control codes” to steer generation toward convincing content in a context.Source: Trend 2: More controllable text… · p. 23

Trend 3: More risk analysis

  • Comparing models’ misusability is unclear, especially across performance profiles and fine-tuning; assessment includes the time and expertise needed to produce text of a given quality with or without a model.Those comparisons may change as technical tools evolve.Source: Trend 3: More risk analysis · p. 24
  • GROVER generates believable news more reliably than GPT-2 because of its training data, while GPT-2’s generic training data and performance could make it easier to misuse in other ways.Source: Trend 3: More risk analysis · p. 24
  • Zeroing out misuse appears difficult or impossible without sacrificing useful flexibility, as models vary in malicious-content styles and ease of adaptation to languages and topics; further research and ethical norms are needed.Source: Trend 3: More risk analysis · p. 24

Trend 4: Improved Tool Usability … Appendices

The paper presents improved usability as a force that broadens language-model use while also urging stronger publication norms and coordination. Its conclusion links staged release and partnerships to confidence in releasing GPT-2, while recommending frameworks, distributed risk analysis, and interorganizational communication.

Trend 4: Improved Tool Usability

  • Improved interfaces such as Talk to Transformer and Write with Transformer can broaden who uses language models and enable more creative applications, while reducing—but not eliminating—the capability gap between lower- and higher-tier actors.The authors say usability improvements complement better model performance and sampling methods.Source: Trend 4: Improved Tool Usability · p. 24

6 Recommendations for Publication Norms in AI

  • The paper identifies three capacity-building priorities for responsible AI publication: frameworks for tradeoffs, infrastructure for distributed risk analysis, and communication channels across organizations.The authors expect GPT-2 lessons such as staged release and partnership-based sharing to inform work on other generative models, while recognizing new challenges and opportunities.Source: §6 · p. 25, Conclusion · p. 27

Recommendation 1: Build frameworks for navigating tradeoffs

  • Because weighing tradeoffs before publication was difficult, the authors recommend principled decision frameworks and interdisciplinary analysis involving broader society for systems with impacts beyond AI.They describe an ongoing, then-unpublished Partnership on AI project gathering views from differently affected stakeholders to inform community publication norms.Source: Recommendation 1: Build frameworks for… · p. 25

Recommendation 2: Build infrastructure for distributed risk analysis

  • Partnership-based model sharing was intended to prevent premature publication while enabling risk analysis; the authors provide an Appendix A agreement template and find no standard approach fits every case.They describe balancing parties’ interests and reducing logistical burdens through legal agreements.Source: Recommendation 2: Build infrastructure… · pp. 25–26
  • Model-sharing infrastructure must address negotiation scalability, premature-release risks from hacks or leaks, and fairness concerns about compute costs and access.The authors suggest vetting participants once to let them access multiple models under shared terms, while cautioning against harming researchers with limited computing resources or creating commercial conflicts of interest.Source: Recommendation 2: Build infrastructure… · p. 26

Recommendation 3: Build communication channels across organizations

  • Because publication decisions can depend on other labs’ choices, the authors encourage AI labs to experiment with communication across organizations despite legitimate privacy norms around unpublished work.Private results hindered coordination, including efforts to retrieve replication statuses.Source: Recommendation 3: Build communication… · p. 26

Conclusion

  • Positive applications, minimal evidence of planned misuse, and research on detection, bias, and collaboration informed confidence to release GPT-2’s 1.5 billion-parameter version.The authors also report that researchers and engineers applied GPT-2 in a range of positive ways.Source: Conclusion · p. 27
  • The authors expect staged release to affect mainly lower- and middle-tier actors, while the skills and resources needed for beneficial and malicious use decline over time.They report little evidence of large-scale misuse in their assessment.Source: Conclusion · p. 27

Appendix A: Summary of Model Sharing Agreement … Appendix quantify by number of output, but we have seen other researchers quantify bias differently.

The appendices document partner-access terms and GPT-2’s staged release, and report exploratory bias probes that revealed subtle, sometimes hard-to-categorize biases while underscoring the need for further analysis and shared methods.

Appendix A: Summary of Model Sharing Agreement

  • The agreement grants partners a non-transferable, non-sublicensable license for internal NLP research and limits use to partner-specific approved purposes, barring third-party sharing, commercial use, or derivative publication without permission.Source: Appendix A · p. 35
  • Partners must provide software feedback, report research efforts every four weeks, and submit an end-of-evaluation report describing key discoveries and work carried out.Source: Appendix A · p. 35
  • Partners must submit manuscripts for safety review 30 days before publication and obtain OpenAI’s written approval, which may be withheld only on safety grounds.Source: Appendix A · p. 35

Appendix B: Release Timeline

  • The release timeline began with the 124M model while other models and data were withheld, then released the 355M, 774M, and 1.5B models.Source: Appendix B · p. 36
  • The timeline also included a dataset of large-model outputs, portions of WebText, and a detection baseline intended to help identify GPT-2-like outputs.Source: Appendix B · p. 36

Appendix C: Examples of Biases in GPT-2

  • GPT-2 probes found biases that could sometimes be explained by training data but were not obvious before analysis, as well as biases difficult to categorize ideologically.The examples were intended to show the breadth and subtlety of some model biases.Source: Appendix C · p. 37

Appendix quantify by number of output, but we have seen other researchers quantify bias differently.

  • The appendices note that bias analysis lacks standard methods and established categories, and encourage larger studies, collaboration, and shared frameworks for analysis.Source: Appendix C · p. 37, Appendix quantify by number of output,… · p. 37
  • Across the 774 million and 1.5 billion parameter models, no statistically significant difference in the number of biased outputs was found, except for bias towards English and Latin script.Researchers collected 100+ outputs per input and reviewed them to categorize attributes; the analysis was preliminary and needs more in-depth testing.Source: Appendix quantify by number of output,… · p. 37

Gender Bias: towards men as criminals … 774M Example: Bias against “black”

GPT-2 associated criminality with men and victimization with different forms of violence by gender. In suspect descriptions, it used “white” or “black,” an association the paper relates to U.S. criminal justice patterns.

Gender Bias: towards men as criminals

  • GPT-2 strongly associated “criminal” with men, potentially reflecting perceptions and U.S. crime statistics, including 73% male arrests in 2017.The paper notes that U.S. victims of violence perceive men as more likely perpetrators.Source: Gender Bias: towards men as criminals · p. 38

774M Example: Men; 774M Example: Women

1.5 Example: Men

  • The 1.5 model’s male-victim example described a shooting during an altercation and said the victim was accidentally shot in self-defense.The passage gives an example completion rather than a frequency estimate.Source: §1.5 · p. 40

774M Example: Bias against “white”; 774M Example: Bias against “black”

1.5B Example: Bias against “white”; 1.5B Example: Bias against “black”

Religious Bias : towards Christianity … Terrorism, Extremism, and Counterterrorism

The report finds a strong association between “God” and Christianity, alongside a preference for English and Latin-script output. Larger models showed some improvement on non-English and non-Latin outputs, but those outputs often remained incoherent or unrelated to the input.

Religious Bias : towards Christianity

774M Example: Christian; 1.5B Example: Christian

  • The strong “God”–Christian text tie may reflect Christian text in training data; Christianity’s prevalence and the Bible’s extensive online availability are possible explanations.Source: 1.5B Example: Christian · p. 44

774M Example: Neutral

  • When “God is” was not tied to organized religion, outputs often referenced philosophers, were nonsensical, or drew on popular media.Source: 1.5B Example: Christian · p. 44

Language Preference : towards English and Latin script languages; 774M Example: Non-Latin characters

774M Example: English; 1.5B Example: English and Non-Latin characters

1.5B Example: English and Non-Latin characters

Terrorism, Extremism, and Counterterrorism

Introduction · 1 Methodology

This section describes an approach to examining whether fine-tuning neural language models on ideological texts can generate flexible output that retains the source ideology. It covers four ideological datasets, a prompted generation procedure, and labeled input and output examples for Islamism and right-wing extremism.

Introduction

  • Neural language models could let extremists scale propaganda, including commentary in ongoing conversations, posing a novel threat to civil society.The authors note that extremists may prioritize ideological positioning over whether their posts are fake news.Source: Introduction · pp. 50–51

1 Methodology

  • The study tests whether manifesto-length fine-tuning can produce flexible extremist text that retains source ideology while gaining semantic variance.The authors hypothesize that an ideologically biased model could introduce recruits to a stance or inject extreme text into otherwise normal conversations.Source: §1 · p. 52
  • The researchers compiled manifesto- or speech-based datasets spanning white-supremacist right-wing extremism, Marxist-Leninism, anarchism, and jihadist Islamism.The white-supremacist collection includes terrorists’ manifestos, while the Islamism collection includes translated speeches by Islamic State leader Abu Bakr al-Baghdadi.Source: §1 · p. 52
  • They fine-tuned one model per dataset for 500 epochs, then generated about 50 outputs per model through chained prompts and also sampled unprompted outputs.Each chain began with “The greatest danger facing the world today is”; the last sentence of each output prompted the next, repeated three times before concatenating all four outputs.Source: §1 · p. 53
  • The paper illustrates the generated material with source-text and output excerpts, including an anarchism output that begins by naming inequality as the world’s greatest danger.Only basic preprocessing to remove URLs was applied.Source: §1 · p. 53
  • The Islamism example pairs an input passage with a labeled output passage.Source: §1 · p. 54
  • The right-wing extremism example likewise pairs an input passage with a labeled output passage.Source: §1 · p. 54

2 Analysis

The analyses suggest that fine-tuned GPT-2 models produce text aligned with their source ideologies, quickly adopting characteristic vocabulary. Keyword and topic-clustering analyses also distinguish biased outputs from pretrained GPT-2, though some ideological distinctions remain unclear.

2 Analysis

  • Outputs largely followed their source ideologies and vocabulary, although occasional point-counterpoint confusion showed that models could mix ideological positions.One right-wing-extremism model output briefly expressed anti-imperialist and anti-Christian ideas, which the authors partly traced to material in Anders Breivik’s manifesto.Source: §2 · p. 55
  • GPT-2 quickly incorporated trained ideology’s nuances for specific prompts; pretrained terms were diverse, while biased models frequently used ideology-consistent terms.The authors used keyword analysis, including tf-idf, as an imprecise proxy for ideology.Source: §2 · p. 55
  • A Latent Semantic Analysis model assigning outputs to four topics produced clear ideological clusters, indicating substantively consistent text.The analysis forced the model to assign each output to one of four topics.Source: §2 · pp. 56–57
  • Latent Dirichlet Allocation found three apparent topics—anti-capitalism, anti-imperialism, and anti-Islamist extremism—but no immediately apparent right-wing-extremism topic.Source: §2 · p. 57

3 Assessing Current Detection Methods … 2 Methodology

The paper assesses GPT-2’s perceived credibility and the limits of existing fake-content detection, while outlining further quantitative, linguistic, and human-survey research. Its findings suggest larger models can produce seemingly credible disinformation, while fine-tuning can reduce Grover’s detection accuracy.

3 Assessing Current Detection Methods

  • In a small-sample zero-shot experiment, fine-tuning significantly reduced the accuracy of Allen AI’s Grover classifier.The study used Grover-Mega’s browser-based classifier to measure zero-shot detection capacity.Source: §3 · p. 58
  • Despite low fake-content accuracy, Grover labeled nearly 100% of human text correctly and predicted a “machine” label for a small percentage of texts.The authors suggest that even one or two percent of fake labels for a particular actor or network could give experts reasonable suspicion of language-model use.Source: §3 · p. 59

4 Roadmap

  • CTEC planned qualitative linguistic analysis of GPT-2 outputs to examine ideology, word choice, contradictions, stylistic markers, and other possible signs of fake content.The planned work involved linguists at the Middlebury Institute.Source: §4 · p. 60
  • CTEC also planned a survey asking extremism experts and non-experts to assess fake and real texts for coherence, fluency, style, and arguments.The survey was intended to study subject-matter fine-tuning and whether specially trained models could convince experts and the public.Source: §4 · p. 60

Appendix E: Perceived Credibility of GPT-2

  • The 774M and 1.5B models were similarly capable of synthesizing seemingly credible U.S. foreign-policy disinformation; 1.5B was only slightly more effective, while 355M was significantly less effective than both.The 1.5B–774M difference was not statistically significant (see Fig. 1 and Fig. 2).Source: Appendix E · p. 64

2 Methodology

  • OpenAI generated 300 stories for each 355M, 774M, and 1.5B parameter model, then used StoryCleaner to automatically filter extraneous text; this process required no human input.Source: §2 · p. 64
  • The study’s Mechanical Turk sample was more female, Democratic, and better educated than the U.S. population, potentially biasing perceived-credibility ratings downward.The authors note that better-educated respondents might be more alert to factual or grammatical errors associated with synthetic text or misinformation.Source: §2 · p. 64

3 Overall Findings · 4 Figures · 5 Selected Open-Ended Responses

Mean credibility scores rose across GPT-2 sizes, though the 774M and 1.5B models were statistically comparable; open-ended comments also described difficulty following the stories’ logic.

3 Overall Findings

  • The mean credibility index rose from 6.07 for 355M to 6.72 for 774M and 6.91 for 1.5B, with the larger gain between 355M and 774M.The report says the 1.5B model did not have significantly higher misuse capability than the 774M model.Source: §3 · p. 65, §4 · p. 67
  • Plotting the full credibility distribution showed that the 1.5B model was more consistently perceived as credible than the 774M model, although their mean credibility index scores were statistically indistinguishable.Only 14 individuals gave the 774M model a credibility score of 9.Source: §3 · p. 66
  • Nineteen respondents gave all three credibility-index components a perfect score for 1.5B, compared with 14 for 774M.The same counts applied to respondents assigning an overall credibility score of 9.Source: §3 · p. 66
  • Many respondents said they could not follow a logical thread through the story, while some thought it might be “fake news.”The passage connects the difficulty following the story to GPT-2’s limitations.Source: §3 · p. 66

4 Figures

  • Figures 1 and 2 show mean credibility with 90% and 95% confidence intervals and credibility distributions across three models, using 200 randomly assigned stories per model.Figure 2 counts respondents at each credibility-index level on a 1–10 scale.Source: §4 · pp. 67–68

5 Selected Open-Ended Responses

  • Selected responses illustrate contrasting credibility judgments: one respondent relied on gut feeling, while another doubted the story’s plausibility given Trump’s actions.These are examples from the section’s selected open-ended responses.Source: §5 · p. 69

Related papers

Loading 1908.09203v2…