PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Societal Impact of Open Foundation Models

Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, Rumman Chowdhury, Alex Engler, Peter Henderson, Yacine Jernite, Seth Lazar, Stefano Maffulli, Alondra Nelson, Joelle Pineau, Aviya Skowron, Dawn Song, Victor Storchan, Daniel Zhang, Daniel E. Ho, Percy Liang, Arvind Narayanan

arXiv:2403.07918v1cs.CYcs.AIcs.LG

TL;DR

Open foundation models raise contested questions about societal impact and governance, while empirical evidence on their benefits and misuse risks remains limited. The paper identifies their distinctive properties and analyzes benefits and risks, including misuse risk relative to existing alternatives. It finds that past studies often do not clearly assess marginal risk and calls for more empirically grounded assessment.

  • Problem

    Disagreement about open foundation models’ societal impact and policy, alongside limited empirical evidence about their benefits and risks, makes their release a central debate.

    Source: §1 · p. 1

  • Method

    The paper identifies five distinctive properties of open foundation models, analyzes how they relate to benefits and risks, and frames misuse assessment around marginal risk relative to closed models or pre-existing technologies.

    Source: §1 · pp. 1–2

  • Results

    Past studies do not clearly assess the marginal risk of open foundation models in most of seven commonly discussed misuse vectors, while the framework helps explain some low-risk cases and contentious findings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors call for empirically grounded work to assess marginal risk as model capabilities and societal defenses evolve.

    Source: §1 · p. 2

    The framework does not account for unknown risks and does not itself coordinate actors needed for some defenses, so other approaches will also be needed.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Foundation models are powerful technologies: how they are released publicly directly shapes their societal impact. In this position paper, we focus on open foundation models, defined here as those with broadly available model weights (e.g. Llama 2, Stable Diffusion XL). We identify five distinctive properties (e.g. greater customizability, poor monitoring) of open foundation models that lead to both their benefits and risks. Open foundation models present significant benefits, with some caveats, that span innovation, competition, the distribution of decision-making power, and transparency. To understand their risks of misuse, we design a risk assessment framework for analyzing their marginal risk. Across several misuse vectors (e.g. cyberattacks, bioweapons), we find that current research is insufficient to effectively characterize the marginal risk of open foundation models relative to pre-existing technologies. The framework helps explain why the marginal risk is low in some cases, clarifies disagreements about misuse risks by revealing that past work has focused on different subsets of the framework with different assumptions, and articulates a way forward for more constructive debate. Overall, our work helps support a more grounded assessment of the societal impact of open foundation models by outlining what research is needed to empirically validate their theoretical benefits and risks.

1. Introduction

The paper addresses the contested societal impact of open foundation models amid policy debate, identifying how their distinctive properties shape benefits and risks. It proposes a framework for assessing misuse risk beyond existing technologies and finds that current studies often do not clearly establish this marginal risk.

1. Introduction

  • The paper identifies five distinctive properties of open foundation models: broader access, greater customizability, local inference, inability to rescind access, and weaker monitoring.These properties underpin its analysis of benefits and risks.Source: §1 · pp. 1–2
  • The paper describes benefits including distributing decision-making power, reducing market concentration, increasing innovation, accelerating science, and enabling transparency, while noting that weights alone do not support every form of scientific research.Some research requires training data, which release of weights does not guarantee.Source: §1 · p. 2
  • The authors frame misuse risk as the additional societal risk from open models relative to closed models or pre-existing technologies, and apply the framework to seven commonly discussed misuse vectors.Examples include disinformation, biosecurity, cybersecurity, non-consensual intimate imagery, and scams.Source: §1 · p. 2
  • Across most misuse vectors, past studies do not clearly assess marginal risk, while the framework explains low-risk cases and why past research has reached contentious conclusions.The authors call for empirical work that accounts for evolving model capabilities and societal defenses.Source: §1 · p. 2
  • The analysis aims to ground discourse and policymaking and direct recommendations to developers, risk researchers, competition regulators, and policymakers.The authors say action by these stakeholders can clarify societal impact and help reap benefits while mitigating risks.Source: §1 · p. 2

2. Background

The paper treats model release as a matter of which assets are available, to whom, for what purposes, and over time. For analysis, it uses a simplified open-versus-closed distinction centered on widely available model weights.

2. Background

  • Model release can vary across assets, access permissions, permitted purposes, and changes in access over time.Examples include user restrictions, use restrictions, staged release, and deprecation.Source: §2 · p. 2
  • The paper defines an open foundation model as one with widely available weights, without requiring open code, data, or compute.Some user restrictions and use restrictions may apply, and release need not be staged.Source: §2 · p. 2
  • Open release is practically irreversible because developers may stop further distribution but cannot revoke existing copies of the weights.Users may also redistribute copies despite developer objections.Source: §2 · p. 2
  • The debate contrasts claims that widely available weights support research, competition, innovation, reproducibility, and transparency with concerns about misuse including disinformation, scams, and bioweapons.The paper notes that these arguments form part of the ongoing debate over open foundation models.Source: §2 · p. 2

3. Distinctive properties of open foundation models

The paper distinguishes open foundation models from closed models through five properties associated with their weights being widely available. These properties affect who can use and adapt models, where inference occurs, whether access can be withdrawn, and how usage can be monitored.

3. Distinctive properties of open foundation models

  • Open models’ weights may effectively be available to the public, although expertise and compute costs can remain barriers to use.User restrictions may be difficult to enforce or verify.Source: §3 · p. 3
  • Weights and related computational artifacts permit broad adaptation, including quantization, fine-tuning, and pruning.This customizability can allow users to fine-tune away alignment interventions, though similar issues can arise with fine-tunable closed models.Source: §3 · p. 3
  • Local deployment lets users run models without transferring their data to the model developer, which can matter when confidentiality or data-protection requirements apply.The passage gives healthcare and finance as examples of relevant domains.Source: §3 · p. 3
  • After weights are widely released, developers can stop further distribution but cannot revoke existing copies, which users may redistribute.The passage gives peer-to-peer distribution as one example.Source: §3 · p. 3
  • Developers do not observe inference by default across local, generic cloud, and dedicated hosting use, making monitoring or moderation challenging.Coordination with dedicated model hosts may allow some forms of monitoring or moderation.Source: §3 · p. 3

4. Benefits of Open Foundation Models

Open foundation models can broaden who shapes model behavior and support innovation, competition, science, and transparency. These benefits have limits: weights alone do not provide all research assets, and openness does not guarantee transparency or continued model improvement.

Distributing who defines acceptable model behavior

  • Broader access and customizability let downstream developers modify model behavior, widening who can define what is acceptable beyond the foundation-model developer.Closed models leave that decision under unilateral developer control.Source: §4 · p. 3

Increasing innovation

  • Local adaptation and inference let developers fine-tune models on proprietary data with fewer data-protection and privacy concerns, supporting innovation across applications.The paper also notes that customization has supported improvements across languages.Source: §4 · p. 4

Accelerating science

  • Broader model access supports inclusion and research across AI interpretability, security, and safety, while ongoing access to specific models matters for reproducibility.Developer safety measures can also complicate or prevent research that requires outputs those measures suppress.Source: §4 · p. 4
  • Weights alone are insufficient for some scientific research: studying how training-data biases propagate into model biases requires access to training data.Training-data information can also help assess contamination between training data and benchmark test sets.Source: §4 · p. 4

Enabling transparency

  • Weights enable some forms of transparency by allowing external researchers, auditors, and journalists to scrutinize models, but documentation and training data would further improve it.The cited transparency index found major open-model developers tended to be more transparent than their closed counterparts.Source: §4 · p. 4
  • Weights do not guarantee transparency, and alone do not reveal upstream resources or downstream impacts of a model.The paper names data sources, labor practices, energy use, affected markets, adverse events, and usage-policy enforcement as areas that may remain opaque.Source: §4 · pp. 4–5

Mitigating monoculture and market concentration

  • Customization may diversify downstream behavior and reduce homogeneous outcomes, while broader access may increase downstream competition; neither necessarily reduces concentration in computing and specialized hardware.The paper also notes that open-model developers may lack user feedback and interaction logs used by closed-model developers to improve models over time.Source: §4 · pp. 4–5

5. Risks of Open Foundation Models

The paper assesses misuse risks by comparing the additional risk from open models with existing risks and defenses, while making assumptions and uncertainty explicit. Its review finds most assessed studies insufficient to establish increased marginal risk, with different outcomes for automated vulnerability detection and digitally altered NCII.

5.1. Risk Assessment Framework

  • The framework assesses a specific misuse vector by identifying the threat and actors, then setting out the existing risk and defenses before evaluating marginal risk.Existing risks and defenses help distinguish newly introduced concerns from risks already present or addressed by current measures.Source: §5.1 · pp. 5–6
  • Marginal risk can be assessed against the status quo or other baselines, including restricted model release, while examining the defenses available under each baseline.The framework cautions against assuming closed releases are intrinsically safer without examining existing safeguards.Source: §5.1 · p. 6
  • Risk analyses should account for how existing defenses may evolve or be supplemented, including defenses that foundation models could help develop.Examples include improved disinformation detectors and code fuzzers.Source: §5.1 · p. 6
  • Assessments should state uncertainties and assumptions, including forecasts about model capabilities, inference costs, and threat-actor adaptation.These assumptions can affect judgments of misuse efficacy and scalability.Source: §5.1 · pp. 5, 8
  • Six of the seven reviewed studies provide incomplete risk analyses, which does not mean they are flawed but leaves them insufficient, on their own, to demonstrate increased marginal societal risk.The two illustrative analyses find low current marginal risk for automated vulnerability detection, but considerable present marginal risk for digitally altered NCII.Source: §5.1 · p. 8
  • The framework does not weigh release benefits against marginal risks or the opportunity cost of withholding release, and it cannot assess unknown unknowns.It can identify coordination opportunities but does not itself coordinate actors.Source: §5.1 · p. 8

6. Recommendations and calls to action

The paper recommends stakeholder-specific action to clarify safety responsibilities, improve evidence on misuse risk, and measure the economic effects of openness. It calls for policy choices to account for both uncertainty and compliance burdens.

AI developers

  • Open-model developers should disclose their responsible AI practices and what they recommend or delegate downstream, while procurers assess existing measures and fill gaps.The paper says unclear safety responsibility could let practices fall through the cracks as downstream applications use upstream models.Source: §6 · p. 8

Researchers investigating AI risks

  • Risk researchers should clarify marginal misuse risks by characterizing the status quo, using realistic threat models, and considering the full misuse supply chain.The recommendation follows the paper’s preliminary finding of uncertainty across several misuse vectors.Source: §6 · p. 8

Policymakers

  • Fund independent risk research, and consider further interventions only after uncertainty falls and evidence shows marginal risks warrant concern.Policymakers should also weigh compliance burdens; strict downstream liability or watermarking duties may be difficult for open developers with little ability to monitor, moderate, or prohibit downstream use.Source: §6 · p. 9

Competition regulators

  • Competition regulators should measure openness’s economic benefits, which remain undocumented without large-scale economic analysis or market surveillance.Benefits may not materialize if model-quality differences, rather than expanded choice or lower costs, more directly drive adoption.Source: §6 · p. 9

7. Conclusion

The paper seeks to clarify open foundation models’ benefits and risks amid conceptual confusion and weak empirical evidence. It urges more evidence-building, while noting that realizing its vision for an AI ecosystem requires action from multiple stakeholders.

7. Conclusion

  • The paper addresses conceptual confusion by defining open foundation models and clarifying their distinctive properties, benefits, and risks.It identifies fragmented understanding and poor empirical evidence as sources of controversy.Source: §7 · p. 9
  • The authors encourage future work to address empirical evidence deficits and say a vibrant AI ecosystem will require significant action from many stakeholders.They note that some underlying philosophical tensions, especially those intertwined with actor incentives, are unlikely to be resolved.Source: §7 · p. 9

A. A brief history of open foundation models

The history traces misuse concerns from earlier image-generation systems to foundation models, alongside release events that drew attention to openness and misuse. The risk-assessment section surveys studies of misuse vectors and evaluates whether they establish increased marginal societal risk.

A. A brief history of open foundation models

  • Earlier generative image models raised deepfake concerns and enabled impersonation, political misinformation, and non-consensual intimate imagery, including through face-swapping algorithms.A Telegram bot was used to generate over a hundred thousand nude images of women.Source: A. A brief history of open foundation… · p. 19
  • GPT-2’s staged release of increasingly large models in 2019, motivated especially by disinformation concerns, brought attention to the connection between release and misuse.OpenAI reported no evidence of model misuse during the staged release.Source: A. A brief history of open foundation… · p. 19
  • Stable Diffusion’s public weights let users circumvent NSFW filters, and pornography made with offshoots—including non-consensual images resembling real people—spread online.The release was among the first text-to-image models widely available outside the research community.Source: A. A brief history of open foundation… · p. 19

B. Risk assessment framework

  • The risk assessment surveys seven misuse vectors and evaluates prior studies, without requiring those studies to compare open models with closed models or existing technologies.Incomplete evidence on a framework element does not by itself mean a study’s analysis is flawed; it means that study does not establish increased marginal societal risk.Source: B. Risk assessment framework · p. 20

B.1. Spear-phishing scams (Hazell, 2023)

The assessed spear-phishing threat involves low-skill actors using LLMs to outsource personalized email writing. The analysis offers limited characterization of pre-existing risk and defenses, and does not examine whether open models circumvent those defenses.

B.1. Spear-phishing scams (Hazell, 2023)

  • The threat is personalized email scams by low-skill actors who can outsource email writing to LLMs.Source: §B.1 · p. 20
  • The analysis gives examples of existing phishing risk but lacks broader characterization of its societal impact, limiting the scope of marginal-risk arguments.Source: §B.1 · p. 20
  • It briefly discusses defenses but does not analyze their effectiveness or whether open models can circumvent protections in operating systems and email services.The analysis also does not examine assumptions such as existing defenses failing or content creation already being cheap without AI.Source: §B.1 · pp. 20–21

B.2. Cybersecurity risk (Seger et al., 2023)

The cybersecurity and disinformation analyses identify threats but leave important parts of the marginal-risk assessment unresolved. In particular, they provide limited grounding in existing risks and defenses, direct comparisons, or the effectiveness of proposed mitigations.

B.2. Cybersecurity risk (Seger et al., 2023)

  • The cybersecurity analysis describes malware creation as the threat but does not clearly identify threat actors or specify the malware threat further.It also does not discuss the prevalence or societal impact of malware-based cyberattacks absent open foundation models.Source: §B.2 · p. 21
  • The cybersecurity discussion mentions bug bounties but does not deeply assess their effectiveness against existing risks or alternative defenses.It also acknowledges arguments that open models may advance safety research without assessing alternatives such as LLM-assisted cyberdefense.Source: §B.2 · p. 21
  • It provides no evidence that open foundation models facilitated malware-related incidents and does not compare them with internet information or closed models.The analysis also does not assess how much interventions reduce marginal risk or analyze core assumptions such as threat-actor resources.Source: §B.2 · p. 21

B.3. Disinformation (Musser, 2023)

  • The disinformation analysis identifies low-cost content creation by propagandists but does not analyze influence effectiveness or the societal impact beyond post counts.It gives examples of existing mass-content-generation risks but does not assess their effects on people at scale.Source: §B.3 · p. 21
  • The analysis references platform and developer defenses but does not evaluate their effectiveness or other existing platform measures such as behavioral moderation and bot detection.Its comparison of monitoring costs focuses on closed versus open models.Source: §B.3 · pp. 21–22
  • It discusses reduced content-generation costs but leaves demand, real-world effectiveness, defense circumvention, and distribution costs unassessed.Distribution costs may overshadow content-creation costs in disinformation operations.Source: §B.3 · p. 21
  • Its assumptions include actor types, relative open- and closed-model costs, and the cost of evading closed-model monitoring.These assumptions shape the analysis of disinformation risk.Source: §B.3 · p. 22

B.4. Biosecurity risk (Gopal et al., 2023)

The biosecurity analysis identifies a threat involving individuals creating pandemic-causing pathogens, but does not fully compare that risk with existing technologies or assess defense effectiveness. Its assumptions about model capabilities and internet information are also unanalyzed.

B.4. Biosecurity risk (Gopal et al., 2023)

  • The identified threat is that individuals without training could obtain and create pandemic-causing pathogens.The analysis does not examine how rogue individuals or non-state actors could engage in bioterrorism without foundation models.Source: §B.4 · p. 22
  • The analysis briefly discusses barriers to leveraging biorisk information, including limited virus information, few capable researchers, and historical immunity.It omits procurement controls and benchtop DNA synthesizers, and does not assess the efficacy of existing defenses.Source: §B.4 · p. 22
  • It does not compare the risk with similar risks from widely available internet information or substantiate its discussion of closed-model safeguards with evidence.It does not analyze risks from jailbreaking or fine-tuning closed models.Source: §B.4 · p. 22
  • Proposed legal-liability and market-based insurance defenses lack evidence of feasibility or effectiveness, especially against well-resourced rogue actors.The analysis also does not assess how effective existing defenses such as procurement controls would be.Source: §B.4 · p. 22
  • The analysis leaves unexamined whether current or future models could generate information about novel pandemics and whether internet information could similarly aid attackers.The former possibility would increase marginal risk, while the latter would lower it.Source: §B.4 · p. 22

B.5. Voice cloning (Ovadya & Whittlestone, 2019)

The reviewed work on voice cloning, digitally altered NCII, and CSAM identifies threats but varies in its treatment of existing risks, comparisons, and defenses. The CSAM analysis also describes concerns linked to open models and challenges in monitoring their use.

B.5. Voice cloning (Ovadya & Whittlestone, 2019)

  • The voice-cloning analysis identifies financial scams using machine-learning voice cloning and considers malicious users with or without ML expertise.It offers some characterization of existing financial-crime risk but little broader analysis of societal impact or defense efficacy.Source: §B.5 · p. 22
  • It discusses openness-related risks and mitigation mechanisms but offers minimal direct comparison with closed models or other technologies and limited analysis of mitigation efficacy.Its assumptions include the current state and release patterns of open and closed models.Source: §B.5 · p. 22

B.6. Non-consensual intimate imagery (NCII) (Lakatos, 2023)

  • The NCII analysis identifies digitally altered imagery created by people who may use easy interfaces without ML skills, and reports tens of millions of visitors to NCII distribution sites.It does not describe preexisting tools such as face swapping.Source: §B.6 · p. 23
  • Although it attributes increased use of NCII-creation services to open foundation models, it does not compare their risk with closed models or tools such as Photoshop.It also does not examine existing technical or legal defenses or substantially analyze defenses against open-model marginal risk.Source: §B.6 · p. 23

B.7. Child sexual abuse material (CSAM) (Thiel et al., 2023)

  • The CSAM analysis identifies hobbyist groups and sharing forums as threat actors, and reports that less than 1% of CSAM on online forums is photorealistic computer-generated CSAM.It outlines legal and platform moderation defenses against CSAM absent open models.Source: §B.7 · p. 23
  • The CSAM analysis links open models to concerns including re-victimization, the liar’s dividend, added moderator burdens, and pressure on enforcement agencies.It reports that most computer-generated CSAM is based on openly available Stable Diffusion models and that watermarking or monitoring is difficult and can be disabled.Source: §B.7 · p. 23
  • It discusses training-data removal, persistent watermarks, legal approaches, and EXIF data as defenses, while noting uncertainty about future model capabilities, watermarking difficulty, and CG-CSAM’s legal status.The analysis also notes shortcomings of the proposed defenses.Source: §B.7 · p. 23

Related papers

Loading 2403.07918v1…