PaperlayerView exact revision on arXiv

Source-linked AI summary

Typology of Risks of Generative Text-to-Image Models

Charlotte Bird, Eddie L. Ungless, Atoosa Kasirzadeh

arXiv:2307.05543v1cs.CY

TL;DR

Research on generative-model harms has examined language technologies, while direct risks from text-to-image models also warrant study. This paper reviews TTI risks and stakeholders, distinguishing anticipated from empirically investigated risks. It presents a risk typology and identifies open questions, including indirect environmental harms outside its scope.

  • Problem

    The paper extends scrutiny of language-technology harms to potential risks and harms from text-to-image models, while noting that many identified risks lack adequate discussion or empirical investigation.

    Source: §1 · pp. 1–2

  • Method

    The authors review literature on text-to-image models and analyze immediate risks, stakeholders, and empirical investigations, including distinctions between anticipated and observed risks.

    Source: §1 · p. 2

  • Results

    The paper presents a typology of TTI risks, reviews mitigation strategies, and discusses open questions about system development and use.

    Source: §7 · p. 16

  • Takeaways & Limitations

    The authors conclude that additional perspectives, expertise, and research are needed to refine the typology and improve understanding of TTI systems’ social implications.

    Source: §7 · p. 16

    The review focuses on direct risks and excludes indirect harms such as environmental unsustainability, which the authors say requires additional research.

    Source: §6 · p. 15

Abstract

from arXiv · show

This paper investigates the direct risks and harms associated with modern text-to-image generative models, such as DALL-E and Midjourney, through a comprehensive literature review. While these models offer unprecedented capabilities for generating images, their development and use introduce new types of risk that require careful consideration. Our review reveals significant knowledge gaps concerning the understanding and treatment of these risks despite some already being addressed. We offer a taxonomy of risks across six key stakeholder groups, inclusive of unexplored issues, and suggest future research directions. We identify 22 distinct risk types, spanning issues from data bias to malicious use. The investigation presented here is intended to enhance the ongoing discourse on responsible model development and deployment. By highlighting previously overlooked risks and gaps, it aims to shape subsequent research and governance initiatives, guiding them toward the responsible, secure, and ethically conscious evolution of text-to-image models.

1 INTRODUCTION

The paper extends discussion of generative-model harms to text-to-image systems through a literature review, organizing immediate risks by category and stakeholder. It distinguishes anticipated from observed risks and identifies gaps for further research.

1 INTRODUCTION

  • The review began with eight seed papers and analyzed collected literature for immediate risks, stakeholders, and empirical investigations.The search was supplemented with manual searching; the detailed methodology is in Appendix A.Source: §1 · p. 2
  • The typology organizes immediate risks into discrimination and exclusion, harmful misuse, and misinformation and disinformation, while noting the categories can overlap.The authors argue that distinguishing categories supports clearer understanding and more robust mitigation strategies.Source: §1 · p. 2
  • The analysis identifies developers, users, regulators, affected parties, data sources, and data subjects as stakeholders, and considers the nature of potential harms.It also touches on environmental harm.Source: §1 · p. 2
  • The paper distinguishes anticipated risks, predicted by researchers, from observed risks investigated empirically, and highlights risks not yet adequately discussed.The distinction underscores the need for empirical investigations into many identified risks.Source: §1 · p. 2
  • The authors review mitigation strategies and open questions, offer policy suggestions, and advocate collaboration among researchers, developers, and policymakers.They aim to improve understanding of potential futures for text-to-image and other generative models.Source: §1 · p. 3

2 GENERATIVE TEXT-TO-IMAGE MODELS

Text-to-image models synthesize images from text prompts through language and vision components. The section sketches architectures behind image synthesis and the recent rise of diffusion models.

2 GENERATIVE TEXT-TO-IMAGE MODELS

  • A text-to-image model synthesizes an image from a textual prompt, with NLP extracting objects, attributes, and relationships and computer vision generating the image.The model is described as a type of generative neural network.Source: §2 · p. 3
  • Flow-based models, autoregressive models, and variational autoencoders have been used for image synthesis, while GANs accelerated generative-model capabilities.Source: §2 · p. 3
  • A typical text-to-image GAN uses a generator to synthesize an image from text and a discriminator to assess authenticity, with adversarial training refining the generator.The section also notes that transformers advanced NLP and vision, early DALL-E versions extended transformer use to vision, and CLIP became pivotal in image-generation tasks.Source: §2 · p. 3
  • Diffusion models reverse noisy data through a neural-network-parameterized Markov chain and have recently achieved state-of-the-art image-synthesis results.Their success has spurred proliferation of popular and open-source diffusion models covered by the taxonomy.Source: §2 · p. 3

3 STAKEHOLDERS AND POWER DYNAMICS

The section examines power dynamics through six stakeholder groups and shows how access, training data, model outputs, and governance can distribute risks and benefits unevenly.

3.1 System Developers

  • Developing state-of-the-art systems requires substantial compute and storage, concentrating development among well-resourced actors; limited team perspectives may reinforce biases and Western or heterosexual output norms [15] [177].Open-access efforts exist, but their competitiveness with for-profit models remains unclear.Source: §3.1 · pp. 3–4

3.2 Data Sources

  • Data collection may deny creators consent or acknowledgment, while inadequate curation obscures attribution, complicates output reasoning, and hinders harm reduction.Models can reproduce near-identical training images, and dataset attribution and consent concerns have prompted legal actions [20] [147].Source: §3.2 · p. 4

3.3 Data Subjects

  • Online availability does not imply intended use: image-subject consent, privacy, and unauthorized explicit depictions remain concerns, including for sex workers [35] [147].The paper notes explicit content in datasets and user retraining to generate specific explicit content.Source: §3.3 · p. 4

3.4 Users

  • Access may be exclusionary where commercial systems restrict territories or use requires input-language fluency; identified malicious users include information operators and cyber-criminals [4].Hobbyists have also created harmful, often pornographic content that can gain viral attention.Source: §3.4 · pp. 4–5

3.5 Affected Parties

  • Creatives may face accidental plagiarism, digital forgery, Western-style dominance, or job displacement, and may feel pressured to use TTI systems to remain competitive [147] [177].The authors connect this pressure to possible exclusion for communities unfamiliar with the technology or unable to compete in a saturated marketplace.Source: §3.5 · p. 5
  • Marginalized groups may be stereotyped or offended by generated images, with repeated negative outcomes potentially contaminating future datasets and downstream technologies [15] [83] [157] [174].Source: §3.5 · p. 5
  • Synthetic misinformation can target women and public figures, while people with less power to contest falsehoods may face heightened risk and unequal access to detection or data protections [96] [106].Source: §3.5 · pp. 5–6

3.6 Regulators

  • Regulatory tools may address privacy, intellectual property, and security concerns, but their effective adaptation to TTI systems remains an open question at national and global scales [49, 70, 138, 161].Source: §3.6 · p. 6

4 RISKS

This section traces TTI risks from biased representation and harmful outputs to misleading media, showing how harms may reach individuals, affected groups, and communities.

4.1 Discrimination and Exclusion

  • Empirical coverage remains limited, often centers salient demographics and dataset bias, while TTI growth may make some visual creators’ roles obsolete.The paper notes that other sources of bias across the model life cycle remain underexplored.Source: §4.1 · pp. 6–7
  • Studies report race-, ethnicity-, culture-, gender-, and class-linked stereotypes, Western-skewed representation, and stereotyped images of non-cisgender identities.The studies also report gender and racial bias around certain occupations or objects.Source: §4.1 · pp. 6–7

4.2 Harmful Misuse

  • 30%+ of images produced by a Stable Diffusion model for prompts mentioning girls as young as 12 were sexualized; non-consensual imagery can harm public participation and image subjects’ identity and bodily agency.Source: §4.2 · p. 7, §4.3 · p. 9
  • Underspecified prompts or poor training data can yield culturally offensive or harmful outputs; malicious users may also exploit poisoned encoders or prompt phrases to bypass safeguards.Source: §4.2 · pp. 7–8
  • TTI models can reproduce source images, including private medical data, and raise privacy and copyright concerns; they may also support spear-phishing or adversarial attacks.One study found about 35% of images duplicated by Stable Diffusion fell under explicit non-permissive copyright notice.Source: §4.2 · p. 8

4.3 Misinformation and Disinformation

  • Individual harms include defamatory deepfakes, identity theft, and likeness misuse; unequal detection across race and gender can compound representational and allocative harms.Source: §4.3 · p. 9
  • Closed online communities can intensify polarisation and resist correction, leaving users reliant on low-credibility news especially vulnerable.Source: §4.3 · pp. 9–10
  • Misleading media can undermine epistemic agency, intensify polarisation and division, misrepresent communities, and threaten democratic, financial, and political systems.Source: §4.3 · pp. 10–11

5 MITIGATION STRATEGIES

The paper presents mitigation as a combination of local system-level measures and broader, multi-stakeholder action for community- and society-level harms. It reviews participatory, operational, technical, and socio-legal strategies, noting limitations in each.

Mitigation strategies

  • Broad community- and society-level harms require multidisciplinary, multi-stakeholder mitigation, while localized harms may be addressed at the system level.Widespread mitigation would require changes to current TTI model and system development and deployment practices.Source: §5 · p. 11

Participatory projects

  • Participatory projects can target underrepresented communities and promote inclusive design, but power dynamics and consent must be handled carefully.Without that care, disparities may persist in consultation and undermine participation; opt-out functions also depend on artists knowing their data is used.Source: §5 · p. 11

Operational solutions

  • Limited releases can enable recall, while open sourcing supports stress testing and transparency; neither remedies biased or harmful underlying data and models.Operational strategies also leave downstream impacts such as job displacement incompletely addressed, so the paper calls for ongoing evaluation and reform.Source: §5 · p. 12

Technical solutions

  • Technical approaches include filtering, steering, and prompt editing, but filtering may exacerbate or fail to address bias, and prompt editing has limited impact.Synthetic-media detection has no perfect approach, and technical measures alone cannot resolve risks such as miscommunication, job loss, or copyright infringement.Source: §5 · p. 12

Socio-legal interventions

  • Socio-legal measures include regulation, media literacy, and corporate accountability, but labelling may affect trust and interventions face corporate resistance or free-speech concerns.The paper also notes that limiting monetary rewards may deter misinformation spread [4].Source: §5 · pp. 12–13

6 OPEN QUESTIONS AND FUTURE RESEARCH

The review identifies neglected risks and research gaps across output bias, labor, cultural communication, and political uses of text-to-image systems. It calls for broader stakeholder input, benchmarks, further empirical work, and policy measures while acknowledging that indirect environmental harms fall outside its main scope.

Output bias

  • The review identifies neglected output biases, including ageism and anti-Asian sentiment, with limited targeted mitigation and research on religious and Indigenous-community bias.These biases can affect users’ ability to generate appropriate images and expose downstream parties to established norms and stereotypes.Source: §6 · p. 13

Dialect bias

  • TTI systems may favor white-aligned American English, and reliance on English or a few languages can require users to conform linguistically, reinforcing the view that other dialects are subpar.The accuracy and quality of translations, especially for prompt nuances, remain uncertain.Source: §6 · p. 13

Pre-release moderation

  • Pre-release moderation can harm workers in traditionally pillaged countries, who may have insufficient support and less power than the companies profiting from the systems.The paper calls for fairer labor practices to reduce moderators’ harm.Source: §6 · p. 13

Job displacement

  • The paper connects TTI-enabled job displacement to existing inequalities, including lost income for marginalized creatives and image subjects who may lack agency over or profit from their images.It raises questions about generated humans replacing paid models and whether apparent diversity in imagery corresponds to compensation for people.Source: §6 · pp. 13–14

Miscommunication

  • Western-centric datasets and fast image generation raise underexplored risks of cross-cultural miscommunication and flooding information environments with synthetic content.The paper calls for multidisciplinary anthropological and technical research and inclusive datasets to address miscommunication.Source: §6 · p. 14

Socio-political instability

  • The possible use of TTI models to support populism and nationalism through false evidence is a widening research gap that would benefit from alignment with deepfake research.The paper cites narratives of “alternative facts” in present-day America as context for this concern.Source: §6 · p. 14

Future research directions

  • The paper calls for TTI benchmarks of safe, ethical, and fair performance, broad stakeholder input, and actionable research such as interviews and case studies with people affected by harms.It also identifies environmental harms as indirect risks excluded from its main focus, requiring research into training energy, emissions distribution, and renewable-energy strategies.Source: §6 · pp. 14–15

Policy recommendations

  • Its five policy recommendations cover multi-stakeholder benchmarks, media literacy, clear communication and consent for data use, copyright and attribution rules, and safeguards against harmful imagery.The recommendations are offered as guidance for general policy initiatives; the paper notes that some may already be covered by existing frameworks.Source: §6 · pp. 15–16

7 CONCLUSION

The paper presents a TTI risk typology alongside a review of mitigation strategies and open questions. It frames these as preliminary contributions that need further perspectives, expertise, and research.

Conclusion

  • The paper presents a typology of risks associated with TTI-induced technologies, followed by a review of mitigation strategies and open questions.Source: §7 · p. 16
  • The paper characterizes its recommendations as preliminary.Source: §7 · p. 16
  • Further perspectives, expertise, and research are needed to refine the typology and improve understanding of TTI systems’ social implications.Source: §7 · p. 16

A TAXONOMY METHODOLOGY

The authors searched academic literature using seed papers and manual searches, then analyzed the collected papers for risks, stakeholders, empirical work, and open questions. They used an inductive-deductive process to build risk subcategories and identify gaps.

Search and collection

  • Semantic Scholar searches seeded with eight papers returned over 300 papers, of which 43 were judged substantial and useful, and manual searches added 40 more.The collected papers were analyzed for stakeholders, risks, empirical investigations, and open research questions.Source: A TAXONOMY METHODOLOGY · p. 25

Taxonomy construction

  • The taxonomy began with three preselected broad categories—discrimination and exclusion, harmful misuse, and misinformation—and derived subcategories from the literature.Source: A TAXONOMY METHODOLOGY · p. 25

Gap identification

  • The authors identified literature gaps retrospectively using analogous research on other technologies and by identifying stakeholders not addressed in the papers.The gaps are identified in the paper’s table.Source: A TAXONOMY METHODOLOGY · p. 25
Loading 2307.05543v1…