PaperlayerView exact revision on arXiv

Source-linked AI summary

Into the LAIONs Den: Investigating Hate in Multimodal Datasets

Abeba Birhane, Vinay Prabhu, Sang Han, Vishnu Naresh Boddeti, Alexandra Sasha Luccioni

arXiv:2311.03449v1cs.CY

TL;DR

The paper examines how dataset scale relates to hateful content in vision-language datasets, a question made important by limited transparency around training data. It audits LAION-400M and LAION-2B-en and finds that scale exacerbates hateful content, while a conservative NSFW threshold does not remove all toxic or hateful alt-text.

  • Problem

    Limited transparency around training datasets restricts independent audits of the data behind widely used models.

    Source: §1 · p. 2

  • Method

    The paper comparatively audits LAION-400M and LAION-2B-en, analyzing alt-text with pysentimiento and examining its relation to scale and image NSFW labels.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    The authors report that scale exacerbates hateful content, and a conservative NSFW threshold of 0.5 still leaves some toxic and hateful alt-text.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    Access to datasets is essential for quantitative and qualitative audits, and image-focused and alt-text analyses are correlated but complementary.

    Source: §6 · p. 10

    The study uses pysentimiento, whose variability across English linguistic variants and specific topics has not, to the authors’ knowledge, been researched.

    Source: §5 · p. 10

Abstract

from arXiv · show

'Scale the model, scale the data, scale the compute' is the reigning sentiment in the world of generative AI today. While the impact of model scaling has been extensively studied, we are only beginning to scratch the surface of data scaling and its consequences. This is especially of critical importance in the context of vision-language datasets such as LAION. These datasets are continually growing in size and are built based on large-scale internet dumps such as the Common Crawl, which is known to have numerous drawbacks ranging from quality, legality, and content. The datasets then serve as the backbone for large generative models, contributing to the operationalization and perpetuation of harmful societal and historical biases and stereotypes. In this paper, we investigate the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. Our results show that hate content increased by nearly 12% with dataset scale, measured both qualitatively and quantitatively using a metric that we term as Hate Content Rate (HCR). We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text. Instead, we found that trace amounts of hateful, targeted, and aggressive text remain even when carrying out conservative filtering. We end with a reflection and a discussion of the significance of our results for dataset curation and usage in the AI community. Code and the meta-data assets curated in this paper are publicly available at https://github.com/vinayprabhu/hate_scaling. Content warning: This paper contains examples of hateful text that might be disturbing, distressing, and/or offensive.

1 Introduction

The paper frames auditing hateful content in vision-language datasets as an important but underexplored issue amid growing datasets and limited transparency about their contents. It compares LAION-400M and LAION-2B-en to examine how scale relates to hateful, targeted, and aggressive content and how existing filters handle it.

1 Introduction

  • The paper situates its question amid growing public use of generative models, including Dall·E 2, Stable Diffusion, and Midjourney.It notes that Stable Diffusion and its variants were trained on open-sourced LAION-family datasets.Source: §1 · pp. 1–2
  • Open and closed vision-language datasets underpin widely used models, but limited access and concealed training-data details restrict independent scrutiny.The paper notes that open datasets are often scraped from the Common Crawl, while details about closed datasets are frequently withheld.Source: §1 · p. 2
  • Scale-first dataset creation and opaque curation can produce datasets with quality and content problems and downstream harms, including discriminatory representations and disparate treatment.The paper cites copyright lawsuits, duplicates, fetishization, and worsening stereotypes among related concerns.Source: §1 · p. 2
  • Hateful, abusive, racist, aggressive, and targeted speech overlap, and minoritized groups are often targets of hate.The cited examples include violent GPT-3 completions prompted with “Muslim” and racial bias in hate-speech datasets.Source: §1 · p. 2
  • Prior findings on biased abusive-language classifiers and the unestablished prevalence of hate in large vision-language datasets motivate auditing their content.The paper calls such auditing a critical first step toward ameliorating downstream effects of hateful content.Source: §1 · p. 3
  • The audit compares LAION-400M and LAION-2B-en to assess how dataset size relates to hateful, targeted, and aggressive content and what existing filters remove.The paper also previews HCR analysis and an examination of the relationship between hateful or targeted speech and NSFW values.Source: §1 · p. 3

2 Scaling Datasets: An Overview

The section contrasts the field’s view of scale as a route to better performance and simpler data work with critical scholarship on its costs. It argues that open-source datasets enable accountability, but require ongoing audits.

2 Scaling Datasets: An Overview

  • Machine-learning research often treats larger scale as a marker of progress and a route to better model performance.The section cites analysis of influential papers and work suggesting that larger datasets can improve performance.Source: §2 · p. 3
  • Scale is also presented as a substitute for curation, filtering, and annotation, with noisy data expected to be offset by greater dataset size.The section describes this as a “scale beats noise” narrative.Source: §2 · p. 3
  • LAION-400M was created as an open-source alternative to opaque datasets and models, but web-sourced data still requires continual rigorous audits.The section presents open access as a prerequisite for independent evaluation and accountability.Source: §2 · p. 3
  • Critical scholarship challenges scale thinking’s compatibility with societal equity and links unchecked scalability to corporate aims such as profit and market power.The section notes that these aims may come at the expense of informed consent, justice, and consideration of societal impacts.Source: §2 · pp. 3–4

3 Dataset Audit: LAION-400M and LAION-2B-en

This section situates the audit among prior dataset studies, then describes an alt-text analysis of LAION-400M and LAION-2B-en using pysentimiento scores and threshold-based quality checks.

3 Dataset Audit: LAION-400M and LAION-2B-en

  • Prior dataset work has documented harmful content and quality problems, while earlier multimodal audits chiefly examined images, sources, or image–text discordance.The cited examples include duplicates in LAION-2B-en and explicit images and stereotypes in LAION-400M.Source: §3 · p. 4

3.1 Audit methodology

  • The audit focuses on alt-text in two LAION datasets, testing its relationship to scale and image NSFW labels with the pysentimiento toolkit.The paper says the toolkit supports GPU inference and multiple languages, and was selected for peer review, documentation, and active maintenance.Source: §3.1 · p. 4
  • Pysentimiento returns probability scores for hateful, targeted, and aggressive speech; each category captures a different aspect of harmful text.The targeted score indicates whether a target is generic or a specific group, while the aggressive score tracks aggression or violence.Source: §3.1 · p. 4

3.2 Experiment design

  • The study samples 3.2 million LAION-400M rows and 12.8 million LAION-2B-en rows because computational resources constrain the audit.It samples 100,000 rows from each of the 160 constituent parquet files in each dataset.Source: §3.2 · p. 5
  • Hate Content Rate measures the percentage of text samples whose category score exceeds a preset threshold; the Any-of-the-three variant fails a sample if any category exceeds it.The study uses these metrics to compare datasets as they scale and to examine the relationship between harmful alt-text and image NSFW content.Source: §3.2 · p. 5
  • The score matrices are used to compare text-quality statistics across dataset sizes and relate harmful alt-text to image NSFW labels.The study uses both the category-specific and Any-of-the-three metrics for these comparisons.Source: §3.2 · p. 5

4 Results

The audit compares HCR across LAION-400M and LAION-2B-en, examines shard-level differences, and tests how harmful alt-text relates to image-based NSFW scores. HCR was higher in the larger dataset, and image-only NSFW filtering left some harmful text in the safe subset.

4.1 Scaling is not benign: comparing LAION 400M and LAION 2B-en

  • 12.26% normalized HCR increase remained when comparing the 2B-en lower-bound estimate with the 400M upper-bound estimate.The 2B-en curves also lie above the 400M curves for hateful, targeted, and aggressive speech at every Pthreshold.Source: §4.1 · pp. 5–6, Figure 1 · p. 6

4.2 Intra-dataset filewise comparisons

  • For all three speech categories, file-wise HCRs were higher in LAION-2B-en than LAION-400M, with Welch-test results supporting the difference.The reported T-values were 14.48, 13.8, and 4.44, with all p-values ≪1e−4.Source: §4.2 · p. 7, Table 1 · p. 7, Figure 2 · p. 7

4.3 Connecting toxic alt-text and NSFW labels

  • In LAION-2B-en, NSFW scores had slight correlations with hateful and targeted alt-text scores, but a much weaker correlation with aggressive text.Pearson correlations were 0.227 for hateful, 0.215 for targeted, and 0.076 for aggressive scores; the analysis focused on 2B-en because 400M NSFW labels were discrete.Source: §4.3 · pp. 7–8
  • Filtering LAION-2B-en at the recommended NSFW threshold of 0.5 left trace amounts of harmful alt-text in the safe subset.The reported residuals included 0.24% likely hateful content and 0.03% targeted content; the authors suggest concurrent text- and image-based filtering.Source: §4.3 · p. 8, Table 2 · p. 8

5 Discussion and Recommendations

The authors argue that scale exacerbates hateful content and that rigorous dataset evaluation and curation are critical. They discuss limits of extrapolation and threshold-based filtering, and recommend more transparent, context-sensitive audits.

5 Discussion and Recommendations

  • Scale exacerbates hateful content in the two audited LAION datasets, underscoring the need for rigorous evaluation, audit, curation, and management.The authors frame these practices as critical for advancing the field transparently and accountably.Source: §5 · p. 8

Computational constraints

  • The authors say resource constraints limited their investigation, including their ability to estimate the prevalence of screenshot images.They cite the costs of image APIs and the 6.2 TB storage needed to download LAION-2B-en, plus resources for analysis, as barriers; they also note that self-audits have limited validity and credibility relative to independent audits.Source: §5 · pp. 8–9

The risks of extrapolation

  • LAION-400M-based extrapolation estimated an upper bound of 7.1 million harmful pairs, below the 7.9–8.6 million interval from LAION-2B-en samples by 0.865 million images.The authors caution that Common Crawl’s graph structure and inter-node correlations may invalidate sample-level i.i.d. assumptions, making such extrapolation imprudent and liable to underestimate content.Source: §5 · p. 9

The dangers of thresholding

  • Even an NSFW threshold of 0.5 leaves some toxic and hateful alt-text, while predefined thresholds assume a universal level of acceptable content.The authors recommend providing raw values from multiple text- and image-based detection approaches and evaluations of what different thresholds filter, leaving final thresholds to dataset users.Source: §5 · p. 9

Appropriate and consistent metrics

  • The authors call for standardized, meaningful metrics to make progress measures comparable and reproducible across dataset scaling studies.They note that cited studies use differing metrics, making comparisons difficult without rerunning analyses.Source: §5 · p. 9

Exploiting multimodality in toxic content detection

  • The authors report that generic multimodal toxic-content filtering approaches suitable for datasets such as LAION were not found, and that filtering without contextual analysis can erase marginalized experiences.They argue that image- and text-focused filtering should be better connected, while noting that no filtering alone is a bulletproof solution.Source: §5 · p. 10

Limitations of pysentimiento

  • The audit used pysentimiento, but its variability across English variants and specific topics had not, to the authors’ knowledge, been researched.They encourage checking whether the hate-scaling finding holds with other hate-detection models.Source: §5 · p. 10

Legal and policy implications

  • The authors present the audit as an initial document for legal and policy experts on multimodal datasets and the negative implications of scale.They connect the datasets’ role in machine-learning systems with questions of consent, rights, evaluation, maintenance, responsibility, and accountability.Source: §5 · p. 10

6 Conclusion

The paper audits LAION-400M and LAION-2B-en and finds that hateful, aggressive, and targeted content is exacerbated by dataset size. It emphasizes open access for audits and calls for due diligence on datasets and models used in real-world systems.

6 Conclusion

  • The audit finds hateful, aggressive, and targeted content in LAION-400M and LAION-2B-en, with such content exacerbated by dataset size.The authors say their analysis of alt-text complements dataset creators’ image-focused analyses.Source: §6 · p. 10
  • The authors argue that open access to datasets and models is essential because quantitative and qualitative dataset exploration hinges on access to the artifacts.They note that an increasing number of machine-learning organizations fail to provide such access.Source: §6 · p. 10
  • The authors urge rigorous audits and due diligence because vision-language models are deployed in real-world systems despite biases and stereotypes that are not well understood or documented.They include recommendation, information-retrieval, semantic-search, and image-captioning systems, and call for necessary action, including refraining from high-stakes use.Source: §6 · p. 10

A The risks of extrapolation

Figure A1 concerns confidence-interval analysis of HCR underestimation when LAION-400M statistics are used.

A The risks of extrapolation

  • Figure A1 presents binomial proportion confidence-interval analysis to establish HCR underestimation when using LAION-400M statistics.The supplied caption gives the figure’s purpose but no values or comparison outcome.Source: Figure A1 · p. 15

B NSFW Analysis

Table A1 reports summary values for harmful-content categories in the LAION-2B-en sample, divided by the suggested NSFW threshold.

B NSFW Analysis

  • Table A1 reports mean, minimum, and maximum values for hateful, targeted, and aggressive content, divided according to the suggested NSFW threshold of 0.5.The caption supplies no cell values or comparison outcome.Source: Table A1 · p. 15

C The origins of the dataset scaling laws: A cartoon sketch emerges

A widely repeated dataset-scaling sketch describes three phases in how generalization error relates to training-set size [29]. The account distinguishes an initial small-data region, a power-law decline, and an eventual error floor.

C The origins of the dataset scaling laws: A cartoon sketch emerges

D Blackbox non-reproducible empirical results

The section questions the reproducibility of empirical evidence used to support dataset scaling. It examines ALIGN’s reported comparison and notes that the experiments have not been replicated elsewhere.

D Blackbox non-reproducible empirical results

  • ALIGN [33] claimed that corpus scale could compensate for noise and produce state-of-the-art representations with a simple learning scheme.The paper’s abstract makes this claim; the section characterizes its empirical support as black-box and non-reproducible.Source: D Blackbox non-reproducible empirical… · pp. 15–16
  • At 3M samples, ALIGN performed much worse than cleaned CC-3M; at 6M and 12M, its model quality rapidly caught up.The comparison used a B7+BERT-base model and randomly sampled ALIGN training data.Source: D Blackbox non-reproducible empirical… · p. 16
  • The section reports that ALIGN data outperformed Conceptual Captions at only 4x the size, but says these experiments have not been replicated elsewhere.The authors question whether the reported scaling ratios hold beyond the cited experiments.Source: D Blackbox non-reproducible empirical… · p. 16

E The tactical template: Fuzzy main section meets non-existent appendices

The section describes a recurring way prominent generative-model papers handle training-data disclosure: sparse dataset details in the main text, risk discussion elsewhere, and an appendix that may not provide construction details. It also contrasts this pattern with BASIC, whose paper did not address safety or dataset auditing.

E The tactical template: Fuzzy main section meets non-existent appendices

  • The described template begins with a small main-text subsection giving minimal dataset sample counts and cross-references to other black-box datasets.Parti and Imagen are cited as using Section 4.1 for this purpose.Source: E The tactical template: Fuzzy main… · p. 16
  • Broader- or societal-impact sections are described as discussing potential dataset problems and downstream risks while citing earlier audits such as [7].The passage gives Parti’s Section 8 and Imagen’s Section 6 as examples.Source: E The tactical template: Fuzzy main… · p. 16
  • The template then points readers to an appendix that may be absent or may omit how the dataset was constructed and sourced.The passage notes that reviewers may not be required to inspect the appendix.Source: E The tactical template: Fuzzy main… · p. 16
  • Unlike the described template, the BASIC model paper is said to omit discussion of model safety and dataset auditing despite using the largest image-text dataset then assembled.The authors also note the paper’s 47-page length and three ArXiv revisions.Source: E The tactical template: Fuzzy main… · p. 16
  • Figure A2 presents Google’s template for (non)declaring training-dataset information alongside paper screenshots.Source: Figure A2 · p. 17

Related papers

Loading 2311.03449v1…