Source-linked AI summary
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Abeba Birhane, Vinay Uday Prabhu, Emmanuel Kahembwe
TL;DR
Large internet-scale multimodal datasets raise concerns about curation, alt-text quality, problematic content, and entrenched bias. This paper audits LAION-400M, a CLIP-filtered CommonCrawl-derived image-alt-text dataset, and finds explicit content, stereotypes, slurs, and unreliable filtering. It argues that the dataset’s release creates serious concerns about downstream harms, data rights, and the feasibility of moderation at this scale.
Problem
Existing large-scale multimodal datasets raise concerns about dubious curation, poor alt-text, problematic CommonCrawl content, and opaque visio-linguistic-model biases.
Method
The paper audits LAION-400M through qualitative search experiments and quantitative analysis of its image-alt-text metadata and filtering behavior.
Results
The audit found explicit rape and pornographic imagery, misogynistic and political stereotypes, racist and ethnic slurs, and offensive text paired with seemingly benign images.
Takeaways & Limitations
The findings expose serious concerns about open large-scale dataset release, downstream harms, data rights, and the responsibilities of AI researchers, regulators, policymakers, and data subjects.
Takeaways & Limitations
Automatic filtering is unreliable in unconstrained visual domains and can censor or erase marginalized experiences without careful contextual analysis.
Abstract
from arXiv · showhide
We have now entered the era of trillion parameter machine learning models trained on billion-sized datasets scraped from the internet. The rise of these gargantuan datasets has given rise to formidable bodies of critical work that has called for caution while generating these large datasets. These address concerns surrounding the dubious curation practices used to generate these datasets, the sordid quality of alt-text data available on the world wide web, the problematic content of the CommonCrawl dataset often used as a source for training large language models, and the entrenched biases in large-scale visio-linguistic models (such as OpenAI's CLIP model) trained on opaque datasets (WebImageText). In the backdrop of these specific calls of caution, we examine the recently released LAION-400M dataset, which is a CLIP-filtered dataset of Image-Alt-text pairs parsed from the Common-Crawl dataset. We found that the dataset contains, troublesome and explicit images and text pairs of rape, pornography, malign stereotypes, racist and ethnic slurs, and other extremely problematic content. We outline numerous implications, concerns and downstream harms regarding the current state of large scale datasets while raising open questions for various stakeholders including the AI community, regulators, policy makers and data subjects.
1 Introduction
Large-scale multimodal datasets combine vast web-sourced image, alt-text, and textual data, but their construction inherits serious curation, accessibility, privacy, and bias concerns. LAION-400M extends this landscape while raising questions about openness, downstream harms, and responsible data governance.
- 1 Introduction: Multimodal AI seeks cross-domain models that jointly represent vision, text, and speech using internet-scale datasets.This shifts data curation toward web-sourced images, alt-text pairs, and corpora such as CommonCrawl.
- 1 Introduction: Large vision datasets have documented curation, gender, racial, geographical, privacy, and problematic-content issues, prompting dataset retractions, image blurring, and label removal.Imagenet curators advocated removing 2674 of 2832 person-subtree synsets.
- 1 Introduction: Alt-text on the web is often poorly written or sparse, despite its intended role in enabling screen-reader descriptions for blind and low-vision users.High-traffic websites reportedly have about 50% alt-text coverage, while only 0.1% of 9.2 million analyzed tweets contained alt-text.
- 1 Introduction: CommonCrawl’s scale makes it attractive for AI training, but audits found 155 million of 1 billion examined URLs in sensitive categories and over 300,000 documents containing hate speech and racism.The April 2021 archive was roughly 320 TB and spanned 3.1 billion pages.
- 1 Introduction: LAION-400M contains hundreds of millions of CommonCrawl-derived image-alt-text pairs filtered with CLIP, bringing these concerns into an openly accessible dataset.The paper frames its audit as an examination of the dataset’s contents and release implications.
2 LAION-400M
LAION-400M is an open, CommonCrawl-derived collection of hundreds of millions of image-text-metadata tuples filtered by CLIP similarity. The authors’ audit found explicit imagery, misogynistic and political stereotypes, and unreliable NSFW filtering, including offensive text paired with benign images.
- 2 LAION-400M: LAION-400M contains 413871335 image-text-metadata tuples collected from random web pages crawled between 2014 and 2021.Raw image-alt-text pairs with CLIP image-text cosine similarity below 0.3 were filtered out.
- 2.1 Misogyny and stereotypes: The dataset’s search portal returned NSFW imagery, including sexual violence, for many non-NSFW queries linked to femininity, relationships, and demographic terms.The audit reports explicit rape imagery and photoshopped images of female celebrities among the results.
- 2.1 Misogyny and stereotypes: Search results risked amplifying hyper-sexualized and misogynist representations alongside Anglo-centric, Euro-centric, and potentially White-supremacist ideologies.Queries included school, appearance, occupation, nationality, presidents, terrorism, and white power.
- 2.2 Search engine bias?: 34%, 16.4%, and 28.2% of matches for Desi, Nun, and Latina respectively contained listed NSFW terms.The corresponding match counts were 34516, 16766, and 37769.
- 2.3 Offensive text. Benign imagery: Seemingly benign images were sometimes paired with NSFW and offensive text tags, exposing a cross-modal failure that image-only inspection could miss.The authors connect this phenomenon to the need for joint image-and-text filtering.
3 How did we get here?
LAION-400M emerged from a shift toward web-crawled, opaque multimodal datasets and relied on CLIP-based similarity filtering. The paper shows that this pipeline can admit offensive associations while excluding accurate descriptions.
- 3.1 Dataset-scale shift: Large-scale multimodal initiatives replaced handheld curation with web crawling, producing datasets of hundreds of millions or billions of image-text pairs.CLIP used 400 million image-text pairs, while ALIGN created a billion-sized image-alt-text dataset.
- 3.2.1 The "scale beat noise" discourse: The “scale beats noise” rationale treats irrelevant captions as noise that scale will average out, while dismissing expensive filtering and post-processing.The paper frames this as a widespread belief driving initiatives such as LAION-400M.
- 3.3 Filtering with CLIP: LAION-400M filters image-alt-text pairs by dropping those with CLIP cosine similarity below 0.3, despite CLIP’s documented biases and problematic associations.The paper also identifies data incest in using CLIP as the filtering model.
- 3.3.3 Cosine-similarity thresholding: For Eileen Collins, CLIP scored a benign astronaut description at 0.28 and a misogynistic housewife description at 0.31, potentially favoring the latter during filtering.The authors attribute this possibility to gender biases built into CLIP.
- 3.3.3 Cosine-similarity thresholding: For Barack Obama’s portrait, the factual description scored below 0.3 while the conspiracy-theoretic description scored above the threshold.These corner cases illustrate how model mis-associations can potentially amplify selection bias toward offensive samples.
4 The asymmetries of course-correction
Large-scale dataset creation is easier to expand than to correct, and detoxification demands substantial technical and human labor. The paper emphasizes that ethical problems often receive slower attention than performance improvements, while cleanup can impose emotional trauma.
- 4.1 Asymmetry of effort: Web mining and aggregation have become highly democratized, but subsequent filtering and model detoxification require markedly greater effort.Common-Crawl tools and asynchronous I/O libraries simplify large-scale collection.
- 4.1 Asymmetry of effort: Specialized image, text, and joint filtering modules have been used in other datasets to address problematic multimodal content.These modules include part-of-speech, sentiment, pornography, and profanity annotations.
- 4.2 Asymmetry of advances: More than 2700 ImageNet-21k synsets were recommended for removal, yet unfiltered versions continued to appear in datasets, repositories, models, and contests.The authors use this history to stress the difficulty of detoxifying much smaller datasets than LAION-400M.
- 4.3 Asymmetry of labour: Dataset cleanup involves emotional trauma in addition to technical labor.Researchers reported discomfort, nausea, and headaches while curating and strategically blurring NSFW imagery.
5 Discussions and open questions
The discussion argues that LAION-400M’s accessibility exposes serious risks involving vulnerable groups, illicit content, data rights, multimodal stereotyping, and unreliable large-scale filtering. It raises questions about accountability, auditing, licensing, and whether the dataset’s harms outweigh its democratization benefits.
- LAION-400M’s openness may expose vulnerable individuals and communities to downstream harms from datasets and models trained on internet-scale data.The discussion emphasizes that democratization must account for rights, welfare, and interests beyond access for AI researchers.
- The dataset contains harmful stereotypes, sexualization, fetishization, racist content, sexual abuse imagery, rape imagery, and non-consensual explicit material.These problems affect representations of women, Black women, Asian women, ethnic minorities, children, and other marginalized groups.
- Large multimodal models may memorize illegal samples or encode illicit content in neurons, raising questions about model inversion and illicit data distribution through model weights.The discussion notes that memorization can occur even when a sample appears only once in the training data.
- LAION-400M’s licensing and scraping practices raise unresolved questions about copyright, privacy, consent, compensation, deletion, and responsibility for removing data.The dataset is released under CC-BY 4.0 while its images remain under copyright, and removal responsibility is delegated to users.
- The authors question whether unrestricted release and commercial use should continue, proposing restrictive research-oriented licensing, independent auditing, and better contextual analysis.They also call for scrutiny of private BigTech datasets because automatic filtering may produce similar problems.
- Automated filtering is unreliable at scale and may censor marginalized experiences, while existing methods may be unable to clean datasets of this size.At a 0.1% incidence rate, a billion-image dataset would contain one million images of minors being sexually assaulted.
6 Conclusion
The conclusion presents LAION-400M as a rare window into the risks of large-scale multimodal datasets and argues for continued research access alongside stronger scrutiny and restrictions. It rejects simple retraction as sufficient and calls for thoughtful audits of both open and private datasets.
- LAION-400M provides first-hand insight into the challenges of handling multimodal visio-linguistic datasets at scale.
- The authors commend the dataset’s grassroots openness while emphasizing serious problems in its release and dissemination.
- The project demonstrates serious CLIP failures and dangers in building semantic search engines with this technology.
- Retraction alone is inadequate because withdrawn datasets can remain available through file-sharing sites, derivatives, and continued research use.The authors instead favor more restrictive licensing to limit non-research use while preserving access for study.
- Large institutions should open private datasets to thoughtful internal and external audits because their potential harms may outweigh competitive advantages.The conclusion also states that the community does not yet fully understand the risks of such datasets.
Warning:
The warning identifies blurred NSFW images and associated offensive textual content.
- Blurred NSFW images are presented with associated offensive textual content.
Appendix A A glimpse into the abyss
The appendix presents hand-blurred collages of screenshots from search-engine queries of LAION-400M. Figure 4 compares results for the queries “Big” and “Small.”
- The appendix contains collages of hand-blurred screenshots from the search-engine-query exercises in Section 2.1.
- Figure 4 compares blurred LAION-400M search results for “Big” and “Small” in panels (a) and (b), respectively.
Appendix B The curious case of "neuron" 1543
The appendix examines how CLIP-related visualizations and LAION-400M retrievals expose problematic associations between image content, captions, and text queries. Neuron-1543 produces phallic activation-maximizing imagery and is associated with an NSFW image landscape in triggering datasets.
- Activation maximization: Activation maximization iteratively changes a white-noise image to maximize a neural-network unit’s activation.The method uses gradient ascent to visualize preferred stimuli.
- LAION-400M retrievals: LAION-400M retrieval examples query gendered, nationality-related, political, and explicitly stereotyped terms, including Asian, Indian, Nigerian, Desi, Nun, Latina, and White power.The figures document retrievals for these terms through blurred screenshots or collages.
- Neuron-1543: Neuron-1543 is examined in the CLIP-Resnet-50-4x model’s image_block_4_5_Add_6_0 layer using activation-maximizing images and text.The figure pairs feature visualization with text that maximizes the neuron’s dot product.
- Neuron-1543: Neuron-1543’s activation-maximizing image contains vividly phallic artifacts, while highly activating ImageNet and YFCC images form a vividly NSFW landscape.The associated text visualization reports text values that maximize the neuron’s dot product.
- LAION-400M retrievals: Figure examples also include unrelated image-caption pairs, illustrating a mismatch between captions and visual content.The mismatch is presented alongside the retrieval and feature-visualization examples.