PaperlayerView exact revision on arXiv

Source-linked AI summary

Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection

Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, Noah A. Smith

arXiv:2201.10474v2cs.CLcs.AI

TL;DR

Language-model training draws on large web corpora that promise diversity but contain undesirable material, while the reasons for selecting some texts as high quality are rarely explicit. The authors analyze the GPT-3 quality filter using U.S. high school newspapers and find it favors some school and regional profiles, with no difference in quality distributions between high- and low-factuality news sources.

  • Problem

    Web dumps offer diverse text but also contain undesirable content, and text-selection criteria affect systems built on language models despite rarely being explicitly justified.

    Source: §1 · p. 1

  • Method

    The authors analyze a replicated GPT-3 quality filter on U.S. high school newspaper articles associated with demographic metadata.

    Source: §3 · p. 3, §3.2 · p. 4

  • Results

    The filter favors newspapers from larger schools and wealthier, more educated, urban areas, and quality distributions do not differ between high- and low-factuality news sources (p = 0.085, two-way Kolmogorov-Smirnov test).

    Source: §1 · p. 1, §4.2 · p. 7, §8 · p. 10

  • Takeaways & Limitations

    Text selection entails a language ideology, so practitioners should document selection criteria and be more intentional about sources and underrepresented authors.

    Source: §1 · p. 2, §5 · p. 9

    The dataset is a niche genre by U.S. students, primarily representing U.S.-centric cultural and political perspectives.

    Source: §6 · p. 9

Abstract

from arXiv · show

Language models increasingly rely on massive web dumps for diverse text data. However, these sources are rife with undesirable content. As such, resources like Wikipedia, books, and newswire often serve as anchors for automatically selecting web text most suitable for language modeling, a process typically referred to as quality filtering. Using a new dataset of U.S. high school newspaper articles -- written by students from across the country -- we investigate whose language is preferred by the quality filter used for GPT-3. We find that newspapers from larger schools, located in wealthier, educated, and urban ZIP codes are more likely to be classified as high quality. We then demonstrate that the filter's measurement of quality is unaligned with other sensible metrics, such as factuality or literary acclaim. We argue that privileging any corpus as high quality entails a language ideology, and more care is needed to construct training corpora for language models, with better transparency and justification for the inclusion or exclusion of various texts.

1 Introduction

Web data promises diversity but also contains substantial undesirable content, so filters select material for training; the criteria behind these selections are value judgments with consequences for whose language enters models. The paper examines the GPT-3 filter and finds its preferences track topical, stylistic, and demographic patterns.

Why quality filtering matters

  • Web dumps promise more diverse text than curated resources but contain boilerplate, code, pornography, and hate speech, prompting automated quality filters to remove undesirable content.The filters include code removers, heuristics, stopword rules, and classifiers.Source: §1 · p. 1
  • Because language varies with social and demographic factors, filtering text as high quality also raises the question of whose language is excluded.The paper frames quality filtering as a value judgment rather than a neutral preprocessing step.Source: §1 · p. 1

Study and findings

  • The GPT-3 quality filter favors newspapers from larger schools and from regions with higher educational attainment, urban centers, and higher-valued homes.The authors also report strong topical and stylistic preferences.Source: §1 · p. 1

Quality and language ideology

  • The term “quality” is often left undefined in NLP, even when papers identify Wikipedia as an example of a high-quality source.The authors argue that choosing a corpus entails a language ideology and that no corpus is truly general-purpose.Source: §1 · pp. 1–2

2 Motivation: Data Sources

The section surveys the limited, often skewed sources that underpin language-model training corpora. Across Wikipedia, books, and popular web content, source concentration and representation patterns constrain whose language and perspectives appear.

Wikipedia

  • Wikipedia has authorship and content biases, including a persistent gender imbalance among editors, limited rural coverage, and Anglo-American perspectives.A relatively small number of people are responsible for most of its content.Source: §2 · p. 2

Books

  • The Toronto BookCorpus used for BERT contains 7,185 self-published novels and has been criticized for imbalanced representation and inadequate documentation.The cited criticisms also include copyright violation and poor quality control.Source: §2 · p. 2
  • Books3 is not a random selection; preliminary analysis finds its most prevalent authors are American and British writers, especially in romance, mystery, and children’s books.The passage notes that the American publishing industry has been criticized as homogeneous.Source: §2 · p. 2

OpenWebText

  • OpenWebText is a replica of GPT-2’s corpus, built from websites featured in popular Reddit submissions; the section notes Reddit users skew male, younger, and liberal.The links tend to appear on highly trafficked subreddits.Source: §2 · pp. 2–3
  • Just 1% of OpenWebText’s 311K unique top-level domains contribute 75% of its documents.Its most common sites include major international news outlets, blogs, sports sites, and technology news.Source: §2 · p. 3

Summary

  • Existing language-model sources disproportionately represent relatively powerful social positions and are less inclusive of women and marginalized groups.Rural, non-dominant gender, sexual, or racial identities and counter-hegemonic perspectives are also less likely to appear.Source: §2 · p. 3

Implications

  • Formal, streamlined sources may omit writing styles and topics important to language technologies’ user communities, even when those texts are less formal or professionally edited.The passage states such writing can remain high quality and essential to users’ communicative lives.Source: §2 · p. 3

3 Measuring the Language Ideology of the GPT-3 Quality Filter

This section evaluates a replicated GPT-3 quality filter on U.S. high-school newspaper articles, relating article scores to document features and school-area demographics. It finds higher scores for larger schools and for schools in wealthier, more educated, urban locations.

3.1 U.S. SCHOOL NEWS

  • The corpus contains 910K articles from 1,410 U.S. high schools, but it is not a representative sample and is limited to evaluation because authors and subjects are minors.The articles span 2010–2019; the corpus covers schools using a particular website template with retrievable archived content.Source: §3.1 · p. 4

3.2 The GPT-3 Quality Filter

  • The replicated GPT-3 quality filter is a binary logistic regression using n-grams to distinguish Books3, Wikipedia, and OpenWebText from random Common Crawl.The three reference corpora form the positive class, and Common Crawl forms the negative class.Source: §3.2 · p. 4

3.3 Document-Level Analysis

  • School newspaper articles are more often classified as low quality than general newswire across categories, especially school announcements.Figure 1 compares histograms built from 10K random documents from each domain.Source: §3.3 · p. 5, Figure 1 · p. 5
  • In U.S. SCHOOL NEWS, documents entirely about Trump and the presidential election score 35 percentage points higher on average, and sports documents 25 points higher, relative to omitted food; first- or second-person pronouns lower scores 5 points, while doubling tokens raises them 9.Source: §3.3 · p. 5, Table 2 · p. 5

3.4 Demographic Analysis

  • Schools in wealthier, urban, more educated ZIP codes and Democrat-leaning counties tend to have higher average quality scores.These are correlations that informed the regression analysis.Source: §3.4 · p. 6, Figure 2 · p. 6
  • A 14-point urban-population rise, 17-point college-educated-adult rise, doubled home values, or quadrupled school size each correspond to a 1-point score increase; public schools score 1.5 points above private schools.These are individual associations; the combined factors account for large score differences across locations.Source: §3.4 · p. 6
  • Controlling for other factors, larger schools and wealthier, more educated, urban locations receive higher quality scores, while rural schools score below urban schools.The reported regression associates these factors with quality scores; the table caption summarizes the demographic regression.Source: §3.4 · p. 6, Table 3 · p. 7
  • The demographic model explains relatively little score variance, uses features from a single time point, and may not generalize beyond this domain.The authors also note possible school-name and ZIP-code alignment errors and that adding features could change the findings.Source: §3.4 · p. 7

4 Alignment with Other Notions of Quality

The section tests whether the GPT-3 quality filter aligns with factuality, essay scores, and literary awards. Across these standards, the filter’s quality judgments conflict with the alternatives examined.

4.1 Data

  • The authors compare filter judgments with human-assigned news factuality, TOEFL essay scores and prompts, and Pulitzer-winning books.They score 9.9K articles from high-factuality outlets and 7.7K from low-factuality outlets, 12.1K TOEFL essays, and Pulitzer-winning books from Books3 and Gutenberg.Source: §4 · p. 7, §4.1 · p. 7

4.2 Results

  • The filter shows no difference in quality scores between articles from news sources rated high and low in factual reliability.Many articles from factually unreliable sources are still considered high quality by the filter.Source: §4.2 · p. 7, Figure 3 · p. 8
  • Among Pulitzer Prize-winning works, the filter favors nonfiction and longer fiction over poetry and dramatic plays.Poetry and drama are also less favored than fiction and fan fiction from BookCorpus.Source: Figure 4 · p. 8, §4.2 · p. 8
  • TOEFL essay prompts predict quality designations more strongly than essay scores do.Essay scores are weakly correlated with quality scores (Pearson r = 0.12, p < 0.001); Prompt 4 essays about products and advertisements receive the highest quality designations.Source: Table 4 · p. 8, §4.2 · p. 8
  • The GPT-3 quality filter conflicts with other standards of text quality.The authors note that the alternative standards also reflect language ideologies, so none is a neutral universal measure.Source: §4.2 · p. 8

5 Discussion

The discussion argues that text selection can exclude sources in undesirable ways, while no single universal standard can define high-quality text. It calls for more deliberate, transparent, and inclusive corpus construction.

5 Discussion

  • The authors argue there is no simple, universal standard for high-quality text because appropriate text depends on use cases.Some selection is inevitable, but current methods may exclude sources counterintuitively or undesirably.Source: §5 · pp. 8–9
  • Researchers can intentionally design systems to reduce the risk of excluding language from underprivileged identities or communities.The authors connect this concern to a history of privileging some people’s language over others.Source: §5 · p. 9
  • NLP practitioners can document selection criteria and explain source choices, while testing the implications of different filters.The authors suggest that text best suited for training may depend on the model’s application.Source: §5 · p. 9
  • The authors recommend more inclusive data collection, including underrepresented viewpoints, diverse genres, and more intentional exclusion criteria.They also suggest interrogating exclusions, checking for prominent inclusion biases, or abandoning the idea of a general-purpose corpus.Source: §5 · p. 9

6 Ethical Considerations & Limitations

The authors identify limits in the school-newspaper sample and its demographic mapping, and describe ethical constraints on collecting and releasing articles.

6 Ethical Considerations & Limitations

  • Sampling depends on a specific publication template and uneven state coverage; the corpus likely misses both the least-resourced and wealthiest schools.The authors speculate that including those schools might continue the reported trends.Source: §6 · p. 9
  • The dataset is a niche genre by U.S. students for local consumption, so it primarily represents U.S.-centric cultural and political perspectives.The authors also acknowledge perpetuating biases by studying English-language text from the United States and call for work in multilingual settings, other domains, and with different authors.Source: §6 · p. 9
  • ZIP codes and counties are imperfect school identifiers, and tracking demographic and author metadata at scale may entail invasive surveillance.The authors point to future work on protecting authors’ rights as data subjects and producers while mapping representation.Source: §6 · p. 9
  • Because the authors did not seek consent and ethical and legal norms around scraping public-facing data, especially minors’ work, remain in flux, they do not release the article corpus.They use the articles only for analysis and evaluation, releasing school-level demographic variables and quality scores for reproducibility.Source: §6 · p. 9

7 Related Work

Related work connects language ideologies to NLP practices and data selection, while raising concerns about institutional archives, filtering tradeoffs, and downstream behavior.

7 Related Work

  • Language ideologies that privilege some language varieties as inherently correct, clear, or objective can function as mechanisms of linguistic discrimination.Prior NLP examples include normalization that strips pragmatic nuance and social signals, and toxicity detection whose behavior aligns with annotator attitudes toward African-American dialects (Sap et al., 2021).Source: §7 · pp. 9–10
  • Institutional archives are an alternative to laissez-faire collection, but their practices may also privilege colonial voices over colonized ones.Jo and Gebru (2020) recommend archival practices; Trouillot (1995) and Decker (2013) provide the example of historical documentation preferences.Source: §7 · p. 10
  • Other filters also encode quality ideologies: bad-word filters disproportionately exclude language about and by minority groups, while popularity filtering leads to data homogeneity.Lightweight filters may emphasize document length over factuality, and the authors call for transparent justification of filtering tradeoffs.Source: §7 · p. 10
  • Prior work reports worse dialogue-agent performance on conversations about race and minority English dialects, GPT-3 stereotypes and harmful outputs, and degraded downstream performance under aggressive filtering.The authors identify closer analysis of how data-selection ideologies relate to model behavior as future work.Source: §7 · p. 10

8 Conclusion

The conclusion reports that the quality reference corpora favor school-newspaper content from larger schools in wealthier, more educated, urban U.S. areas. It calls for more intentional curation and documentation of possible exclusions.

8 Conclusion

  • The quality reference corpora implicitly favor content from larger schools in wealthier, more educated, and urban areas of the United States.The finding comes from applying the filter to a new dataset of U.S. school newspapers.Source: §8 · p. 10
  • The authors recommend intentionally curating text from underrepresented authors and groups, drawing from multiple genres and writing styles, and documenting curation procedures and possible exclusions.They note that no single action will solve the issue.Source: §8 · p. 10

A.1 Language Model Training Corpora

The appendix points readers to a table listing popular language-modeling corpora.

A.1 Language Model Training Corpora

  • Table 5 displays a list of popular language-modeling corpora.The passage does not specify the listed corpora.Source: §A.1 · p. 15

A.2 Datasheet

The U.S. SCHOOL NEWS datasheet is linked.

A.2 Datasheet

  • The U.S. SCHOOL NEWS datasheet is available at the linked URL.https://bit.ly/3rLrmwVSource: §A.2 · p. 15

A.3 Quality Filter Hyperparameters

Appendix analyses describe the filter setup and examples, then test topic, demographic, factuality, and TOEFL relationships. Robustness checks largely preserve the main demographic findings, with qualifications around home values and political covariates.

A.7 Additional Regressions

  • The race-adjusted regression largely matches the main findings, but home values are no longer significant and Asian is the only significant racial category.Percentage of Asian students correlates with median home values (Pearson r =0.32, p < 0.001), potentially absorbing part of the wealth association.Source: §A.7 · p. 15, Table 8 · p. 18
  • Including 2016 GOP vote share yields a strong negative association with quality scores, while home values and rurality are no longer significant.The authors retain the model without GOP vote share because voting may be idiosyncratic and correlated with rurality; they consider wealth and geography more plausible explanations than parents’ political preferences.Source: §A.7 · p. 15, Table 9 · p. 18

A.3 Quality Filter Hyperparameters

  • The appendix reports the logistic-regression hyperparameters for its reproduction of the GPT-3 quality filter in Table 6.The table summarizes the hyperparameter search space and best assignments.Source: §A.3 · p. 15, Table 6 · p. 16

A.5 Topic Modeling

  • For 10K U.S. SCHOOL NEWS opinion pieces, the filter prefers topics more prevalent in Wikipedia or newswire.The appendix points to a topic-level quality-score distribution.Source: §A.5 · p. 15, Figure 5 · p. 16

A.6 Demographic Features

  • The appendix reports displaying the demographic features used in the analyses in Table 7.The passage identifies the table as a description of features included in the demographic analyses.Source: §A.6 · p. 15, Table 7 · p. 17

A.9 TOEFL Exam Responses

  • TOEFL exam scores correlate weakly with filter quality scores across prompts (Pearson r=0.12 ± 0.05, p ≈0), while prompts appear more predictive.The appendix also orders TOEFL prompts by their quality scores.Source: Figure 6 · p. 18, Table 10 · p. 19

A.4 Example Articles

  • Table 11 presents example student newspaper articles, noting that lower-rated student-life pieces often differ in style from higher-rated documents.The appendix also points readers to examples and their quality scores.Source: §A.4 · p. 15, Table 11 · p. 19

A.8 Low Factuality News Considered High Quality

  • The GPT-3 filter assigns high quality scores to examples from low-factuality news sources, including articles containing COVID disinformation.The table caption identifies the sources as low factuality according to MediaBiasFactCheck.com.Source: §A.8 · p. 15, Table 12 · p. 19

Related papers

Loading 2201.10474v2…