NLPositionality: Characterizing Design Biases of Datasets and Models

• Researchers’ lived experience, identity, culture, and background shape design choices that can make NLP systems work differently across populations.Source: §1 · pp. 1–2
• NLPositionality re-annotates data with diverse LabintheWild volunteers and compares demographic agreement with dataset labels or model predictions.Source: §1 · p. 2
• The framework supports continuous annotation collection over long periods and participant learning through comparing their responses with AI systems and similar demographic groups.Source: §1 · p. 2
• Across two tasks, datasets and models aligned most with White, educated young people from English-speaking countries, and datasets closely aligned with their original annotators.Source: §1 · p. 2

• A person’s positionality comprises perspectives shaped by demographics, identity, and life experiences.Source: §2 · p. 2
• Cambo and Gergle (2022) define model positionality as a model’s social and cultural position relative to stakeholders; the authors extend the concept to datasets.Source: §2 · pp. 2–3
• Design biases arise when choices based partly on researchers’ positionality make datasets or models work better for some populations than others.Source: §2 · p. 3
• Following Friedman and Nissenbaum (1996), social biases pre-exist in society, whereas design biases emerge from computing systems.Source: §2 · p. 3
• Prior examples include NLP’s English emphasis and toxicity blocklists that can marginalize minority voices.Source: §2 · p. 3

3 NLPositionality: Quantifying Dataset and Model Positionality

• The framework compares demographic responses on identical instances with dataset labels or model predictions, using demographics as an initial proxy for positionality.Source: §3 · p. 3

3.1 Collecting Diverse Annotations

• LabintheWild provides a diverse participant pool and motivates participation through opportunities to learn how one’s judgments compare with AI systems and demographically similar people.Source: §3.1 · pp. 3–4
• For each task, the study selects 300 instances through stratified sampling across relevant dataset metadata and re-annotates them under instructions similar to the original task.Source: §3.1 · p. 4

3.2 Quantifying Positionality

• For each demographic, the analysis aggregates same-instance annotations by their mean, computes Pearson’s r against dataset labels or model predictions, applies Bonferroni correction, and ranks alignment.Source: §3.2 · p. 4

4.1 Social Acceptability

• For social acceptability, 612 participants annotated 300 Social Chemistry examples, producing 11,294 annotations compared with the dataset, Delphi, and GPT-4.Source: §4.1 · p. 5

4 Case Studies

• Across both tasks, datasets and models generally aligned more with English-speaking, college-educated, White, younger groups, though some also correlated strongly with other populations.Source: §4.1 · p. 5, §4.2 · p. 7

4.1 Social Acceptability

• Social Chemistry’s highest reported correlations were English-speaking (r = 0.76∗), college-educated (r = 0.74∗), White (r = 0.73∗), and ages 20-30 (r = 0.74∗).Source: §4.1 · p. 5
• Delphi showed a similar but weaker pattern, while also correlating more with Baltic-country-raised people (r = 0.72∗) than English-speaking groups; GPT-4 showed a similar pattern.Source: §4.1 · p. 5
• Non-binary people aligned less with Social Chemistry, Delphi, and GPT-4; professional-school-educated people and Black, Latinx, and Native American populations ranked least in their respective categories.Source: §4.1 · p. 7

4.2 Hate Speech Detection

• For hate speech detection, 484 participants annotated 300 Dynahate examples, producing 5,005 annotations compared with the dataset, two APIs, HateRoBERTa, and GPT-4.Source: §4.2 · p. 7
• In hate speech detection, Dynahate, Perspective API, Rewire, HateRoBERTa, and GPT-4 showed alignment with Western-associated groups, while some systems also correlated with other populations.Source: §4.2 · p. 7
• Non-binary people aligned less across hate-speech datasets and models, and professional-school-educated, Black, Latinx, and Native American groups ranked least in their categories.Source: §4.2 · p. 7

5 Discussion

• Across tasks, datasets, and models, statistically significant moderate correlations appear with Western, educated, White, and young populations.The strengths of these correlations vary.Source: §5 · p. 8
• Non-binary people and people who are not native English speakers consistently rank lowest in alignment with datasets and models across both tasks.The passage also identifies Black, Latinx, and Native American people among populations historically marginalized by technological systems.Source: §5 · p. 8

Datasets Tend to Align with Their Annotators

• Social Chemistry and Dynahate datasets align with demographic characteristics reported for their annotators.Social Chemistry annotators were largely women, White, college-educated, ages 30–39, and from the U.S.; Dynahate annotators were mostly women, White, ages 18–29, native English speakers, and British.Source: Datasets Tend to Align with Their… · p. 8
• Researchers should document design choices and report their positionality and assumptions to improve reproducibility and clarify decision rationales.The authors also recommend documenting annotator demographics when recruiting diverse annotators, since that recruitment process can introduce new design biases.Source: Datasets Tend to Align with Their… · p. 9
• Researchers should center harmed communities through participatory or value-sensitive design, experiential-expert panels, and diverse research teams.These are presented as ways to reduce design biases.Source: Datasets Tend to Align with Their… · p. 9
• Inclusive NLP need not make every technology work for everyone; specialized and culturally adaptive systems can uplift underrepresented communities.The authors also urge sharing disaggregated annotations and using methods that handle disagreement rather than forcing a single label.Source: Datasets Tend to Align with Their… · p. 9

6 Conclusion

• NLPositionality quantifies dataset and model positionality by collecting annotations from diverse global annotators and comparing them with dataset labels and model predictions.The conclusion describes re-annotating a sample of task data before assessing alignment.Source: §6 · p. 9
• In social acceptability and hate speech detection, examined datasets and models align better with Western, White, college-educated, and younger populations.The conclusion presents these findings as evidence of design biases and positionality.Source: §6 · p. 9
• The authors conclude that NLP needs more inclusive datasets and models so research can benefit all people.This is the conclusion's stated aim, not a demonstrated outcome of the case studies.Source: §6 · p. 9

Positionality measures

• Demographics may not capture positionality reliably because beliefs can vary within demographic groups; individual attitudes or beliefs may be a better measure for future work.Source: §7 · pp. 9–10

Participant sample

• The LabintheWild sample is not representative of the world’s population because participation requires internet access and may be affected by self-selection.Source: §7 · p. 10

Analytical and task scope

• Pearson’s r does not account for interactions between demographics, group averages may mask individual variation, and small groups may limit generalizability or risk tokenizing marginalized communities.The analysis covers only two tasks with relatively straightforward annotation schemes; tasks needing more annotator explanation may be harder to generalize to.Source: §7 · p. 10

Annotation and task design

• A single Likert scale may not adequately represent the complex phenomena of social acceptability and toxicity, including their situational context.The approach was evaluated for classification tasks and classifiers; generation tasks would require more careful annotator training, difficult to achieve on a voluntary platform without adequate incentives.Source: §7 · p. 10

Inclusive NLP

• The authors describe their work as a step toward reducing disparities by recognizing dataset and model positionalities and identifying design biases.They regard cross-disciplinary research as crucial to building inclusive AI systems.Source: §8 · p. 10

Ethical considerations

• Collected demographics capture only a small part of positionality, and assigning labels to people is inherently reductionist.A single Likert scale also cannot capture all the complexity of social acceptability and toxicity, such as situational context.Source: §8 · p. 10

Responsible use

• Quantifying an existing system’s positionality is not an endorsement and does not remove researchers’ responsibility to examine whether it should exist or to adjust it further.Source: §8 · p. 10

Study ethics

• The study received University of Washington IRB approval; participants gave informed consent and were not financially compensated.Participants included lay people from varied ages, including minors, and were informed of risks such as exposure to toxic or mature content.Source: §8 · p. 10

Research team positionality

• The team acknowledges that its quantitative, computer-science perspective and members’ backgrounds shape how it approaches positionality and design bias.The team resides in the United States and notes that its nationalities and ethnicities are overrepresented in NLP development; its understanding of impacts is largely based on research rather than personal experience.Source: §8 · pp. 10–11

A Data

• The appendix describes decisions made when sampling data points from the different datasets.Source: A Data · p. 16
• The described sampling decisions cover data points from different datasets.Source: A Data · p. 16
• The appendix also describes post-processing decisions for the sampled data.Source: A Data · p. 16

A.1 Sampling

• Social Chemistry instances were sampled from items labeled “Controversial (∼50%)” and balanced across moral foundation labels; Dynahate samples came from rounds 3 and 4 and were balanced by hate label and target type.Source: §A.1 · p. 16

A.2 Post-Processing

• For Social Chemistry, instance scores were averaged across annotators before comparison with annotators’ aggregated scores.Source: §A.2 · p. 16

B Study Design

• The social acceptability and hate speech studies launched in April and August 2022, respectively, after preliminary user studies; collected annotations were anonymous and securely stored.Source: B Study Design · p. 16

B.1 LabintheWild Study Flow

• The study flow included consent, demographic collection, task instructions, annotation, feedback, and results; participants consented after reviewing study procedures and risks.Source: §B.1 · p. 16
• Participants reported demographics including age, native language, religion, education, and ethnicity, while only longest-lived country and prior test participation were required.Ethnicity was displayed only for participants in the United States.Source: §B.1 · p. 16
• Instructions defined social acceptability as an ethical judgment and used Dynahate’s hate-speech definition with examples distinguishing hate speech from related non-hate content.Source: §B.1 · p. 16
• Participants rated social acceptability on a five-point Likert scale and hate speech as “Hate speech,” “Not sure,” or “Not hate speech,” with optional rationales.After annotation, they could view how AI and participants from the same country responded; final results also showed agreement with AI and same-demographic participants by instance type.Source: §B.1 · pp. 16–17
• Instance presentation included items with no annotations from a demographic, items already sampled by that demographic, and an even split across instance types.Participants could also provide open-text feedback and report technical difficulties or cheating.Source: §B.1 · p. 17

• The section reports additional results from analyses of LabintheWild data.Source: C Additional Results · p. 17

D Cultural Spheres

• Figure 4 shows participants rating an action’s ethics and then seeing responses from others in their country.Source: Figure 4 · p. 18
• Figures 5–8 cover ethical-reflection instructions, toxicity examples, hate-speech ratings, and AI-alignment results by demographic and moral-foundation category.Source: Figure 5 · p. 18, Figure 6 · p. 19, Figure 7 · p. 19

C.1 p-values

• Table 3 reports associated p-values for Pearson’s r correlations, with Bonferroni-corrected α = 2.04e-05.The listed α before correction is 0.001.Source: §C.1 · p. 17, Table 3 · p. 21

D Cultural Spheres

• The cultural-sphere table marks which countries were collected, while the text cautions that continent-based groupings can obscure country differences.Asia, for example, includes Japan and Saudi Arabia.Source: D Cultural Spheres · p. 17, Table 4 · p. 22