Source-linked AI summary
WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identities
Jiska Beuk, Gerasimos Spanakis
TL;DR
Dutch language models remain understudied for LGBTQ+ bias, motivating a culturally adapted Dutch benchmark. The paper translates and validates WinoQueer with Dutch queer participants and evaluates Dutch and multilingual MLMs and ARLMs. Although overall bias is near neutral, transgender and non-binary identities show the highest bias scores in some models, while benchmark scores should not be treated as proof that a model is unbiased.
Problem
Bias toward LGBTQ+ identities is well studied in English, but evidence for Dutch language models remains limited.
Method
The paper adapts WinoQueer to Dutch language and culture, preserves identity-based sentence pairs, and validates the dataset through a Dutch queer community survey before evaluating MLMs and ARLMs.
Results
Overall mean bias was close to neutral, but transgender and non-binary identities consistently received high scores, reaching up to 97% for some models.
Takeaways & Limitations
Culturally grounded benchmarks can reveal identity-specific disparities that aggregate bias scores may obscure.
Takeaways & Limitations
A low WinoQueer-NL score does not prove that a model is free of bias, and the benchmark cannot guarantee that a model is truly unbiased.
Abstract
from arXiv · showhide
While English language models have been widely examined for anti-queer bias, Dutch models remain understudied. To address this gap, we developed a culturally and linguistically adapted Dutch dataset based on the English WinoQueer benchmark, containing pairs of stereotypical and counter-stereotypical sentences. To validate and expand it, we conducted an online survey with 43 Dutch queer participants, confirming 145 of 171 stereotypes as culturally relevant and identifying 22 new biases through free-text responses. The final released dataset, comprising 42,906 sentences, was evaluated using a range of Dutch-specific and multilingual models, including both masked language models (MLMs) and autoregressive language models (ARLMs), with bias measured via a score comparing log-likelihoods of stereotypical versus counter-stereotypical sentences. While the mean bias score across models appeared neutral (~50%), closer analysis revealed significant disparities: some models favored stereotypical sentences up to 97% of the time for transgender identities, but only 6% of the time for gay-related pairs, with transgender and non-binary identities consistently receiving the highest bias scores. Our findings highlight the importance of culturally grounded datasets for evaluating and mitigating biases that disproportionately impact marginalized groups in Dutch language models.
1 Introduction
The paper addresses limited evidence on LGBTQ+ bias in non-English language models by creating and evaluating a Dutch, culturally adapted benchmark. It focuses on differences in bias toward LGBTQ+ identity groups using Dutch and multilingual masked and autoregressive models.
- LLMs can inherit training-data biases that resurface in outputs and harm marginalized groups.
- Dutch queer people continue to face social bias despite high reported acceptance of homosexuality.
- Research on language-model bias has largely focused on observable characteristics, while bias in non-English models remains understudied.
- The paper creates a Dutch benchmark based on English WinoQueer and adapted to Dutch language and culture.
- The evaluation covers Dutch and multilingual models, including both masked and autoregressive language models.
- A key contribution is analyzing differences in model bias across LGBTQ+ identity groups.
2 Background and Related Work
Prior work shows that language models can reproduce queer and other social biases, but most queer-bias benchmarks remain English-focused. The paper therefore emphasizes linguistic and cultural adaptation and participatory dataset design for cross-cultural evaluation.
- Queer Bias and Stereotypes in Language Models: WinoQueer evaluates LGBTQ+ bias with stereotype and counter-stereotype sentence pairs sourced from queer individuals, but is limited to English and American culture.
- Dutch Language Models and Bias: Dutch masked and autoregressive language models have also been found to reflect societal stereotypes and perpetuate social biases.
- Dutch Language Models and Bias: Identical prompts can produce different bias levels across languages, underscoring that linguistic and cultural context affects how bias is expressed.
- Challenges in Multilingual and Cross-Cultural Bias Evaluation: Directly translating English benchmarks into Dutch risks distortion because grammar, identity terms, and cultural stereotypes do not always transfer equivalently.
- Bias Mitigation and Participatory Approaches: Participatory dataset design involves affected communities in benchmark creation, while complete debiasing remains challenging because overcorrection can introduce new biases.
3 Dataset Creation
WinoQueer-NL was built by translating and culturally validating WinoQueer with Dutch-speaking LGBTQ+ participants, then releasing sentence-pair and stereotype-metadata datasets. The process preserved identity-based counterfactual comparisons while adding Dutch-specific biases and relevance assessments.
- Dataset Creation: The dataset creation process translated WinoQueer, surveyed Dutch-speaking LGBTQ+ participants, and compiled the validated final datasets.
- Dataset Translation: Identity replacement creates counterfactual sentences so that the identity term is the intended difference between stereotypical and counter-stereotypical variants.
- Dataset Translation: Four native Dutch LGBTQ+ speakers manually checked predicate translations, leading to the exclusion of six predicates for similarity or unnaturalness.
- Dutch Queer Community Survey Design: The survey combined validation of translated stereotype pairs, harmfulness ratings, and free-text reports of additional biases.
- Dutch Queer Community Survey Design: 22 new Dutch-specific biases were added, covering appearance, personality, family roles, and linguistic or identity-based assumptions.
- Final Datasets: The sentence-pair dataset contains 42,906 pairs built from 11 templates, 167 predicates, 60 names, and 9 identities.
- Final Datasets: The metadata dataset records templates, predicates, identities, relevance judgments, harmfulness ratings, and relevance tiers for 193 stereotypes.
- Relevance and Harmfulness Validation: 26 stereotypes were excluded as culturally irrelevant, leaving 167 relevant stereotypes distributed across Tiers 1, 2, and 3.
4 Bias Assessment
The study evaluates Dutch-specific and multilingual masked and autoregressive language models using bias scores based on sentence-pair likelihoods. It reports weighted and identity-averaged measures, while cautioning that MLM and ARLM scores are not directly comparable.
- Model Selection: The evaluation covers Dutch-specific and multilingual language models, including both masked and autoregressive models.The selected models include BERTje, RobBERT, DistilmBERT, mBERT, XLM-RoBERTa, and several Dutch and multilingual autoregressive models.
- Bias Score Calculation: MLM scoring predicts unmodified tokens while conditioning on the modified identity tokens, preventing differing identity frequencies from directly affecting masked-token likelihoods.The method compares stereotypical and counter-stereotypical sentence variants using pseudo-log-likelihoods.
- Bias Score Calculation: ARLM scoring evaluates each unmodified token from left to right using all preceding tokens, including modified and unmodified tokens.The total score is computed separately for both sentences in each pair.
- Bias Score Interpretation: The bias score is the percentage of sentence pairs in which a model favors the stereotypical sentence.Two variants account for uneven identity representation: usage-weighted bias gives more weight to frequent identities, while identity-averaged bias weights identities equally.
- Bias Score Interpretation: A score of 50 indicates perfect neutrality, whereas scores above 50 indicate stereotypical completions and scores below 50 may signal overcorrection or indirect bias.Because MLM and ARLM evaluation methods differ, their scores cannot be directly compared, although percentages allow relative comparison.
5 Experiments & Results
Experiments reveal that near-neutral aggregate bias scores conceal substantial identity- and model-specific disparities. Dutch and multilingual models show especially high bias toward transgender and non-binary identities, while predicate harmfulness is not a strong predictor of model bias.
- Overall results: 50.53% is close to the neutral baseline, but individual models and identity groups show substantial variation in bias.Usage-weighted scores can obscure pronounced identity-specific disparities.
- Masked language models: 88.12% for transgender versus 19.31% for queer identities shows that RobBERT’s aggregate score conceals divergent group-level outcomes.The identity-averaged RobBERT score was 67.12%.
- Autoregressive language models: 96.80% for transgender and 90.14% for non-binary identities made GPT2 Dutch large strongly favor stereotypical sentences.GPT2 Dutch large had an overall bias score of 65.29%, while Fietje-2 scored 46.83%.
- Autoregressive language models: 96.42% for non-binary versus 5.85% for gay identities demonstrates extreme identity-specific variation in Llama-3.2-1B.All multilingual autoregressive models had usage-weighted scores below 50%.
- Dutch–English comparison: 78.97% for non-binary and 74.01% for transgender identities contrasted with a roughly 42% Dutch identity gap versus about 16% in English.The Dutch benchmark therefore showed more uneven identity-specific patterns than the English benchmark.
- Predicate harmfulness: |r| < 0.12 and r2 ≈0.014 indicate that stereotype harmfulness does not predict model bias.The largest reported effect was r = 0.1189 for XLM-RoBERTa (Base).
6 Conclusion
The study introduces a Dutch benchmark for language-model bias and finds strong variation across models and identity groups despite an overall mean near neutrality.
- The benchmark was created through tokenization, translation, and validation, resulting in 167 stereotypes and 42,906 sentence pairs.
- 43 Dutch-speaking queer participants helped ensure the dataset’s cultural relevance.
- Transgender and non-binary identities consistently showed high bias scores, reaching up to 97% for some models, while gay and queer identities had much lower scores.
- Usage-weighted and identity-averaged scores indicated that overrepresented identities tended to receive lower bias scores.
Broader Impact and Limitations
The study’s limitations concern survey coverage, dataset structure, and how benchmark scores should be interpreted. These constraints limit representativeness, stereotype coverage, and claims about models being unbiased.
- Survey limitations: The survey’s small and unevenly represented sample limited identification of new biases across identity groups.The authors suggest expanding the survey to improve representation of the Dutch queer community.
- Survey limitations: Privacy-by-design preserved participant anonymity but prevented checking whether the 43 respondents represented the broader LGBTQ+ community.The survey therefore provides information about stereotype relevance without establishing community-wide representativeness.
- Dataset limitations: Template constraints excluded some open-text biases because they could not be expressed in the dataset’s sentence formats.This limits the range of stereotypes that WinoQueer-NL can capture.
- Interpretation: A low WinoQueer-NL score should not be interpreted as proof that a model is free of bias.The benchmark may indicate progress but cannot guarantee that a model is truly unbiased.
A Survey respondents
Table 8 reports the self-identified sexual and gender identities of the 43 survey respondents. Because participants could select multiple identities, the percentages do not sum to 100%.
- Survey respondents: Table 8 summarizes self-identified sexual and gender identities among 43 survey respondents.Multiple identities could be selected by each person.
- Survey respondents: The reported identity percentages do not add up to 100% because respondents could select multiple identities.
C Subset Analysis by Tier
Filtering the dataset by stereotype relevance had little effect on overall bias scores, but identity-specific disparities remained pronounced. Transgender and nonbinary categories stayed highly biased, whereas gay identities remained substantially lower.
- Subset construction: 42,906 sentence pairs formed the full evaluation set, alongside Tier 1–2 and Tier 1 subsets containing 36,084 and 21,670 pairs.Tier 1 included only stereotypes with at least 70% respondent agreement, while Tier 1–2 retained the more reliable tiers.
- Overall results: 50.53% usage-weighted bias on the full set changed only slightly to 50.33% for Tier 1–2 and 50.61% for Tier 1.Identity-averaged scores similarly shifted from 55.63% to 55.40% and 55.58%, indicating limited impact from filtering.
- Identity-specific results: Nonbinary identities received scores just below 80% across all subsets, while transgender scores remained around 74% to 74.2%.These identity-specific patterns persisted despite restricting the dataset to more clearly agreed-upon stereotypes.
- Identity-specific results: Gay identities received bias scores just above 36% in all three subsets.This remained substantially lower than the transgender and nonbinary scores.
D Bias Score Results for Dutch Stereotypes
The section reports bias scores for Dutch stereotype sentence pairs across model and identity-group analyses, including a separate analysis of newly added Dutch stereotypes.
- 5,790 sentence pairs comprise the analysis of newly added Dutch stereotypes.
- Table 9 reports bias scores per model and identity group using Tiers 1 and 2.
- Table 10 reports bias scores per model and identity group using Tier 1 only.
- Table 11 reports bias scores per model and identity group for newly introduced Dutch stereotypes.