Source-linked AI summary
Queer inclusion in speech datasets: An audit and taxonomy of practical tensions
Brooklyn Sheppard, Anaelia Ovalle, Adina Williams, Levent Sagun
TL;DR
Speech datasets provide limited representation of queer voices, while collecting sensitive identity data raises ethical challenges. The paper audits speech datasets and analyzes collection practices, finding a lack of queer representation and identifying tensions between AI data-collection customs and queer community considerations.
Problem
The paper addresses the lack of queer voices in speech technology datasets and the challenges of collecting sensitive identity data safely.
Source: §1 · p. 1Method
The authors audit six speech technology datasets and two queer-focused speech-science datasets, examining collection practices and developing a taxonomy of tensions.
Source: §2.1 · p. 2, §2.2 · p. 2, §4 · p. 3Results
Takeaways & Limitations
The authors say future work would benefit from directly engaging queer speakers to examine these tensions through lived experience.
Source: §5 · p. 4
Abstract
from arXiv · showhide
In this paper, we examine speech datasets for their inclusion of LGBTQIA+, or queer, voices and provide a taxonomy of tensions to better understand why there is a lack of such voices in current speech technology datasets. Through an audit of six diverse speech datasets, we find that measurable queer representation is low (0-1.4% of speakers) - insufficient for robust disparity measurement. We take this community as a case study to consider what challenges and tensions are associated with collecting speech data from marginalized communities. For comparison, we audit an additional two datasets from the speech sciences that were created by, for, and with the queer community. We note that many customs in speech dataset collection efforts in AI and speech technology research may conflict with values emphasized in participatory approaches with marginalized communities, and provide a taxonomy describing these tensions.
1. Introduction
Speech datasets face both representation gaps and heightened ethical challenges when collecting sensitive identity information. Using queer voices as a case study, the paper audits dataset inclusion and identifies tensions between AI collection practices and queer community considerations.
1. Introduction
- Speech datasets require recruiting and recording diverse speakers, while sensitive identity annotation is harder to anonymize safely than text data.The paper notes risks of unintended harms, including re-identification and voice misappropriation.Source: §1 · p. 1
- Because voice is linked to identity, limited queer representation risks systematic underperformance, while labeled queer speech may enable harms such as “gaydar” technologies.The paper also notes that queer speech has been studied for phonetic, prosodic, lexical, and identity-related variation.Source: §1 · p. 1
- Speech-bias research has linked speaker identity to model associations and downstream errors, including higher ASR error rates [31] [34] [36].Related fairness datasets have examined gender, ethnicity, first language, and dialect.Source: §1 · p. 1
- Prior studies found significant performance degradation for speakers identifying as “other” across tested models, but samples numbered only 8 to 86 depending on language.The authors describe that inquiry as preliminary and note the need for larger samples of varied gender identities.Source: §1 · p. 1
- The paper audits queer representation and collection practices, then frames tensions between common AI data-collection customs and queer community considerations.Its representation analysis focuses on gender identity because datasets commonly annotate gender but not other queer identities; the authors state that identified tensions apply to the queer community as a whole.Source: §1 · p. 2
2. Methodology
The methodology audits six English-language speech technology datasets and compares them with two datasets designed for queer inclusion. It assesses dataset selection and six dimensions of annotation, distribution, institutional practice, ethics disclosure, recruitment, and access.
2.1. Dataset selection
- Six English-language speech technology datasets were selected for diverse speaker backgrounds, prior fairness assessment, or speech-fairness design; two queer-focused speech-science datasets were also audited.The latter were MAGES, with 14 gender-expansive speakers, and PTMV, with 20 masculine-identifying participants spanning several identities.Source: §2.1 · p. 2
2.2. Axes of auditing
- The audit examined gender annotation options and label distributions, institutional affiliation, ethics-review disclosure, recruitment methods, and data access and permitted use.Questions included whether labels were self-provided or predefined, whether recruitment involved community engagement, and whether licenses allowed AI training or evaluation.Source: §2.2 · p. 2
3. Comparison of datasets
The audit compares gender-label options and distributions, institutional and ethics-review disclosures, recruitment practices, and access terms across eight speech datasets. The comparison includes two datasets created specifically for queer representation.
Gender annotation options
- Gender annotation ranges from binary-only and predefined mutually exclusive categories to self-expression, but self-provided labels are used across the audited datasets.MAGES and PTMV allow free-text gender identities; EdAcc is the only speech-technology dataset with free-text write-ins, while Fairspeech, ARCTIC, and English Dialects use binary categories.Source: §3 · p. 2
Distributions of gender labels
- Only CCV2 and EdAcc include speakers with gender labels outside the binary; CV and Fairspeech had such options but contain no speakers identifying with them.The authors report that Fairspeech excluded non-binary speakers because their numbers were not significant, while CV had no contributors openly identifying as transgender or non-binary.Source: §3 · p. 2
Institutional affiliations & ethics review disclosure
- Only three datasets publicly disclose an ethics or institutional review process: MAGES, PTMV, and EdAcc.The authors caution that non-disclosure does not establish that internal review was absent, particularly at large industrial research organizations.Source: §3 · pp. 2–3
Participant recruitment methods
- Most datasets recruit through crowdsourcing or open calls, whereas MAGES and PTMV use queer-community word-of-mouth and social-media recruitment.EdAcc and English Dialects report origins in personal connections; Fairspeech and CCV2 report paid participants but not detailed selection methods.Source: §3 · p. 3
Data access and usage
- All datasets except MAGES are freely available online, but permitted uses vary from evaluation-only to training and evaluation, while PTMV specifies clinical use.MAGES requires a meeting with the lead researcher before release; its access process is described as honoring community vulnerability and safety.Source: §3 · p. 3
4. Why is there a lack of queer voices in diverse speech datasets?
The authors identify four tensions between common AI data-collection practices and queer-community needs or preferences. Their audit describes tensions involving scale, engagement, data control, and identity categories.
4. Why is there a lack of queer voices in diverse speech datasets?
- The taxonomy names four tensions: scaling versus inclusion, efficiency versus engagement, access versus autonomy, and static categories versus fluid identities.These tensions are drawn from the audit of speech datasets and their collection practices.Source: §4 · p. 3
4.1. Scaling versus inclusion
- The open Common Voice corpus had less queer representation than EdAcc and CCV2 despite having 778 and 17 times their speaker counts, respectively.The authors argue that scaling in practice may still underrepresent small or stigmatized groups, and suggest smaller community-focused collection may alleviate this tension.Source: §4.1 · p. 3
4.2. Efficiency versus engagement
- Crowdsourcing collects data efficiently, while trust-building and collaborative engagement with affected communities can be slow and patient.MAGES and PTMV recruited through direct community engagement following ethics review, and are significantly smaller than AI-community datasets.Source: §4.2 · pp. 3–4
4.3. Access versus autonomy
- Open access can conflict with queer speakers’ autonomy over speech data, which may be difficult to anonymize and may reflect personal identity.MAGES restricts access to honor community vulnerability; PTMV is open access but intended for clinical purposes, unlike the other open-access datasets designed for speech technology research.Source: §4.3 · p. 4
4.4. Static categories versus fluid identities
- Predefined, mutually exclusive gender labels can discourage people whose identities combine categories and may imply distinctions between transgender and cisgender men or women.The authors note that categorizing queer voices may also open the possibility of harmful automatic gender or sexuality identification.Source: §4.4 · p. 4
5. Conclusion
The paper reports a lack of queer representation in diverse speech datasets and offers a taxonomy of tensions between AI-community customs and queer-community needs and preferences. It calls for future participatory engagement with queer speakers to explore these tensions through lived experience and across communities under the queer umbrella.
5. Conclusion
- The authors demonstrate a lack of queer representation in diverse speech datasets, including datasets theoretically open to all.They draw on the audit and dataset-creation analysis to provide a taxonomy of tensions between AI-community customs and queer-community needs and preferences.Source: §5 · p. 4
- The authors hope the work draws attention to this underrepresentation and inspires participatory research with the queer community in ethical speech and language technology development.The conclusion frames this as an aspiration for future research.Source: §5 · p. 4
- Future work should directly engage queer speakers to examine these tensions through lived experience and how they differ across communities under the queer umbrella.Source: §5 · pp. 4–5