PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing the Voice, Preserving the Self: A Participatory Design Approach to Deaf-Centric Text-to-Speech

Shela Atemnkeng, Patrick Boudreault, Paige DeVries, Lloyd May, Christian Vogler

arXiv:2609.10199v1cs.HC

TL;DR

DHH users have limited ways to verify TTS output without hearing it, and generic voices may not reflect their identity. This paper uses participatory design with DHH stakeholders to explore Deaf-centric TTS, surfacing requirements for accessible verification, expressive customization, and authentic representation.

  • Problem

    DHH users may have difficulty verifying TTS output without auditory access, and generic voices may not reflect personal identity or vocal characteristics.

    Source: §1 · p. 2

  • Method

    The paper explores Deaf-centric TTS through participatory design with DHH users, including focus groups and co-design activities.

    Source: §3.1 · p. 7, §5 · p. 18

  • Results

    The study surfaces requirements for non-auditory verification, expressive customization, and authentic representation in Deaf-centric design.

    Source: §5 · p. 18, §10 · p. 22

  • Takeaways & Limitations

    The resulting design framework provides a foundation for developing TTS that preserves Deaf identity and supports verification without relying on listening.

    Source: §10 · p. 22

    The small number of participants limits the authors’ ability to draw conclusions about a preferred design direction.

    Source: §8 · p. 22

Abstract

from arXiv · show

We describe a participatory design approach toward developing Deaf-centric text-to-speech (TTS) technologies. While TTS is growing rapidly in the mainstream, it has received little attention to date in the deaf and hard of hearing (DHH) technology space. Critical problems have remained unaddressed for DHH users, including the ability to manipulate tone, emotions and delivery via non-auditory means. Verifying that the generated speech matches intent and is appropriate for a given situation without having to listen to it is another challenge. Respecting cultural and identity factors in the generated speech is also important. This work explores the design space with DHH participants through two focus groups, three co-design sessions, and four one-on-one early-stage design evaluation sessions. Participants included people both familiar and unfamiliar with TTS, as well as DHH content creators. We describe key findings, design ideas, results, and implications for future Deaf-centric TTS development. We also identify unmet technology requirements that pose barriers to adoption of Deaf-centric TTS technology.

1 Introduction

The paper frames Deaf-centric TTS as a design space where DHH users need ways to verify speech without listening and shape output to reflect their intent and identity. It explores that space through participatory design and early-stage evaluation with DHH users.

1 Introduction

  • Existing TTS can be difficult for DHH users to assess for appropriateness, naturalness, or emotional alignment, while generic voices may not reflect personal identity or vocal characteristics.The paper highlights the need for non-auditory quality control, emotional control, and identity-centered customization.Source: §1 · p. 2
  • DHH content creators report reduced social-media engagement when videos lack voiceovers, and DHH presenters may struggle to add intelligible narration to signed content.In conversation, challenges with prosody and auditory turn-taking can also contribute to communicative inequities.Source: §1 · p. 2
  • The study uses focus groups, co-design activities, and early-stage design evaluation with DHH users to explore Deaf-centric TTS design.Its aims include control over delivery, tone, and emotion, alongside voice cloning capabilities.Source: §1 · p. 2
  • The work surfaces design requirements, ethical considerations, and user-defined criteria for appropriate, authentic voice output to guide future Deaf-centric TTS development.The authors present the design space as a foundation for future research and prototyping of identity-aware technologies.Source: future work · p. 2
  • Voice cloning should be user-driven: it may reproduce a person’s own voice or adjust deaf-accented speech, but should not presume one desired outcome.The stated aim is to expand DHH users’ agency without forcing speech to conform to hearing-centric expectations.Source: future work · pp. 2–3

1.1 Positionality Statement

  • The work is rooted in the research team’s firsthand frustrations working with TTS, including experiences of a deaf graduate student and Deaf professors.The team also includes hearing researchers with accessible-HCI and interpreting experience.Source: §1.1 · p. 3

2 Related Work

Prior TTS and speech technologies offer voice customization and have explored speaker adaptation, but DHH users have had little role in defining or evaluating their output. This paper situates Deaf-centric TTS within gaps in non-auditory verification, identity representation, and participatory design.

2 Related Work

  • TTS customization often assumes users can aurally verify output, leaving many features inaccessible to DHH users who cannot evaluate speech through auditory means.Emotional TTS also relies on auditory evaluation of emotional fidelity.Source: §2 · p. 3
  • Speaker-adaptive TTS has shown technical feasibility for retaining vocal identity, but prior work focused mainly on intelligibility for hearing listeners rather than DHH users’ priorities.This work instead treats DHH users as primary stakeholders in defining appropriate and authentic voice output.Source: §2 · p. 4
  • Research on DHH speech describes prosodic and timing patterns linked to limited auditory feedback, which the paper frames as identity-linked characteristics to accommodate rather than correct.The cited work motivates modeling atypical prosody while supporting user-controlled adjustments.Source: §2 · p. 4
  • Commercial ASR can perform poorly on deaf-accented speech, while TTS systems also lack mechanisms to represent DHH users’ identity, tone, and expressive intent.The authors describe this as a gap at both ends of the voice communication pipeline.Source: §2 · p. 4
  • Although co-design has been used in DHH technologies, the authors report that prior co-design work had not addressed TTS as a design domain.They position their study as an initial contribution toward a framework for Deaf-centric voice technologies.Source: §2 · p. 5

3 Methods

The methods use a staged participatory design process to explore DHH users’ independent, nonauditory control over emotional tone and voice personalization in TTS. They document and analyze co-design work and prepare and evaluate three early TTS visualization concepts with DHH participants.

3 Methods

  • The study aimed to support broader emotional expression, independent control of speech tone without auditory feedback, and voice personalization reflecting users’ vocal characteristics.It also sought to improve emotional feedback to reduce miscommunication and promote inclusive communication.Source: §3 · p. 5
  • The study included 27 participants and followed Deaf-led participatory ethics emphasizing communication autonomy, respect for sign language, and harm mitigation.Three early-stage evaluation participants had also taken part in co-design.Source: §3 · p. 6

3.1 Focus Groups

  • Two remote focus groups explored DHH users’ TTS experiences and needs, but self-reported expertise proved unreliable and both groups included mixed experience levels.The groups were intended to include frequent or familiar TTS users and those with little or no familiarity.Source: §3.1 · p. 7

3.2 Focus Group Procedures

  • Each 120-minute Zoom focus group used written-English prompts presented alongside ASL, with participants discussing TTS topics and proposing improvements.Topics included written communication with hearing people, AI, voice quality and personalization, and evaluating and trusting TTS.Source: §3.2 · p. 7

3.2.1 Focus Group Analysis Methods.

  • Two analysts used inductive coding, reconciled their codebooks, and conducted frequency analysis to identify dominant and conceptually meaningful themes.The analysis drew on verified transcripts and informed the focus-group findings reported later in the paper.Source: §3.2.1 · p. 7

3.3 Co-Design Sessions

  • Three participatory co-design sessions translated focus-group insights into interface concepts; personas surfaced content-creator perspectives, prompting a third session with DHH creators.The first two sessions were in person, and the third was remote.Source: §3.3 · pp. 7–8

3.3.1 Co-Design Session Procedures.

  • A voiced-video demonstration differed from participants’ expectations and helped them recognize the need to control and verify TTS emotions, style, and delivery.Participants then worked in small groups on TTS interface designs through sketches and drawings.Source: §3.3.1 · p. 8
  • Co-design groups reviewed one another’s sketches, added comments and starred promising ideas, then reconvened to identify commonalities, divergences, and concepts for prototyping.In session 2, groups also explained sketches in video-recorded ASL to document their intent.Source: §3.3.1 · p. 8

3.3.2 Co-Design Analysis Methods.

  • Researchers triangulated visual artifacts, participant comments, CART transcripts, and recordings through recurrent-pattern and thematic analysis to contextualize design concepts and derive technical prerequisites.Analysis also identified current and future prerequisites across TTS, AI, sign-language technology, and UI/UX.Source: §3.3.2 · pp. 8–9

3.4 Early-Stage Design Evaluation

  • Applying the 80/20 rule, researchers prioritized high-impact non-auditory communication ideas and selected three visualization concepts for four one-on-one DHH evaluation sessions.The sessions examined how DHH users interpreted early visual representations of prosody, tone, and delivery.Source: §3.4 · p. 9

3.4.1 Short Demo Prototypes Preparation.

  • Figma wireframes became lightweight browser demos using Figma Make, Codex, and a Node.js connection to OpenAI TTS, simulating text entry and speech preview without a deployed system.The wireframes were also arranged as printed storyboard-style sheets to support comparison across interaction stages.Source: §3.4.1 · p. 9

3.4.2 Design Evaluation Procedure.

  • Evaluation moved from five-second intuitive reactions to prototype demonstrations and a side-by-side comparison focused on how visualizations conveyed prosody and helped judge intended voice output.Scenarios grounded the activity in realistic communication, such as sending a class voice message intended to sound excited and confident.Source: §3.4.2 · pp. 9–10

3.4.3 Design Evaluation Data Analysis.

  • Researchers compared evaluation data across participants using a goal-driven approach, synthesizing facilitator notes and participant comments around clarity, expressiveness, confidence, and identity alignment.Source: §3.4.3 · p. 10

4 Results

Focus groups organized DHH participants’ perspectives around six themes, from inclusion and vocal characteristics to adoption barriers and usability. Participants emphasized that TTS must fit communication contexts and identities while remaining controllable and verifiable.

4.1 Focus Group Results

  • The analysis identified six themes—access and inclusion, vocal characteristics, communication, identity and personalization, adoption barriers, and usability—across 152 codes.The themes provided a structure for organizing participant perspectives.Source: §4.1 · p. 10

4.1.1 Access and Inclusion.

  • Participants stressed language access for people whose first language is ASL or another language, DeafBlind inclusion, and the possible need for trained acoustic expertise.One participant suggested an audiologist-like expert could help with voice modification.Source: §4.1.1 · p. 10, §4.1.2 · p. 10

4.1.2 Vocal Characteristics.

  • Participants wanted adaptable, identity-matching delivery—including situation-specific saved voices—and noted that voice modification may require trained acoustic expertise.They described tone as important to mutual understanding and sought voices that match intent and context.Source: §4.1.2 · p. 10

4.1.3 Communication.

  • Participants wanted phrase-based delivery choices, but loud settings and important exchanges could make TTS difficult or interpreters preferable.A participant suggested choices populated from the phrase being said; another noted listeners may not hear TTS in noisy environments.Source: §4.1.3 · p. 10

4.1.4 Identity and Personalization.

  • Participants wanted voices and expression to reflect intersecting identities and context, while some preferred an alternative voice over using their own for privacy.Some participants also questioned whether they could verify that a TTS voice matched its person.Source: §4.1.4 · p. 11

4.1.5 Barriers to Adoption.

  • Participants cited delays and listener impatience, AI-related risks, and situations where pen-and-paper or readable text seemed preferable.One participant specifically preferred pen and paper in medical settings, while another thought listeners might prefer reading a message.Source: §4.1.5 · p. 11

4.1.6 Key Usability Requirements.

  • Usability depended on understandable, nonmechanical speech, tone presets, offline access, clear operating status, and UI control over modulation.One participant described stopping TTS use after others struggled to understand its mechanical-sounding voice.Source: §4.1.6 · p. 11

4.2 Co-Design Results

Co-design participants moved from quick personalization toward ways to verify delivery and preserve Deaf identity. Their ideas included visual and AI-supported verification, identity-conscious voice controls, and creator-focused workflows.

4.2 Co-Design Results

  • The first session prioritized autonomy and quick personalization through presets, sliders, toggles, emotional wheels, and scenario cards.Doctor visits were one suggested preset scenario.Source: §4.2 · p. 12, Figure 1 · p. 12
  • Participants proposed emojis and speaking avatars alongside TTS, as well as haptic and waveform cues, to confirm emotional intent without listening.A smiling emoji could signal friendliness.Source: §4.2 · p. 12, Figure 2 · p. 12
  • ASL avatars were proposed especially by participants who preferred ASL, reflecting a visual-spatial approach to verifying speech that parallels their communication practices.Participants suggested sign-based cues rather than relying only on text or audio.Source: §4.2 · p. 12
  • The second session proposed culturally and demographically marked voice profiles, expressive avatars, layered emotional tags, and preservation of Deaf accents.Participants saw preserving Deaf accents as important to retaining individuality in synthetic voices.Source: §4.2 · pp. 12–13
  • Verification ideas included AI-generated tone summaries, audience polling, and facial-expression recognition to assess whether output aligned with user intent and Deaf identities.Facial expression recognition was also proposed as a way to attach tone and emotion while typing.Source: §4.2 · pp. 11, 13
  • Participants framed Deaf accent cloning as a way to maintain authenticity, identity, and community belonging—not as a corrective tool.They raised concerns that hearing-normative synthetic speech could erase meaningful Deaf speech characteristics.Source: §4.2 · p. 13
  • Creators noted that captions miss personality and emotional nuance, and proposed TTS to reach hearing viewers while preserving expressive intent.Their current audiences were primarily DHH because their content was mainly in sign language.Source: §4.2 · p. 13
  • Creator ideas included workflows for control and editing, scenario wheels for combining traits across contexts, speculative sign-to-voice, visual timing cues, AI verification, and persistent voices across platforms.Karaoke captions and animated waveforms were proposed to track timing, pitch, and emphasis.Source: §4.2 · pp. 13–14

4.3 Early Stage Design Evaluation Results

Four one-on-one sessions evaluated emoji, karaoke-style, and waveform visualizations through immediate interpretation, short demos, and deeper comparisons. Participants found distinct strengths and shortcomings, with no single design meeting all study goals.

4.3.1 Immediate interpretability (five-second test).

  • Static wireframes left most participants uncertain, and the karaoke design’s purpose was not discernible at first glance.Emoji viewers also struggled to understand its function, while waveform reactions included both audio-related associations and confusion.Source: §4.3.1 · p. 16

4.3.2 Understanding of Emotional Tone and Expressiveness Through Short Demos.

  • In short demos, emojis were most immediately legible for emotional tone, while karaoke was better suited to rhythm and pace than emotional nuance.Waveforms were strongest for prosodic variation, but needed richer labels and emotional annotation; no single visualization alone supported expressiveness and confidence.Source: §4.3.2 · pp. 16–17

4.3.3 Deeper Comparison of Tone, Pitch, and Emotion Representation Across Designs

  • No single visualization met all four goals: participants favored waveforms for prosodic detail, emojis for emotional clarity, and sometimes karaoke for rhythm and pace.The findings suggest combining approaches may better support verification that generated speech matches intent.Source: §4.3.3 · p. 17, §4.3.2 · p. 17

4.3.3 Deeper Comparison of Tone, Pitch, and Emotion Representation Across Designs.

  • Participants differed on control preferences: some wanted fine-grained editing, while others preferred a “close enough” voice with one or two clicks.Suggestions included sliders for emotional intensity, reflecting how communication needs shaped design priorities.Source: §4.3.3 · p. 18

5 Discussion

The discussion frames Deaf-centric TTS as supporting verification, identity, and varied communication needs, while surfacing ethical concerns around voice cloning. It connects participant findings to gaps in current TTS research and design.

5 Discussion

  • DHH participants envision TTS not just as a communication aid, but as a way to express identity, convey emotions, and manage interactions.The discussion also extends participatory design practices with DHH communities into TTS.Source: §5 · p. 18

5.1 Non-auditory Verification as a Fundamental Design Requirement

  • Non-auditory verification remains a fundamental design requirement: waveforms convey prosodic detail but can confuse emotion with loudness, emojis can be ambiguous, and karaoke better conveys rhythm than nuance.The findings suggest integrating visualization types; the lack of non-auditory evaluation mechanisms is also a structural gap in broader TTS research [1, 16].Source: §5.1 · p. 18

5.2 Identity, Representation and the Limits of Generic Voice Designs

  • Participants treated preserving Deaf-accented speech as a design value, not a problem to correct, and wanted voices reflecting intersectional identities and context.Proposed mechanisms included cultural and demographic voice-profile markers, accent-preserving cloning, and layered emotion tags.Source: §5.2 · p. 19

5.3 Usability, Workflows and the Diversity of DHH Use Cases

  • The evaluation surfaced a tension between word-level control and speed, supporting layered interfaces from simple presets to deeper editing.Content creators also prioritized workflow integration, platform continuity, and expressive tools for reaching hearing audiences.Source: §5.3 · p. 19

5.4 Ethics, Trust, and Voice Cloning

  • Participants raised privacy and misrepresentation concerns about voice cloning and proposed explicit consent, synthetic-voice transparency, and accountability for misuse.The work identifies a need for a framework for Deaf-centric voice technologies.Source: §5.4 · p. 20

6 Technical Requirements for Implementing the Designs

The paper translates participant needs into technical requirements and researcher-interpreted implementation challenges, spanning emotion controls, voice cloning, sign-language technologies, and AI-based verification.

6.1 Participant-Driven Needs

  • Participant-driven needs include adjustable emotions, formal-to-informal tone, context-dependent presets, per-word emotion tagging, privacy, Deaf-accented voice options, and non-auditory verification.The requirements are organized as participant-driven needs and researcher-interpreted insights.Source: §6.1 · p. 20

6.2 Researcher-Interpreted Insights

  • The researcher-interpreted requirements were identified in consultation with a TTS expert.Emotion adjustment on a continuum would require Valence-Arousal-Dominance modeling.Source: §6.2 · p. 20, §6.2.1 · p. 20

6.2.1 Emotion Customization.

  • Formal-to-informal tone control is challenging because TTS is typically trained on discrete tone states, and blending tone with emotion requires additional work.Per-word emotion tagging requires word-level annotated training data, whose availability is unclear; local processing for privacy is constrained by model size on many consumer devices.Source: §6.2.1 · p. 20

6.2.2 Voice Cloning.

  • Retaining identifying Deaf-accented characteristics in cloned voices is a major challenge because typical cloning uses normative speech datasets.Variable DHH accents would first need to be included in those datasets.Source: §6.2.2 · p. 20

6.2.3 AI Prerequisites.

  • Conveying emotion and tone in ASL requires advances in sign recognition and avatars; current avatars cannot sufficiently express detailed emotional nuance alongside grammatical markings.The paper also notes gaps in translating emotion-tagged speech or writing into functionally equivalent ASL expressions.Source: §6.2.3 · p. 21

6.2.4 Supporting AI-based Verification.

  • AI-based verification requires ASR that can recognize emotion, tone, and delivery, which remains an active research area.Even reliable recognition leaves open how to express those attributes in understandable written English or ASL.Source: §6.2.4 · p. 21

7 Overall Implications and contributions

The paper contributes a DHH-participatory approach to TTS design, identifies unmet verification and identity requirements, and outlines broader applications, ethics concerns, and research needs.

7 Overall Implications and contributions

  • The work establishes participatory design with DHH users as a productive methodology for TTS development, extending prior DHH co-design work into a previously unaddressed domain.The cited prior practices are.Source: §7 · p. 21
  • It identifies non-auditory verification as an unmet Deaf-centric TTS requirement and evaluates three visualization approaches as a basis for future prototyping.The exploration has not yet found a good approach; a combination may be effective.Source: §7 · p. 21
  • The paper reframes Deaf-accent preservation as a design value for how speaker-adaptive TTS systems should be developed and evaluated for DHH users.Source: §7 · p. 21
  • It expands DHH TTS applications beyond interactive communication to content creation and professional media production, with implications for tools described by Cao et al..Source: §7 · p. 21
  • It identifies voice-cloning ethics as an open DHH research and design problem and offers user-grounded requirements that complement theoretical FATE frameworks for sign language.Source: §7 · p. 21
  • It identifies further foundational research and development needs in identity-preserving speech synthesis and sign-language technologies.Source: §7 · p. 21

8 Limitations

The study’s limitations include a narrow participant and geographic scope, non-interactive early-stage evaluation with only four participants, and unresolved technical and AI-related risks.

8 Limitations

  • The personas omitted DeafBlind and DeafDisabled perspectives, and all participants were U.S.-based, making the results Western-centric.Most co-design participants were local to Washington, DC; only focus groups and one co-design session included remote participants from across the United States.Source: §8 · pp. 21–22
  • The four-person evaluation used brief, non-interactive demos, so feedback reflects concept interpretation rather than hands-on use and cannot establish a preferred design direction.Participants could not manipulate controls, generate speech, or experience the full setup-to-output workflow.Source: §8 · p. 22
  • The study warns that uncritical AI use could perpetuate stereotypes and stigma, while unrealistic expectations of AI capabilities also carry risks.Bias and insufficient diversity in training data may preclude AI applications in DHH contexts; proposed facial-expression recognition and AI-based tone application or verification are currently difficult to implement.Source: §8 · p. 22
  • Deaf-accented voice cloning that removes accent characteristics remains uncharted, and AI training-data biases may preclude applications in DHH contexts.The paper also identifies risks around intersectional identities and speech.Source: §8 · p. 22

9 Future Work

Future work includes implementing and comparing the design concepts in hands-on evaluations, while addressing technical challenges in emotion detection, synthetic voice construction, and sign-language-driven TTS.

9 Future Work

  • Concrete prototypes and hands-on testing would let participants compare concepts side by side, discuss trade-offs, and assess whether one design or a combination best meets Deaf-centric TTS goals.Further co-design sessions and design evaluations are also planned.Source: §9 · p. 22
  • Emotion detection and synthetic voice construction need further research to balance reliable subtle cues across voices and context with naturalness, personalization, situational context, and safety.The paper identifies emotion detection as an active research frontier.Source: §9 · p. 22
  • Sign-language-driven TTS remains a long way off and requires advances in machine learning and culturally sensitive translation pipelines.Source: §9 · p. 22

10 Conclusion

The paper presents a participatory design framework for Deaf-centric TTS grounded in work with DHH participants and creators. It identifies layered design requirements and positions participatory design as a path for future inclusive TTS development.

10 Conclusion

  • The framework grounds Deaf-centric TTS in identity, authenticity, and non-auditory verification, drawing on focus groups, co-design, and design evaluation with DHH participants and creators.Source: §10 · p. 22
  • The surfaced requirements include low-complexity entry points, identity-driven customization that preserves Deaf identity, and visual checks of alignment with communicative intent.Evaluation of visualization concepts grounds these priorities in concrete design trade-offs.Source: §10 · p. 22
  • The findings lay groundwork for DHH technology research and establish participatory design as a productive path toward TTS built with the communities it is meant to serve.Source: §10 · p. 22
  • The authors describe this as uncharted territory and a first work of its kind, while explicitly not claiming a complete solution.They offer guiding principles for future TTS technologies inclusive of DHH individuals.Source: §10 · pp. 22–23

AI Use Statement

AI-assisted tools supported rapid prototype preparation; the authors state that research decisions, analyses, interpretations, and writing were developed by the authors.

AI Use Statement

  • Figma Make translated designed wireframes into basic interaction flows for the Emoji-style prototype.Source: AI Use Statement · p. 23
  • OpenAI Codex translated Karaoke and Waveform wireframes into lightweight web-based demo prototypes, including simulated typing and speech previews.Source: AI Use Statement · p. 23
  • The tools supported rapid prototyping only; authors developed the study design, research materials, analyses, interpretations, and paper text.Source: AI Use Statement · p. 23
Loading 2609.10199v1…