PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Communication in Speech Therapy: Exploring the Cognitive Synergy Between Gesture and Speech

Paul-Peter Arslan, Xiao Xiao

arXiv:2609.08486v1cs.HC

TL;DR

The paper asks whether a rhythm-based interface can support speech therapy by combining gesture and speech. It adapts a manual-dexterity tool for finger-tapped synthesized speech and reports increased verbal output and improved motivation and therapeutic engagement, while noting that personalization takes time in short sessions.

  • Problem

    The study asks whether gesture-rhythm interfaces can support speech therapy, where children's engagement can fluctuate during sessions.

    Source: §1 · p. 2, §5.1 · p. 7

  • Method

    The authors adapted a rhythm-based manual rehabilitation interface for finger-tapped control of synthesized vocal phrases and gathered therapist interviews and feedback from children.

    Source: §1 · p. 2, Introduction to the Interview · p. 16

  • Results

    Children using the interface showed increased verbal output, and the authors report improved motivation and therapeutic engagement.

    Source: §1 · p. 2, §8 · p. 12, §9 · p. 13

  • Takeaways & Limitations

    Preliminary findings suggest rhythmic gestures can strengthen gesture-verbal communication in speech therapy, with broader and longer-term contexts requiring further study.

    Source: §9 · p. 13

    Personalizing the interface required setup time that was challenging to balance with direct interaction in short therapy sessions.

    Source: §7.1 · p. 10, §7.3 · p. 11

Abstract

from arXiv · show

This paper examines the adaptation of a rhythm-based interface, originally designed for manual dexterity rehabilitation, for use in speech therapy. The interface allows users to control synthesized vocal phrases through finger tapping, leveraging the cognitive link between gesture and speech. Through interviews with four therapists and pilot tests with one speech therapist and eight children with speech impairments (autism, Down syndrome, verbal apraxia, dyslexia), we found that the interface improves motivation and therapeutic outcomes by facilitating more interactions between verbally challenged patients and the therapist. Our findings suggest that rhythmic gestures can enhance verbal communication, offering potential for broader therapeutic and educational applications.

1 Introduction

The paper explores whether rhythmically tapping to control synthesized speech can use the connection between gesture and speech in therapy. Therapist interviews and child testing indicated improved motivation and therapeutic outcomes, with patient-specific adaptations emerging as important.

1 Introduction

  • Rhythm supports coordination and motor skills in rehabilitation and can aid fluency, intonation, and articulation in speech therapy.The introduction frames rhythm as relevant to both movement and speech practice.Source: §1 · p. 2
  • The study investigates gesture, rhythm, and speech through an interface that lets users control synthesized voice rhythm by finger tapping.The interface was originally developed to improve manual dexterity in stroke patients.Source: §1 · p. 2
  • Interviews with two physical and two speech therapists informed a preliminary exploration with eight children who had speech disorders.The authors identify speech therapy as promising because hand gestures often accompany verbal communication.Source: §1 · p. 2
  • Therapists reported that combining gestures with rhythm helped children memorize and pronounce words, while the interface improved motivation and therapy outcomes.The research also prompted refinements tailored to individual patients.Source: §1 · p. 2
  • The authors present the adapted interface as effective in speech therapy and suggest potential for broader therapeutic applications.They propose that rhythmic integration could improve outcomes in motor and speech rehabilitation.Source: §1 · p. 2

2 Related Work

Related work describes rhythm as a support for motor coordination and speech practice, and gestures as closely connected to language production. Prior studies motivate combining rhythmic and gesture-based approaches, while identifying adaptive interfaces as a future direction.

2.1 The Role of Rhythm in Rehabilitation

  • Rhythmic interventions support motor coordination and learning, while rhythm-based speech approaches aim to improve fluency and organize speech sequences [4, 36, 38, 45, 53].Rhythmic Auditory Stimulation and the Dextrain Manipulandum are cited as tools for motor recovery.Source: §2.1 · pp. 2–3

2.2 How Gestures and Speech are Connected

  • Gesture and speech are described as integrated in language production, with shared cognitive frameworks and overlapping neural circuits.Gesture-based therapies have also been used for neurogenic communication disorders such as aphasia.Source: §2.2 · p. 3

2.3 New Therapies Combining Rhythm and Gestures

  • Prior studies report that combining rhythm and gestures can support word recall, pronunciation, engagement, articulation, and fluency in speech rehabilitation.The cited work attributes this potential to integrating motor actions with linguistic elements.Source: §2.3 · p. 3
  • Future work is proposed on more interactive, adaptive gesture-speech interfaces for wider therapeutic and educational settings.The authors describe these contexts as opportunities for development.Source: §2.3 · p. 3

3 Development & Description of the Interface

The interface adapts a finger-measuring device for rhythmic speech practice, pairing finger presses with timed syllable playback and visual highlighting. A preparation pipeline segments speech and converts corrected labels into files for the interface.

3.1 Interface Design

  • The Dextrain Manipulandum is paired with Unity and FMOD to support rhythmic repetition and generate timed synthesized speech segments.The adapted system is intended to support speech and fine motor skills, with practice focused on rhythmic accuracy and articulation.Source: §3.1 · p. 3

3.2 Specific Features

  • Each index-finger press triggers a syllable, turns its displayed text red, and lets the user track progress through synchronized visual and audible feedback.Figure 2 shows the interface, tapping device, and software components.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • Pre-recorded words are segmented into timed syllables so users can synchronize finger movements with sound and practice rhythm and articulation.The system plays the corresponding syllable when a button is pressed.Source: §3.2 · p. 4
  • The preparation pipeline separates vocals, segments phonemes, manually corrects timestamps in Praat, and converts the TextGrid labels into the interface format.Figure 3 presents the pipeline; Figure 4 shows corrected segmentation and the labeling file with manually added highlight letters.Source: §3.2 · p. 4, Figure 3 · p. 4, Figure 4 · p. 5

4 Methodology

The study combined therapist interviews, interface adaptations, and supervised sessions with eight children already receiving speech therapy. Interviews and observations were transcribed or recorded and analyzed to assess practice fit, engagement, and speech-exercise performance.

4.1 Participants

  • Four therapists participated in interviews, while testing involved one pediatric speech therapist and her eight patients aged six to twelve with varied speech disorders.The children had autism, Down syndrome, verbal apraxia, or dyslexia.Source: §4.1 · p. 5

4.2 Procedures

  • The study proceeded through interviews and an interface demonstration, feedback-based adaptations, and testing with the therapist and eight children.These were the three phases of the study.Source: §4.2 · p. 6

4.2.1 Interviews and Demonstration

  • Therapist interviews covered current practices, interface feedback, possible adjustments, new content, and criteria for evaluating effectiveness.Open questions were asked consistently, and responses were recorded and transcribed verbatim.Source: §4.2.1 · p. 6

4.2.2 Implementation of Adaptations

  • Interview feedback led to changes intended to improve usability in therapy sessions and add exercises tailored to therapists’ practices.Source: §4.2.2 · p. 6

4.2.3 Testing Sessions

  • Over two days, the speech therapist supervised eight children in 30-minute sessions using the adapted interface.Researchers observed engagement and progress, and the therapist reported practical challenges and performance changes.Source: §4.2.3 · p. 6

4.3 Data Collection and Analysis

  • Interview data were analyzed thematically, while session observations and therapist feedback were examined for engagement, motivation, and speech-exercise performance.The analysis also considered how the interface fit existing practice and where it could be improved.Source: §4.3 · p. 6
  • A single researcher conducted coding and analysis; data were anonymized and consent was obtained from therapists and children’s parents.Source: §4.3 · p. 7

5 Findings

Therapists described rehabilitation as requiring continual adaptation to changing patient needs and engagement. They identified flexible strategies and gesture–speech–rhythm combinations as useful ways to support participation and communication.

5.1 Specific Challenges in Sessions

  • Therapists said sessions require real-time adaptation because children may disengage, patients may produce spontaneous words, and physical or motivational states can fluctuate.They emphasized responding to these shifts rather than relying on a fixed session plan.Source: §5.1 · p. 7

5.2 Adaptation Strategies

  • Adjusting activity difficulty to patient performance and using tailored tools were described as ways to keep patients challenged and maintain functionality.T3 described simplifying an activity for a struggling patient or increasing its complexity when the patient was doing well.Source: §5.2 · pp. 7–8
  • Encouragement, educational games, and visual aids were among the strategies therapists used to sustain motivation and attention.T2 highlighted praise, while another therapist described games and visual aids as keeping sessions lively.Source: §5.2 · pp. 7–8
  • Therapists described external rhythm as useful when patients could no longer maintain a rhythm themselves, and real-time visual progress as motivating.They also noted that a constant metronome beat may not fit real-life scenarios.Source: §5.2 · p. 8

5.3 Interaction Between Motor Gestures and Speech

  • Therapists observed that gestures can support communication when words are difficult and can pair syllable production with visible and felt rhythm.T1 described children tapping one piece per spoken syllable; T3 also noted gestures accompanying verbal instructions in physical therapy.Source: §5.3 · pp. 8–9
  • The interface showed promise for linking motor gestures, speech, and rhythm, which therapists associated with stronger therapeutic outcomes for patients facing motor and verbal coordination challenges.Therapists described tapping syllable pieces and pairing rhythm with recitation as supporting memorization.Source: §5.3 · pp. 8–9

6 Interface Adaptations Based on Therapist Feedback

Therapist feedback led to a more flexible interface, including personalized exercises and clearer user experience. Its web workflow lets therapists upload audio and text, review timestamps, and prepare materials for patient sessions.

6 Interface Adaptations Based on Therapist Feedback

  • Therapist feedback prioritized flexibility and personalization, while interface refinements targeted intuitiveness, visual clarity, and fit with therapeutic practice.Source: §6 · pp. 9–10
  • The web interface processes uploaded audio and text, then lets therapists review and adjust waveform timestamps before downloading a timestamp file for later use.Controls support listening to segments, moving to the next segment, playing from the current timestamp onward, and downloading the output.Source: Figure 5 · p. 9, §6 · p. 9
  • The exercise library was expanded with common words, number sequences, and daily-life or school terms, categorized by speed and length to adjust difficulty.Source: §6 · p. 9
  • Figure 6 depicts a syllable-by-syllable repetition exercise in which each button press highlights and triggers synthesized pronunciation of the selected syllable.Source: Figure 6 · p. 10

7 Experience and Evaluation with Children

In sessions with eight children, the adapted rhythmic interface attracted attention and showed promise for motivation and language skills, although responses varied and personalization took time. The observations point to tailoring exercise content and pacing to each child.

7 Experience and Evaluation with Children

  • Testing involved eight children in sessions with T1, a speech therapist, following interface modifications.Source: §7 · p. 10

7.1 Motivation and Engagement

  • The interface quickly drew the children’s attention, but their interest varied; adding words matched to a child’s preferences allowed further personalization.T1 reported that some children became bored if an exercise did not engage them.Source: §7.1 · p. 10

7.2 Evaluation of Progress

  • Although concentration and participation generally improved, some children lost interest over time; adding a preferred word could renew attention, and a successful new pronunciation could boost confidence.Source: §7.2 · p. 11

7.3 The Concept of Dose-Effect

  • Short, regular exercise sequences and gradually introduced words were favored to avoid overwhelming children, but personalization required balancing setup time against direct interaction.T1 noted that too many exercises in a row could cause children to disengage.Source: §7.3 · p. 11

7.4 Impact on Verbal Skills

  • The eight-child pilot suggested the interface could improve motivation and language skills, but children varied in verbal and motor performance and some rushed repetitions without accurate pronunciation.New, more complex words also supported targeted work on sounds for one child.Source: §7.4 · p. 11

8 Discussion

The discussion reports motivational, speech, and engagement benefits from rhythmic gestures, while identifying personalization and long-term validation as important considerations. It also proposes possible extensions to language learning, other rehabilitation settings, and neuroeducation.

Discussion

  • The interface enhanced children’s motivation and engagement, while adding preferred words made sessions more flexible and better suited to individual needs.The authors also report that these customizations improved session efficacy.Source: §8 · pp. 11–12
  • The adaptation of a motor-rehabilitation tool to speech therapy leverages the cognitive link between gesture and language, extending the tool’s application beyond motor skills.The discussion connects this finding to McNeill.Source: §8 · p. 12
  • Controlled tempo facilitated smoother speech production, which the authors suggest gave children more time to articulate clearly and reduced cognitive load.The authors relate this approach to slow speech-rate techniques used in stuttering therapy.Source: §8 · p. 12
  • The interface’s synchronized auditory, visual, and motor stimuli were associated with stronger engagement, while real-time feedback reinforced the association between actions and verbal expressions.The discussion relates multisensory input to learning and memory retention, and immediate auditory and visual feedback to learning.Source: §8 · p. 12
  • Children showed increased verbal output; the authors suggest that interactivity and personalized exercises may have encouraged participation and spontaneous speech.The discussion connects movement in language learning to verbal production in Asher’s work.Source: §8 · p. 12
  • Whether progress during sessions produces durable gains in language acquisition and fluency remains a question for future research.The discussion cites research on lasting neural connectivity effects from rhythmic and motor-based interventions.Source: §8 · p. 12
  • The authors propose exploring rhythmic gestures for vocabulary learning, aphasia rehabilitation, neurostimulation, and neuroeducation, while also suggesting speech cues as a possible aid to motor rehabilitation.These are proposed applications, not outcomes established by this study; the discussion cites motor activities in language learning, Total Physical Response, aphasia, neurostimulation, and physical activity and executive functions.Source: §8 · pp. 12–13

9 Conclusions

The conclusion presents the adaptation of a manual-dexterity rehabilitation interface for speech therapy as a way to engage children with speech disorders through the connection between gestures and speech. It describes the findings as preliminary and calls for broader, long-term validation.

9 Conclusions

  • The study adapted a rhythmic interface originally designed for manual-dexterity rehabilitation for use in speech therapy.The authors report that it enhanced motivation and therapeutic engagement in children with speech disorders.Source: §9 · p. 13
  • The authors identify possible applications in speech improvement, neurorehabilitation, and education, where synchronized gestures and speech could support language learning and cognitive development.These are proposed cross-domain possibilities, not validated outcomes in those settings.Source: §9 · p. 13
  • The findings are preliminary, and further studies are needed to validate the interface in broader, long-term therapeutic contexts.Source: §9 · p. 13

A Interview Protocol

The interview protocol was designed to gather clinicians’ perspectives on integrating a rhythm-based system into therapy and to inform its development. It moved from current practice to an interface demonstration and feedback, including suggestions for patient testing and evaluation.

Introduction to the Interview

  • The introduction framed interviews as a way to learn how a finger-tapping system for synthesized vocal phrases might fit therapeutic practice and meet clinician and patient needs.The protocol also sought clinicians’ views on potential benefits and challenges.Source: Introduction to the Interview · p. 16

Part 1: Questions about Current Practice

Part 2: Demonstration

Part 3: Questions about the Interface

  • The final part solicited interface feedback, examples of suitable patients and content, and suggestions for real-world testing, evaluation criteria, and success measures.It also asked whether the interview questions were relevant and what else should be included.Source: Part 3: Questions about the Interface · p. 17

B Challenges and Adaptive Strategies in Rehabilitation Sessions

Table 3 summarizes challenges, successful strategies, and adaptations observed in speech and physical therapy sessions.

B Challenges and Adaptive Strategies in Rehabilitation Sessions

  • Table 3 covers challenges, successful strategies, and adaptations observed during speech and physical therapy sessions.Source: Table 3 · p. 18
Loading 2609.08486v1…