PaperlayerView exact revision on arXiv

Source-linked AI summary

Silent Speech Interfaces for Speech Restoration: A Review

Jose A. Gonzalez-Lopez, Alejandro Gomez-Alanis, Juan M. Martín-Doñas, José L. Pérez-Córdoba, Angel M. Gomez

arXiv:2009.02110v3eess.AScs.HCcs.LG

TL;DR

Severe speech disorders can make ordinary verbal communication impossible, motivating alternatives that do not rely on acoustic speech. This review surveys SSI biosignal sensing and speech decoding, finding evidence of feasibility alongside persistent speaker- and session-dependence.

Abstract

from arXiv · show

This review summarises the status of silent speech interface (SSI) research. SSIs rely on non-acoustic biosignals generated by the human body during speech production to enable communication whenever normal verbal communication is not possible or not desirable. In this review, we focus on the first case and present latest SSI research aimed at providing new alternative and augmentative communication methods for persons with severe speech disorders. SSIs can employ a variety of biosignals to enable silent communication, such as electrophysiological recordings of neural activity, electromyographic (EMG) recordings of vocal tract movements or the direct tracking of articulator movements using imaging techniques. Depending on the disorder, some sensing techniques may be better suited than others to capture speech-related information. For instance, EMG and imaging techniques are well suited for laryngectomised patients, whose vocal tract remains almost intact but are unable to speak after the removal of the vocal folds, but fail for severely paralysed individuals. From the biosignals, SSIs decode the intended message, using automatic speech recognition or speech synthesis algorithms. Despite considerable advances in recent years, most present-day SSIs have only been validated in laboratory settings for healthy users. Thus, as discussed in this paper, a number of challenges remain to be addressed in future research before SSIs can be promoted to real-world applications. If these issues can be addressed successfully, future SSIs will improve the lives of persons with severe speech impairments by restoring their communication capabilities.

I. INTRODUCTION

The introduction frames silent speech interfaces as an assistive route to restoring communication when ordinary speech is unavailable. It reviews biosignal sensing and how disorder characteristics can affect sensor suitability.

I. INTRODUCTION

  • Speech and language impairments disrupt daily communication and can contribute to isolation, social withdrawal, and depression, –.Assistive and augmentative communication devices range from paper-and-pencil resources to picture boards and text-to-speech software.Source: §I · p. 1
  • SSIs decode speech from non-acoustic biosignals, including signals captured through imaging, articulography, surface EMG, and EEG.EEG records neural activity involved in speech production; surface EMG captures electrical activity driving facial muscles.Source: §I · pp. 1–2
  • This review surveys recent SSI advances for communication restoration, including disorders, decoding approaches, sensing modalities, and current challenges.The review focuses on communication restoration for speech-disabled individuals.Source: §I · p. 2

II. SPEECH AND LANGUAGE DISORDERS

  • Sensor suitability depends on the disorder; electrical brain or facial-muscle signals may suit dysarthria better than camera-based articulator tracking.The review notes that dysarthria can impair movement and coordination of the lips and tongue.Source: §II · p. 2
  • The review describes speech disorders and their causes, effects, and currently available communication aids, including SSI investigations.Speech can become difficult or impossible when vocal organs, speech-related brain areas, or neural pathways are damaged or altered.Source: §II · p. 2

A. Aphasia

The review describes communication challenges across aphasia, motor speech disorders, and voice disorders. It summarizes existing communication aids and SSI studies, noting both demonstrated feasibility and practical constraints.

A. Aphasia

  • Aphasia affects language comprehension and formulation, while communication aids such as voice-output devices can be slow and unsuitable for deeply paralysed patients.Aphasia may follow stroke, head injury, cerebral tumours, or neurodegenerative disorders; post-stroke aphasia was the focus of one five-person visual-speller pilot, with participants spelling individual words at up to 100% accuracy.Source: A. Aphasia · p. 2
  • P300 visual-speller systems enabled aphasic patients to communicate, but were slow, cognitively demanding, and unintuitive; reported rates reached 10–12 words/min.Controls achieved significantly higher spelling accuracy than aphasic participants, who nevertheless used the system successfully.Source: A. Aphasia · p. 3

B. Apraxia and dysarthria

  • 96% average word recognition accuracy was reported for combined audio and EMG, compared with 95% for audio-only and 85% for EMG in speaker-dependent dysarthric-speaker ASR.Speaker-independent systems performed very poorly; speaker-dependent acoustic models tailored to each subject yielded the best scores.Source: B. Apraxia and dysarthria · p. 3
  • Speaker-dependent systems require substantial training data, while articulatory normalisation was effective for reducing speaker variation, especially with combined acoustic and articulatory data.The reviewed studies support the viability of silent speech recognition using articulatory data from dysarthric speakers.Source: B. Apraxia and dysarthria · p. 4

C. Voice disorders

  • After laryngectomy, substitute voices can be difficult to understand and modulate, while SSI speech recognition and synthesis have been demonstrated across several biosignal modalities.The reviewed SSI proposals were validated mostly with healthy users, with some notable exceptions.Source: C. Voice disorders · p. 4

III. SILENT SPEECH INTERFACES

SSIs decode speech from biosignals generated by processes that remain unimpaired for a person. Speech-to-text systems turn biosignal features into word sequences using ASR models adapted to the biosignal.

III. SILENT SPEECH INTERFACES

  • SSIs can use biosignals from unimpaired speech-production processes, including neural, peripheral motor, articulatory, phonation, and pulmonary activity.The available signals depend on which processes remain intact for a person.Source: §III · p. 4
  • An SSI can decode biosignal features into text with ASR, then optionally synthesize the decoded text, or generate audible speech directly without an intermediate recognition step.Direct synthesis commonly uses DNNs trained on time-aligned speech and biosignal recordings.Source: §III · p. 4

A. Silent speech to text

  • Silent ASR converts raw biosignals into framewise feature vectors, whose representation depends on the sensing modality, such as image features for lipreading or articulator coordinates for EMA and PMA.The feature sequence is evaluated against word sequences using acoustic and pronunciation lexicon models.Source: A. Silent speech to text · p. 5
  • ASR searches for the most likely word sequence by combining a word-sequence prior with the likelihood of the observed feature vectors given that sequence.The language model is biosignal-agnostic; the acoustic model depends on the biosignal.Source: A. Silent speech to text · p. 5
  • HMM acoustic likelihoods marginalize over state sequences, commonly using a first-order Markov assumption for state transitions.The Viterbi algorithm is one efficient way to search for the most likely state sequence; silent-speech subword units can be difficult to distinguish, and multimodal signals may be asynchronous.Source: A. Silent speech to text · pp. 5–6

B. Direct speech synthesis

Direct synthesis maps biosignal features to acoustic speech features, most often through data-driven regression. Approaches range from Gaussian mixture models to neural networks, while mapping ambiguity and computational demands constrain some methods.

B. Direct speech synthesis

  • Direct synthesis models a mapping from biosignal feature vectors to acoustic feature vectors; the mapping can be nonlinear and non-unique for some biosignals.Limited spatial or temporal sensing resolution can leave aspects of speech production uncaptured.Source: B. Direct speech synthesis · p. 6
  • Model-based synthesis can infer vocal-tract shape from articulator measurements and simulate airflow to produce speech, but accurate physical models can be computationally expensive.Reported 3D finite-element simulations took 70–80 hours for 20 ms vowels in one work and an average of six hours for 0.2 s diphthongs in another.Source: B. Direct speech synthesis · p. 6
  • Data-driven synthesis estimates a mapping from time-synchronous biosignal and speech recordings, typically collected while the speaker’s voice is intact or not severely impaired.The target vectors represent a compact acoustic parametrisation, and a vocoder can synthesize the final signal from predicted acoustic features.Source: B. Direct speech synthesis · pp. 6–7
  • Gaussian mixture regression predicts acoustic features from biosignal features using the posterior mean, but framewise conversion can produce perceivable discontinuities.Maximum-likelihood parameter generation incorporates static and dynamic speech features to produce better acoustics than conventional GMM mapping.Source: B. Direct speech synthesis · p. 7
  • Trajectory-based conversion jointly models source, static speech, and dynamic speech features, then optimizes the sequence of static speech features.The MLPG algorithm solves this optimization using an iterative EM-based method.Source: B. Direct speech synthesis · p. 7
  • Studies report that DNNs outperform GMM and HMM mappings in conversion quality for EMG and PMA, with several neural architectures explored for direct synthesis.Reported architectures include feed-forward, convolutional, and recurrent networks.Source: B. Direct speech synthesis · p. 7

C. Comparison of the two SSI approaches

Silent speech-to-text and direct synthesis trade off vocabulary constraints, paralinguistic information, training demands, and latency. Direct synthesis supports language-independent output and is the approach associated with low-latency SSI operation.

C. Comparison of the two SSI approaches

  • Silent speech-to-text can benefit from language and pronunciation models but cannot recognize words outside its training vocabulary; direct synthesis is language-independent and not vocabulary-limited.ASR also loses paralinguistic features such as speaker identity or mood, which direct synthesis could recover.Source: C. Comparison of the two SSI approaches · p. 7
  • Silent speech-to-text is difficult to train at large vocabulary scale because enough silent-speech data is hard to record, whereas direct synthesis usually requires only a few hours of training data.The passage attributes the difference to biosignal-to-speech mapping being arguably easier than training a full speech recognizer.Source: C. Comparison of the two SSI approaches · p. 7
  • Direct synthesis is the approach that can achieve the low latency sought for SSI feedback; the review suggests an ideal latency of 50 ms and that up to 100 ms may be acceptable.The review connects longer delays with hesitation or disruptive effects on speech production.Source: C. Comparison of the two SSI approaches · pp. 7–8

IV. SENSING TECHNIQUES

Speech production involves coordinated brain, muscle, articulator, glottal, and pulmonary processes. Sensing techniques can measure these processes at their origin, their muscular effects, or the resulting movements.

IV. SENSING TECHNIQUES

  • Speech production involves articulator movements that shape the vocal tract, driven by muscles and ultimately initiated and coordinated by the brain.The glottis and lungs also contribute to speech production.Source: §IV · p. 8
  • Sensing can record biosignals at their origin through brain-activity measures or at their destination by monitoring muscle activity.The section reviews available sensor technology and SSI research using these approaches.Source: §IV · p. 8
  • A third sensing strategy measures the effects of brain and muscle activity by tracking articulator movements.The relevant biosignal depends on which speech-production processes are being measured.Source: §IV · p. 8

A. Brain activity

Brain-activity sensing spans neural origin signals and downstream articulator control, but brain processes remain imperfectly understood and high-resolution recording is difficult. The section reviews speech-related cortical organization and sensor technologies with distinct spatial, temporal, and practical trade-offs.

A. Brain activity

  • Brain activity sensing may address voice disorders, dysarthria, apraxia, and some aphasia cases, though speech-related brain processes remain imperfectly understood and high-resolution recording is problematic.The section contrasts sensing at speech-production origin with monitoring muscle activity or articulator movement.Source: A. Brain activity · p. 8

1) Neuroanatomy of speech production:

  • The dual-stream model links the ventral temporal stream to comprehension and the dorsal stream to mapping acoustic speech onto frontal articulatory networks.The dorsal stream is left-dominant; the ventral sensorimotor cortex coordinates articulator representations for speech motor patterns.Source: 1) Neuroanatomy of speech production · p. 8

2) Brain activity sensors:

  • Brain sensors measure either haemodynamic responses or electrical and magnetic activity generated by neuronal currents.Haemodynamic methods track oxygenation changes; electrical recordings include EEG, ECoG, and LFPs, while MEG measures ultra-weak magnetic fields.Source: 2) Brain activity sensors · pp. 8–9
  • Haemodynamic methods offer noninvasive sensing and excellent spatial resolution, but their responses lag at least 12 s and yield limited speech-recognition results.Trial repetition can improve temporal resolution to 1–2 s; few speech-recognition studies have used these methods.Source: 2) Brain activity sensors · p. 9
  • EEG records neural potentials noninvasively with sub-millisecond timing, but spatial smoothing, artefacts, and movement distort its signals.These limitations lead EEG-based AAC to rely mainly on time-locked averages or broad neural features.Source: 2) Brain activity sensors · p. 9
  • Invasive ECoG and LFP recordings are considered promising for chronic neural speech prostheses, but insertion can damage tissue and electrode performance may degrade.A neurotrophic electrode can support lifetime recordings, but its probe is limited to four wires.Source: 2) Brain activity sensors · pp. 9–10

3) SSIs based on brain activity signals:

Brain-activity SSIs have progressed from EEG-based synthesis and small-vocabulary recognition toward higher-resolution invasive recordings. ECoG studies report speech decoding and synthesis results, including intelligible closed-vocabulary outputs and real-time approaches.

3) SSIs based on brain activity signals:

  • EEG-based speech approaches remain limited by low spatial resolution, with early recognition studies covering only small vocabularies, phonemes, or syllables.One synthesis approach decoded imagined limb movements rather than evoking speech production.Source: 3) SSIs based on brain activity signals · p. 10
  • ECoG word and phone error rates below 25% and 50%, respectively, were reported for a brain-to-text system.A later seq2seq ECoG system reported WERs as low as 3%.Source: 3) SSIs based on brain activity signals · p. 10
  • A wireless LFP-based BCI drove two formant frequencies for vowel synthesis, completing decoding within 50 ms in a locked-in participant.The system used a Kalman filter and provided auditory feedback.Source: 3) SSIs based on brain activity signals · p. 10
  • Listeners readily identified and transcribed ECoG-synthesised speech for closed vocabularies of 25 and 50 words.The system decoded articulatory kinematics first, then acoustic features.Source: 3) SSIs based on brain activity signals · p. 10
  • ECoG-to-speech systems reported decoded-signal intelligibility above 50% in some cases, and TTS strategies yielded more intelligible speech with real-time implementation [211],.The approaches mapped ECoG recordings to speech features before synthesis.Source: 3) SSIs based on brain activity signals · p. 11

B. Muscle activity

EMG captures muscle activity associated with articulation, including during inaudible speech, and is widely used in SSI investigations. Research spans recognition and direct synthesis, while session dependence, speaking-mode mismatch, and speaker dependence remain challenges.

B. Muscle activity

1) EMG and speech production:

  • EMG patterns from tongue and facial muscles can retrieve speech-related information even when words are spoken inaudibly, and EMG signals precede articulatory motion by 60 ms.This timing supports low-latency EMG-to-speech conversion and may suit voice disorders such as laryngectomy, or some speech disorders such as dysarthria.Source: 1) EMG and speech production · p. 11

2) SSIs based on EMG signals:

  • EMG recognition research progressed from small-vocabulary studies to sentence recognition at about 70% in a single-speaker setup and later large-vocabulary methods,.Bundled phonetic features model interdependences among phonetic features using decision-tree clustering,; later work explored hybrid EMG-DNNs and transfer learning,.Source: 2) SSIs based on EMG signals · pp. 11–12
  • Direct EMG speech synthesis has advanced with array sensors and deep learning, while the signal’s 60 ms lead over articulation supports low-latency feedback,,,.A study assessed effects of DNN size, training-data amount, and frame shift on generated speech quality.Source: 2) SSIs based on EMG signals · p. 12
  • Session differences persist despite array sensors, and audible-versus-silent mismatch can degrade ASR; spectral mapping with multi-style training improved recognition rates by 14.3%,,.The reported average WERs were 34.7% for silently mouthed speech and 16.8% for audibly spoken speech.Source: 2) SSIs based on EMG signals · p. 12
  • Most EMG SSI studies remain speaker-dependent, training ASR or synthesis models on data from the end user.The review identifies speaker independence as an unresolved issue that could reduce the amount of user-specific training data required.Source: 2) SSIs based on EMG signals · p. 12

C. Articulator motion capture

Articulator-motion sensing provides speech-related biosignals by tracking movement, and is particularly suited to voice disorders when glottal activity is not captured. Magnetic articulography offers EMA and PMA variants with different portability and processing trade-offs.

C. Articulator motion capture

  • Articulator-motion techniques are best suited to voice disorders such as laryngectomy when they do not capture glottal activity.The section covers kinematic sensors attached to the vocal tract and imaging techniques that visualize articulator movement.Source: C. Articulator motion capture · p. 12

1) Magnetic articulography:

  • EMA tracks receiver coils in three-dimensional space with high temporal resolution and little preprocessing, but wired connections and external transmitters restrict it to laboratory use,.The captured data include coil coordinates, velocity, and acceleration.Source: 1) Magnetic articulography · p. 12
  • EMA-based ASR with bidirectional RNNs and normalization achieved state-of-the-art performance against other ASR models on a dataset including healthy and laryngectomised speakers.The dataset included twelve healthy and two laryngectomised English speakers.Source: 1) Magnetic articulography · p. 12
  • PMA uses magnets on vocal articulators and external magnetic sensors, avoiding wired connections but requiring preprocessing because sensor signals combine the magnets’ fields.This design makes PMA more portable and comfortable than EMA.Source: 1) Magnetic articulography · p. 13, Figure 3 · p. 13
  • PMA speech-synthesis models produced speech averaging about 75% intelligibility, reaching about 92% for some participants,.Recognition studies also reported 97% word and 94% phoneme accuracy for an isolated-word and phone task [24].Source: 1) Magnetic articulography · p. 13

2) Palatography:

The review surveys palatography and imaging approaches for capturing articulator movement, alongside their use in speech recognition and synthesis. It also identifies technical and clinical challenges that keep SSIs from real-world use.

2) Palatography:

  • EPG records tongue–palate contact timing and location, while OPG uses optical sensors to track tongue position and lip movements without palate contact.EOS combines EPG and OPG; reviewed work applies these techniques to articulation recognition and tongue-contour reconstruction.Source: 2) Palatography · pp. 13–14

3) Imaging techniques:

  • Imaging-based systems have mapped ultrasound and lip features to speech, while deep-learning methods have supported recognition, synthesis, and joint optimization.One multimodal CNN achieved 80.4% recognition accuracy, and multitask learning reported about 7% relative error-rate reduction for both recognition and synthesis.Source: 3) Imaging techniques · p. 14

V. CURRENT CHALLENGES AND FUTURE RESEARCH DIRECTIONS

  • Most SSIs remain laboratory-tested, and consistent intelligible speech for large vocabularies or conversation, training without acoustic data, and clinical deployment remain unresolved.The review also calls for improved sensing and decoding, online user-in-the-loop evaluation, and determining the role of SSIs in speech-language therapy; that clinical question lies beyond its scope.Source: §V · pp. 14–15

A. Improved sensing techniques

The review describes practical sensing limits and the requirements for more capable sensors, then considers training approaches for clinical cases without time-aligned biosignal and speech recordings.

A. Improved sensing techniques

  • Current sensing is often uncomfortable or fragile in real-world use, can miss parts of the vocal tract, and may recover little prosody for laryngectomised patients.The review identifies sensor misalignment, limited tongue-motion capture with sEMG, and difficulty sensing back-of-vocal-tract phones with EMA/PMA; combining sensors may broaden coverage.Source: A. Improved sensing techniques · p. 15
  • Wireless, robust, low-power sensors are needed, while invasive BCI systems additionally require biocompatible probes for long-term recordings across multiple brain areas.The review describes dense neural probes and approaches to reducing tissue reaction and probe–tissue motion, but notes that some technologies have only been tested in animals.Source: A. Improved sensing techniques · pp. 15–16

B. Training with non-parallel data

  • When biosignal and speech data are not time-aligned, proposed options include time-warping, sequence-to-sequence mapping, and non-parallel adversarial training.Sequence-to-sequence methods may preclude real-time synthesis because they normally process the whole sequence at once; CycleGAN-based approaches can avoid parallel data and alignment.Source: B. Training with non-parallel data · p. 16
  • Model-based synthesis can avoid requiring time-synchronous user speech recordings, but needs vocal-tract shape from biosignals or a mapping to articulator kinematics.The review notes that direct recovery of vocal-tract shape is available only for certain capture techniques, such as EMA.Source: B. Training with non-parallel data · p. 16

C. Online and incremental training algorithms

The review calls for incremental, feedback-driven learning and examines how changes across sessions, speaking modes, and speakers affect SSI performance. It also notes that speaker-independent direct synthesis remains an open problem.

C. Online and incremental training algorithms

  • Reinforcement learning is proposed to adapt SSI control from user feedback as skills or illness change, while requiring little manual intervention and minimal labelled data.The review points to prior applications in robotic prosthesis control and computational models of infant vocal learning.Source: C. Online and incremental training… · p. 16
  • Incremental SSI training should begin with a useful limited system, then expand as practice supplies data and improves user control.The proposed “virtuous circle” aims to reduce frustration and early abandonment while gradually adding vocabulary or continuous speech.Source: C. Online and incremental training… · pp. 16–17

D. Robustness against inter- and intra-subject variability

  • Wearable SSIs remain far from session-independent, and mismatched audible and silent speaking modes can reduce reconstruction accuracy.Multi-style training and signal mapping are discussed as ways to address speaking-mode differences; an ECoG experiment attributed lower covert-speech accuracy to those differences.Source: D. Robustness against inter- and… · p. 17
  • Speaker-independent EMA recognition accuracy rose from 68.63% to 95.90% after Procrustes matching, while significant progress in speaker-independent direct synthesis remains absent.The review suggests adapting an average voice model toward the target user's voice as a possible direction for direct synthesis.Source: D. Robustness against inter- and… · pp. 17–18

E. Validation in a clinical population

The review finds that SSI evidence is concentrated in healthy users and offline tests, while clinical validation, realistic evaluation, and larger shared datasets remain important gaps. It concludes that speech decoding is possible, but current systems are not yet robust enough for widespread use outside laboratories.

E. Validation in a clinical population

  • SSI studies have mostly involved healthy users, with clinical recordings limited by difficult data collection, lack of parallel training data, patient variability, recruitment, and ethics.Implanted sensors in clinical participants were generally placed for other neurological conditions, without optimizing placement for language-processing areas.Source: E. Validation in a clinical population · p. 18

F. Evaluation in more realistic scenarios

  • Most SSI evaluations use offline, prerecorded data, whereas real-world assessment calls for online, prolonged tests with diverse users and goal-oriented dialogue.Dialogue can involve contextual and visual cues that help resolve word meaning.Source: F. Evaluation in more realistic… · p. 18

G. Availability of public datasets

  • Small, fragmented datasets make technical comparisons difficult, and TORGO is the only large database identified with aligned acoustic and EMA data from dysarthric speakers and controls.The review describes TORGO as about 23 hours of data from eight dysarthric speakers and seven controls.Source: G. Availability of public datasets · p. 18

VI. CONCLUDING REMARKS

  • The review covers sensing from articulator movement to brain activity and concludes that present-day SSI performance and robustness are insufficient for large-scale use outside laboratories.It identifies better sensors, new training algorithms, non-parallel and zero-data scenarios, and clinical validation as needs before SSIs could become a communication option for people with severe deficits.Source: §VI · p. 18
Loading 2009.02110v3…