PaperlayerView exact revision on arXiv

Source-linked AI summary

Myovox: Reading Speech from the Muscles of the Face

Varshith Madishetty

arXiv:2609.17548v1cs.CL

TL;DR

Myovox asks whether vocalized English text can be decoded from facial-muscle sEMG without relying on audible speech at inference. It restores the missing decode, replaces the causal encoder with a full-context Conformer, and adds ensembling, n-best union, and language-model reranking. The resulting system reaches 18.53% WER on the emg2speech General Corpus, while the remaining limit is the acoustic phone error rate rather than reranking.

  • Problem

    Myovox addresses open-vocabulary English text decoding from facial-muscle activity during vocalized speech, with the broader silent-speech goal left beyond this healthy-speaker demonstration.

  • Method

    The report makes isolated, staged changes: restores missing open-vocabulary decoding settings, uses a bidirectional Conformer with audio-feature distillation, then ensembles acoustic models and reranks their n-best union with a QLoRA-fine-tuned language model.

  • Results

    18.53% WER is the best reported result on the emg2speech General Corpus, improving from the published 51.17% WER through three separable moves.

  • Takeaways & Limitations

    The report’s central practical conclusion is that decode-time gains eventually stop because correct words are absent from the acoustic posteriors, making acoustic phone accuracy the binding constraint.

  • Takeaways & Limitations

    All results come from one healthy speaker, so generalization to other people, anatomies, and speaking styles is untested.

Abstract

from arXiv · show

Myovox, from myo (muscle) and vox (voice), decodes open-vocabulary English text from 31-channel surface electromyography (sEMG) recorded from the muscles of the face during vocalized speech. It takes the single-subject emg2speech General Corpus from a published 51.17% word error rate to 18.53%, in three separable moves, each measured in isolation. First, I recover the open-vocabulary decode settings missing from the public release and reach a faithful 40.63% WER / 39.02% PER baseline whose phone error rate matches the published one to within 0.8 points, so the acoustic model is reproduced faithfully. Second, I replace the causal encoder with a bidirectional Conformer trained by a four-term cross-modal distillation against the parallel audio's WavLM-Large layer-9 features, reaching 26.14% WER / 22.34% PER from the electromyography alone. Third, I ensemble two acoustic models, union their multi-scale n-best lists, and rerank with a QLoRA-fine-tuned 7B language model, reaching 18.53% WER, the best result reported on this corpus, though not the best reported for sEMG-to-text on other corpora (Section 2). I then report the negative result that bounds the whole approach: reranking is exhausted at 18.5% because the binding constraint is the electromyographic acoustic phone error rate (~20.9%), not the language model. The correct words are simply absent from the acoustic posteriors, so no reranker can reach the 9.30% n-best oracle. All test numbers are on the 400-sentence held-out test set under the authors' official 8,500 / 760 / 400 sequential split; every hyperparameter is tuned once on validation and applied once to test.

1 Introduction

Myovox decodes English text from facial-muscle electrical activity during vocalized speech, while the report emphasizes a disciplined, decomposable path from published baseline to final result and identifies phone errors as the system-wide limit.

  • Myovox converts 31-channel facial, jaw, and throat muscle voltages into open-vocabulary English text during vocalized speech.The sensors record faint electrical activity produced as speech-related muscles fire.
  • The long-term silent-speech medical goal is not demonstrated because training used one healthy speaker who vocalized normally.The demonstrated use case is inaudible communication by a healthy person, treated as a stepping stone toward silent speech.
  • The report’s held-out evaluation uses 400 test sentences under the authors’ 8,500 / 760 / 400 sequential split.Table 1 states that all test numbers use this split.
  • 18.53% WER is the final reported result on this corpus, decomposed into a decode correction, modeling gain, and extraction gain.The report presents this progression as separable changes rather than one undifferentiated improvement.
  • PER measures decoder-independent acoustic phone errors, whereas WER measures word errors after substitutions, insertions, and deletions.The report argues that PER cannot be improved by a decoder or language model and therefore limits the full system.

2 Related work

Myovox is positioned among non-invasive muscle- and brain-based speech decoders, with comparisons constrained by differing corpora, sensors, subjects, speaking modes, and metrics.

  • Surface EMG taps facial and neck muscles near articulation, whereas electro- and magnetoencephalography tap cortex earlier in the motor chain.Myovox belongs to the first non-invasive route; invasive intracortical systems are excluded.
  • Table 2 groups non-invasive speech and language decoders by corpus and warns that their reported numbers are not comparable across groups.The caption cites differences in corpora, electrode counts, speaking modes, subject counts, and metrics.
  • 18.53% WER is the best reported result on the emg2speech General Corpus, but not the best sEMG-to-text result across corpora.MONA LISA reports 3.7% on a different corpus, so the values are not directly comparable.
  • Brain2Qwerty provides a related non-invasive brain-decoding comparison, including Conformer-plus-LoRA convergence from a different modality.Its reported results span 32% CER from MEG, 67% from EEG, and 39% WER from MEG across versions.
  • Myovox contributes missing open-vocabulary decode settings, a control separating full-context modeling from audio distillation, and evidence locating decode-time limits.The report states that none of these contributions is a new architecture.

3 The data

The evaluation uses a single healthy speaker’s General Corpus with parallel audio, an open-vocabulary lexicon, and a fixed sequential train-validation-test split.

  • The entire system is trained and evaluated on one healthy speaker, so transfer to other people, anatomies, and speaking styles is unknown.The results demonstrate what is possible for one speaker rather than cross-subject generalization.
  • Parallel microphone audio is available during vocalized speech and serves as a training-time teacher, but not as a silent-speech inference input.Per-sentence EMG and audio durations correlate at 1.000, and the report checks dependence on this teacher.
  • The decoder uses a 34,546-word LibriSpeech-derived lexicon, enabling words unseen during training.Twelve of 2,429 test tokens fall outside the lexicon, creating a 0.49% WER floor.
  • All hyperparameters are tuned on 760 validation sentences and applied once to 400 test sentences in recording order.The split is kept exactly as authored: 8,500 training, 760 validation, and 400 test sentences.

4 Baseline: reproduction and a decode correction

The baseline keeps the released acoustic model but restores omitted open-vocabulary decode settings, producing a faithful 40.63% WER / 39.02% PER reproduction.

  • Acoustic model: The released DualHeadTDSCTC uses covariance features, dual CTC heads for HuBERT units and phonemes, and an HLG weighted finite-state decoder.The dual-CTC loss is 0.8 · CTCunit + 0.1 · CTCphone + 0.1 · consistency.
  • What was missing: The public release omits decode-time settings and reaches roughly 75% WER despite reproducing the acoustic model.The missing configuration separates good posteriors from good word outputs.
  • Decode correction: A blank penalty is the largest baseline lever, reducing validation WER from 77.6% to 60.6% when swept from 0 to 2.The released CTC posteriors are dominated by blank probability, approximately 0.92, and no penalty was applied.
  • Decode correction: Selecting checkpoints by validation PER lowers test PER from 42.9% to 39.0%, while the tuned baseline acoustic scale remains 1.0.The missing words.txt was regenerated, and the blank/scale pair applied to test was (2.0, 1.0).
  • Result: 40.63% WER / 39.02% PER is the corrected test baseline, 10.5 WER points below the published 51.17% while matching PER within 0.8 points.The matched phone error rate supports faithful acoustic-model reproduction; the WER gain is attributed to the corrected decode rather than model changes.

5 The acoustic model: a full-context Conformer taught by the audio

A bidirectional Conformer with cross-modal audio distillation reduces acoustic error substantially, while a control shows that full context—not the audio teacher—drives word-level gains.

  • Full context: The released causal encoder is replaced with a bidirectional Conformer because offline transcription can use the whole sentence rather than only past frames.The replacement retains the covariance front-end and two CTC heads while adding full-context self-attention and convolution.
  • Cross-modal distillation: The four-term objective aligns projected EMG features with parallel audio through regression, contrastive learning, and phoneme CTC alongside the baseline acoustic losses.A frozen WavLM-to-phoneme recognizer makes feature matching target phoneme-decodable representations rather than smooth but uninformative audio-like features.
  • Results: 26.14% WER / 22.34% PER: the full-context distilled Conformer improves over the 40.63% / 39.02% baseline by 14.49 and 16.68 points.The improvement appears in greedy phone error rate without a language model and is significant under paired bootstrap testing.
  • Control: 26.10% WER: an otherwise identical electromyography-only Conformer matches the distilled model’s 26.14% WER at the word level.Distillation still improves PER from 23.71 to 22.34, but the word-level gain comes from the full-context encoder.

6 The final pipeline: ensemble, union, and a language-model reranker

The final pipeline improves decoding without changing the acoustic model by combining two encoders, unioning multi-scale candidates, and reranking them with a fine-tuned language model.

  • Pipeline design: The acoustic model remains fixed during decode-time optimization; only ensemble averaging, candidate-pool construction, and language-model reranking change.The reranker maps candidate lists plus detected phonemes to the reference, with free and constrained variants selected on validation.
  • Acoustic ensemble: 23.47% WER: averaging two acoustic models improves over the 26.14% distilled model, while ensemble decoding ranges from 23.5% to 25.1% across acoustic scales.The ensemble lowers greedy PER to 20.9%, although the augmented member does not have better phone error than the distilled member.
  • Multi-scale n-best union: 9.30% oracle WER: multi-scale n-best union lowers the oracle from 11.94%, creating recoverable headroom above the union’s 23.26% single-best result.The reranker recovers 4.7 of those points, while the remaining nine are analyzed as out of reach in Section 7.
  • Language-model reranking: 18.53% WER: QLoRA reranking improves the union’s 23.26% single-best output by 4.7 points, with bootstrap intervals excluding zero.The reranker is trained only on training data, uses cross-decoded training candidates, and passes a zero-count verbatim-recall audit.

7 Why it stops at 18.5%

Reranking stops near 18.5% because acoustic phone errors, not language modeling, determine which words enter the candidate pool. Multiple diagnostics show that improving downstream decoding cannot recover words absent from the acoustic posteriors.

  • Downstream interventions changed PER only incrementally: audio distillation improved it from 23.71 to 22.34, while ensembling reduced it from 22.34 to 20.90.Together, post-encoder changes moved PER by 1.44 points, compared with 16.68 points from full context alone.
  • The electromyography-only Conformer reached 26.10% WER versus 26.14% for the distilled model, showing that the audio teacher did not transfer a word-level advantage.The teacher itself was only an approximately 10% PER recognizer.
  • 18.53% WER remains above the 9.30% n-best oracle because the correct words are absent from the acoustic candidate pool.Free generation would amount to hallucination, and the audit reports zero recall for those missing words.
  • 20.9% acoustic PER is the binding constraint, so reaching roughly 10% WER requires a better electromyographic acoustic model rather than a larger language model.The report identifies more data, multiple subjects, or a stronger front-end as possible directions.
  • The same upstream-encoder bottleneck is reported for Brain2Qwerty v2, whose authors identify encoder quality as the dominant term.That system’s fine-tuned language model improved WER while worsening CER.

8 Limitations •

The evaluation is limited by a single healthy speaker, vocalized speech, corpus-specific comparisons, and characteristics of the fixed sequential split. Several reported gains and scores also leave important questions unresolved.

  • All results come from one healthy speaker, and the encoder memorizes training sentences, leaving cross-subject robustness untested.Training PER is far below the approximately 27% validation PER.
  • 18.53% is the best result on the emg2speech General Corpus, but MONA LISA reports 3.7% on a different vocalized-EMG corpus.The recordings differ, so the gap cannot be assigned to corpus or model from this report.
  • The headline score is measured on the easier test segment of the fixed sequential split because both systems perform better on test than validation.The report therefore gives both columns rather than only the more favorable one.
  • WavLM distillation uses parallel audio during training, which is unavailable during silent-speech inference, although the EMG-only encoder matches it at word level.
  • The ensemble’s 1.4-point PER and 2.7-point WER gains do not separate sharper argmax effects from posterior reshaping.The report explicitly leaves that attribution unresolved.
  • Six of the 400 test sentences duplicate training text, producing 18.53% with duplicates versus 18.75% without them.
  • The 18.53% reranking result is bounded by acoustic PER, while reranking still contributes a 4.7-point WER reduction.This is a negative result about saturation, not a claim that reranking is useless.
  • Because the subject spoke aloud, the report validates the easier vocalized case rather than silent speech.It is presented as a stepping stone toward silent speech.

9 The project website

The project website presents Myovox as a visual explainer for surface-EMG speech decoding. It uses animation to show how facial-muscle electrical signals become text at an 18.53% word error rate.

  • The website is a cinematic, scroll-driven explainer aimed at readers new to speech decoding and electromyography.It documents the project separately from the terse technical report.
  • Animations walk through each pipeline stage instead of presenting equations, while reporting an 18.53% word error rate.

10 Glossary

The glossary defines the evaluation metrics, decoding components, signal representations, model architectures, and speech-related terms used throughout Myovox. Together, these entries clarify how facial sEMG is converted into phoneme and word predictions.

  • WER measures word substitutions, insertions, and deletions, while PER measures greedy phoneme errors independently of the decoder and language model.WER is the reader-facing metric; PER isolates acoustic-model quality.
  • An n-best oracle selects the best candidate in a hypothesis pool, giving a lower bound for reranking over that pool.The union pool has a 9.30% oracle against 18.53% achieved WER.
  • Acoustic posteriors are per-frame probabilities over output symbols, and missing probability mass for correct phonemes prevents later recovery.
  • The causal TDS encoder sees only past and present frames, whereas the bidirectional Conformer uses full-sentence context through attention and convolution.The Conformer combines long-range self-attention with local depthwise convolution.
  • WavLM-Large layer 9 supplies parallel-audio teacher features for cross-modal distillation, while LoRA adapts the 7B reranker with low-rank updates.QLoRA applies LoRA on a 4-bit quantized base model.
  • sEMG is non-invasive EMG recorded from skin electrodes; Myovox uses a 31-channel facial array sampled at 5 kHz during vocalized speech.Silent speech is the eventual target but is not evaluated here.
  • The vec(E) front-end represents each analysis window by a regularized covariance matrix across the 31 EMG channels.This representation captures channel co-activation associated with facial articulation.
  • CTC trains frame-level predictions against unaligned sequences by summing over alignments with a blank symbol.This removes the need for frame-by-frame EMG labels.

Dual-CTC

The section defines the decoding and evaluation pipeline, then explains how candidate generation, acoustic ensembling, and reranking improve word-level recognition while PER remains acoustic-only.

  • Dual-CTC: The baseline uses two CTC heads on one encoder, predicting HuBERT units and phonemes with a consistency penalty based on P(phone | unit).A fixed conditional table penalizes disagreement so the unit head’s richer signal informs the phone head.
  • Dual-CTC: Cross-modal distillation trains the sEMG student to match representations or outputs from an audio teacher observing the same event, with contrastive matching across parallel moments.The teacher is used only during training; at inference, the student uses EMG alone.
  • Dual-CTC: PER is measured by greedy framewise argmax with repeats and blanks collapsed, so it excludes lexicons, language models, and search.This makes PER a measure of the acoustic model rather than downstream decoding.
  • Dual-CTC: The WFST graph H ◦L ◦G maps CTC frame posteriors through phonemes and a pronunciation lexicon into word sequences scored by a language model.The pronunciation dictionary determines which words can be emitted, while the open-vocabulary setup uses a lexicon much larger than the corpus vocabulary.
  • Dual-CTC: Unioning n-best hypotheses from multiple acoustic scales widens the reranker’s candidate pool and lowers oracle WER without changing the acoustic model.The reranker prompts a fine-tuned language model with candidates and the detected phoneme sequence; constrained decoding selects a candidate, while free decoding may rewrite it.
  • Dual-CTC: 20.90% PER accompanies 23.47% WER after averaging two independently trained encoders, improving over 26.14% WER and 22.34% PER.The ensemble averages per-frame phone log-probabilities before decoding; the report does not separate sharpened argmax effects from posterior reshaping.

11 Frequently asked questions

The FAQs separate decoder improvements from acoustic modeling, clarify the scope of the vocalized healthy-speaker experiment, and identify acoustic PER as the binding ceiling.

  • 11 Frequently asked questions: Matching 39.02% baseline PER to the published 38.19% PER shows the initial 10.5-point WER recovery came from missing decoder settings, chiefly the blank penalty.Later gains are separated into acoustic modeling and decode-time extraction rather than treated as one undifferentiated improvement.
  • 11 Frequently asked questions: 18.53% WER is the best result on this corpus, but not the general sEMG-to-text state of the art because MONA LISA reports 3.7% on a different corpus.The corpora differ in channels, recordings, speakers, speaking modes, and training resources, so the results are not directly comparable.
  • 11 Frequently asked questions: The 400-sentence test set is easier than validation by roughly nine points because the official split is sequential rather than randomly sampled.All hyperparameters were tuned on 760 validation sentences and applied once to the held-out test set; six test sentences duplicate training text and are reported separately.
  • 11 Frequently asked questions: The experiment demonstrates vocalized silent communication by decoding facial muscle activity, not speech from a person who has lost the ability to articulate.It uses a healthy speaker who vocalized normally, so transfer to ALS or laryngectomy patients is not established.
  • 11 Frequently asked questions: Parallel audio is absent from inference: it serves only as a training-time teacher, and removing distillation changes word error from 26.14% to 26.10%.Distillation still improves PER from 23.71% to 22.34%, showing a phonetic benefit that does not translate into word-level improvement in this control.
  • 11 Frequently asked questions: The acoustic ceiling is approximately 20.9% PER because downstream reranking cannot recover correct words absent from acoustic posteriors.The report leaves open whether ensemble WER gains reflect phonetic improvement or posterior diversity, but either interpretation leaves the ceiling intact.
  • 11 Frequently asked questions: The report attributes future progress to more data, multiple subjects, or a front-end that preserves phonetic information from raw 31-channel sEMG rather than covariance summaries.Additional speakers are preferred because they address both the acoustic ceiling and the single-subject generalization limitation.
Loading 2609.17548v1…