MyoCodec: A Streaming Neural Codec for Electromyography

1 INTRODUCTION

• Existing EMG approaches lack a shared, task-agnostic representation, and a practical interface must also be compact, streamable, and lightweight.The introduction contrasts this gap with neural audio codecs and notes BioCodec's EEG focus and lack of frame-wise streaming in its experimental design and implementation.Source: §1 · pp. 1–2
• The codec uses causal Transformers and residual vector quantization to encode 2 kHz EMG into representations up to 50 Hz and 2400 bits/s per channel.It is trained on twelve public EMG datasets and reports 0.48 ms per frame, or 41× real-time throughput, on one NVIDIA RTX PRO 6000 Blackwell GPU.Source: §1 · p. 2
• MYOCODEC achieves strong intrinsic codec quality and outperforms or closely matches existing frameworks across typing, hand-pose, speech-decoding, and speech-to-EMG tasks.The introduction frames these evaluations as evidence of generalization across applications.Source: §1 · p. 2
• MYOCODEC's discrete tokens could support language- or speech-language-model systems that map muscle activity directly to language, speech, or actions.The introduction also identifies silent-speech communication, hands-free text entry, gesture interaction, prosthetic control, and wearable interfaces as potential applications.Source: §1 · p. 2

2.1 ARCHITECTURE

• Causal Transformer encoder and decoder process EMG through a residual vector quantizer, with six 256-entry codebooks and 50 Hz frames from 2 kHz input.Each Transformer has eight layers; the encoder uses a 64-frame sliding attention window and cache.Source: §2.1 · p. 3

2.2 TRAINING CURRICULUM

• Phase 1 jointly trains reconstruction and quantization objectives; Phase 2 retains them while gradually adding adversarial and feature-matching objectives.The adversarial and feature-matching weights rise linearly from zero during the first 50k Phase 2 steps.Source: §2.2 · p. 4

2.3 TRAINING DATA AND PREPROCESSING

• Training uses twelve public EMG datasets with approximately 24k channel-hours from 250k recordings, applying across-channel normalization while preserving relative electrode amplitudes.For datasets used in downstream evaluation, codec pretraining uses only their official training partitions.Source: §2.3 · p. 5

2.4 DOWNSTREAM TASKS

• Four downstream tasks test typing, hand-pose estimation, speech decoding, and speech-to-EMG synthesis, replacing each task's EMG representation module with MYOCODEC.Depending on the task, evaluation uses quantized outputs or continuous pre-quantization features; tasks are trained causally unless stated otherwise.Source: §2.4 · p. 5

2.5 EVALUATION

• Evaluation spans SI-SDR/SNR, waveform and envelope correlations, codebook statistics, and task metrics; emg2qwerty performance improves at −20 ms, dips at zero, then rises with positive lookahead.Task metrics include character error rate, pose errors, speech error rates and quality, and synthesized-EMG envelope correlation.Source: §2.5 · pp. 5–6, Figure 3 · p. 6

3.1 INTRINSIC CODEC QUALITY

• MyoCodec achieves the best reconstruction across reported metrics and outperforms BioCodec and TinyMyo on all four metrics with only two codebooks (800 bits/s).Its waveform reconstructions also better preserve transient peaks that the other models often miss.Source: §3.1 · p. 6
• All six codebooks are fully utilized, with perplexity increasing across RVQ stages in a coarse-to-fine decomposition.The first stage captures dominant signal structure; later stages encode finer residual details.Source: §3.1 · p. 7

3.2 DOWNSTREAM TASKS

• For emg2qwerty, MYOCODEC ranks best on seen subjects and second-best on unseen subjects while operating at 2400 bits/s per channel.The comparison omits the original framework’s SpecAugment and temporal-jittering augmentation methods.Source: §3.2 · p. 7
• For emg2pose, MYOCODEC performs best across joint-angle error, landmark distance, and fingertip distance.All models use the same LSTM pose decoder; only the input EMG representation differs.Source: §3.2 · p. 7, Table 4 · p. 7
• For emg2speech, MYOCODEC outperforms all baselines on CER, WER, PER, PFER, and UTMOS; Figure 4 identifies 100–140 ms as the best lookahead range.The reported downstream models are causally trained with 100 ms lookahead.Source: §3.2 · p. 7, Table 5 · p. 7, Figure 4 · p. 8
• For speech2emg, non-causal STE-GAN performs best, while causal MYOCODEC closely matches performance with 120 ms of lookahead.Generated-signal quality is assessed using envelope correlation with 20 ms and 50 ms smoothing windows.Source: §3.2 · p. 8, Table 6 · p. 8

3.3 POTENTIAL FUTURE APPLICATIONS

• MYOCODEC takes 0.482 ms to encode and decode each 20 ms frame on the tested GPU; its compact, causal, low-bitrate design is presented as promising for on-device and token-model applications.The discussion also identifies potential uses in wearable interaction and synthetic EMG generation, while noting that synthetic applications require further validation.Source: §3.2 · p. 8, §3.3 · pp. 8–9

3.4 LIMITATIONS

• The training data are concentrated in typing and pose datasets, while downstream comparisons measure plug-in compatibility rather than maximum performance with architectures optimized for codec tokens.The two datasets account for 71% of collected data, while speech-related EMG contributes 1.9%; most task models may not fully exploit discrete tokens.Source: §3.4 · p. 9

4 CONCLUSION

• MyoCodec encodes EMG at 50 Hz and 2400 bits/s using causal Transformer blocks and residual vector quantization.Source: §4 · p. 9

REPRODUCIBILITY STATEMENT

• The authors state that code and model weights will be publicly released upon publication.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

• The study uses publicly released EMG datasets under their original authors’ consent and IRB arrangements, with no additional human-subject data collected.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

• The authors report using AI for literature summarization, implementation assistance, and writing refinement, while taking responsibility for the final content.Source: AI USE STATEMENT · p. 10

Biosignal foundation models

• TinyMyo is a roughly 3.6 M-parameter EMG foundation model with 100 Hz, 192-dimensional continuous embeddings and support for varying electrode layouts.Its learned bank of 16 channel embeddings lets one model serve different layouts; it is used as a baseline across four downstream tasks.Source: A RELATED WORK · p. 14

Neural audio codecs

• Neural audio codecs combine an encoder, residual vector quantization, and a decoder, with work spanning streaming inference, codebook utilization, and language-model-useful tokens.SoundStream introduced RVQ-GAN, EnCodec incorporated streaming inference, DAC improved codebook utilization, and Mimi distilled phonetic information into its first RVQ stage.Source: A RELATED WORK · p. 14

BioCodec

• BioCodec adapts an EnCodec-style causal-convolution and six-stage RVQ architecture for biosignal tokenization, with its EMG model operating at 27.78 Hz and 1333 bits/s per channel.Its EMG model was pretrained on emg2qwerty and evaluated primarily as a tokenizer for downstream gesture classification.Source: A RELATED WORK · p. 14

emg2qwerty

• The emg2qwerty task decodes wrist-worn EMG into keystrokes, and its published front end transfers a generic model across participants to held-out ones.The front end uses a log-spectrogram, rotation-invariant band MLP, and time-depth-separable convolutional encoder trained with CTC.Source: A RELATED WORK · p. 14

emg2pose

• The emg2pose corpus pairs wrist EMG with optical hand tracking across 193 participants to decode hand joint angles.Salter et al. release the corpus and a recurrent decoder.Source: A RELATED WORK · p. 14

emg2speech and speech2emg

• Prior emg2speech work decodes speech from facial and neck EMG, while speech2emg reverses the direction by synthesizing articulatory EMG from speech features.Scheck & Schultz (2023) introduce STE-GAN for speech-conditioned EMG generation.Source: A RELATED WORK · pp. 14–15

Codec comparison

• Two-codebook MYOCODEC outperforms TinyMyo and BioCodec on reconstruction quality.Figure 5 compares reconstruction quality across bitrates.Source: Figure 5 · p. 15

Bitrate comparison

• At 800 bits/s, approximately 60% of BioCodec’s 1333 bits/s, MYOCODEC outperforms BioCodec across all four reconstruction metrics.This uses two codebooks.Source: B BITRATE COMPARISON · p. 15
• Using one codebook at 400 bits/s, MYOCODEC leads in SI-SDR, SNR, and rwav but trails in renv.This comparison shows the quality trade-off at the lowest reported bitrate.Source: B BITRATE COMPARISON · p. 15
• Using the first Nq codebooks reduces bitrate in increments of 400 bits/s.The reported bitrate–quality comparison covers Nq from 1 through 6.Source: B BITRATE COMPARISON · p. 15

Encoder

• The encoder patchifies 2 kHz EMG into 40-sample frames at 50 Hz and applies an eight-layer causal Transformer with a 64-frame sliding window.Each layer retains a 64-frame key–value cache, keeping per-frame computation independent of sequence length.Source: §C.1 · p. 15

Quantizer

• The encoder projects its 256-dimensional output to 32 dimensions before a six-stage residual vector quantizer with 256 entries per codebook.A learned projection maps the quantized representation back to 256 dimensions before decoding.Source: §C.1 · pp. 15–16
• Low-use codebook entries are reinitialized from current-batch encoder outputs every 250 optimization steps.The procedure is intended to maintain codebook utilization and help prevent codebook collapse.Source: §C.1 · p. 16

Decoder

• The decoder mirrors the encoder’s causal Transformer configuration and projects each output to 40 waveform samples.It has approximately 6.3 M parameters; the complete codec has 12.72 M parameters.Source: §C.1 · p. 16

Discriminators

• Adversarial training uses multi-period waveform and multi-scale STFT discriminator branches, which are discarded at inference.The multi-period branch uses periods {2, 3, 5, 7, 11}; the multi-scale STFT branch uses FFT and window sizes {64, 128, 256, 512}.Source: §C.1 · p. 16

Phase 1

• Phase 1 combines time-domain Huber, multi-scale spectral, vector-quantization, and commitment losses.The spectral objective uses FFT sizes {64, 128, 256}; its phase term is weighted by 0.1.Source: §C.2 · p. 16

Huber loss

• The Huber loss uses a quadratic region for small reconstruction errors and a linear region that limits the influence of high-amplitude bursts and motion artifacts.The time-domain term is the mean Huber penalty between the target and reconstruction, with β = 1.Source: §C.2 · p. 16

Vector-quantization losses

• Residual vector-quantization stages select nearest codebook entries from successive residuals, with VQ and commitment losses summed over all six stages.The VQ loss updates selected entries, whereas the commitment loss updates the encoder and discourages outputs from drifting from the codebook; the commitment weight is 0.25.Source: §C.2 · p. 16
• Because nearest-neighbor assignment is nondifferentiable, the decoder receives a straight-through estimate equal to the quantized representation in the forward pass.Detaching selected entries in the residual recursion also prevents later stages from propagating gradients through earlier codebooks.Source: §C.2 · pp. 16–17

Phase 2

• Phase 2 retains the Phase 1 objective and adds adversarial and feature-matching terms.Their weights increase linearly from zero over the first 50,000 steps of Phase 2; Phase 2 runs for an additional 100,000 steps.Source: §C.2 · p. 17

emg2qwerty

• For typing, MYOCODEC and BioCodec provide quantized RVQ outputs, while TinyMyo provides continuous encoder features to the downstream model.The shared model retains a multi-band rotation-invariant MLP and CTC output layer, followed by a four-layer causal Transformer; augmentation is omitted to isolate representation and lookahead effects.Source: D DOWNSTREAM IMPLEMENTATION · p. 17

emg2pose

• For hand-pose estimation, the codec replaces the TdsNetwork featurizer, and the downstream model uses encoder features before quantization.Within each frame, attention mixes information across 16 electrodes without additional lookahead; the published velocity-integrating LSTM decoder is retained.Source: D DOWNSTREAM IMPLEMENTATION · p. 17

emg2speech

• For speech decoding, frozen pretrained front ends predict JHCodec’s continuous 1024-dimensional pre-quantization latent, with temporal components shared across four systems.Training uses the vocalized Gaddy and Klein corpus; its encoder is bidirectional and serves as an offline reference.Source: D DOWNSTREAM IMPLEMENTATION · p. 17

speech2emg

• For speech-to-EMG synthesis, a causal temporal Transformer and bidirectional channel Transformer predict continuous EMG features, with targets varying by encoder.MYOCODEC and BioCodec use bottleneck projections, while TinyMyo uses encoder outputs; training combines mean-squared error with STE-GAN’s multi-time-domain loss.Source: D DOWNSTREAM IMPLEMENTATION · pp. 17–18

Evaluation metrics

• Evaluation uses CER for typing; joint-angle, landmark, and fingertip errors for pose; speech-recognition and perceptual metrics for speech decoding; and envelope correlation for synthesis.Typing CER is reported with greedy and beam-search decoding; emg2speech reports CER, WER, PER, PFER, and UTMOS.Source: D DOWNSTREAM IMPLEMENTATION · p. 18

• The corpus table covers twelve public EMG corpora, and its channel-hour counts include only channels and recordings retained after filtering.Source: Table 8 · p. 18
• Only DB2, DB4, DB6, and DB7 are used.Source: E PRETRAINING CORPUS · p. 18