PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning

Ulysse Côté-Allard, Cheikh Latyr Fall, Alexandre Drouin, Alexandre Campeau-Lecours, Clément Gosselin, Kyrre Glette, François Laviolette, Benoit Gosselin

arXiv:1801.07756v5cs.LGstat.ML

TL;DR

sEMG gesture recognition lacks the large single-user datasets commonly needed by deep learning. The paper uses transfer learning from aggregated multi-user recordings with ConvNets across three signal modalities, and reports improved performance across datasets, including 98.31% and 68.98% average accuracies. The study’s main scope boundary is evaluation only with able-bodied participants and related recording conditions.

  • Problem

    Single users cannot reasonably produce the tens of thousands of examples needed for deep learning, motivating the use of aggregated recordings to learn features for new subjects.

  • Method

    The paper pre-trains ConvNets on aggregated inter-user sEMG data and transfers them across raw EMG, spectrogram, and CWT input modalities.

  • Results

    Transfer-learning ConvNets systematically and significantly outperform non-augmented versions, reaching 98.31% average accuracy over 17 participants and 68.98% over 10 participants.

  • Takeaways & Limitations

    The learned features generalize across participants and never-seen-before gestures, supporting reuse of pre-trained weights for Myo Armband tasks with different gestures.

  • Takeaways & Limitations

    The study does not test amputees, electrode shifts, varied limb positions, or users wearing fewer electrodes than the pre-training group.

Abstract

from arXiv · show

In recent years, deep learning algorithms have become increasingly more prominent for their unparalleled ability to automatically learn discriminant features from large amounts of data. However, within the field of electromyography-based gesture recognition, deep learning algorithms are seldom employed as they require an unreasonable amount of effort from a single person, to generate tens of thousands of examples. This work's hypothesis is that general, informative features can be learned from the large amounts of data generated by aggregating the signals of multiple users, thus reducing the recording burden while enhancing gesture recognition. Consequently, this paper proposes applying transfer learning on aggregated data from multiple users, while leveraging the capacity of deep learning algorithms to learn discriminant features from large datasets. Two datasets comprised of 19 and 17 able-bodied participants respectively (the first one is employed for pre-training) were recorded for this work, using the Myo Armband. A third Myo Armband dataset was taken from the NinaPro database and is comprised of 10 able-bodied participants. Three different deep learning networks employing three different modalities as input (raw EMG, Spectrograms and Continuous Wavelet Transform (CWT)) are tested on the second and third dataset. The proposed transfer learning scheme is shown to systematically and significantly enhance the performance for all three networks on the two datasets, achieving an offline accuracy of 98.31% for 7 gestures over 17 participants for the CWT-based ConvNet and 68.98% for 18 gestures over 10 participants for the raw EMG-based ConvNet. Finally, a use-case study employing eight able-bodied participants suggests that real-time feedback allows users to adapt their muscle activation strategy which reduces the degradation in accuracy normally experienced over time.

I. INTRODUCTION

The paper addresses limited single-user training data in sEMG gesture recognition by transferring features learned from aggregated inter-user recordings. It introduces ConvNet architectures using raw signals, spectrograms, and CWT representations, alongside a publicly available Myo dataset.

  • Motivation: Single users cannot realistically generate the tens of thousands of examples often needed to train deep learning gesture recognizers.Aggregating recordings from multiple participants may provide data for learning mappings that generalize to new subjects.
  • Contribution: The proposed transfer-learning scheme uses inter-user data with ConvNets for sEMG-based hand-gesture recognition.The work also reduces computational load and improves performance relative to an earlier conference version.
  • Contribution: Three ConvNet input modalities are evaluated: raw sEMG, short-time Fourier transform spectrograms, and continuous wavelet transforms.The authors identify CWT as a newly applied feature representation for sEMG-based hand-gesture classification.
  • Contribution: The paper releases a public Myo dataset with 19 participants for pre-training and 17 participants for final evaluation.The authors describe it as the largest published dataset using the commercially available Myo Armband.
  • Contribution: The study extends the work with a fourteen-day use-case experiment examining real-time feedback and online classifier performance without recalibration.Experiments are generally reported as averages over 20 runs because the algorithms are stochastic.
  • Data and setting: The Myo Armband offers convenient, non-invasive acquisition but records lower-quality signals at 200 Hz, below the recommended sEMG sampling range.Dry electrodes are less robust to motion artifacts, and the limited sampling rate can impair gesture discrimination.

2) Time-Window Length:

The study defines gesture acquisition and real-time processing around seven Myo Dataset gestures, variable armband placement, and a 260 ms input window. It also evaluates NinaPro DB5 using a broader gesture set.

  • 2) Time-Window Length:: A 260 ms sliding window was selected to leave 40 ms for preprocessing and classification while remaining within the 300 ms latency target.The window contains 52 samples at the Myo’s sampling rate and overlaps adjacent windows by 235 ms.
  • 3) Labeled Data Acquisition Protocol:: The Myo Dataset contains seven gestures, each held for five seconds, with rest periods omitted to balance the gesture classes.A cycle comprises all seven gestures, and four cycles form a round.
  • 3) Labeled Data Acquisition Protocol:: Participants positioned the armband themselves, creating realistic variation in placement and orientation across recordings.The protocol sought to emulate end-user placement without prior knowledge of optimal electrode positioning.
  • NinaPro DB5: NinaPro DB5 contributes 10 participants performing 17 gestures plus a neutral gesture in its second exercise set.That exercise set contains the gestures considered in the rest of the work and adds movements such as wrist pronation and supination.

A. Feature Sets

The paper compares conventional engineered feature sets and classifiers with deep-learning approaches. It uses dimensionality reduction to support computationally inexpensive and robust sEMG classification.

  • Feature Sets: Four literature-based feature sets provide comparison baselines for the proposed deep-learning methods.They are evaluated with SVM, ANN, RF, KNN, and linear discriminant classifiers.
  • Dimensionality Reduction: LDA is used for feature projection because it is computationally inexpensive, hyperparameter-free, and associated with robust sEMG gesture-classification accuracy.The paper compares accuracy with and without dimensionality reduction in an appendix.
  • Feature Sets: The time-domain baseline contains MAV, ZC, SSC, and WL features and is presented as a common reference for new classification techniques.
  • Feature Sets: A larger feature set combines time-domain features with skewness, RMS, IEMG, AR coefficients, and Hjorth parameters.
  • Feature Sets: The NinaPro feature set combines RMS, mDWT, EMG histogram, and time-domain features, while another set combines SampEn, cepstral coefficients, RMS, and WL.The latter was selected from fifty evaluated features, with SampEn ranked first.
  • Network Design: Although ConvNets are computationally expensive, the paper designs them with embedded-system deployment in mind.

B. Proposed Convolutional Network Architectures

The paper develops ConvNets for raw EMG, spectrogram, and CWT inputs, adapting slow-fusion designs to the time-structured representations of multichannel sEMG. Transfer learning aggregates multiple users’ data to learn features applicable to new subjects.

  • Architecture motivation: Slow-fusion ConvNets separate temporal information into parallel branches and gradually fuse those branches throughout the network.This design was adapted from video classification because spectrograms and CWTs encode time alongside channel and frequency or scale.
  • Input modalities: Spectrogram inputs are represented as Time x Channel x Frequency matrices after windowed Hann-transform processing across eight Myo channels.The resulting spectrogram matrix is 4x8x14 after removing the first frequency band.
  • Input modalities: A raw ConvNet processes raw EMG with a distinct architecture because raw signals constitute a different input modality from time-frequency representations.The raw architecture was selected from prior work using a Myo Armband hand-gesture dataset; an enhanced version uses 549 091 learnable parameters.
  • Transfer-learning preparation: The proposed approach aggregates multiple individuals’ data so ConvNets can learn more general features without requiring one person to record tens of thousands of examples.Transfer learning is used to apply features learned from aggregated users’ signals to new subjects.
  • Transfer-learning preparation: Automatic channel alignment is required for transfer learning because unconstrained armband orientation varies widely between subjects.Channels are circularly shifted using the most active gesture-related channel of the first subject as the reference.
  • Input modalities: The spectrogram ConvNet splits inputs into two temporal parts, while the CWT ConvNet splits them into four and fuses branch feature maps by element-wise summation.The spectrogram and CWT architectures use 67 179 and 30 219 learnable parameters, respectively.

B. Adaptive Batch Normalization

The adaptive batch-normalization scheme shares a pretrained network across participants while allowing batch-normalization parameters to adapt to new users. Its target network adds a second network through scaled, layer-wise residual connections.

  • Adaptive Batch Normalization: AdaBatch freezes pretrained network weights during target training except for batch-normalization parameters, which are treated as domain-related information.The same network is used for source and target tasks, unlike the separate-network PNN architecture.
  • Adaptive Batch Normalization: The shared Source Network learns the general mapping between muscle activity and gestures across participants.This shared source-task formulation avoids training one source network per participant.
  • Target architecture: The Second Network receives scaled Source Network outputs through layer-wise element-wise summation to learn the residual between source and target tasks.Learnable scalar coefficients control the transferred outputs before each sum connection.
  • Target architecture: The combined Source and Second Networks form the Target Network, with different dropout rates used during source pretraining and target training.MC Dropout is set to 35% for Source Network pretraining and 50% for Target Network training.
  • Evaluation: Evaluation compares each ConvNet with its transfer-learning version across one to four training cycles and uses participant-level Wilcoxon tests.Additional comparisons use Friedman ranking followed by Holm post-hoc testing against the best-ranked method.

B. NinaPro Dataset

The NinaPro evaluation tests transfer learning across participants, including an out-of-sample-gesture setting and a 14-day real-time case study. Feedback was associated with counteracting accuracy degradation over time, while muscle fatigue did not appear to reduce accuracy.

  • NinaPro evaluation: The NinaPro DB5 evaluation averages accuracy over ten participants, using the other nine participants for each participant’s pretraining.ConvNets and their transfer-learning versions are evaluated for one to four training cycles.
  • NinaPro evaluation: A separate NinaPro experiment evaluates eleven gestures absent from the pretraining dataset, directly testing transfer to never-seen-before gestures.The CWT ConvNet is compared with its transfer-learning-augmented version in this out-of-sample setting.
  • Real-time case study: The real-time case study follows eight participants for fourteen days using the CWT+TL classifier, with four receiving feedback and four receiving none.Each daily session contains 30 randomly selected gestures, and the first second after each requested gesture is excluded from results.
  • Real-time case study: 95.42% was the average accuracy across participants during the first recording session.The reported value provides the initial-session reference for the 14-day comparison.
  • Real-time case study: Without feedback, accuracy degraded over fourteen days, whereas the feedback group was seemingly able to counteract that degradation.The comparison is reported as an observation from Fig. 10 rather than as a definitive causal result.
  • Fatigue analysis: A positive accuracy-regression slope suggests that muscle fatigue did not negatively affect the proposed ConvNet’s accuracy during the five-minute sessions.Many participants nevertheless reported experiencing muscular fatigue during recordings.

IX. DISCUSSION

The proposed transfer-learning ConvNets consistently outperformed non-augmented versions and generalized across participants and gestures, while several deployment and scope limitations remain. Evaluation reached 98.31% on seven gestures across 17 participants and 68.98% on 18 gestures across 10 participants.

  • Performance comparison: TL-augmented ConvNets significantly outperformed non-augmented versions across training-cycle conditions.The advantage held across tested datasets and remained significant with a single training cycle.
  • Generalization: The TL method also outperformed non-augmented ConvNets in the out-of-sample experiment, with significance reported for all training repetitions.The authors associate this with learning features that generalize across participants and previously unseen gestures.
  • Limitations and future work: Real-time accuracy averaged 95.42%, below the Evaluation Dataset’s 98.31%, likely because gesture transitions were absent from training data.The paper notes that transition data may be poorly classified and proposes unsupervised transition detection as future work.
  • Limitations and future work: The study did not test amputees, electrode shifts, or variability from different limb positions, limiting its evaluated scope.The proposed transfer-learning scheme also has difficulty adapting when a new user wears a different number of electrodes.
  • Performance comparison: 98.31% average accuracy was achieved on the evaluation dataset across 17 participants.This result was reported for the TL-augmented ConvNet on seven gestures.
  • Performance comparison: 68.98% average accuracy was achieved on NinaPro DB5 across 10 participants for 18 hand/wrist gestures.The proposed classifier used a single Myo Armband.

APPENDIX A DATA AUGMENTATION

The appendix evaluates signal-specific augmentation methods to address limited sEMG training data and overfitting, selecting sliding windows as the only statistically improved technique. It also considers computational feasibility for embedded deployment, with CWT and spectrogram networks requiring different computational budgets.

  • Augmentation methods: The baseline uses non-overlapping 260ms windows, while sliding windows add overlapping temporal translations without synthetic examples.This preserves recorded signal content while increasing the number of training examples.
  • Augmentation methods: Other tested augmentations emulate muscle fatigue, electrode displacement, Gaussian noise, or their aggregation.These transformations modify spectrogram frequency content, redistribute power across channels, add white Gaussian noise, or combine methods.
  • Augmentation evaluation: Sliding-window augmentation was the only tested technique that significantly improved accuracy over the baseline.The appendix therefore used sliding windows as the paper’s sole data-augmentation technique.
  • Embedded implementation: CWT and spectrogram ConvNets require approximately 2,274,000 and 14,728,000 FLOPs per classification, respectively.The paper estimates that suitable hardware and network compression could support embedded systems below 100mW.
  • Embedded implementation: Without optimization, CWT classification takes 21.42ms per example, compared with 2.94ms for spectrogram and 3.70ms for raw EMG ConvNets.Transfer learning adds 0.57ms, 0.90ms, and 0.14ms to the CWT, spectrogram, and raw EMG pipelines, respectively.

APPENDIX C FEATURE ENGINEERING

The section introduces windowed feature engineering for sEMG signals, covering time-domain descriptors and their definitions. It also includes a comparison table for proposed data augmentation techniques.

  • Features are calculated from overlapping windows of signal x, whose kth sample in the ith window is x_i,k.
  • Mean Absolute Value returns the mean of a fully rectified signal.
  • Slope Sign Changes measures how frequently the signal slope changes sign across three consecutive samples.A threshold ϵ ≥0 reduces the impact of noise.
  • Zero Crossing counts how frequently the signal passes through zero, using threshold ϵ ≥0 to lessen noise effects.Depending on the slope and selected threshold, a crossing may not be detected.
  • Waveform Length provides a simple characterization of the signal waveform.
  • Table V compares five proposed data augmentation techniques.

5) Skewness:

This section describes distributional and signal-dynamics features, including skewness, RMS, Hjorth parameters, and integrated EMG. These features characterize asymmetry, amplitude, frequency, complexity, and rectified signal magnitude.

  • Skewness measures the overall asymmetry of a distribution as its third central moment.Its calculation uses the signal distribution and standard deviation σ.
  • RMS, also called the quadratic mean, is closely related to standard deviation and equals it when the signal mean is zero.
  • Hjorth parameters: Hjorth Activity represents the surface of the frequency-domain power spectrum and corresponds to signal variance.
  • Hjorth parameters: Hjorth Mobility represents the mean frequency of the signal.
  • Hjorth parameters: Hjorth Complexity represents the change in frequency.
  • Integrated EMG returns the sum of the fully rectified signal.

9) Autoregression Coefficient (AR):

The section presents autoregressive, entropy, histogram, cepstral, and multiresolution wavelet features. It also documents the implementation and a boundary-effect limitation of the mDWT feature.

  • Autoregression Coefficient (AR): An autoregressive model predicts future data from a weighted average of previous samples plus white noise.The number of coefficients trades computational complexity against predictive power.
  • Autoregression Coefficient (AR): The AR model uses order P, coefficients ρ_j, and residual white noise ϵ_t.
  • Sample Entropy (SampEn): Sample Entropy estimates the complexity and randomness of a system.
  • Histogram (HIST): HIST quantifies how often EMG amplitude deviations from baseline occur across n equal-width amplitude bins.The symmetric amplitude range is centered around the baseline, with n as a hyperparameter.
  • Cepstral coefficients: Cepstral coefficients are derived from the inverse Fourier transform of the log power-spectrum magnitude and can be obtained directly from AR.
  • Modified Discrete Wavelet Transform (mDWT): mDWT removes time information from the discrete wavelet transform and instead calculates cumulative energy at each decomposition level.The implementation uses a db7 wavelet at level 3.
  • Modified Discrete Wavelet Transform (mDWT): The mDWT is affected by boundary effects because level 3 is combined with a signal length of 52.This choice matches the wavelet and level used in earlier work with a shorter signal length.

C. Time-Frequency Domain Features

Time-frequency features address the non-stationarity of sEMG by analyzing localized signal segments. STFT spectrograms use a fixed window, whereas wavelets analyze signals across multiple scales with frequency-dependent resolution.

  • Short Term Fourier Transform based Spectrogram (Spectrogram): Fourier analysis has limited use for non-stationary sEMG because it cannot identify when signal components occur.
  • Short Term Fourier Transform based Spectrogram (Spectrogram): The STFT applies a sliding window and computes a Fourier transform for each segment.
  • Short Term Fourier Transform based Spectrogram (Spectrogram): A spectrogram is the squared magnitude of the STFT computed from signal s(t) and window width w.
  • Short Term Fourier Transform based Spectrogram (Spectrogram): For STFT, wider windows improve frequency resolution at the expense of time resolution, and vice versa.
  • Continuous Wavelet Transform (CWT): Wavelets provide frequency-dependent time-frequency resolution by analyzing signals at different scales of a mother wavelet.A wavelet has limited duration, varying frequency, and zero mean; CWT frequency is a pseudo-frequency.

APPENDIX D HYPERPARAMETERS SELECTION FOR STATE OF THE ART

The appendix describes classifier hyperparameter searches and compares feature-set accuracies with and without dimensionality reduction. It also reports a transfer-learning adaptation for differing electrode counts, which significantly improves modified ConvNet accuracy.

  • Hyperparameter selection: SVM hyperparameters included RBF and linear kernels, with logarithmic searches over C and γ.C ranged from 10^-3 to 10^3, while γ ranged from 10^-5 to 10^2 for the RBF kernel.
  • Hyperparameter selection: ANN hyperparameters covered hidden-layer sizes, sigmoid, tanh, and ReLU activations, learning rates, L2 penalties, Adam optimization, and early stopping.Ten percent of the training data was used for validation during early stopping.
  • Hyperparameter selection: KNN searches varied neighbor counts and evaluated Manhattan, Euclidean, and third- or fourth-degree Minkowski distances.The candidate neighbor counts were 1, 2, 3, 4, 5, 10, 15, and 20.
  • Feature-set evaluation: Dimensionality reduction reduced computation from more than two and a half months to one week on the Evaluation dataset.The comparison concerned average accuracies for state-of-the-art feature sets with and without dimensionality reduction.
  • Transfer-learning adaptation: 67.65% average accuracy was achieved by the transfer-learning-enhanced modified ConvNet versus 61.47% without augmentation across 17 participants.The difference was significant under a one-tail Wilcoxon signed-rank test (p-value=0.00494), and the adaptation accommodates different electrode counts.
Loading 1801.07756v5…