PaperlayerView exact revision on arXiv

Source-linked AI summary

ECG Heartbeat Classification Using Multimodal Fusion

Zeeshan Ahmad, Anika Tabassum, Ling Guan, Naimul Khan

arXiv:2107.09869v1cs.LGcs.NEeess.SP

TL;DR

ECG heartbeat classification faces challenges from conventional expert-dependent feature extraction. The paper proposes MIF and MFF, fusing three ECG-derived image representations. The proposed frameworks outperform existing methods on reported classification metrics, with MFF favoring classification performance and MIF inference speed.

  • Problem

    Conventional ECG heartbeat classification relies on manually extracted features that require expert knowledge and can be time-consuming and sensitive to noise, scaling, and translation.

    Source: §I · p. 2

  • Method

    MIF combines GAF, RP, and MTF images into one CNN input, while MFF fuses CNN penultimate-layer features for an SVM classifier.

    Source: §I · p. 2

  • Results

    The proposed frameworks outperform existing methods in accuracy, precision, and recall; MFF has higher classification metrics on both datasets, while MIF has faster inference.

    Source: §V · p. 9

  • Takeaways & Limitations

    The frameworks are designed to extract and fuse complementary, discriminative features while reducing dimensionality.

    Source: §I · p. 2

    MIF requires exactly three grayscale images, while MFF's three separate AlexNets require more training and inference time.

    Source: §V · p. 10

Abstract

from arXiv · show

Electrocardiogram (ECG) is an authoritative source to diagnose and counter critical cardiovascular syndromes such as arrhythmia and myocardial infarction (MI). Current machine learning techniques either depend on manually extracted features or large and complex deep learning networks which merely utilize the 1D ECG signal directly. Since intelligent multimodal fusion can perform at the stateof-the-art level with an efficient deep network, therefore, in this paper, we propose two computationally efficient multimodal fusion frameworks for ECG heart beat classification called Multimodal Image Fusion (MIF) and Multimodal Feature Fusion (MFF). At the input of these frameworks, we convert the raw ECG data into three different images using Gramian Angular Field (GAF), Recurrence Plot (RP) and Markov Transition Field (MTF). In MIF, we first perform image fusion by combining three imaging modalities to create a single image modality which serves as input to the Convolutional Neural Network (CNN). In MFF, we extracted features from penultimate layer of CNNs and fused them to get unique and interdependent information necessary for better performance of classifier. These informational features are finally used to train a Support Vector Machine (SVM) classifier for ECG heart-beat classification. We demonstrate the superiority of the proposed fusion models by performing experiments on PhysioNets MIT-BIH dataset for five distinct conditions of arrhythmias which are consistent with the AAMI EC57 protocols and on PTB diagnostics dataset for Myocardial Infarction (MI) classification. We achieved classification accuracy of 99.7% and 99.2% on arrhythmia and MI classification, respectively.

I. INTRODUCTION

ECG beat classification is difficult amid noisy, variable recordings, while conventional feature extraction is expert-dependent and time-consuming. The paper proposes MIF and MFF to fuse ECG-derived image modalities while reducing feature dimensionality.

I. INTRODUCTION

  • ECG recordings vary across environments and subjects and are noisy, while conventional feature extraction is expert-dependent, time-consuming, and may generalize poorly.These challenges make beat-by-beat examination laborious for cardiologists.Source: §I · p. 2

II. RELATED WORK

  • Prior ECG deep-learning work includes both 1D signal models and 2D CNN approaches using transformed ECG representations.The introduction notes that 2D representations have been experimentally reported to classify heartbeats more accurately than 1D representations.Source: §I · p. 2

I. INTRODUCTION

  • Existing deep-learning fusion methods are described as lacking robust fusion frameworks and relying mainly on concatenation or decision-level fusion.Source: §I · p. 2
  • The paper proposes MIF and MFF, which transform ECG beats into GAF, RP, and MTF images and fuse modalities while keeping combined-feature size similar to an individual modality.The introduction states that these transformations conserve spatial-domain correlated information and improve performance over spectrogram and time-frequency approaches.Source: §I · p. 2

A. ONE-DIMENSIONAL CNN APPROACHES

Prior work spans 1D ECG classifiers, image-based 2D CNNs, and multimodal fusion methods. The paper positions MIF and MFF as alternatives to fusion approaches that depend on concatenation and its dimensionality costs.

A. ONE-DIMENSIONAL CNN APPROACHES

  • 1D CNN studies cover arrhythmia and MI classification, including patient-specific, transfer-learning, and deeper CNN models.Other cited approaches combine CNNs with LSTMs, GAN-based augmentation, or layered classifiers.Source: A. ONE-DIMENSIONAL CNN APPROACHES · pp. 2–3

B. TWO-DIMENSIONAL CNN APPROACHES

  • 2D ECG approaches transform signals into spectrograms, wavelet images, recurrence plots, and other image or matrix representations for CNN classification.The cited work reports higher accuracy for 2D-CNN than 1D-CNN in one comparison and 83.82% accuracy for AlexNet on spectrograms in another.Source: B. TWO-DIMENSIONAL CNN APPROACHES · p. 3

C. FUSION BASED APPROACHES

  • Fusion methods combine multiscale, spatial, or layer-specific features, but existing methods are described as relying heavily on concatenation, which can increase complexity and degrade accuracy.The paper presents MIF and MFF as frameworks that fuse features while reducing dimensionality.Source: C. FUSION BASED APPROACHES · pp. 3–4

III. MATERIALS AND METHODS

  • The methods section introduces ECG-to-image transformation as common to MIF and MFF, then outlines image fusion, feature fusion, gated fusion, and CNN architecture.Source: §III · p. 4

A. ECG SIGNAL TO IMAGE TRANSFORMATION

The framework converts each ECG heartbeat into GAF, RP, and MTF images, which encode different properties of the signal. These image representations preserve distinct temporal or recurrence information, with MTF specifically unable to restore the original ECG.

A. ECG SIGNAL TO IMAGE TRANSFORMATION

  • GAF maps normalized ECG samples to angular cosines and timestamps to radii, preserving sample positions so the original heartbeat can be restored.The encoding is described as bijective and preserves spatial-domain affiliations.Source: 1) Formation of Images by Gramian… · p. 4
  • RP images encode the spacing between time points to visualize recurrent behavior in non-stationary ECG signals.The paper shows RP images for five MIT-BIH categories.Source: 2) Formation of Images by Recurrence… · p. 4
  • MTF maps transitions between quantile bins onto spatial locations, but its transition-based construction prevents restoration of the original ECG.The transition likelihoods are stretched across locations in the signal; the paper uses 10 bins for normalized data.Source: 3) ECG to Markov Transition Field… · pp. 4–5
  • The transformations use full-length heartbeats, so ECG signals of any length can be converted to images and resized by interpolation.Source: 3) ECG to Markov Transition Field… · p. 5
  • GAF, RP, and MTF images show different patterns across the five MIT-BIH categories, and their grayscale versions are described as more interpretable.The image coordinates correspond to pixel values.Source: 3) ECG to Markov Transition Field… · p. 5

B. MULTIMODAL IMAGE FUSION FRAMEWORK

MIF combines GAF, RP, and MTF images into a triple-channel compound image that carries information from all three transformations. The compound image is used with a CNN for feature extraction and classification.

B. MULTIMODAL IMAGE FUSION FRAMEWORK

  • MIF combines GAF, RP, and MTF grayscale images as three orthogonal channels in a compound image that carries their statistical dynamics.The paper characterizes the transformations as preserving temporal information and being lossless.Source: B. MULTIMODAL IMAGE FUSION FRAMEWORK · p. 5
  • The fused image is used with AlexNet for feature extraction and classification in an end-to-end CNN approach.Source: B. MULTIMODAL IMAGE FUSION FRAMEWORK · p. 5

C. MULTIMODAL FEATURE FUSION FRAMEWORK

MFF extracts features from three ECG image modalities and uses a gated fusion network to combine them before SVM classification. The network applies high-boost filtering and learned weights to fuse modality features.

C. MULTIMODAL FEATURE FUSION FRAMEWORK

  • MFF extracts fc-7 features from separate AlexNets for GAF, RP, and MTF images, then fuses them with a Gated Fusion Network before SVM classification.The fused features retain dimensionality considerations during fusion.Source: C. MULTIMODAL FEATURE FUSION FRAMEWORK · p. 6
  • The GFN takes features from the second-last fully connected layer, fc-7, of each AlexNet as its inputs.The authors state that the network was adapted from their previous work.Source: 1) Gated Fusion Network · p. 6
  • The GFN convolves each modality's features with a high-boost kernel, then applies sigmoid-generated weights and point-wise products to produce fused features.The filter is described as the difference between a scaled input and its low-pass version; c = 1 is reported as the best-performing setting.Source: 1) Gated Fusion Network · pp. 6–7
  • The described CNN architecture has three convolutional layers, two pooling layers, and a fully connected layer, with 16 kernels in the first convolutional layer and 32 in each later layer.The convolutional kernels are 5x5; the pooling layers are 2x2 with stride 2.Source: 2) CNN Architecture · p. 7

D. CLASSIFICATION TASK AND CLASSIFIER

The classification task covers ECG heartbeat classification for arrhythmia and MI detection. MIF uses Softmax, while MFF uses SVM, and the paper reports accuracy, precision, and recall as classification metrics.

D. CLASSIFICATION TASK AND CLASSIFIER

E. TRAINING AND OPTIMIZATION

The experiments use both AlexNet and a smaller CNN, with AlexNet adapted for the datasets' class counts. Stochastic Gradient Descent with Momentum is used to optimize the deep networks.

E. TRAINING AND OPTIMIZATION

  • AlexNet inputs are resized to 227 x 227, and the experiments also use a smaller, computationally efficient CNN to assess whether comparable performance is possible.The authors state that computational costs of the two CNN models are compared in Table 11.Source: E. TRAINING AND OPTIMIZATION · p. 8
  • AlexNet is fine-tuned by reducing fc7 from 4096 to 512 units and fc8 from 1000 units to the number of dataset classes.The datasets have five MIT-BIH classes and two PTB classes.Source: E. TRAINING AND OPTIMIZATION · p. 8
  • Stochastic Gradient Descent with Momentum (SGDM) is used to optimize the deep networks.The paper describes SGDM as accelerating gradient vectors in the right directions toward faster convergence.Source: E. TRAINING AND OPTIMIZATION · p. 8

A. ECG DATABASES

The experiments use MIT-BIH for five AAMI EC57 heartbeat categories and PTB for MI classification using lead II recordings. Both datasets are standardized and denoised; the study does not model specific noise types.

A. ECG DATABASES

  • PTB experiments use lead II recordings and focus on healthy controls and MI cases; the input is described as resampled to 125 Hz.The dataset collection included 290 subjects, including 148 diagnosed with MI and 52 healthy controls.Source: A. ECG DATABASES · p. 8, 2) PTB Diagnostic ECG dataset · p. 9
  • Both datasets are supplied in standardized, denoised form with training and testing splits and heartbeat labels already provided; the study does not model specific noise types.Source: A. ECG DATABASES · p. 8
  • MIT-BIH heartbeat categories follow the five-class AAMI EC57 standard, with beats annotated by at least two experts.The dataset was collected from 47 subjects at 360 Hz.Source: 1) PhysioNet MIT-BIH Arrhythmia Dataset · p. 9
  • SMOTE is applied to upsample MIT-BIH training classes other than N because the training data are class-imbalanced.The resulting heartbeat counts are referenced in Table 3.Source: 1) PhysioNet MIT-BIH Arrhythmia Dataset · p. 9

V. DISCUSSION

The proposed frameworks outperform prior methods on accuracy, precision, and recall, while comparisons reveal trade-offs between fusion methods, accuracy, and computational efficiency. The authors also identify limitations in image requirements and processing time.

V. DISCUSSION

  • The proposed frameworks outperform prior methods on accuracy, precision, and recall across MIT-BIH heartbeat and PTB MI comparisons.Tables 9 and 10 present the dataset comparisons; the passage does not provide their individual values.Source: §V · p. 9, Table 9 · p. 10, Table 10 · p. 10
  • Average fusion assigns equal weights of 1 to the three modalities; trainable weight coefficients are left for future investigation.Equal weights correspond to 0.333 per modality.Source: §V · p. 9
  • Concatenation fusion performs poorly relative to other methods and creates high-dimensional features associated with added computation and information deterioration during classification.The fusion comparisons include concatenation and average fusion methods.Source: §V · p. 9
  • MFF yields higher accuracy, precision, and recall than MIF on both datasets, while MIF has faster inference speed.Inference speed is the time taken to recognize one test sample, measured in microseconds.Source: §V · p. 9, Table 12 · p. 10
  • The CNN shown in Fig. 4 is less accurate than AlexNet but computationally efficient.The comparison uses the MFF framework on MIT-BIH.Source: §V · p. 10, Table 11 · p. 10
  • The authors prefer SVM to softmax, citing prior experimental evidence and the margin-based nature of SVM classification.They describe the more rigorous nature of classification as the reason for SVM's better performance over softmax.Source: §V · p. 10
  • Some state-of-the-art comparisons use slightly different testing conditions, although the authors consider the comparisons appropriate.Source: §V · p. 10
  • MIF requires exactly three statistical grayscale images, whereas MFF's use of three separate AlexNets increases training and inference time.Source: §V · p. 10

VI. CONCLUSION

The paper presents MIF and MFF for ECG heartbeat classification, converting signals into three image representations and fusing them through image- or feature-level approaches. Across MIT-BIH arrhythmia and PTB MI classification, the authors report better accuracy, precision, and recall than previous state-of-the-art methods.

VI. CONCLUSION

  • MIF combines three ECG-derived images into one CNN input, while MFF fuses penultimate-layer CNN features for an SVM classifier.The inputs are GAF, RP, and MTF images; the experiments cover five MIT-BIH arrhythmias and PTB MI classification.Source: §VI · p. 10
  • The proposed frameworks beat previous state-of-the-art methods in accuracy, precision, and recall, and multimodal fusion outperforms using modalities individually.Source: §VI · p. 10
Loading 2107.09869v1…