PaperlayerView exact revision on arXiv

Source-linked AI summary

BRIDGE-EEG: Bridging Self-Supervised Pretraining and Efficient Deployment for Cross-Dataset EEG Classification

Meghna Roy Chowdhury, Chengwei Zhou, Haotian Yu, Gourav Datta, Shreyas Sen

arXiv:2609.12218v1cs.HCcs.LGeess.SP

TL;DR

EEG analysis must transfer across noisy, heterogeneous recordings while remaining practical on constrained devices. BRIDGE-EEG preprocesses diverse recordings into a common representation, pretrains a teacher on unlabeled EEG, and distills compact students. Across six benchmarks, students retained useful performance where the teacher representation was strong, while motor imagery remained limited by pretraining diversity.

  • Problem

    EEG pipelines often remain tied to a dataset, montage, or task despite real-world variation in signal quality and recording setups, while large pretrained models' energy costs on constrained hardware remain undercharacterized.

    Source: §I · p. 1

  • Method

    BRIDGE-EEG maps heterogeneous recordings to a common 62-channel representation, pretrains an SE-ResNet18 teacher with SimCLR on unlabeled EEG, and distills it into compact students.

    Source: §I · p. 2

  • Results

    Across six benchmarks, students retained useful performance where the teacher representation was strong, while motor imagery remained limited by pretraining diversity.

    Source: §VII · p. 10

  • Takeaways & Limitations

    The evaluation provides an empirical step toward future wearable EEG deployment using compact students with accuracy comparable to larger EEG foundation models.

    Source: §I · p. 2

    Motor-imagery performance remained limited by pretraining diversity rather than compression itself.

    Source: §VII · p. 10

Abstract

from arXiv · show

The growing use of electroencephalography (EEG) motivates automated analysis that is accurate, transferable, and deployable on constrained hardware. Recent EEG foundation models learn general representations from large-scale pretraining, but their size and computational cost limit edge and wearable deployment. We introduce BRIDGE-EEG, an efficient multi-task EEG classification pipeline that preserves the benefits of pretraining while reducing model size. A unified preprocessing scheme maps heterogeneous recordings with different channel counts, montages, and sampling rates to a device-agnostic 62-channel time--frequency representation. We pretrain an SE-ResNet18 teacher (11.84 M parameters) with SimCLR on unlabeled EEG from five heterogeneous datasets, then compress it into SE-ResNet8 (1.56 M) and SE-ResNet4 (0.48 M) students using task-agnostic and task-specific distillation. We evaluate six benchmarks spanning abnormality detection, motor imagery, and emotion recognition. For abnormality detection and emotion recognition, the students achieve accuracy comparable to or better than several recent EEG foundation models with 10--1,000$\times$ more parameters. Motor imagery shows a remaining representation gap, highlighting the importance of pretraining diversity. Inference profiling on a server GPU, desktop CPU, and NVIDIA Jetson Orin Nano shows up to 3.0$\times$ lower edge energy per inference (15.64 mJ vs. 46.67 mJ). The compact models further support future deployment on MCU-class wearables.

I. INTRODUCTION

EEG models must generalize across noisy recordings, diverse acquisition setups, and tasks while remaining practical for constrained devices. BRIDGE-EEG combines multi-dataset pretraining, distillation, and cross-tier profiling to investigate that trade-off.

I. INTRODUCTION

  • The central gap is between large pretrained models that generalize across heterogeneous EEG data and compact embedded models that are practical but usually task-specific.The work asks whether foundation-model benefits can be retained in compact models while characterizing accuracy–energy trade-offs across server GPU, desktop CPU, and edge hardware.Source: §I · p. 2
  • BRIDGE-EEG combines unified preprocessing, self-supervised multi-dataset pretraining, post-pretraining distillation, and measured inference profiling in one pipeline.The authors describe it as the first EEG framework to combine these elements in a single pipeline.Source: §I · p. 2
  • Its 62-channel time–frequency representation maps heterogeneous recordings to a common input for a shared backbone across acquisition setups.The pipeline addresses differing channel counts, montages, and sampling rates.Source: §I · p. 2
  • The present study extends SSL-SE-EEG from a single SSL architecture study to a full SSL-to-KD pipeline and multi-task evaluation.It also expands profiling from GPU-only latency and power to measured energy per inference across three hardware tiers.Source: §I · p. 2
  • The study evaluates six downstream applications across server, desktop, and edge hardware, reporting student accuracy comparable to larger EEG foundation models.The introduction describes the results as an empirical step toward future wearable EEG deployment.Source: §I · p. 2

II. RELATED WORK

EEG research has shifted from task-specific supervised models toward pretrained representations intended to transfer across tasks. Yet deployment efficiency remains less characterized than predictive performance.

A. EEG Learning Models: From Task-Specific to FMs

  • Earlier EEG networks were generally trained from scratch on single datasets and did not explicitly target cross-dataset generalization.The related-work discussion names EEGNet, ShallowConvNet, and DeepConvNet among these task-specific approaches.Source: A. EEG Learning Models: From… · p. 2
  • Large unlabeled EEG corpora enabled self-supervised models to learn reusable representations; BENDR was an early example supporting transfer across multiple EEG tasks.Later EEG foundation models vary in architecture, tokenization, and pretraining objective.Source: A. EEG Learning Models: From… · pp. 2–3
  • Although EEG foundation-model and self-supervised studies report accuracy and theoretical compute, deployment efficiency is less characterized by measured on-device energy.The related work also notes growing use of Mamba-based EEG models and continued development of EEG-specific SSL objectives.Source: A. EEG Learning Models: From… · p. 3

B. Knowledge Distillation for Efficient Biosignal Inference

Prior EEG distillation work has mostly been task-specific, while BRIDGE-EEG combines shared pretrained representations with compact students and cross-tier profiling. Its unified preprocessing aligns heterogeneous recordings to a common model input.

B. Knowledge Distillation for Efficient Biosignal Inference

  • Prior EEG distillation studies have focused on channel reduction, sleep staging, or emotion recognition rather than multi-task compression from a shared self-supervised backbone.The cited related work also notes that these studies generally do not report measured hardware efficiency.Source: B. Knowledge Distillation for… · p. 3

III. PROPOSED FRAMEWORK

  • BRIDGE-EEG combines multi-dataset pretraining, knowledge distillation, and inference profiling across a server GPU, desktop CPU, and Jetson Orin Nano.The framework targets accuracy comparable to or above recent EEG foundation models on abnormality detection and emotion recognition while maintaining edge-class efficiency.Source: §III · p. 4

A. Unified Preprocessing

  • The preprocessing maps recordings to 62 standard channels, zero-filling absent channels, then converts nonoverlapping 4-second windows into 128×128 spectrograms covering 1–50 Hz.Signals are filtered, baseline-corrected, and z-normalized before STFT conversion; the common tensor shape is (62, 128, 128).Source: A. Unified Preprocessing · p. 4, Figure 3 · p. 4

B. Self-Supervised Pretraining with SE-ResNet18

The paper uses SimCLR to pretrain an SE-ResNet18 encoder on unlabeled EEG, learning shared representations across datasets without task labels. Augmented spectrogram views and a contrastive loss support this training, while SE blocks recalibrate channel-wise features.

B. Self-Supervised Pretraining with SE-ResNet18

  • SimCLR pretrains a shared SE-ResNet18 encoder on pooled, unlabeled EEG to learn representations across tasks, populations, devices, and cognitive conditions.The encoder uses SE blocks after each of four residual stages and maps (62, 128, 128) spectrograms to 512-d embeddings.Source: B. Self-Supervised Pretraining with… · pp. 4–5
  • SE blocks recalibrate channel-wise feature responses after each residual stage through global average pooling and a learned gating mechanism.They can emphasize informative channels and suppress noisy or interpolated ones.Source: B. Self-Supervised Pretraining with… · p. 5, Figure 4 · p. 5
  • Two augmented views use channel dropout, time- and frequency-bin masking, and Gaussian noise before passing through the shared encoder and projection MLP.The projection head is discarded after pretraining; the encoder’s 512-d output is retained for downstream tasks.Source: B. Self-Supervised Pretraining with… · p. 5
  • NT-Xent maximizes agreement between views of the same spectrogram, while minibatch negatives from any dataset encourage discrimination across recording conditions.The denominator sums over all 2N minibatch embeddings except the anchor embedding; lower temperature sharpens the similarity distribution.Source: B. Self-Supervised Pretraining with… · p. 5

C. Knowledge Distillation (KD)

The pipeline compresses the 11.84 M-parameter pretrained teacher into smaller students using task-agnostic or task-specific distillation. The strategies differ in whether distillation precedes or follows task-specific teacher fine-tuning.

C. Knowledge Distillation (KD)

  • The 11.84 M-parameter teacher is compressed into SE-ResNet8 (1.56 M) and SE-ResNet4 (0.48 M) students using two distillation strategies.Both students retain SE blocks; the strategies span different points on the accuracy–efficiency tradeoff.Source: C. Knowledge Distillation (KD) · p. 5
  • Strategy A distills the SSL-pretrained teacher once into a shared SE-ResNet8 backbone, then fine-tunes that backbone for downstream tasks.The student’s 256-d embeddings are projected to the teacher’s 512-d space during distillation; the projection is later discarded, and the loss combines NT-Xent with MSE.Source: C. Knowledge Distillation (KD) · pp. 5–6
  • Strategy B fine-tunes the teacher for each task before distilling into separate SE-ResNet8 and SE-ResNet4 students with a standard Hinton KD loss.It uses soft distributional knowledge and direct supervision; its effectiveness depends on sufficiently large downstream datasets for reliable teacher class boundaries.Source: C. Knowledge Distillation (KD) · p. 6

IV. EXPERIMENTAL SETUP

The evaluation uses five datasets for pretraining and six separate downstream datasets across emotion recognition, motor imagery, and abnormality detection. It describes training protocols, baselines, and batch-size-one profiling across three hardware platforms.

IV. EXPERIMENTAL SETUP

  • Five pretraining datasets span 3–64 channels and 128–512 Hz, while six separate datasets—two per application category—support downstream evaluation.The pretraining pool and evaluation sets share no datasets in common.Source: §IV · p. 6
  • The teacher is pretrained for 50 epochs on four NVIDIA L40S GPUs with total batch size 1024, then teacher and students are fine-tuned for 50 epochs per downstream dataset.Fine-tuning uses layer-wise learning rates, class-weighted cross-entropy, and early stopping; external comparisons include published EEG foundation-model and task-specific results when available.Source: §IV · p. 6
  • Inference is profiled at batch size 1 on an NVIDIA L4, an Intel i7-8700 CPU, and an NVIDIA Jetson Orin Nano.CPU power is estimated with a utilization-scaled TDP proxy; the GPU and edge device use monitored power measurements.Source: §IV · p. 6

V. RESULTS

  • The results section compares models with state-of-the-art foundation models, examines performance retention under compression, and assesses efficiency gains across hardware tiers.The passage introduces these analyses but does not report their outcomes.Source: §V · p. 6

A. EEG-FM like accuracy

Across six benchmarks, compact students compare favorably with foundation models on abnormality detection and emotion recognition, while motor imagery remains the largest performance gap. The results vary by task and dataset, with the strongest comparisons on binary clinical tasks.

Abnormality detection

  • 90% accuracy with 1.56 M parameters on TUAB puts the Strategy B SE-ResNet8 above benchmarked EEG foundation models, including REVE and LaBraM-Huge.The student uses approximately 44× and 237× fewer parameters than those respective models; SE-ResNet4 reaches 88% with 0.48 M parameters.Source: A. EEG-FM like accuracy · pp. 6–7, Table III · p. 7

Emotion recognition

  • On SEED, SE-ResNet8 students score 80% and 78%, above the benchmarked foundation models; on EmoEEG, Strategy A reaches 77%, above the teacher’s 76%.On EmoEEG, Strategy B scores 60% and SE-ResNet4 scores 35% on the seven-class task.Source: A. EEG-FM like accuracy · p. 7, Table IV · p. 7

Motor imagery

  • On generalized BCI-IV-2a, the best model reaches 39% against 64% for REVE, whereas BCI-IV-2b results are closer to EEGPT: 69% for the student versus 72%.Students remain close to their teacher on both datasets; the paper links the remaining gap to limited motor-imagery pretraining diversity and zero-filled channels.Source: A. EEG-FM like accuracy · pp. 7–8, Table V · p. 7

A. EEG-FM like accuracy

  • Students match or exceed the teacher on binary clinical tasks and retain moderate emotion-recognition performance, but show the largest declines on motor imagery.The paper concludes that performance is comparable to or above EEG foundation models with 10–1,000× more parameters on abnormality and emotion tasks, while a motor-imagery gap remains.Source: A. EEG-FM like accuracy · p. 8, Figure 6 · p. 8

B. Accuracy Retention Under Compression

Compression preserves useful performance unevenly across tasks: SE-ResNet8 retains performance across all three categories, while macro-F1 exposes imbalances that accuracy alone can miss.

B. Accuracy Retention Under Compression

  • 7.6× smaller, the 1.56 M-parameter SE-ResNet8 preserves useful performance across all three task categories, while the 0.48 M SE-ResNet4 remains competitive on binary clinical tasks.Across six benchmarks, compression is most successful on binary clinical tasks, remains viable for emotion recognition, and is less effective for motor imagery.Source: B. Accuracy Retention Under Compression · p. 8
  • On TUAB, students have higher accuracy than the teacher (90% vs. 87%) but lower F1 (0.83 vs. 0.87), showing accuracy can overstate performance on an imbalanced task.The paper notes that missing abnormal recordings is costly in clinical deployment.Source: B. Accuracy Retention Under Compression · p. 8
  • On SEED, Strategy A has 78% accuracy and 0.77 F1, versus Strategy B’s 80% accuracy and 0.60 F1, indicating more balanced emotion-class predictions for Strategy A.Motor-imagery students remain below the teacher in both accuracy and F1, while the F1 analysis finds the gap smaller than the 7.6× compression might suggest.Source: B. Accuracy Retention Under Compression · p. 8

C. Inference Efficiency Across Deployment Hardware

Inference profiling across a server GPU, desktop CPU, and edge device shows that compression can lower runtime costs, with SE-ResNet8 offering a favorable accuracy–energy trade-off for desktop and edge deployment. The measurements use batch size 1, and CPU power and energy are proxy estimates.

GPU Tier

CPU Tier

  • On the i7-8700 CPU, SE-ResNet8 latency is 12.22 ms versus 19.19 ms for SE-ResNet18, but the CPU power figures are proxy estimates that may overestimate dynamic power.The profiling evaluates batch-size-1 inference; dynamic energy is computed from dynamic power multiplied by latency.Source: C. Inference Efficiency Across… · pp. 8–9

Edge Tier

  • 15.64 mJ per inference for edge SE-ResNet8 compares with 46.67 mJ for the SE-ResNet18 teacher, and the paper identifies SE-ResNet8 as a favorable edge accuracy-to-energy trade-off.On the Jetson Orin Nano, SE-ResNet8 latency is 5.73 ms; the teacher’s is 12.93 ms.Source: C. Inference Efficiency Across… · p. 9, Table VI · p. 9

VI. DISCUSSION

The discussion examines how pretraining transfers to compact students and how their representations change under task-specific fine-tuning. KD Strategy A brings students near the SSL teacher before labels, while fine-tuning pulls students toward task-specific representations, with weaker alignment on harder, data-scarce settings.

VI. DISCUSSION

  • The discussion connects representation transfer from teacher to student with a practical pathway toward wearable-class deployment.Source: §VI · p. 9

A. Knowledge Transfer from Teacher to Student

  • CKA ≈0.85–0.92: before labels, KD-pretrained students start extremely close to the SSL teacher’s representation space.Table VII reports linear CKA similarity between model variants.Source: A. Knowledge Transfer from Teacher to… · p. 9, Table VII · p. 10
  • CKA ≈0.19–0.25: fine-tuning substantially reorganizes the teacher’s feature space, while students only partially follow the fine-tuned teacher.Student alignment with the fine-tuned teacher is moderate on TUAB (0.48 for SE-ResNet8; 0.37 for SE-ResNet4) and weak on EmoEEG (≈0.22–0.24).Source: A. Knowledge Transfer from Teacher to… · pp. 9–10
  • TUAB t-SNE shows clearer normal/abnormal clusters, whereas EmoEEG emotion classes remain more entangled.The passage associates the stronger student–teacher divergence with harder, data-scarce settings.Source: A. Knowledge Transfer from Teacher to… · p. 10

B. Towards Wearable Deployment

Across six benchmarks, compact students retain useful performance where the teacher’s pretrained representation is strong, while motor imagery remains limited by pretraining diversity. Profiling shows reduced latency and energy, and the discussion frames SE-ResNet4 as a wearable candidate while noting that MCU deployment needs further work.

B. Towards Wearable Deployment

  • SE-ResNet4, the 0.48 M-parameter wearable candidate motivated by the MCU’s roughly 256 kB–1 MB on-chip memory, requires approximately 1.9 MB for 32-bit weights; this estimate covers weight storage only.Source: B. Towards Wearable Deployment · p. 10
  • Quantization to 16-bit or 8-bit would shrink SE-ResNet4 weights to roughly 0.96 MB or 0.48 MB; MCU-class deployment remains future work.The authors also identify channel-count reduction and subject-adaptive training as future work.Source: B. Towards Wearable Deployment · p. 10, §VII · p. 10

VII. CONCLUSION

  • Across six benchmarks, students retain useful performance where the teacher’s pretrained representation is strong, but motor imagery remains limited by pretraining diversity rather than compression.The conclusion also reports that compression reduced latency and energy per inference across server GPU, desktop CPU, and Jetson Orin Nano profiling.Source: §VII · p. 10
  • Strategy A provides shared initialization and more balanced per-class predictions, while Strategy B yields stronger task-specific accuracy, especially for abnormality detection.The conclusion cautions that accuracy alone can overstate compression success on class-imbalanced tasks.Source: §VII · p. 10

Related papers

Loading 2609.12218v1…