PaperlayerView exact revision on arXiv

Source-linked AI summary

Automating Learner Assessment: Benchmarking Machine Learning and Deep Learning Models for EEG-Based Familiarity Prediction

Isuru Nanayakkara, Thilina Halloluwa

arXiv:2608.16541v1eess.SPcs.HCcs.LG

TL;DR

Objective learning assessment needs direct measures beyond quizzes and exams. This paper benchmarks EEG-based familiarity prediction across faces and equations, finding that trial-independent validation reduces peak performance to 60.38% using CNN while remaining above chance.

  • Problem

    Objective learning assessment needs direct measures beyond quizzes and exams because traditional methods rely on learners’ ability to articulate their knowledge.

  • Method

    The study records EEG while participants view familiar or unfamiliar faces and mathematical equations, then benchmarks machine-learning and deep-learning familiarity classifiers.

  • Results

    60.38% peak classification performance using CNN under trial-independent validation, down from 98.53% under standard splits, remained significantly above chance.

  • Takeaways & Limitations

    Trial-independent Group K-Fold validation provides a more realistic benchmark for EEG-based familiarity prediction, while temporal-frontal Gamma and Beta features support neurophysiological interpretation.

  • Takeaways & Limitations

    Participant identifiers were unavailable, so Group K-Fold could not prevent trials from the same subject appearing in both training and testing sets.

Abstract

from arXiv · show

Objective assessment of learning remains a fundamental challenge in education. Electroencephalography (EEG) provides a direct, non-invasive window into the neural correlates of knowledge acquisition, including cognitive familiarity. This study benchmarks fifteen machine learning (ML) and deep learning (DL) models for EEG-based familiarity prediction across two cognitive domains: faces (factual knowledge) and mathematical equations (conceptual knowledge). Using continuous EEG data from 23 participants, we extract spectral features (Power Spectral Density) across six frequency bands. We show that while standard stratified cross-validation yields artificially high classification performance (up to 0.9853 F1-score using CNN) due to temporal leakage across neighboring epochs, a rigorous trial-independent validation (Group K-Fold) drops the peak performance to 0.6038 F1-score (using CNN), which is still statistically significant above the 25% chance level. This highlights the critical necessity of trial-independent evaluation to avoid overestimating model generalizability. Furthermore, feature importance and SHAP analysis reveal that temporal and frontal Gamma and Beta oscillations are the most critical biomarkers for familiarity. This work establishes a realistic benchmark for EEG-based cognitive monitoring in educational technologies.

I. INTRODUCTION · II. RELATED WORKS

Traditional learning assessments often provide indirect and articulation-dependent proxies for understanding, motivating direct neural measures. This study positions EEG and comparative model evaluation as foundations for objective familiarity-based learning assessment.

  • I. INTRODUCTION: Examinations can be poor proxies for true understanding, creating a need for objective, direct measures of knowledge acquisition.
  • I. INTRODUCTION: The paper proposes measuring neural correlates of learning with EEG, focusing on familiarity as a foundational element of learning.
  • II. RELATED WORKS: Learning evaluation research has developed and validated diverse assessment methods, including extensively studied quizzes and examinations.
  • II. RELATED WORKS: Traditional quizzes and exams rely on learners’ ability to articulate knowledge, motivating alternative approaches that directly and objectively measure learning.
  • II. RELATED WORKS: Psychophysiological signals offer alternative measures of learning-related cognitive states, including eye-tracking for attention and comprehension and skin conductance for emotional arousal.
  • II. RELATED WORKS: Machine learning has decoded cognitive states from EEG, while prior work examined familiarity-related ERP signatures including the FN400 component.
  • II. RELATED WORKS: The study compares fifteen models for familiarity classification using human faces and mathematical equations, establishing a benchmark for reliable EEG-based learning evaluation.

III. METHODOLOGY · A. Participants and Experimental Design

The study elicited familiarity-related brain activity by presenting human faces and mathematical equations, with recognition labels established before EEG recording. Twenty-three participants completed counterbalanced face and equation blocks while viewing randomized familiar and unfamiliar stimuli.

  • III. METHODOLOGY: The experiment recorded brain activity while participants viewed human faces and mathematical equations to elicit familiarity-related cognitive responses.The two stimulus categories represented distinct familiarity domains: faces and equations.
  • III. METHODOLOGY: Face stimuli included recognized public figures and unfamiliar individuals, while equations ranged from well-known high-school formulas to obscure advanced equations.Examples included the Pythagorean theorem, quadratic formula, and Maxwell’s equations.
  • III. METHODOLOGY: Participants’ recognition responses produced familiar or unfamiliar labels for each face and equation, verified through a pre-experiment questionnaire.The questionnaire required participants to confirm recognition of every stimulus before the main experiment.
  • A. Participants and Experimental Design: Twenty-three healthy volunteers participated: 12 female and 11 male, aged 20-30, with a mean age of 24.5.Participants had science and mathematics backgrounds, provided informed consent, and the sample’s homogeneity limited generalizability.
  • A. Participants and Experimental Design: The main experiment used two counterbalanced blocks, one for faces and one for equations, each presenting 40 familiar and 40 unfamiliar images in random order.Each image lasted 2.5 seconds, followed by a 1.5-second interstimulus interval; participants silently counted familiar items.
  • A. Participants and Experimental Design: EEG was recorded with an ECI Electro Cap using 14 channels selected for relevance to cognitive and visual processing.The channels were Fp1, Fp2, F3, F4, F7, F8, C3, C4, T3, T4, P3, P4, O1, and O2.
  • A. Participants and Experimental Design: An Arduino-based system synchronized familiar/unfamiliar event markers with EEG recordings by detecting invisible color strips on the slideshow screen.The synchronization markers were embedded in the presentation and were not visible to participants.

B. Data Pre-processing · C. Feature Extraction and Validation Protocol

The study used a multistep EEGLAB preprocessing pipeline to clean EEG recordings before extracting PSD features. It compared leakage-prone stratified validation with trial-independent Group K-Fold validation while using cost-sensitive learning and weighted F1-score.

  • B. Data Pre-processing: EEG recordings were preprocessed and analyzed using the MATLAB EEGLAB toolbox.The pipeline cleaned raw data, removed artifacts, and prepared recordings for feature extraction.
  • B. Data Pre-processing: 128 Hz recordings were bandpass filtered from 1 Hz to 50 Hz after downsampling from 256 Hz.Noisy channels were manually removed and interpolated to address poor scalp contact.
  • B. Data Pre-processing: Electrode locations were remapped to the standard international 10-20 system for the ECI Electro Cap configuration.Mean baseline subtraction also corrected for DC offset.
  • C. Feature Extraction and Validation Protocol: PSD features were calculated with Welch’s method using 64-sample windows, 87.5% overlap, 32-sample nperseg, and 50% overlap across six frequency bands.Features were extracted from cleaned EEG epochs using sliding windows.
  • C. Feature Extraction and Validation Protocol: Class-balanced weights inversely proportional to class frequencies addressed domain-size differences during model fitting.The weighted F1-score was used as the primary performance metric.
  • C. Feature Extraction and Validation Protocol: Two validation protocols were implemented to address temporal data leakage.The comparison contrasted randomly shuffled epoch-level validation with trial-independent grouping.
  • C. Feature Extraction and Validation Protocol: Standard 5-fold Stratified K-Fold shuffled epochs randomly, allowing neighboring windows from the same trial into training and testing sets.This leakage-prone setup could artificially inflate performance.
  • C. Feature Extraction and Validation Protocol: Trial-independent 5-fold Group K-Fold kept epochs from each trial together using GroupID across 37 unique blocks.Grouping by unique trial block ID prevented epochs from the same trial from being split across training and testing sets.

1) Deep Neural Network (DNN) Architecture and Hyperparameter Tuning: · IV. RESULTS

The DNN was optimized with regularization and early stopping, while the results section frames EEG and deep learning as feasible for automated familiarity assessment. Results combine ERP scalp-map analysis with quantitative machine-learning evaluation.

  • 1) Deep Neural Network (DNN) Architecture and Hyperparameter Tuning:: The final DNN uses an input layer with 84 features and hidden dense layers containing 32 and 16 neurons.Both hidden layers use ReLU activation.
  • 1) Deep Neural Network (DNN) Architecture and Hyperparameter Tuning:: Dropout (0.5) was incorporated into the two hidden dense layers to help prevent overfitting on spectral features.The architecture was explicitly optimized for this purpose.
  • 1) Deep Neural Network (DNN) Architecture and Hyperparameter Tuning:: The DNN uses a softmax output layer for four classes and was trained with categorical cross-entropy loss.The passage specifies the output structure and training objective.
  • 1) Deep Neural Network (DNN) Architecture and Hyperparameter Tuning:: The Adam optimizer was configured with learning rate = 0.001, alongside early stopping with patience = 15.These settings were part of the final DNN training configuration.
  • 1) Deep Neural Network (DNN) Architecture and Hyperparameter Tuning:: Similar hyperparameter tuning was conducted for the CNN and LSTM models.The passage places CNN and LSTM tuning alongside DNN optimization.
  • IV. RESULTS: The results assess EEG and deep learning as feasible approaches for predicting a learner’s familiarity with educational content.The study presents this feasibility finding as relevant to objective and automated learner evaluation.
  • IV. RESULTS: ERP scalp maps provide neurophysiological context for the classification results and support examination of familiarity-related neural correlates.The analysis also identifies brain regions most involved in this cognitive process.
  • IV. RESULTS: Quantitative machine-learning analysis reports classification performance and establishes a benchmark for future work.The results are organized into visual ERP analysis followed by quantitative evaluation.

1) Mathematical Equations: · 2) Human Faces:

EEG scalp maps distinguished familiar from unfamiliar mathematical equations and human faces through opposing activation patterns. Familiar equations showed frontal positivity, whereas unfamiliar faces showed increased positivity over centro-parietal regions associated with novel, task-relevant stimuli.

  • 1) Mathematical Equations:: Familiar mathematical equations elicited more positive activations, particularly over frontal scalp regions.These activations increased with higher amplitudes.
  • 1) Mathematical Equations:: Unfamiliar mathematical equations produced more negative activations than familiar equations.The contrasting patterns are shown in the ERP scalp maps for familiar and unfamiliar equations.
  • 1) Mathematical Equations:: Frontal positivity for familiar equations may reflect working memory and semantic retrieval processes.This interpretation is consistent with the role of frontal lobes in higher-order cognitive functions.
  • 2) Human Faces:: Unknown human faces elicited higher activations over frontal, central, and parietal scalp regions.ERP scalp maps compare familiar and unfamiliar human-face responses.
  • 2) Human Faces:: Familiar human faces produced more negative activations than unfamiliar human faces.This contrast was observed when participants viewed the two face categories.
  • 2) Human Faces:: Unfamiliar faces showed increased positivity, particularly over centro-parietal sites, reminiscent of the P300b component.P300b is often associated with processing novel and task-relevant stimuli.

A. Quantitative ERP and PSD Statistical Analysis · B. Machine Learning-Based Classification

Trial-level aggregation addressed pseudoreplication from overlapping sliding windows, showing robust domain differences but subtler familiarity-related spectral shifts. Fifteen ML and DL models were then evaluated under stratified and trial-independent cross-validation to assess data leakage effects.

  • A. Quantitative ERP and PSD Statistical Analysis: Trial-level aggregation was used to address pseudoreplication caused by highly autocorrelated overlapping sliding windows.PSD features were averaged across sliding windows belonging to each trial.
  • A. Quantitative ERP and PSD Statistical Analysis: High-frequency spectral features differentiated stimulus domains robustly, whereas familiarity-driven spectral shifts lacked trial-level significance.The passage characterizes familiarity-related changes as more subtle than domain-related differences.
  • A. Quantitative ERP and PSD Statistical Analysis: Larger participant cohorts or trial-independent deep learning models were identified as necessary to resolve familiarity-driven spectral shifts.This limitation follows the absence of trial-level significance for familiarity effects.
  • B. Machine Learning-Based Classification: Fifteen machine learning and deep learning models were evaluated under both cross-validation protocols to illustrate data leakage effects.The protocols are contrasted through stratified and trial-independent evaluation in the accompanying figure.
  • B. Machine Learning-Based Classification: Model performance was summarized using weighted F1-score ± SD with 5-fold cross-validation.Table I reports comparative performance across the evaluated models.
  • B. Machine Learning-Based Classification: Stratified K-Fold with temporal leakage was compared against Group K-Fold trial-independent validation.Figure 5 presents the comparison of model weighted F1-scores under both protocols.

C. Domain-Separated and Within-Category Classification

Domain-separated LOGO evaluations showed robust decoding of familiarity within equations and faces, while permutation testing confirmed that all familiarity-related tasks were significantly above chance. The findings indicate that multivariate spectral patterns, rather than isolated frequency-band shifts, support familiarity classification.

  • Evaluation design: Secondary evaluations separated domain-only classification from equations-only and faces-only familiarity classification to reduce domain–familiarity conflation in the 4-class task.The evaluations used separate binary classifiers for Equations versus Faces, Familiar versus Unfamiliar Equations, and Familiar versus Unfamiliar Faces.
  • LOGO performance: 0.9491 ± 0.0867 peak average LOGO Mean Recall was achieved for domain-only classification using CNN.HistGradient Boosting reached 0.8998 ± 0.1025 for the same task.
  • LOGO performance: 0.8992 ± 0.0850 peak average LOGO Mean Recall was achieved for faces-only familiarity using HistGradient Boosting, versus 0.8875 ± 0.2347 using CNN Optimized.These results support decoding familiarity within the face category rather than only discriminating visual domains.
  • Permutation significance: 0.0020 faces-only familiarity p-value and 0.0120 equations-only familiarity p-value remained significant after Bonferroni correction.The adjusted values were 0.0060 for faces-only and 0.0360 for equations-only familiarity after 1000 permutations.
  • Interpretation: Classifiers exploited a significant joint multivariate pattern of spectral power across channels and bands, although individual frequency bands showed no robust standalone familiarity shifts after FDR correction.This reconciles the classification results with the univariate statistical tests.
  • Permutation significance: p ≤0.0010 was observed for the 4-class familiarity task, with True Mean Recall = 0.6976 and Mean Null Recall = 0.3229.All three familiarity-related tasks survived Bonferroni correction, establishing performance above chance.

D. Feature Importance and SHAP Interpretability Analysis

Interpretability analysis used Random Forest feature importances and SHAP values from models trained on the original 4-class configuration. High and Low Gamma, Beta, and Gamma/Beta power at specified channels emerged as important features influencing familiarity predictions.

  • Analysis design: Feature rankings and SHAP values were derived from models trained on the original 4-class configuration spanning familiarity levels across equation and face domains.This configuration was used instead of domain-separated binary classifiers to reflect the overall decision structure.
  • Random Forest feature importance: 32.2% High Gamma and 26.2% Low Gamma were the most critical Random Forest spectral features, followed by 15.8% Beta.Random Forest was selected because it natively ranks nonlinear feature interactions in high-dimensional EEG spectral data.
  • SHAP interpretability: Higher Gamma and Beta power at T4, F8, and F7 strongly drove the DNN toward predicting familiar states.SHAP values revealed the direction of feature impact and provided interpretability for the deep learning model.

V. DISCUSSION · A. Contributions and Implications

The study demonstrates that temporal leakage inflates EEG familiarity-prediction performance, while trial-independent validation provides a more realistic benchmark. It also identifies ensemble-model generalization and high-frequency temporal-frontal oscillations as important implications for practical assessment tools.

  • V. DISCUSSION: 98.53% to 60.38%: peak classification performance fell under trial-independent validation, or to 55.10% using Gradient Boosting.The rigorous result remained significantly above chance, supporting the feasibility of objective familiarity assessment.
  • V. DISCUSSION: Temporal data leakage inflated performance when standard cross-validation was compared with trial-independent validation.The study frames this inflation as a central challenge in decoding cognitive familiarity from EEG signals.
  • A. Contributions and Implications: Ensemble methods, specifically Gradient Boosting and Random Forest, generalized better to unseen trials than deep learning architectures on small EEG datasets.This finding motivates benchmarking procedures that explicitly account for temporal leakage and limited sample sizes.
  • A. Contributions and Implications: 0.6038 F1-score: CNN achieved the peak trial-independent benchmarking performance on the full sliding-window dataset of 3960 samples.LSTM reached 0.4816 under the same full-dataset configuration.
  • A. Contributions and Implications: ± 0.0780 and ± 0.2174: CNN and LSTM, respectively, showed relatively high standard deviations, indicating limited training stability across evaluations.The passage links larger sample size with more robust spatial-spectral representations than smaller configurations, where deep architectures overfit heavily.
  • A. Contributions and Implications: Gamma and Beta oscillations in temporal-frontal regions were identified as key neurophysiological markers through quantitative t-tests, SHAP, and Random Forest importances.These findings provide a scientific basis for lighter, real-time familiarity assessment tools using simplified feature sets.

B. Limitations and Future Directions

The study is limited by a small, homogeneous sample, possible cross-subject leakage, unstable domain-separated metrics, coarse spectral resolution, and subjective preprocessing. Future work should validate models across subjects in larger populations, automate artifact handling, and incorporate explicit temporal features.

  • Sample Size and Homogeneity: 23 participants, all with STEM backgrounds, limit generalizability, especially for faces-only results based on a 13-block subset.The domain-separated face findings rely on a small subset.
  • Cross-Subject Leakage: Trial-level Group K-Fold validation resolves temporal leakage, but missing participant identifiers leave cross-subject leakage possible.Trials from the same subject could appear in both training and testing sets, particularly affecting domain-separated binary models.
  • LOGO Metric Instability: 0.8670±0.2392 Mean Recall for equations and 0.8875±0.2347 for faces under CNN show instability in LOGO evaluation.With only 12–13 blocks per domain, atypical or noise-heavy blocks can strongly affect the mean.
  • Spectral Resolution Constraints: 4 Hz frequency resolution makes narrow bands such as Delta (1–4 Hz) difficult to isolate precisely, limiting low-frequency feature interpretation.This constraint results from a 128 Hz sampling rate and 32-sample Welch segment length.
  • Manual Pipeline Subjectivity: Manual ICA selection and artifact rejection introduce subjectivity; future work should use tools like ICLabel, larger cross-subject populations, and explicit ERP amplitude and latency features.ERP markers are proposed specifically for the face condition.

VI. CONCLUSION · VII. DECLARATION OF GENERATIVE AI AND AI-ASSISTED TECHNOLOGIES IN THE MANUSCRIPT

The study benchmarks fifteen machine learning and deep learning models for EEG-based familiarity prediction and shows that validation protocols critically affect reported performance. Trial-independent Group K-Fold validation yields realistic peak performance using CNN, while temporal-frontal Gamma and Beta bands emerge as significant features.

  • VI. CONCLUSION: 15 models are benchmarked for EEG-based familiarity prediction, emphasizing validation protocols in brain-computer interfaces.
  • VI. CONCLUSION: Temporal leakage inflates accuracy under standard data splits.
  • VI. CONCLUSION: 60.38% realistic peak performance is achieved using CNN under trial-independent Group K-Fold validation.
  • VI. CONCLUSION: 55.10% performance is achieved using Gradient Boosting under trial-independent Group K-Fold validation.
  • VI. CONCLUSION: Trial-independent Group K-Fold validation reveals more realistic performance than standard splits.
  • VI. CONCLUSION: Temporal-frontal Gamma and Beta bands are identified as significant features for EEG-based familiarity prediction.
Loading 2608.16541v1…