PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Subject Statistical Shift Estimation for Generalized Electroencephalography-based Mental Workload Assessment

Isabela Albuquerque, João Monteiro, Olivier Rosanne, Abhishek Tiwari, Jean-François Gagnon, Tiago H. Falk

arXiv:1906.08823v2cs.LGeess.SPstat.ML

TL;DR

EEG workload models can generalize poorly across subjects, while domain-adaptation methods may rely on assumptions that do not hold for EEG. The paper estimates conditional and marginal shifts across subjects and examines normalization strategies; its results report that z-score normalization mitigated both shifts and improved assessment on unseen subjects.

  • Problem

    EEG workload correlates vary across subjects and physical strain, while domain-adaptation methods may rely on distributional assumptions that do not hold for EEG.

    Source: §I · pp. 1–2

  • Method

    The paper estimates cross-subject conditional and marginal shifts and evaluates normalization strategies alongside leave-one-subject-out workload classification.

    Source: §I · p. 2, D. Cross-subject mental workload… · p. 5

  • Results

    Z-score normalization mitigated both conditional and marginal shifts and improved assessment on unseen subjects; across subjects, it correctly classified roughly 70% of points.

    Source: B. Generalization gap · p. 7, §VI · p. 10

  • Takeaways & Limitations

    Shift estimates can help assess dataset assumptions and quantitatively investigate the effectiveness of domain-adaptation strategies.

    Source: §I · p. 2, §III · p. 3

    Z-score-normalized models had worse intra-subject classification performance than models using other strategies.

    Source: B. Generalization gap · p. 8

Abstract

from arXiv · show

Assessment of mental workload in real-world conditions is key to ensure the performance of workers executing tasks that demand sustained attention. Previous literature has employed electroencephalography (EEG) to this end despite having observed that EEG correlates of mental workload vary across subjects and physical strain, thus making it difficult to devise models capable of simultaneously presenting reliable performance across users. Domain adaptation consists of a set of strategies that aim at allowing for improving machine learning systems performance on unseen data at training time. Such methods, however, might rely on assumptions over the considered data distributions, which typically do not hold for applications of EEG data. Motivated by this observation, in this work we propose a strategy to estimate two types of discrepancies between multiple data distributions, namely marginal and conditional shifts, observed on data collected from different subjects. Besides shedding light on the assumptions that hold for a particular dataset, the estimates of statistical shifts obtained with the proposed approach can be used for investigating other aspects of a machine learning pipeline, such as quantitatively assessing the effectiveness of domain adaptation strategies. In particular, we consider EEG data collected from individuals performing mental tasks while running on a treadmill and pedaling on a stationary bike and explore the effects of different normalization strategies commonly used to mitigate cross-subject variability. We show the effects that different normalization schemes have on statistical shifts and their relationship with the accuracy of mental workload prediction as assessed on unseen participants at training time.

I. INTRODUCTION

Real-world workload assessment is difficult when demanding mental tasks coincide with physical strain, while EEG workload models vary across subjects and generalize poorly. Because domain adaptation can depend on distributional assumptions that may fail for EEG, the paper proposes estimating cross-subject shifts and studies their relation to normalization and workload prediction.

I. INTRODUCTION

  • EEG-based workload models show high cross-subject variability and poor generalization, while user calibration can be costly and impractical.This limits their applicability to real-world settings where workers may also experience physical strain.Source: §I · p. 1
  • Covariate shift cannot be assumed for EEG: conditional label distributions may differ across subjects even for the same feature vector.Domain adaptation methods rely on distributional assumptions, so checking which shifts occur in a dataset is important.Source: §I · pp. 1–2
  • The paper estimates cross-subject conditional and marginal shifts, tests whether normalization mitigates them, and examines 18 WAUC participants across physical-activity conditions.The study also evaluates workload classification and the effects of different ways to modulate physical activity.Source: §I · p. 2

II. DOMAIN ADAPTATION AND CROSS-SUBJECT GENERALIZATION

The section formalizes learning across domains using feature vectors, labeling functions, hypotheses, and risk. It then introduces empirical risk minimization as a practical approach when the underlying distribution is unavailable and defines the generalization gap.

II. DOMAIN ADAPTATION AND CROSS-SUBJECT GENERALIZATION

  • A domain is a distribution over a feature space of vectors computed from data, and a labeling function maps that space to labels.A hypothesis is a candidate mapping from the feature space to the label space; risk measures its loss on a domain.Source: §II · p. 2
  • True risk is generally intractable because the domain is unavailable; supervised learning therefore seeks a minimum-risk hypothesis over the hypothesis class.The risk is defined using a loss that measures disagreement with the true labeling function.Source: §II · p. 2

A. Empirical risk minimization

  • Empirical risk minimization uses an observed i.i.d. sample as a practical alternative, and the generalization gap is the absolute difference between true and empirical risks.The gap is small when empirical risk approximates true risk, supporting low risk on new samples from the domain.Source: A. Empirical risk minimization · p. 2

B. Domain adaptation

Domain adaptation bounds target-domain risk through source-domain risk, divergence between domains, and the compatibility of a hypothesis across them. An extension makes labeling-function mismatch explicit, showing why marginal alignment alone may not suffice.

B. Domain adaptation

  • A target-risk bound depends on source risk, HΔH-divergence between source and target, and the minimum combined risk achievable across both domains.The latter term captures how adaptable the hypothesis class is to both domains.Source: B. Domain adaptation · pp. 2–3
  • An extended bound replaces the adaptability term with a divergence accounting for possible mismatch between source and target labeling functions.The labeling mismatch is measured by the minimum disagreement between the functions under either domain's distribution.Source: B. Domain adaptation · p. 3
  • Good generalization requires both low marginal-distribution divergence and low labeling-function mismatch; reducing marginal divergence alone may hurt target performance when labels differ.Prior work often focused on marginal mismatch while assuming labeling functions were shared across domains.Source: B. Domain adaptation · p. 3

C. Cross-subject generalization as domain adaptation

The section treats each subject as a domain with its own labeling function and frames prediction for an unseen subject as a source-to-target adaptation problem. Source risk can be estimated from training subjects, but the other terms in the risk bound lack a straightforward estimate.

C. Cross-subject generalization as domain adaptation

  • For M subjects, the first M−1 form the pooled source domain, while the M-th subject is the unseen target for which risk is to be predicted.Each subject is associated with its own domain and labeling function.Source: C. Cross-subject generalization as… · p. 3
  • The unseen-subject risk bound includes source-to-target distribution divergence and a term for mismatch between the source and target labeling functions.The setup applies the two-domain bound to the pooled training subjects and the held-out subject.Source: C. Cross-subject generalization as… · p. 3
  • Source risk is estimated by aggregating risks across training subjects, but the other two terms in the bound have no straightforward estimate.The next section proposes a strategy for computing those remaining terms.Source: C. Cross-subject generalization as… · p. 3

III. ESTIMATING SHIFTS ACROSS MULTIPLE

The section proposes estimates of conditional and marginal shifts across subjects to characterize distributional mismatch and inform domain-adaptation choices. Conditional shift is estimated through pairwise comparisons of learned labeling functions and summarized across subjects.

III. ESTIMATING SHIFTS ACROSS MULTIPLE

  • Shift estimates can check assumptions such as covariate shift and compare feature spaces for their expected performance on unseen subjects.The paper presents these as uses of the estimates, not as guarantees of improved performance.Source: §III · p. 3

A. Conditional shift

  • Conditional shift is estimated across subjects by comparing learned labeling functions pairwise and aggregating the resulting disparities.The approach uses learned predictors because true labeling functions and the required expectations are unavailable in practice.Source: A. Conditional shift · pp. 3–4
  • For each subject pair, a k-nearest-neighbor labeler trained on one subject classifies labeled examples from the other, producing directional mismatch estimates.The two directional estimates are combined by taking their minimum, yielding a symmetric disparity value for the pair.Source: A. Conditional shift · pp. 3–4
  • Pairwise disparities form a symmetric matrix, whose Frobenius norm aggregates conditional shift and is rescaled to [0, 1] for feature-space comparisons.The matrix is symmetric because each pair's estimate is set using the smaller of its two directional estimates.Source: A. Conditional shift · p. 4

B. Marginal shift

Marginal shift is estimated from how well a classifier distinguishes data from different subjects, with pairwise estimates aggregated across the subject group.

B. Marginal shift

  • The H-divergence captures differences between marginal domain distributions and can be estimated from a binary classifier's error distinguishing samples from the two domains.Lower classifier error corresponds to a higher H-divergence estimate; labeled samples are not required.Source: B. Marginal shift · p. 4
  • For multiple subjects, pairwise domain divergences are estimated by classifying features from one subject against another and recording average error rates.The approach uses five-fold cross-validation to estimate those rates.Source: B. Marginal shift · p. 4
  • A rescaled Frobenius norm of the symmetric matrix of pairwise errors provides an aggregate marginal-shift value.The matrix collects the pairwise subject-classification error rates.Source: B. Marginal shift · p. 4

IV. EXPERIMENTAL SETUP

The experiments use WAUC EEG recordings collected during mental tasks and physical activity, extract frequency-band features, and compare normalization strategies for cross-subject workload assessment.

A. WAUC dataset

  • WAUC recordings cover 18 subjects performing MATB-II mental tasks while running or cycling, with EEG collected using an eight-channel wearable headset.Half the subjects performed treadmill activity and half used a stationary bike; the dataset also includes two types of baseline periods.Source: A. WAUC dataset · p. 4

B. Feature extraction

  • EEG was downsampled to 250 Hz, band-pass filtered from 0.5–45 Hz, and segmented into four-second epochs with three seconds of overlap.The extracted power spectral density features cover delta, theta, alpha, and beta bands.Source: B. Feature extraction · p. 4

C. Normalization

  • Baseline normalization uses each subject's baseline feature statistics to emphasize task-related changes, with baseline choice determining which stimuli are represented.Baseline 1 reflects mental and physical stimuli, whereas baseline 2 reflects changes associated with mental workload while physical effort is present.Source: C. Normalization · p. 5
  • The comparison includes no normalization, per-subject whitening, and normalization using either of the two baseline periods.The paper assesses how normalization relates to workload performance and conditional and marginal shifts.Source: C. Normalization · p. 5

D. Cross-subject mental workload classification

Cross-subject workload classification uses leave-one-subject-out evaluation, while the experiments estimate shifts without applying domain adaptation.

D. Cross-subject mental workload classification

  • Leave-one-subject-out cross-validation pools the other subjects for training and holds one subject out as the target, without applying domain adaptation.The setup evaluates distributional shifts and their relationship with out-of-distribution generalization.Source: D. Cross-subject mental workload… · p. 5

E. Implementation details

  • Experiments used 30 repetitions with random 300-of-600-example partitions; Random Forests used 20 estimators for marginal-shift estimation and 30 for workload prediction.The random seed was set to 10.Source: E. Implementation details · p. 5

V. RESULTS AND DISCUSSION

Normalization effects differed across conditional and marginal shifts, and some conditional-shift contrasts depended on activity type. Whitening reduced both shift estimates, while baseline normalization had mixed conditional effects and increased marginal shift.

A. Statistical shifts estimation

  • Baseline normalization slightly reduced conditional shift for bike, while treadmill baseline 1 increased it and baseline 2 brought it closer to whitening.The treadmill baseline 1 increase could potentially negatively affect workload assessment on unseen subjects.Source: A. Statistical shifts estimation · p. 5
  • Pairwise disparity matrices compare unnormalized and whitened features by activity; their diagonal entries are single estimates based on disjoint training and test sets.The diagonal values inform the quality of the labeling-function approximation but do not show estimate variability as Figures 1 and 2 do.Source: A. Statistical shifts estimation · pp. 5–6
  • Whitening significantly reduced aggregate conditional-shift estimates for both treadmill and bike participants.The figures show 30 independent estimates for each activity condition; lower values represent smaller estimated conditional shift.Source: A. Statistical shifts estimation · p. 5, Figure 1 · p. 6, Figure 2 · p. 6
  • Bike participants had much higher cross-subject conditional shift than treadmill participants, alongside lower reported classification performance for bike.The paper relates this pattern to more subject-specific EEG responses during stationary-bike activity; lower performance is reported in.Source: A. Statistical shifts estimation · p. 6
  • Subject-wise whitening decreased estimated marginal shift for both activities, whereas baseline 1 and 2 normalization increased it.Higher marginal shift indicates greater accuracy in pairwise cross-subject classification and thus easier subject discrimination in PSD feature space.Source: A. Statistical shifts estimation · p. 6

B. Generalization gap

Across treadmill and bike participants, z-score normalization had the smallest estimated source-to-target generalization gap and the best average accuracy, while its within-subject accuracy was lower than other strategies. The relationship between shifts and generalization was not uniform across normalization strategies or participants.

B. Generalization gap

  • The study estimated training and test accuracy and absolute generalization gaps per left-out subject, using 30 independent runs for treadmill and bike conditions.Evaluation used leave-one-subject-out cross-validation for low/high workload classification.Source: B. Generalization gap · p. 6, Table I · p. 8, Table II · p. 9
  • Bike participants had considerably lower source-domain accuracy than treadmill participants, whose labeling functions appeared easier to approximate.The authors note that this pattern is also reflected in diagonal values of the pair-wise disparity matrices.Source: B. Generalization gap · p. 7, Figure 3 · p. 7, Figure 4 · p. 7
  • Z-score normalization had the smallest approximated generalization gap in both activity conditions and the best average accuracy, correctly classifying roughly 70% of unseen-subject points.The two conditions were treadmill and stationary bike; results were averaged across subjects.Source: B. Generalization gap · p. 7, Table I · p. 8, Table II · p. 9
  • For treadmill participants, baseline 1 had a slightly smaller average gap than baseline 2 despite a considerably higher conditional shift, while their average marginal shifts were close.The authors suggest conditional and marginal shifts may differ in their contributions to the generalization bound.Source: B. Generalization gap · p. 7
  • Z-score normalization required no information beyond task features and reduced data-collection time, but its intra-subject classification performance was worse than with other strategies.The compared baseline-recording normalization strategies require a calibration step before the task.Source: B. Generalization gap · p. 8
  • For treadmill subjects 2, 3, 4, 5, 7, and 8, lower average conditional shift coincided with a smaller median estimated gap; subject 8 did not benefit from z-score normalization.For subject 8, conditional shift and generalization gap increased while accuracy decreased.Source: B. Generalization gap · pp. 8–9

C. Main takeaways

The conclusions report that z-score normalization mitigated conditional and marginal shifts while improving workload assessment on unseen subjects. They recommend reducing conditional shift, while noting an accuracy trade-off on data from training subjects.

C. Main takeaways

  • Prioritize normalization that decreases overall cross-subject conditional shift for smaller generalization gaps, while noting lower accuracy on unseen data from training subjects.Source: C. Main takeaways · p. 9

VI. CONCLUSIONS

  • Feature normalization and the equipment used to induce physical workload both affected the estimated conditional shift.Source: §VI · p. 9
  • Z-score normalization mitigated both conditional and marginal shifts and improved mental workload assessment on unseen subjects at training time.Source: §VI · p. 10
Loading 1906.08823v2…