PaperlayerView exact revision on arXiv

Source-linked AI summary

Artifact Annotations Partially Substitute for Per-User Calibration: SAFE-EDA and a Normalization-Controlled Evaluation of Wrist-EDA Affect Recognition

Haochen Chai, Xinbi Luo, Zining Liu, Fangfang Jiang

arXiv:2610.01692v1cs.LGeess.SP

TL;DR

Wrist-EDA affect models normalize signals whose amplitudes vary across people, but studies often omit whether held-out-user data supplied those statistics. This paper compares artifact-supervised pretraining with scratch training across normalization protocols and datasets. Pretraining was better in 12 of 13 configurations, while normalization altered its gain differently across datasets.

  • Problem

    Studies of held-out-subject EDA recognition often do not report whether normalization statistics come from training subjects or the evaluated user, obscuring the information available at prediction time.

    Source: §I · pp. 1–2

  • Method

    The study compares SAFE-EDA pretrained on expert EDABE artifact labels with the same network trained from scratch, crossing normalization sources and window hops on WESAD and testing the factors on a second dataset.

    Source: §I · p. 2, C. Datasets and Cohorts · p. 4, G. Protocol axes and evaluation controls · p. 5

  • Results

    Artifact-supervised pretraining outperformed scratch training in 12 of 13 configurations; on WESAD its gain was 0.078 to 0.227, while per-user normalization increased the gain on the second dataset.

    Source: §VI · p. 10, §V · p. 10

  • Takeaways & Limitations

    Reporting the normalization-statistics source alongside the split, window length, hop, formula, and data-availability timing helps distinguish first-use evaluation from calibrated retrospective evaluation.

    Source: §VI · p. 10

    Both downstream datasets measure laboratory-evoked affect, so the results do not establish performance in daily-life or clinical use.

    Source: C. Scope and limitations · p. 10

Abstract

from arXiv · show

Wrist electrodermal activity (EDA) differs in amplitude from one person to the next, so affect-recognition models normalize their input before classification. Studies that test such models on held-out subjects seldom report where the normalization statistics come from, yet statistics computed from the held-out subject's own recording give the model information that a device does not have when it is first worn. We asked how this choice alters the measured benefit of pretraining. A compact convolutional network, SAFE-EDA, was pretrained on expert artifact annotations from 43 subjects and compared with the same network trained from scratch on the Wearable Stress and Affect Detection (WESAD) dataset (15 subjects, leave-one-subject-out), with two normalization sources crossed with four window hops. When the statistics came only from training subjects, pretraining raised macro-F1 by 0.078 to 0.227; when they came from the held-out user's full recording, the gain fell to between 0.020 and 0.050 and was no longer significant. Artifact supervision was far more useful than self-supervised pretraining on the same recordings (0.078 versus 0.008). Across 13 configurations in two datasets, the pretrained network was better in 12, but on the second dataset (26 subjects) per-user normalization increased the gain instead of reducing it, so the interaction depends on the data. Only five of 50 published WESAD studies state which data were used for normalization. Reporting this choice is necessary to separate first-use performance from performance after calibration.

I. Introduction

The paper examines how the source of normalization statistics affects the measured benefit of artifact-supervised pretraining for wrist-EDA affect recognition. It distinguishes first-use evaluation from retrospective per-user calibration and argues that normalization-source reporting is often missing.

I. Introduction

  • Per-user calibration uses the held-out user’s complete recording to set normalization statistics, whereas first-use evaluation relies on statistics learned from other people.The authors emphasize that neither protocol is inherently wrong, but they represent different information available at prediction time.Source: §I · p. 1
  • Only five of 50 surveyed WESAD studies stated where normalization statistics came from, compared with 26 that reported window length.The respective reported proportions were 10% (95% Wilson interval 4–21%) and 52% (39–65%).Source: §I · p. 1
  • The study compares identical network architectures and training procedures that differ only in initialization, crossing two normalization sources with four window hops on WESAD.It also tests a second dataset, contrasts artifact supervision with self-supervision and shuffled labels, and ablates backbone components.Source: §I · p. 2
  • On WESAD, artifact-supervised pretraining improved over scratch by 0.020–0.227, with the largest gain using training-subject statistics and less-overlapping windows; dependence reversed in direction on the second dataset.Pretraining on artifact labels gave a gain about ten times larger than self-supervision on the same recordings.Source: §I · p. 2

II. Related Work

  • Prior work distinguishes artifact recognition as a pretraining target from artifact detection used to clean signals, while subject-independent evaluation alone does not specify whether unlabeled test-subject data set the scale.The paper frames its question narrowly as varying normalization-statistics sources while holding the architecture fixed.Source: §II · p. 2

A. Study design

The study isolates initialization and normalization source while keeping the target backbone and training budget fixed. Its protocol combines a structured survey, paired dataset comparisons, and explicit distinctions between training-derived and held-out-user statistics.

A. Study design

  • The design compares correct artifact labels, shuffled labels, and self-supervision on the same source recordings, with scratch training indicating the architecture’s contribution.The authors define partial substitution operationally as one source of information yielding a smaller measured gain when another is present.Source: A. Study design · p. 3

B. Methods-text survey

  • The WESAD survey counted a protocol field only when Methods text specified it completely, and its proportions describe a fixed, non-exhaustive sample coded by one author.A scaler name without its fitting data did not count as reporting the normalization-statistics source.Source: B. Methods-text survey · p. 3

C. Datasets and Cohorts

  • WESAD provides wrist recordings from 15 subjects, and its main analysis uses 15-fold leave-one-subject-out evaluation; the Wearable dataset retains 26 identifiers with readable EDA files across all three sessions.All comparisons are paired within subject.Source: C. Datasets and Cohorts · p. 4

D. Normalization Regimes and Model

  • Global-train normalization fits statistics on training subjects, while per-subject-z uses all unlabeled samples from the held-out user and therefore evaluates retrospectively.A causal deployment would instead use training statistics or a defined pre-use segment; the segment duration and online updating remain unstudied.Source: D. Normalization Regimes and Model · p. 4
  • SAFE-EDA has 359,739 parameters, transferring 296,822 stem and residual-block parameters while reinitializing the target classification head in each fold.One 60-second window takes 140.6 MFLOPs and has a measured single-window CPU latency of 9.97 ms.Source: D. Normalization Regimes and Model · p. 4

E. Signal representation and transfer boundary

SAFE-EDA transfers temporal layers between source and target tasks while adapting kernel support to different sampling rates. A fixed decomposition supplies physiological features and reconstruction confidence, and the evaluation varies split, normalization source, and window hop.

E. Signal representation and transfer boundary

  • The temporal-stem and block weights transfer from 32-Hz source to 4-Hz, 60-s target data; source examples are not stretched, and target pooling and classification layers are initialized afresh.At 4 Hz, the three shorter experts share a 1.75-s support, while the longest reaches 3.25 s. Fixed kernel width keeps parameter shapes unchanged across sampling rates, enabling transfer.Source: E. Signal representation and transfer… · p. 4, Figure 2 · p. 5

F. Fixed decomposition and diagnostic confidence

  • The fixed decomposition branch receives no target-task gradients; its reconstruction residual yields deterministic confidence, not classification correctness or artifact probability.Only the source-task artifact head supplies artifact probabilities.Source: F. Fixed decomposition and diagnostic… · p. 4

G. Protocol axes and evaluation controls

  • The WESAD protocol crosses two normalization sources with four hops under leave-one-subject-out evaluation, with both initialization arms, three seeds, and 15 held-out subjects per cell.Hops of 1, 4, 60, and 240 samples correspond to nominal overlaps of 99.6%, 98.3%, 75%, and 0%; smaller hops increase redundant windows without adding participants.Source: G. Protocol axes and evaluation controls · p. 5

H. Training and Statistical Analysis

The analysis uses subject-level paired comparisons and reports that normalization sources are rarely specified in published WESAD studies. Window length is reported more often than the source of normalization statistics, although the latter determines whose observations shape the input scale.

H. Training and Statistical Analysis

  • Training used AdamW at a 0.001 learning rate for 12 epochs, with three seeds averaged within each subject before inference.No held-out value selected a cell, checkpoint, seed, epoch, or hyperparameter; neighboring-window results were descriptive, and configurations sharing subjects were not treated as independent tests.Source: H. Training and Statistical Analysis · p. 5
  • Subject-level intervals used 20,000 bootstrap resamples, while equivalence was summarized descriptively against a prespecified 0.05 macro-F1 margin.Expected calibration error and Brier score were computed without post-hoc temperature or threshold fitting.Source: H. Training and Statistical Analysis · pp. 5–6

A. Reporting in published WESAD studies

  • Only 5 of 50 studies identified the data used for normalization, compared with 28 reporting a split and 26 reporting window length.Four of the five normalization-reporting studies used the evaluated user’s own window or baseline.Source: A. Reporting in published WESAD studies · p. 6
  • Normalization source determines whose observations, and potentially which future observations, shape an example’s numerical representation, even when window length is reported.A formal comparison of the reporting proportions would require their joint pattern across studies.Source: A. Reporting in published WESAD studies · p. 6

B. Effect of split and normalization

On WESAD, pretraining’s measured benefit varied with both normalization source and window hop. Global-train normalization produced substantially larger gains than per-subject normalization, while a random-window split inflated scores and a single fixed holdout reversed the arm ordering.

B. Effect of split and normalization

  • Random-window assignment yielded transfer macro-F1 0.9966 versus 0.5794 under LOSO, while one fixed holdout reversed the ordering (0.4787 transfer, 0.5258 scratch).The random-window result is treated only as a leakage reference; inference rests on LOSO.Source: B. Effect of split and normalization · p. 6
  • Under LOSO, per-user normalization raised scratch macro-F1 from 0.5015 to 0.6762 and transfer macro-F1 from 0.5794 to 0.6964.At hop 1, wrist-only transfer reached accuracy 0.7815 (95% CI [0.7040, 0.8485]); comparison with the multimodal WESAD report is not fully protocol-aligned.Source: B. Effect of split and normalization · p. 6

C. The normalization–hop grid on WESAD

  • With global-train normalization, transfer gains over scratch were 0.0779, 0.1098, 0.2268, and 0.2145 across hops 1, 4, 60, and 240; all remained significant after Holm correction.Under per-subject-z normalization, gains were 0.0202–0.0495 and none was significant; the regime difference peaked at hop 60 (+0.1773, p = 0.000427).Source: C. The normalization–hop grid on WESAD · pp. 6–7, Figure 4 · p. 7
  • As hop increased from 1 to 60, scratch macro-F1 fell by 0.1350 under global-train normalization while transfer changed by only 0.0139.At hop 240, with fewer than 500 training windows per fold, the global-train gain remained 0.2145.Source: C. The normalization–hop grid on WESAD · p. 7
  • Transfer effects varied by participant, and some participants’ effect direction changed with normalization source.All 15 participants benefited at global-train hop 60; under per-subject-z there were 10 or 11 wins out of 15.Source: C. The normalization–hop grid on WESAD · p. 7

D. Stability across hops and configurations

Across hops, pretraining was more stable than scratch training, especially under global-train normalization. Transfer exceeded scratch in 12 of 13 configurations across WESAD and Wearable, but those configurations share subjects and are not independent replications.

D. Stability across hops and configurations

  • Across four hops, scratch macro-F1 varied by 0.1350 under global-train and 0.0572 under per-subject-z, versus transfer ranges of 0.0211 and 0.0455.Scratch’s across-hop standard deviation was 6.34 times transfer’s under global-train, but 1.22 times under per-subject-z.Source: D. Stability across hops and… · p. 7
  • Relative to global-train statistics, per-user normalization raised scratch macro-F1 more than transfer macro-F1 at every hop.At hops 1, 4, 60, and 240, the respective increases were 0.1747, 0.2087, 0.2911, and 0.2409 for scratch, versus 0.1170, 0.1331, 0.1138, and 0.0689 for transfer.Source: D. Stability across hops and… · p. 8
  • Transfer exceeded scratch in 12 of 13 configurations: 6 of 6 WESAD and 6 of 7 Wearable configurations.The one-sided sign test was nominally significant (p = 0.00171), but configurations share subjects, so the summary indicates consistency rather than independent replication.Source: D. Stability across hops and… · p. 8

E. What the pretraining contributes

SAFE-EDA’s pretraining objectives and transfer controls show that artifact supervision contributes substantially more than self-supervision on the same recordings. Correct artifact labels also outperform shuffled labels, and transferring the full trunk yields the strongest reported transfer score.

E. What the pretraining contributes

  • Artifact-label pretraining reached macro-F1 0.5794, a paired gain of 0.0779 over scratch, while self-supervised pretraining reached 0.5090, a difference of 0.0075.The comparison with self-supervision also changes the objective, so shuffled-label controls more directly test annotation content.Source: E. What the pretraining contributes · p. 8, Figure 6 · p. 9
  • Transfer scores rose from 0.5380 with the whole trunk frozen to 0.5794 with the full trunk transferred.Stem-only and block-only transfer reached 0.5463 and 0.5507, respectively; scratch architecture variants ranged from 0.5005 to 0.5094.Source: E. What the pretraining contributes · p. 8
  • Artifact recognition and affect classification have different outputs, so the transferred trunk can carry useful signal structure while the target head learns its own decision rule.The interpretation does not require artifact detection to clean the signal or reconstruct a clean waveform.Source: E. What the pretraining contributes · p. 8
  • Correct artifact labels contributed a further +0.0325 beyond shuffled-label supervision, while supervised pretraining with shuffled labels contributed +0.0455.These contrasts account for 41.7% and 58.3% of the total gain, respectively.Source: Figure 6 · p. 9

F. Specificity controls

Wearable results show that normalization source can change transfer gains in the opposite direction from WESAD, while the section’s controls examine artifact-task performance, class effects, calibration, and transition windows.

F. Specificity controls

  • Artifact F1 was 0.5792 versus 0.4234 for the released post-processed reference on the official EDABE test set.The difference was +0.1558 (95% CI [0.1196,0.1915]); the validation scores were 0.4654 versus 0.3453.Source: F. Specificity controls · p. 9
  • At global-train, hop 1, shuffled-label pretraining retained 0.0455 of the gain and correct labels added 0.0325.These were 58.3% and 41.7% of the gain, respectively; the passage attributes contributions to pretraining itself and label meaning.Source: F. Specificity controls · p. 9
  • The matched logistic-regression baseline reached 0.5729 (n = 15), 0.0065 below the transfer arm.Classical methods were evaluated only at the dense-window setting; sparse-hop scratch-versus-transfer comparisons used the same architecture.Source: F. Specificity controls · p. 9

G. Replication on the Wearable dataset

H. Class-Level Effects, Probability Calibration, and Transition Windows

  • Under global-train, transfer changed WESAD amusement accuracy by −5.94 points, versus +11.96 for baseline and +9.80 for stress.Subject-level expected calibration error changed little under per-subject-z (0.2116 to 0.2031), while differencing reduced it by 0.0320; excluding transition windows improved accuracy in both arms.Source: H. Class-Level Effects, Probability… · pp. 9–10

I. Signal-level diagnostics

The signal-level examples illustrate what the decomposition branch computes, while the discussion relates pretraining and per-user calibration as partly substitutable information on WESAD.

I. Signal-level diagnostics

  • In the lower-confidence window, departures from the smooth tonic component coincide with residual structure and reduced reconstruction confidence, but are not identified as artifacts.The target windows have no artifact annotations, and the examples show extremes of confidence.Source: §I · p. 10

V. Discussion

  • On WESAD, the gain over scratch ranged from 0.020 to 0.227, with normalization source accounting for much of the range under fixed architecture and training budget.The largest gain occurred with global-train normalization; per-subject-z uses the held-out user’s complete recording, including samples recorded after each prediction.Source: §V · p. 10

A. Annotations and calibration as partially substitutable information

  • Artifact annotations and per-user calibration partly supply the same information: per-user normalization helped scratch more than pretrained models at every WESAD hop.The proposed amplitude-invariance account is a hypothesis consistent with the comparisons, not a measured quantity; transfer retained positive point estimates under per-subject-z.Source: A. Annotations and calibration as… · p. 10

B. First-use monitoring and protocol disclosure

The paper distinguishes first-use monitoring from complete-record calibration and recommends disclosing the normalization data and their availability. Its downstream datasets are laboratory-evoked, and the study analyzes public data without recruiting participants.

B. First-use monitoring and protocol disclosure

  • The authors recommend reporting the split, window length and hop, normalization formula, statistics source, and when user data become available.A per-user baseline is legitimate when the intended workflow provides it, but complete-record normalization should be identified as retrospective.Source: B. First-use monitoring and protocol… · p. 10

C. Scope and limitations

  • Both downstream datasets record laboratory-evoked affect, so the results do not establish performance in daily-life or clinical use.The Wearable scans vary normalization and hop separately, so they cannot separate their interactions.Source: C. Scope and limitations · p. 10

VI. Conclusion

  • Artifact-supervised pretraining improved wrist-EDA affect recognition in 12 of 13 configurations across two datasets, but gain size depended on normalization source.On Wearable, the effect went the other way; shuffled-label controls indicate that supervised pretraining itself contributes alongside label content.Source: §VI · p. 10

Data and Ethics Statement

  • The study used public EDABE, WESAD, and PhysioNet Wearable releases; it recruited no new participants and redistributed no raw dataset data.The study is a secondary analysis of public data, with reproducibility code and non-sensitive derived materials available through the stated repositories.Source: Data and Ethics Statement · p. 11

Supplementary Material

The supplement documents a fixed, non-exhaustive sample of 50 studies, its Methods-text coding rules, and the released study and experimental data.

Supplementary Material

  • The 50-study sample is fixed and non-exhaustive, and one author coded it without an independent second coder.These limits describe the sample and coding process, not the soundness of the studies’ procedures.Source: Supplementary Material · p. 13
  • Fields were coded only when Methods or experimental-setup text established them; unresolved fields do not establish the information.A scaler name alone does not identify the statistics source, and a train/test ratio alone does not identify the allocation unit.Source: Supplementary Material · p. 13
  • The data supplement includes the search strategy, study identities, coded values, quotations, PDF page locations, and experimental-value CSV files.Paired inference uses seed-averaged subjects rather than windows or seeds; diagnostic subsets and table notes are defined in the data dictionary.Source: Supplementary Material · p. 13
  • Table S1 lists study-level reporting status for window length, hop, overlap, split type, LOSO, and metric aggregation.It also distinguishes training-data normalization from evaluated-user window or baseline normalization.Source: Table S1 · p. 13
  • Table S2 summarizes stated-field counts across the 50 studies, with 95% Wilson score intervals where computed.The counts describe reporting, not implementation quality.Source: Table S2 · p. 13
Loading 2610.01692v1…