PaperlayerView exact revision on arXiv

Source-linked AI summary

W-JEPA: Wavelet Joint-Embedding Predictive Architecture for Wearable Activity Recognition

Yogesh Prabhu, Loki Natarajan, Jingjing Zou

arXiv:2610.19411v1cs.LG

TL;DR

Wearable activity recognition must handle heterogeneous sensor setups and participants, while distinguishing activities that may differ only in subtle, scale-specific signal details. W-JEPA uses fixed wavelet tokens as units of masked latent prediction; across six benchmarks it outperforms evaluated baselines and remains strongest when participants are unseen.

  • Problem

    Wearable datasets differ in sensor placement, channel count, sampling characteristics, participants, and activity vocabularies, while similar activities can differ mainly in fine temporal structure.

    Source: §1 · pp. 1–2

  • Method

    W-JEPA organizes inertial windows into fixed wavelet tokens indexed by scale, sensor axis, and time, then predicts masked latent features.

    Source: §1 · p. 2

  • Results

    With participants held out, W-JEPA achieves 80.6 average macro-F1, a 12.1% relative improvement over the strongest published baseline at 71.9.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results suggest that physically grounded multiscale tokenization and latent prediction provide a promising foundation for transferable representations of heterogeneous wearable data.

    Source: §5 · p. 11

Abstract

from arXiv · show

A major challenge in wearable device activity recognition is the heterogeneity across datasets in sensor placement, channel count, sampling rate and subjects. Moreover, the motion patterns that distinguish activities often appear as subtle, scale-specific variations within otherwise similar waveforms. As a result, raw signals are often ambiguous, and models trained on them tend to rely on cues specific to a particular sensor setup or participant, which limits both cross-dataset transfer and generalization to unseen participants. Addressing this requires representations that preserve such activity-relevant, scale-specific variation. We introduce W-JEPA, a self-supervised framework that exploits the multiscale structure of inertial motion and leverages a joint-embedding predictive architecture, by organizing windows into a fixed grid of wavelet tokens indexed by scale, sensor axis, and time. These wavelet tokens help expose activity-specific detail within broadly similar raw signals, and serve as the units of masked latent prediction. W-JEPA predicts masked token representations supplied by an exponential-moving-average (EMA) target encoder. Because recovering a masked token requires relating motion at one scale and axis to motion at other scales and axes, the model learns the cross-scale, cross-axis structure of human movement. Across six wearable activity benchmarks, W-JEPA outperforms state-of-the-art methods under random-window splits and remains the strongest when subjects are held out, improving average macro-F1 by 8.7 percentage points over the strongest published baseline. These results indicate that physically grounded multiscale tokenization, combined together with latent predictive learning, provides a promising foundation for transferable representations of heterogeneous wearable inertial data.

1. Introduction

Wearable datasets vary in sensor setups and participants, while activities can differ only in subtle motion details that raw signals obscure. W-JEPA uses fixed wavelet tokens for masked latent prediction and achieves strong results across benchmarks, including when participants are held out.

1. Introduction

  • The model predicts masked-token latent features from an EMA target encoder, with masks drawn independently across channels and bands.Recovering a token encourages relationships across scales and axes, without reconstructing raw signal values.Source: §1 · pp. 2–3
  • With participants held out, W-JEPA reaches 80.6 average macro-F1, a 12.1% relative improvement over the strongest published baseline at 71.9.The paper reports that it achieves the highest mean accuracy and macro-F1 on every dataset under both subject-disjoint and random-window splits.Source: §1 · p. 3
  • Replacing wavelet tokens with raw temporal patches lowers average macro-F1 by 7.6 points under participant holdout, versus 2.9 points under random windows.The introduction says this is consistent with wavelet tokens reducing reliance on participant-specific cues.Source: §1 · p. 3
  • W-JEPA uses a fixed tokenizer to organize each inertial window into tokens indexed by scale, sensor axis, and time.Each band covers the same frequency range across datasets, while only the channel embedding depends on sensor layout.Source: §1 · pp. 2–3
  • Across six benchmarks, W-JEPA is reported to outperform eight baselines under both subject-disjoint and random-window splits.The evaluation excludes each target dataset from pretraining; its lead also holds with fine-tuning restricted to wrist sensors.Source: §1 · p. 3
  • Without wavelet tokens, 41.5% of horizontal exercise-bike windows are classified as vertical cycling, compared with 0.8% with them.The paper presents this as evidence that wavelet tokens help distinguish similar activities.Source: §1 · p. 3

2. Related Work

Prior wearable self-supervision uses raw temporal patches, learned codes, or other prediction targets, while wavelets have been used for features, forecasting, and reconstruction. W-JEPA combines scale-indexed wavelet units with masked latent prediction for multi-axis inertial data.

2. Related Work

  • Prior wavelet methods use wavelets as supervised features, forecasting tokens, or masked-coefficient reconstruction targets; W-JEPA instead uses them as units of masked latent prediction for multi-axis inertial data.The cited prior work includes Tian et al. (2020), Vuong et al. (2023), Masserano et al. (2024), and Thukral et al. (2026).Source: §2 · pp. 3–4
  • Wearable masked-modeling methods commonly reconstruct sensor values or temporal patches, while MoPFormer predicts codes from a learned motion-primitive vocabulary.The passage notes that reconstruction requires reproducing amplitude and phase details that vary with participant and sensor placement, while learned vocabulary units depend on pretraining data.Source: §2 · p. 4
  • HAR-JEPA applies joint-embedding prediction to activity recognition with temporal-patch units, which still mix motion scales within each interval.W-JEPA retains latent targets but replaces the unit with wavelet tokens indexed by scale, sensor axis, and time.Source: §2 · p. 4
  • Prior approaches to cross-dataset generalization alter the training signal through augmentation or language alignment, whereas W-JEPA changes the representation unit.The paper describes these directions as complementary and treats participant-disjoint results as the primary evaluation.Source: §2 · p. 4

3. Method

W-JEPA uses fixed multiscale wavelet tokens as prediction units, then predicts masked latent targets from visible context to model relationships across scales and sensor axes.

3. Method

  • W-JEPA tokenizes inertial windows into channel–band–time positions and predicts masked latent targets from visible context using an EMA target encoder.The EMA target encoder computes targets from the full token sequence.Source: §3 · p. 4

3.1. Wavelet tokenization

  • Each channel yields four detail and one approximation bands, split into P=2 bins whose mean and RMS statistics form addressed tokens in a 5CP grid.Learned channel, band, and bin embeddings provide token addresses; only the channel embedding depends on sensor layout.Source: §3.1 · pp. 5–6

3.2. Masked latent pretraining

  • The objective predicts target-encoder features rather than reconstructing raw samples and adds a per-window variance hinge across masked predictions.Replacing the variance hinge with SIGReg while holding the rest of the pipeline fixed costs accuracy.Source: §3.2 · pp. 6–7
  • Each channel–band pair produces two temporal tokens masked together, while pairs are selected asynchronously across channels and bands.This leaves complementary visible context and encourages the predictor to exploit dependencies across sensors, axes, and scales.Source: Figure 2 · p. 5, §3.2 · p. 6
  • For downstream fine-tuning, W-JEPA discards the target encoder and predictor and fine-tunes the context encoder with a classification head on all available channels.The target dataset’s channel count determines the tokenizer’s token positions.Source: §3.2 · p. 7

4. Experiments

W-JEPA leads the evaluated methods across both evaluation protocols, and wavelet tokens contribute especially under participant holdout and limited participant diversity. The experiments also examine similar-activity confusion, attention across timescales, and restricted wrist-sensor settings.

4.1. Main Results

  • W-JEPA achieves the highest mean accuracy and macro-F1 on every dataset under subject-disjoint and random-window splits, and loses the least under participant holdout.Table 1 reports the subject-disjoint comparison; Table 2 reports the random-window comparison.Source: §4.1 · p. 7, Table 1 · p. 8, Table 2 · p. 9

4.2. What the wavelet tokens contribute

  • Using wavelet tokens raises average held-out macro-F1 to 80.6, versus 73.1 for W-JEPA with temporal patches and 71.9 for HAR-JEPA.Wavelet tokens also outperform raw patches on all six subject-disjoint datasets.Source: §4.2 · p. 8
  • With the test cohort fixed, wavelet tokens generally outperform learned temporal patches as training participants are added, especially with few participants.Gains persist across displayed training-set sizes on PAMAP2, DSADS, and MHealth, but are smaller and less consistent on RealWorld.Source: §4.2 · p. 8, Figure 3 · p. 10
  • For horizontal exercise-bike cycling, confusion with vertical-bike cycling falls from 41.5% without wavelets to 0.8% with wavelets, while some errors shift to other activities.The learned representations also show more separation between the two classes with wavelets.Source: §4.2 · p. 9
  • On clean DSADS inputs, the coarse A4 band receives the largest attention share for most activities, while D1 leads for running and jumping.At 30% noise, A4 attention falls by 6.1% relative to clean inputs, while D2 and D1 rise by 11.1% and 9.4%; held-out metrics remain within about one point of clean values.Source: §4.2 · p. 9

4.3. Ablation under restricted sensor settings

  • With fine-tuning restricted to wrist inertial signals, W-JEPA achieves the highest mean score on both metrics for PAMAP2 and MHealth under both split protocols.Participant holdout lowers scores for every method, but W-JEPA retains its mean-performance lead.Source: §4.3 · p. 10

5. Conclusion

W-JEPA organizes inertial windows into wavelet tokens and learns by predicting masked latent features. The reported findings support its promise as a foundation for transferable representations of heterogeneous wearable data.

5. Conclusion

  • W-JEPA organizes inertial windows into tokens indexed by scale, sensor channel axis, and time.It learns by predicting masked latent features.Source: §5 · p. 11
  • W-JEPA outperforms the evaluated baselines and loses the least performance when evaluation uses unseen participants.Replacing wavelet tokens with temporal patches weakens performance and can increase confusion between similar activities.Source: §5 · p. 11
  • The results suggest that physically grounded multiscale tokenization and latent prediction together provide a promising foundation for transferable representations of heterogeneous wearable data.Source: §5 · p. 11

A. Datasets and Splits

The appendix describes six wearable activity datasets, which vary in sensor locations, channel types, sampling rates, participants, and activity labels. Their subject-disjoint splits are documented with sample and participant counts.

A. Datasets and Splits

  • Appendix Table 5 reports dataset statistics for subject-disjoint training and testing, with sample counts defined as segmented sensor windows.Source: Table 5 · p. 16
  • PAMAP2 records inertial signals from wrist, chest, and ankle devices across nine participants; DSADS records 19 activities from eight participants using five sensor units.PAMAP2 includes 12 specified activities and six optional labels (Reiss, 2012; Reiss & Stricker, 2012); DSADS provides 45 original sensor channels at 25 Hz.Source: A. Datasets and Splits · p. 16
  • MHealth covers 12 activities with sensors on the chest, right wrist, and left ankle, while RealWorld covers eight activities with devices at seven body positions.MHealth also includes two chest ECG leads (Banos et al., 2014; 2015); RealWorld records acceleration at 50 Hz and provides additional measurements.Source: A. Datasets and Splits · pp. 16–17
  • UCI-HAR provides six activities from 30 volunteers wearing a waist-mounted smartphone, whereas USC-HAD covers 12 daily activities from 14 participants using a hip-mounted device.UCI-HAR supplies filtered inertial signals in 128-sample windows; USC-HAD participants repeat each activity five times.Source: A. Datasets and Splits · p. 17

B. Implementation Details

The implementation uses fixed temporal-bin and masking settings, Transformer encoder and predictor stacks, and an EMA target encoder. Fine-tuning uses a linear classifier and adapts channel embeddings to the target channel count.

B. Implementation Details

  • Pretraining uses two temporal bins per band, a 0.65 mask ratio, 50 epochs, and batch size 256; fine-tuning uses 70 epochs and batch size 64.Both stages use AdamW with weight decay 0.05 (Loshchilov & Hutter, 2019).Source: B. Implementation Details · p. 17
  • The encoder has eight Transformer blocks of width 384, while the four-block predictor has width 192.Both use six attention heads, fixed sinusoidal position embeddings, and the specified shared normalization and activation settings.Source: B. Implementation Details · p. 17
  • The target encoder starts from the context encoder, has gradients disabled, and receives EMA updates with momentum increasing from 0.998 toward 1.Learning rates warm up from 10^-6 and then cosine-decay toward 10^-6.Source: B. Implementation Details · p. 17
  • Classification flattens encoder tokens before layer normalization, dropout, and a linear classifier, while channel embeddings are reused or reinitialized according to target channel count.Other compatible encoder weights transfer to fine-tuning, and fixed position embeddings are regenerated for the target token count.Source: B. Implementation Details · pp. 17–18

C.1. Regularizer ablation

The regularizer ablation changes only the variance regularizer, while the sensitivity study varies pretraining and input-design choices and the labeled fine-tuning fraction. The supplied passages describe these study designs but do not report their comparative outcomes.

C.1. Regularizer ablation

  • The regularizer ablation keeps the model, pipeline, and JEPA prediction loss unchanged while replacing the variance regularizer with SIGReg or a VICReg-style regularizer.Table 6 summarizes five-seed means and standard deviations across six benchmarks.Source: §C.1 · p. 18, Table 6 · p. 18

C.2. UCI-HAR sensitivity study

  • The UCI-HAR sensitivity study varies mask ratio, temporal bins, window length, and labeled fine-tuning fraction around a baseline of 65% masking, two bins per band, and 500-sample windows.Figure 5 reports top-1 test accuracy from one run; nominal bin sizes are not literal wavelet receptive fields.Source: Figure 5 · p. 18
  • The sensitivity sweeps cover 5–95% masking, 1–10 temporal bins, 100–1000-sample windows, and 1% or 10–100% of the training pool for fine-tuning.The experiments assess hyperparameter and data sensitivity, not whether wavelets or the JEPA objective are necessary.Source: §C.2 · pp. 18–19

D.1. Exercise-bike example details

The exercise-bike example uses left-leg gyroscope z-axis signals from DSADS and compares horizontal with vertical cycling. The two signals have a raw Pearson correlation of 0.992 and are processed into 500-sample windows after resampling.

D.1. Exercise-bike example details

  • The example uses the DSADS left-leg gyroscope z-axis (LL_zgyro), with panels (a) and (b) showing horizontal and vertical exercise-bike cycling from WHAR subject 7.Source: §D.1 · p. 19
  • The horizontal- and vertical-cycling raw signals have a Pearson correlation of 0.992.The signals are from the selected windows in sessions 119 and 127.Source: §D.1 · p. 19
  • Signals are resampled from 25 to 100 Hz and divided into 500-sample windows that remain within session boundaries.No amplitude normalization or post-extraction time shifting is applied.Source: §D.1 · p. 19

D.2. Attention by timescale under input noise

The attention analysis measures final-layer attention across wavelet bands on held-out DSADS participants and examines changes under added noise. It reports band-specific attention shifts, largely intact performance across tested noise levels, and representation projections across six datasets.

D.2. Attention by timescale under input noise

  • The analysis averages final-layer token attention over heads and query positions, sums it within each band, and compares noise-related changes against a fixed training-derived per-channel RMS reference.The middle heatmap normalizes each band's clean attention by the uniform allocation of 1/5; noise changes use paired participant-bootstrap intervals.Source: §D.2 · pp. 19–20, Figure 6 · p. 19
  • Full-test performance remains largely intact across tested noise levels, and the shift toward detail bands accompanies preserved classification performance without establishing causation.Table 7 reports DSADS full-test performance under fixed input noise on a single seed.Source: Table 7 · p. 20, §D.2 · p. 20
  • At 30% added noise, pooled A4 attention falls by 6.1% relative to clean inputs, while D2 and D1 rise by 11.1% and 9.4%.On clean inputs, A4 receives the largest attention share for most activities, while D1 leads for running and jumping.Source: §D.2 · p. 20

D.3. Visualization of learned representations

  • PCA and UMAP project W-JEPA's final-block representations across six activity-recognition datasets, with test windows colored by activity class within each dataset.The figure describes the representation visualization but does not state a comparative outcome.Source: Figure 7 · p. 21

Related papers

Loading 2610.19411v1…