Source-linked AI summary
TransfHAR: Self-Supervised Wrist Representations for On-Demand Activity Recognition
Aidan Bradshaw, Riku Arakawa, Xin Liu, Karan Ahuja
TL;DR
Fine-grained wrist activity recognition lacks scalable adaptation to user-defined activities because labeled data are scarce. TransfHAR uses self-supervised pretraining on broad coarse wrist IMU data, then adapts a frozen representation with a lightweight probe, exceeding supervised baselines by 6.2 balanced-accuracy points on average across three held-out benchmarks.
Problem
Fine-grained wrist activity recognition lacks scalable adaptation to arbitrary user-defined activities because public wrist IMU datasets and fixed label vocabularies provide limited relevant labeled data.
Method
TransfHAR learns transferable wrist motion representations through self-supervised pretraining on pooled coarse IMU data, then adapts a frozen encoder with a lightweight linear probe.
Results
6.2 balanced-accuracy points on average: TransfHAR exceeded supervised baselines across three held-out manipulative, procedural, and gesture-like benchmarks.
Takeaways & Limitations
Broad self-supervised wrist pretraining supports rapid personalization to new fine-grained activities without retraining a full model.
Takeaways & Limitations
The window-based formulation cannot model longer temporal structure, limiting performance on procedural steps that are locally similar but sequentially distinct.
Abstract
from arXiv · showhide
Fine-grained wrist activity recognition can support applications such as procedural step guidance and context-aware assistance, yet acquiring labeled data for every new task, user, and activity granularity remains a bottleneck. We present TransfHAR, a self-supervised wrist IMU framework for on-demand, fine-grained activity recognition by learning transferable motion priors from global, unlabeled activities. We show that self-supervised pretraining on coarse wrist IMU activities (e.g., sitting, walking, exercise) learns motion structure rich enough to transfer to fine-grained manipulative, gestural, and procedural activities (e.g., snapping, stirring, waving) that are absent from pretraining. We implement TransfHAR as a real-time smartwatch application that lets users define and expand their own activity set for personalized recognition from only a few demonstrations. Across three offline cross-dataset evaluations, TransfHAR matches or exceeds fully supervised baselines that use complete label sets with equal or additional sensor channels, by 6.2 balanced-accuracy points on average. In an in-lab study with 10 participants each performing seven novel wrist activities, TransfHAR reaches 86.7% balanced accuracy across participants with five examples per class and 90.4% when updated from a single one-minute recording per class. These results indicate that broad self-supervised wrist pretraining provides an effective foundation for on-demand fine-grained activity recognition.
1 Introduction
TransfHAR addresses the mismatch between fixed, benchmark-driven activity sets and users’ diverse routines by learning transferable wrist-IMU motion representations for personalized, on-demand recognition. Across offline cross-dataset and interactive evaluations, it transfers to held-out fine-grained activities and supports rapid smartwatch adaptation from limited demonstrations.
- Motivation: Fixed activity sets rarely match users’ actual routines, while collecting labeled data and training task-specific classifiers for every new activity does not scale.Limited-example training from scratch is ineffective, and prior customization approaches generally extend fixed vocabularies within one interaction domain.
- Approach: TransfHAR uses self-supervised learning on pooled public wrist-IMU data to learn reusable motion structure for user-defined activity recognition.A ViT-1D encoder learns patterns such as oscillations, impacts, rotations, and cross-axis coordination that recur in manipulations, gestures, and task steps.
- Interactive evaluation: 86.7% balanced accuracy is achieved with 12.8 seconds of demonstration per class, rising to 90.4% using one minute per class in the real-time smartwatch system.Users define activity labels, provide demonstrations, train a lightweight personalized classifier on-device, and receive real-time predictions.
- Offline evaluation: 6.2 balanced-accuracy points on average: frozen self-supervised wrist representations exceed supervised baselines across held-out manipulative, procedural, and gesture-like benchmarks.The offline evaluation spans three wrist-IMU benchmarks whose activity regimes are excluded from pretraining.
2 Related Work
Prior work establishes self-supervised learning and wrist sensing as promising foundations for transferable and fine-grained activity recognition. However, existing evaluations usually assume aligned tasks or predefined activity sets, leaving broad coarse-motion pretraining and on-demand customization open.
- Self-supervised wearable sensing: Self-supervised learning learns transferable representations from unlabeled data across language, vision, and time series, using masked prediction and contrastive objectives.Wearable sensing is especially suitable because inertial datasets are costly to annotate and often heterogeneous in sensors and collection protocols.
- Self-supervised wearable sensing: Wearable self-supervised methods include transformation prediction, cross-dimensional motion prediction, contrastive learning, and masked reconstruction, with recent work expanding corpora and evaluation pipelines.These efforts demonstrate that inertial signals contain substantial latent structure learnable without semantic labels.
- Transfer limitations: Most evaluations use aligned pretraining and downstream settings, while models are commonly fine-tuned end to end instead of used as fixed feature extractors.This leaves unresolved whether broad pretraining on predominantly coarse wrist motion can produce a frozen representation for different activity regimes, datasets, or label spaces.
- Transfer limitations: Most HAR systems train supervised models for fixed datasets, sensor configurations, and activity vocabularies, while transfer learning typically targets cross-dataset or cross-user generalization within the same granularity.These transfer settings generally involve relatively small label spaces of fewer than 10 classes.
- User customization: Wrist sensing supports fine-grained recognition of predefined finger motions, hand and object interactions, and daily activities such as writing and stirring.This literature generally assumes that the activity set is specified during system design and training.
- User customization: User customization research uses demonstrations, template- and distance-based recognition, and lightweight personalized models to support recognition or adaptation from few examples.Examples include uWave’s user-defined gestures and smartwatch systems extending existing recognizers with one to five examples.
3 Datasets
TransfHAR standardizes heterogeneous public wrist IMU datasets into a common smartwatch-oriented representation and organizes activities by wrist-signal motor complexity. The pooled data are concentrated in lower-complexity posture, locomotion, and exercise activities, while fine-grained manipulative and procedural motions are scarcer.
- Sensor standardization: All sources are converted to wrist-only IMU streams sampled at 50 Hz with a shared FLU axis convention and consistent physical units.The dominant wrist is used as the canonical reference when available.
- Motor-complexity taxonomy: TransfHAR defines four motor-complexity levels based on dominant wrist signal characteristics rather than activity names or class counts.The taxonomy addresses inconsistencies in how prior HAR datasets describe coarse and fine-grained activities.
- Motor-complexity taxonomy: L1 activities are gravity-dominated, L2 activities are quasi-periodic, L3 activities contain aperiodic variable-amplitude bursts, and L4 activities require temporal context across ordered steps.L3 activities typically require learned representations, while L4 activities are locally similar to L3 within short windows.
- Dataset coverage: Most large public wrist IMU datasets emphasize L1/L2 posture, locomotion, and exercise, whereas L3/L4 manipulative and procedural motions remain much scarcer.Figure 3 describes the pooled data landscape as dominated by L1/L2 activities, with comparatively limited L3 and negligible L4 coverage.
4 TransfHAR Framework
TransfHAR uses self-supervised masked reconstruction to learn transferable wrist-IMU representations from coarse activities, then freezes the encoder for lightweight, user-defined fine-grained activity classification. The framework supports on-device personalization and real-time inference on a smartwatch.
- Two-stage framework: TransfHAR pretrains a ViT-1D encoder on pooled public wrist IMU data using masked reconstruction over coarse L1/L2 activities, then transfers its representations to user-defined L3/L4 classes.Stage A uses unlabeled public data, while Stage B freezes the encoder and trains a lightweight linear probe for downstream activities.
- Self-supervised pretraining: The masked autoencoder hides 65% of patch tokens, processes visible tokens with the transformer, and reconstructs the masked patches using a lightweight decoder.The reconstruction loss is mean squared error computed over masked patches.
- Signal processing and encoder: The wrist signals are segmented into 2.56-second windows at 50 Hz with 50% overlap, normalized per channel, low-pass filtered at 24 Hz, and patch-tokenized independently by channel.The 3-axis and 6-axis settings produce 96 and 192 tokens respectively, with 384-dimensional embeddings.
- On-demand personalization: Stage B uses a frozen encoder that produces a 384-dimensional mean-pooled embedding, while only a single linear probe is updated as users add examples or activity classes.Probe training uses inverse-frequency class-weighted cross-entropy, validation balanced accuracy, and early stopping.
- On-device implementation: 23.8 ms is required for encoder inference and probe prediction per available window on Apple Watch Series 10, with a 1.3 s refresh interval including stride.The full capture, training, and classification loop runs locally on the watch without a paired phone or remote server.
5 Offline Evaluation · 5.1 Datasets · 5.2 Protocol
The offline evaluation tests whether frozen wrist-IMU representations pretrained on broad motion transfer to held-out fine-grained activities across three public benchmarks. It compares few-shot and full-split probe training under dataset-specific protocols using balanced accuracy and motion-only supervised baselines.
- 5 Offline Evaluation: The evaluation probes a frozen encoder on three public wrist-IMU benchmarks covering manipulative, procedural, and gesture-like activities held out from pretraining.The goal is to assess transfer from broad L1/L2 wrist motion to fine-grained L3/L4 activities.
- 5.1 Datasets: SAMoSA contains 27 everyday activity classes from 20 participants, with 50 Hz wrist IMU and audio collected in home and workshop environments.Only the motion stream is used, and the activities are primarily L3 manipulations spanning kitchen, bathroom, and workshop settings.
- 5.1.1 SAMoSA.: PrISM-Tracker contains eight multi-step procedures; the latte-making subset includes 19 labeled steps across 23 sessions from 14 participants.TransfHAR is evaluated on the latte-making procedure for both few-shot and full-split comparison, alongside motion-only baselines.
- 5.1.2 PrISM-Tracker.: UTD-MHAD provides 27 actions from 8 subjects across multiple modalities, but evaluation uses only its 21-class inertial wrist subset and the published inertial baseline.The subset contains short gesture and exercise motions such as arm swings, drawing motions, knocks, and throws.
- 5.2 Protocol: Evaluation uses each dataset’s published protocol, reproduces PrISM and SAMoSA motion-only baselines from their original code, and tests 3-axis and 6-axis encoders with matching checkpoints.This establishes comparisons under consistent dataset-specific procedures while preserving each encoder’s corresponding pretrained input configuration.
- 5.2.1 Few-shot and full-split conditions.: Few-shot probes use K∈{1, 5, 10, 20} randomly sampled labeled windows per class, corresponding to 2.56, 12.8, 25.6, and 51.2 seconds per class.Only the training split is subsampled; validation and test sets remain unchanged.
- 5.2.1 Few-shot and full-split conditions.: In the full-split regime, probes train on all windows in each training partition, while the encoder remains frozen and only the probe is optimized.Results are reported as means over five random seeds with standard deviations in parentheses.
- 5.2.2 Metrics. Í𝐶: Balanced accuracy is the primary model-selection metric, and supervised comparisons use fully supervised motion-only baselines with 9-axis input for SAMoSA and PrISM and 6-axis input for UTD-MHAD.Balanced accuracy accounts for class imbalance across benchmarks.
5.3 Offline Results
Offline evaluations show that TransfHAR’s 6-axis representation improves with additional labeled shots, exceeds supervised baselines under full-split transfer, and generalizes unevenly across procedural activities. Longer temporal context and gyroscope information are especially important for locally similar procedural steps.
- Shot scaling: 6-axis balanced accuracy rises sharply from K=1 to K=5, improves more gradually through K=20, and reaches its highest value in the full-split setting across all three datasets.On SAMoSA, performance increases from 21.8% at K=1 to 42.6% at K=5; on UTD-MHAD, it increases from 31.2% at K=1.
- Shot scaling: At K=20, 6-axis performance reaches 55.4% on SAMoSA, 61.7% on UTD-MHAD, and 27.6% on PrISM latte, rising to 64.9%, 69.1%, and 36.5% full-split.PrISM latte shows smaller gains, increasing from 14.5% at K=1 to 27.6% at K=20 and 36.5% with the full training split.
- Ablations and interpretation: Longer 5.12 s windows benefit PrISM latte-making, while training from scratch falls below pretrained variants and fine-tuning offers no advantage over freezing the encoder.These findings indicate that temporal context and the learned representation, rather than architecture alone, drive transfer performance.
- Full-split comparison: 64.9% vs. 56.8% on SAMoSA, 36.5% vs. 31.0% on PrISM latte, and 69.1% vs. 64.1% on UTD-MHAD show full-split 6-axis TransfHAR exceeding supervised baselines.The reported TransfHAR values are balanced accuracy with standard deviations of 0.7, 1.1, and 2.3, respectively.
- Cross-procedure transfer: The 6-axis encoder exceeds the fully supervised PrISM 9-axis baseline by 6.2 points on average across all eight procedures, with largest gains of +9.7 pp on MakeStencil and +9.2 pp on MakeCoffee.The 3-axis encoder is close to parity at 0.9 points and falls below the baseline on three procedures.
- Cross-procedure transfer: Performance varies by procedure, reaching 49.9% on Cooking and 48.1% on Skin Care but only 29.6% on MakeStencil and 35.3% on MakeSandwich.The lower results are associated with procedural steps that are locally similar within short windows and depend more on temporal context.
6 On-Demand Personalization Study
The on-demand personalization study evaluated TransfHAR for user-defined wrist activities using a frozen encoder and participant-specific linear probes trained from one-minute recordings or a few labeled windows. Across seven activities per participant, the 6-axis encoder achieved 90.4% balanced accuracy with full-session supervision, while accuracy declined gradually as pooled vocabulary size increased and errors concentrated in a small number of activity pairs.
- Study design: 10 participants each selected or defined 7 fine-grained wrist activities from a bank spanning gestures, object manipulation, workspace actions, and personal motions.Participants wore Apple Watches on their dominant wrists and recorded three separate 1-minute sessions per activity, with 30-second breaks between sessions.
- Evaluation: The study used 2.56-second windows with 50% overlap and evaluated each participant separately under cross-session round-robin testing.Each participant’s probe trained on one session and tested on the two held-out sessions; K-shot conditions sampled K ∈ {1, 5, 10, 20} windows per class.
- Few-shot adaptation: 90.4% balanced accuracy was achieved by the 6-axis encoder when the probe used one 1-minute recording per class.The same setting produced 89.9% macro-F1; the 3-axis encoder reached 87.3% balanced accuracy and 86.6% macro-F1.
- Vocabulary scaling: Balanced accuracy declined gradually as the pooled non-personalized vocabulary grew from 10 to 70 classes, with most decline occurring between 10 and 30 classes.The 6-axis model varied by less than 1 point from 30 to 60 classes, suggesting users can expand label vocabularies without retraining the encoder.
- Error structure: Most participant-activity pairs were recognized near ceiling, while low-performing classes clustered mainly within P7 and errors concentrated among a few locally similar activities.For P7, Phone Scroll, Lotion, and Scratch Nails formed the main confusion subset, with Lotion misclassified as Scratch Nails 25% of the time.
7 Discussion
TransfHAR’s broad wrist-motion pretraining transfers to fine-grained activities absent from pretraining, enabling users to define personalized activity sets with few examples. Its effectiveness is bounded by overlapping wrist motions and limited modeling of longer procedural sequences.
- Transferable motion structure: Broad wrist motion anchors fine-grained recognition, allowing representations trained on coarse activities to transfer to unseen manipulative, gestural, and procedural classes.Downstream labels refine distinctions latent in diverse global-motion representations rather than requiring entirely new motion structure.
- On-demand personalization: Users can introduce or revise activity classes at deployment time from only a few examples instead of relying on a fixed activity vocabulary.This supports wearable applications whose activity sets evolve with user-specific routines, assembly workflows, or personalized procedures.
- Boundary conditions: On-demand recognition works best for kinematically distinct L3 activities with different burst structures or rotational dynamics.Pour-over steps separate well when each involves a distinct tool motion, but additional examples do not resolve genuinely overlapping wrist motions.
- Boundary conditions: Practical systems should warn users when a new class lies too close to an existing embedding and offer to merge the classes.This representation-driven feedback addresses failures caused by overlapping wrist motion.
- Limitations and future work: The window-based formulation cannot model longer temporal structure, limiting performance on procedural L4 tasks whose steps are locally similar but sequentially distinct.A proposed remedy is a lightweight temporal aggregation layer over window embeddings, such as a recurrent or state-based model.
8 Conclusion
TransfHAR is an on-demand wrist activity-recognition framework that uses self-supervised pretraining to learn broad motion priors and adapt them through a lightweight linear probe. Its frozen representation transfers across manipulative, gestural, and procedural activities in three held-out offline benchmarks without retraining a full model.
- TransfHAR is a framework for on-demand wrist activity recognition built on self-supervised pretraining.
- Broad motion priors from heterogeneous public wrist IMU data enable rapid personalization through a lightweight linear probe.Personalization to new fine-grained activities does not require retraining a full model.
- Across three held-out offline benchmarks, the frozen representation transfers strongly to manipulative, gestural, and procedural activities.
A Window Length Ablation
Under the full-split protocol, the 2.56 s window used throughout the paper performs best on both L3 benchmarks, whereas the L4 procedural task benefits from a longer 5.12 s window.
- A Window Length Ablation: The 2.56 s window used throughout the paper is best on both L3 benchmarks.The comparison is conducted under the full-split protocol.
- A Window Length Ablation: The L4 procedural task benefits from the longer 5.12 s window.This contrasts with the 2.56 s window’s advantage on both L3 benchmarks.
- A Window Length Ablation: Table 4 evaluates three window lengths using a frozen 6-axis encoder with a linear probe under the full-split protocol.Balanced accuracy is reported as the mean (SD) over five seeds.
B Confusion Matrices and Error Maps
This section presents visual error analyses for the personalization study and held-out benchmarks. The figures show per-class accuracy, confusion structure, and per-participant matrices.
- Figure 13: Figure 13 maps per-class accuracy for every participant-activity pair in the personalization study.The 6-axis full-session setting highlights classes below 70%.
- Figure 14: Figure 14 shows per-class confusion structure on the three held-out benchmarks.
- Figure 15: Figure 15 provides per-participant confusion matrices.
C Activity Bank and Participant Details
The study presented participants with a 55-activity bank while allowing them to define personalized activity sets. Across participants, the 70 selected activities included bank items, participant-specific variants, and entirely new activities.
- Activity Bank: The activity bank contained 55 candidate activities illustrating the target granularity of on-demand wrist activity recognition.Participants could select activities from the bank or define their own.
- Participant Details: Of the 70 selected activities, 43 came directly from the bank, 12 were participant-specific variants, and 15 had no bank equivalent.Participant-specific activity sets are documented in Table 6 using semantic labels corresponding to Act 1–Act 7.