PaperlayerView exact revision on arXiv

Source-linked AI summary

IMU-Bench: Benchmarking Wrist IMU Representations Across Kinematic, Semantic and Clinical Tasks

Yiquan Li, Aidan Bradshaw, Chenfeng Gao, Karan Ahuja

arXiv:2610.19481v1cs.ROcs.AI

TL;DR

Wearable benchmarks leave open whether wrist-IMU tasks share representations and how supervision transfers among them. IMU-Bench compares six self-supervised methods across ten tasks from eight datasets, finding broad frozen-probe and fine-tuning performance alongside trade-offs between task supervision.

  • Problem

    Existing wearable benchmarks evaluate multiple tasks but leave unresolved how supervision for one task affects transfer to others and whether the tasks benefit from a shared representation.

    Source: §1 · pp. 1–2

  • Method

    IMU-Bench evaluates ten tasks from eight datasets, comparing six self-supervised methods pretrained on unlabeled recorded and simulated wrist motion and testing transfer and joint training.

    Source: §1 · pp. 2–3

  • Results

    InrtlWrist achieves the best or joint-best frozen primary score on nine tasks, and fine-tuning outperforms training from scratch on nine tasks.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    Source-task supervision transfers between kinematic tasks, but joint-training gains can cost performance on full-body pose and language retrieval, distinguishing broad downstream usefulness from mutually beneficial supervision.

    Source: §8 · p. 11

    The scaling study does not isolate motion coverage from data volume and source composition, and the benefit of larger corpora at longer training budgets remains unresolved.

    Source: §8 · p. 11

Abstract

from arXiv · show

Wrist-worn inertial measurement units (IMUs) provide a common sensing modality for tasks ranging from activity and gesture recognition to pose estimation, inertial odometry and clinical motor assessment. Existing wearable benchmarks evaluate performance across multiple tasks, but leave open how supervision for one task affects transfer to others and whether these tasks benefit from a shared representation. We present IMU-Bench, a benchmark of ten tasks from eight datasets for studying IMU multi-task performance and cross-task transfer. We compare six self-supervised methods pretrained on unlabeled recorded and simulated wrist motion, evaluating classification, pose and velocity estimation, and IMU-language retrieval. We find that our masked autoencoder, InrtlWrist, achieves the best or joint-best frozen-probe performance on nine of ten tasks, and fine-tuning improves on training from scratch on nine of ten tasks, including full-body pose, odometry and retrieval. Across transfer and joint-training experiments, supervision that benefits one target can reduce performance on others, with supervised sources transferring within kinematic estimation while, under frozen probing, none approaches the pretrained encoder on language retrieval, and joint-training gains that come at a cost to the paired task. These findings distinguish broad downstream usefulness from complementary task supervision. Code, task cards and a leaderboard are publicly available.

1. Introduction

Wrist IMUs offer a shared signal for studying activity, clinical assessment, interaction, pose, and odometry, but it is unclear whether these tasks share a representation or how their supervision transfers. IMU-Bench evaluates ten tasks across eight datasets and finds broad downstream usefulness from InrtlWrist alongside trade-offs between task supervision.

1. Introduction

  • The benchmark asks whether tasks reveal distinct aspects of wrist motion through a shared representation and whether supervision for one target helps or compromises others.Earlier wearable studies establish transfer across classification tasks and sensing conditions, while support for joint rotations, metric velocity, and motion–language correspondence is less understood.Source: §1 · pp. 1–2
  • IMU-Bench compares six self-supervised methods pretrained on unlabeled recorded and simulated wrist motion across classification, pose and velocity estimation, and IMU–language retrieval.Evaluation includes frozen probing, fine-tuning, supervised source-task transfer, and selected joint-training pairs.Source: §1 · p. 2, Figure 1 · p. 2
  • InrtlWrist achieves the best or joint-best frozen-probe performance on nine of ten tasks, while fine-tuning improves over training from scratch on nine, including full-body pose and odometry.The benchmark also examines transfer and joint training, finding that gains for one target can come at a cost to another.Source: §1 · p. 2
  • The paper analyzes cross-task transfer and complementarity, distinguishing beneficial transfer among kinematic tasks from trade-offs in joint training and the gap to pretrained features on frozen motion–language retrieval.Additional analyses examine pretraining volume and motion coverage.Source: §1 · pp. 2–3

2. Related Work

Prior wearable-IMU benchmarks and self-supervised studies largely emphasize activity and health classification, while pose and odometry are often studied separately. IMU-Bench adds cross-task supervision analysis and matched comparisons of representation methods.

2. Related Work

  • Wearable benchmarks mostly assess activity recognition, with some extending to gesture, gait, sports, or physiological signals; sparse-IMU pose and inertial odometry are generally studied separately.Task-specific systems also cover clinical classification and IMU–language alignment.Source: §2 · p. 3
  • Wearable self-supervised methods use varied objectives and individually chosen backbones, data, and protocols, motivating matched-capacity comparison on a shared corpus.InrtlWrist alternates frame-wise and global attention to retain channel coupling and patch-level resolution relevant to pose and odometry.Source: §2 · p. 3
  • Task relationships can be studied through transfer, joint task selection, or direct representation comparison; this work uses transfer and controls probe comparisons with fixed light decoders and untrained-encoder and raw-input baselines.Its data analysis varies motion coverage at matched volume.Source: §2 · p. 3

3. Datasets and Tasks

IMU-Bench pretrains on recorded and simulated wrist motion while excluding evaluation datasets, then evaluates ten tasks spanning activity, clinical, kinematic, gesture, and language targets.

3.1. Pretraining Data

  • Pretraining uses 197,094 unlabeled 10.24-second windows with 50% overlap, sampled from about 280 hours of recorded and simulated wrist motion.The five sources include an activity-recognition pool, OCDetect, EatingSpeed, and simulated motion from Motion-X++ and BONES-SEED; evaluation datasets are excluded.Source: §3.1 · pp. 3–4

3.2. Tasks

  • The task suite evaluates arm and full-body pose, wrist-frame velocity-derived odometry, microgesture recognition, clinical classification, and bidirectional IMU–language retrieval.Nymeria supports pose and odometry; OpenWatch supports microgesture, PADS supports clinical classification, and simulated HumanML3D clips are paired with CLIP text embeddings.Source: §3.2 · p. 4
  • Four activity tasks classify 2.56-second windows by macro-F1; HHAR, MHEALTH, and PRISM use subject-disjoint splits, whereas TransfHAR uses chronological splits because most activities were performed by only one of its ten participants.The other task-specific measures include pose errors, odometry ATE, microgesture macro-F1, participant macro-F1, and mean bidirectional Recall@1 for retrieval.Source: §3.2 · p. 4

4. Methods

InrtlWrist encodes wrist accelerometer and gyroscope windows with alternating channel-wise and global attention, then learns by masked reconstruction. Evaluation compares frozen pretrained features, fine-tuning, and training from scratch under controlled task protocols.

4.1. Architecture

  • A 12-layer transformer represents 0.32-second patches from six accelerometer and gyroscope channels, alternating attention across channels at each patch with global attention across all channels and positions.A learnable summary token is included in global attention.Source: §4.1 · p. 5

4.2. Training

  • InrtlWrist is pretrained by encoding visible patches and reconstructing masked measurements, after which the reconstruction decoder is discarded.The comparison includes five self-supervised baselines and a plain-attention InrtlWrist variant, using a shared corpus and matched encoder capacity.Source: §4.2 · p. 5

4.3. Evaluation Protocols

  • Frozen probing fixes the pretrained encoder while fine-tuning and training from scratch optimize encoder and decoder, initialized from pretrained and random weights respectively.Within each task, comparisons share decoder architecture, optimizer, training budget, split, and model-selection rule; classification uses linear heads and kinematic tasks use specified attention decoders.Source: §4.3 · p. 5

5. Results

InrtlWrist performs strongly across the benchmark, while results vary by task and evaluation protocol. Fine-tuning improves kinematic and retrieval performance, but not every task benefits.

5. Results

  • Nine of ten tasks are best or joint-best for InrtlWrist under frozen probing, and fine-tuning beats scratch on nine of ten; gesture is the exception (54.2 vs 60.8).InrtlWrist is best or joint-best on eight tasks after fine-tuning.Source: Figure 2 · p. 6, §5 · p. 6

5.1. Activity, Gesture, and Clinical Recognition

  • InrtlWrist leads all four activity datasets under frozen probing, with 57.8 mean macro-F1 versus 56.2 for LIMU-BERT and 53.3 for scratch; fine-tuning raises the mean to 63.9.For gesture, frozen InrtlWrist reaches 64.7 clip macro-F1, while fine-tuning lowers it to 54.2; SimMTM leads matched methods at 61.9.Source: §5.1 · p. 6
  • On PADS, frozen InrtlWrist reaches 55.2 participant macro-F1, below SSL-Wearables at 57.4; fine-tuning lowers macro-F1 to 51.9 but raises balanced accuracy from 60.0 to 63.9.All scores use participant-level evaluation.Source: §5.1 · pp. 6–7

5.2. Kinematic Estimation

  • Fine-tuning reduces full-body MPJPE from 97.9 to 88.1 mm and odometry ATE from 0.200 to 0.154 m, surpassing scratch on both kinematic tasks.For arm pose, fine-tuning reduces position error from 8.14 to 8.01 cm, while angular error changes from 9.89° to 9.86°.Source: §5.2 · pp. 7–8

5.3. Semantic Alignment

  • InrtlWrist has the highest mean IMU–language Recall@1 among the six matched methods under both protocols, rising from 27.9 frozen to 38.8 after fine-tuning.In the full gallery, fine-tuned scores are 5.1 IMU-to-text and 4.4 text-to-IMU Recall@1, versus 2.1 in both directions for scratch.Source: §5.3 · p. 8

6. Cross-Task Transferability

The transfer experiments test whether supervision on one task benefits other targets, using frozen and fine-tuned source encoders. Transfer can help within kinematic estimation, but gains do not consistently extend across task families.

6. Cross-Task Transferability

  • The protocol trains each source encoder from scratch on one of seven tasks, then evaluates it on the other six using a new decoder and benchmark target splits and budgets.Frozen transfer is compared with a random frozen encoder; fine-tuned transfer is compared with target training from scratch.Source: §6 · p. 8
  • Frozen supervised sources outperform a random encoder in 28 of 42 source–target pairs, but 35 of 42 transfers remain below target training from scratch.Five of six kinematic transfers remain positive on disjoint participant thirds, at roughly half their original magnitude; MAE leads on full-body pose and odometry.Source: §6 · pp. 9–10, Figure 3 · p. 9
  • With target fine-tuning, 22 source–target pairs improve over scratch and 20 decline; every source improves IMU–text and PADS, while every source reduces gesture F1.No source improves odometry, and only odometry supervision improves arm pose.Source: Figure 3 · p. 9, §6 · p. 10

7. Architecture and Training Ablations

The ablations examine decoder capacity, pretraining-data amount and motion coverage, and joint task supervision. Their results show that measured pretraining gains and task-level outcomes depend on these choices.

7. Architecture and Training Ablations

  • Adding temporal decoder layers reduces the pretrained encoder’s body-MPJPE advantage below 1 mm, and a 12-layer odometry decoder lets a random encoder outperform every pretrained encoder.Increasing decoder width alone changes the gap little.Source: §7 · p. 10
  • At 50,000 updates, 0.01% pretraining subsets trail 0.1% subsets at every coverage level, while uniform draws from 0.1% to 100% perform alike.Increasing motion coverage generally improves performance, particularly when few clusters are available; these findings are conditional on the training budget and sampling scheme.Source: §7 · p. 10
  • Joint training improves odometry ATE from 0.154 to 0.149 m and gesture F1 from 54.2 to 65.0, but increases body MPJPE and reduces frozen IMU–text Recall@1.The partner-task gains accompany losses on the jointly trained pose task and language retrieval.Source: §7 · pp. 10–11

8. Conclusion

IMU-Bench evaluates ten wrist-IMU tasks and presents InrtlWrist as a shared reference encoder. The conclusion distinguishes broad downstream performance from complementary task supervision.

8. Conclusion

  • InrtlWrist achieves the best or joint-best frozen primary score on nine tasks and outperforms training from scratch on nine after fine-tuning.Supervised source-task transfer occurs between kinematic tasks, but no frozen supervised source approaches the pretrained encoder on retrieval; joint training gains come at a cost to full-body pose and language retrieval.Source: §8 · p. 11

Reproducibility statement

  • A registry-driven harness fixes each task’s inputs, decoder, split, metric and budget, with encoder cards documenting each model.The project page provides code, task cards and a leaderboard; per-cell result files and pretrained weights are slated for release there.Source: Reproducibility statement · p. 11, A. Implementation Details · p. 18

Statement on the use of AI

  • The authors used Claude through Claude Code to assist with coding, data cleaning, experimental analysis and paper preparation, while retaining responsibility for the final work.They state it was not used to create synthetic datasets, translate text or conduct qualitative analysis; simulated wrist IMU was computed from motion capture using a physical model.Source: Statement on the use of AI · p. 11

A. Implementation Details

  • All tasks use six-channel wrist inputs z-scored per channel; pretraining additionally applies randomly drawn proper 90°-multiple rotations to both sensors.Downstream augmentation is absent except for IMU–text alignment, which applies random 3-D rotations to every encoder.Source: A. Implementation Details · p. 19
  • Self-supervised encoders share a 12-layer, 384-wide ViT, corpus and optimizer recipe, while differing in objective and some attention configurations.The random frozen control uses InrtlWrist’s architecture with random initialization; the raw-input control removes the transformer layers and preserves input patches for the decoder.Source: A. Implementation Details · p. 19

B. Tasks and Metrics

This section defines task inputs, prediction targets, splits and scoring across wrist-IMU datasets. It also states where evaluation protocols constrain the scope of comparisons.

B. Tasks and Metrics

  • Evaluation uses subject-disjoint splits for several datasets, session-disjoint splits for Nymeria, the dataset’s own split for retrieval and a time split for TransfHAR.Published task-specific systems use differing inputs and recipes, so they are reported for orientation rather than ranked against the controlled encoders.Source: B. Tasks and Metrics · pp. 19, 22
  • Nymeria pose tasks predict five right-arm joints or 23 SMPL body joints, with position error or root-relative MPJPE scoring, respectively.The session split evaluates new recordings of people also represented in training; a participant-disjoint split would remove 31% of training sessions.Source: B. Tasks and Metrics · p. 19
  • Nymeria odometry predicts pelvis velocity in the wrist frame and scores reconstructed trajectories by absolute trajectory error.Scoring uses the true initial orientation, making this easier than full inertial navigation; results are not comparable to world-frame odometry systems.Source: B. Tasks and Metrics · p. 19, Table 7 · p. 22
  • OpenWatch microgesture recognition uses five gestures plus background, with clip labels derived from window predictions and a canonical-identity subject split.The test split is reported; Table 8 compares the encoder with scratch training and task-specific systems.Source: B. Tasks and Metrics · pp. 19–20, Table 8 · p. 23
  • HumanML3D retrieval aligns simulated right-wrist IMU with CLIP text space and averages Recall@1 across both directions over 200 galleries of 100 test clips.The evaluation covers 3,701 eligible clip ids from the official test split.Source: B. Tasks and Metrics · p. 20, Table 9 · p. 23
  • PADS classifies participants across three diagnostic classes using averaged window posteriors from separate wrist-step sessions and reports subject-level macro-F1.This metric is coarse: an untrained frozen control scores 51.7, above six of nine pretrained encoders, so close models are not ranked by it.Source: B. Tasks and Metrics · pp. 20–21

B.1. TransfHAR smartwatch dataset

  • HHAR, MHEALTH and PRISM use subject-disjoint activity splits, while TransfHAR splits recordings in time because most activities were performed by one participant.TransfHAR therefore measures recognition later in the same recording, not generalization to new people.Source: B. Tasks and Metrics · p. 21, §B.1 · pp. 23–24

C. Data Audit

The data audit documents exposure risks and dataset overlap, while decoder-capacity tests show that probe design can obscure differences between frozen representations.

C. Data Audit

  • The reference pretraining corpus excludes commonly used evaluation datasets, and the audit cannot rule out participant overlap between PRISM and SAMoSA.The authors report no Nymeria source overlap and describe the reference corpus as separate from benchmark evaluation sets.Source: C. Data Audit · pp. 24, 26
  • OpenWatch’s 50 participant ids correspond to 43 identities, including 35 mirrored copies; splitting by release ids can place the same person in training and test.The original split includes a copy of training participant 46 as test id 48, but the score shift between splits also reflects different test-set composition.Source: C. Data Audit · p. 25, Table 12 · p. 25
  • HumanML3D’s official split has 437 of 2,192 test clips sharing source-recording frames with training or validation clips, making retrieval partly transductive.The audit also notes a gyroscope-unit mismatch in EatingSpeed, addressed by unit conversion on load.Source: C. Data Audit · p. 26

D. Controls and Decoder Capacity

  • Deeper pose decoders narrow the gap between InrtlWrist and an untrained encoder to 0.3 mm, and a 12-layer odometry decoder lets the untrained encoder beat every pretrained one.The authors therefore retain light decoders for kinematic probes; raw-input control scores characterize the chosen readout rather than raw-IMU learning in general.Source: D. Controls and Decoder Capacity · p. 26

E. Cross-Task Transfer on Disjoint Participants

On participant-disjoint Nymeria thirds, most frozen kinematic transfers persist at reduced gains, but the comparison with InrtlWrist changes. The thirds control participant exposure, though their smaller size leaves data volume confounded.

E. Cross-Task Transfer on Disjoint Participants

F. Joint Task Supervision

Joint supervision can improve partner-task performance while reducing full-body pose and language scores; the embedding and pretraining analyses characterize where activity structure, corpus coverage, and motion diversity matter. These results are bounded by the tested architecture, data pool, and training budget.

F. Joint Task Supervision

  • Full-body MPJPE rises from 88.1 mm alone to 91.5 mm with odometry and 94.4 mm with microgesture, while frozen IMU–text Recall@1 falls from 27.9 to 21.0 and 17.2.Partner tasks gain; gesture F1 also exceeds training from scratch. Each target selects its own checkpoint, so scores need not share one checkpoint.Source: F. Joint Task Supervision · p. 27, Table 14 · p. 28

G. Embedding Space of the Reference Encoder

  • 38.8% of windows’ ten nearest neighbors share an activity label and 65.6% share a semantic family, yet neither labeling clusters globally.Across datasets, locomotion, cycling, and stairs travel at 8.7 to 18 times chance, while household, hand, desk, posture, gesture, and gym activities travel at 1.7 to 4.5 times chance.Source: G. Embedding Space of the Reference… · p. 28, Figure 6 · p. 28
  • Mocap-derived windows have 0.96 of their ten nearest neighbors in their own dataset, versus 0.59 for recorded windows; the exposure contrast is only +0.027 and changes sign under leave-one-source-out.The authors caution that synthetic inertial data remains separated even when included in pretraining.Source: G. Embedding Space of the Reference… · p. 29
  • Dataset identity remains the strongest neighborhood signal (0.632 against chance 0.058), so shared embedding regions do not establish dataset-invariant representations.The authors state that task measurements, not the embedding visualization, establish downstream value.Source: G. Embedding Space of the Reference… · pp. 28–29

H. Pretraining Volume and Motion Coverage

  • A uniform 0.1% pool draw (1,959 windows) contains 82% of motion modes, and a 0.5% draw contains 99.7%; draws of at least 5% cover all modes.The reference corpus covers every mode at 10% of the pool.Source: H. Pretraining Volume and Motion… · p. 29
  • With all modes available, downstream probes show no consistent gains across the tested data volumes, while restricting the 1,959-window sample to one mode collapses classification probes.At one mode, gesture F1 is 30.3 against 59.5 with all modes; from ten modes onward, kinematic probes are within 5% of the 1,000-mode model in raw error. Mode count also shifts source composition, so it does not isolate mode number.Source: H. Pretraining Volume and Motion… · p. 29, Figure 7 · p. 30
  • The volume and coverage findings apply to one architecture, one data pool, and 50,000 steps; the experiments do not separate diversity from volume.Whether more data pays at larger budgets remains open, and the motion modes are data-driven clusters rather than a movement taxonomy.Source: H. Pretraining Volume and Motion… · p. 29
  • An earlier eleven-source sweep lowered reconstruction loss by only 0.4% (0.7604 to 0.7572), but its downstream results were excluded because the corpus contains evaluation participants.Source: H. Pretraining Volume and Motion… · p. 30

Related papers

Loading 2610.19481v1…