Source-linked AI summary
WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition
Marius Bock, Hilde Kuehne, Kristof Van Laerhoven, Michael Moeller
TL;DR
Datasets combining egocentric video and inertial sensor data for HAR remain scarce, limiting multimodal evaluation. WEAR provides egocentric video and 3D acceleration data from 22 participants performing 18 sports activities and evaluates inertial-based, vision-based, and combined HAR scenarios. The modalities exhibit complementary prediction strengths and weaknesses, while single-stage TAL models process raw inertial data and fuse both modalities using simple concatenation.
Problem
Datasets combining egocentric video and inertial sensor data for HAR remain scarce, limiting multimodal evaluation.
Method
WEAR provides egocentric video and 3D acceleration data from 22 participants performing 18 sports activities and evaluates inertial-based, vision-based, and combined HAR scenarios.
Results
The modalities exhibit complementary prediction strengths and weaknesses, while single-stage TAL models process raw inertial data and fuse both modalities using simple concatenation.
Takeaways & Limitations
WEAR provides a common challenging benchmark for assessing combined inertial- and vision-based HAR approaches.
Takeaways & Limitations
The participant selection is biased toward young, healthy people, and future extensions are needed for broader ages, physical impairments, locations, and seasons.
Abstract
from arXiv · showhide
Research has shown the complementarity of camera- and inertial-based data for modeling human activities, yet datasets with both egocentric video and inertial-based sensor data remain scarce. In this paper, we introduce WEAR, an outdoor sports dataset for both vision- and inertial-based human activity recognition (HAR). Data from 22 participants performing a total of 18 different workout activities was collected with synchronized inertial (acceleration) and camera (egocentric video) data recorded at 11 different outside locations. WEAR provides a challenging prediction scenario in changing outdoor environments using a sensor placement, in line with recent trends in real-world applications. Benchmark results show that through our sensor placement, each modality interestingly offers complementary strengths and weaknesses in their prediction performance. Further, in light of the recent success of single-stage Temporal Action Localization (TAL) models, we demonstrate their versatility of not only being trained using visual data, but also using raw inertial data and being capable to fuse both modalities by means of simple concatenation. The dataset and code to reproduce experiments is publicly available via: mariusbock.github.io/wear/.
1 INTRODUCTION & MOTIVATION
WEAR addresses the scarcity of datasets combining egocentric video and inertial data by introducing an outdoor sports HAR benchmark designed for complementary multimodal evaluation. Its benchmarks show that single-stage TAL models can process raw inertial data and fuse modalities effectively.
- The dataset uses application-driven sensor placement and changing outdoor locations to provide complementary information beyond the captured video stream.WEAR is collected across different outdoor locations with varying visual and surface conditions.
- WEAR is intended to assess methods that combine the strengths of inertial- and vision-based approaches in challenging outdoor recordings.
- Single-stage TAL models process raw inertial data and fuse modalities, significantly outperforming the best single-modality and oracle late-fusion approaches in mAP.
2 RELATED WORK
The related work situates WEAR among multimodal HAR datasets, inertial HAR architectures, visual TAL models, and fusion strategies. Existing datasets and methods motivate a benchmark centered on outdoor activity recognition and direct multimodal fusion.
- 2.1 Inertial- and Video-based HAR datasets: WEAR is compared with multimodal datasets by size, participants, activities, environment, sensor placement, and trimmed versus untrimmed recording format.The comparison includes egocentric vision datasets that also provide inertial data.
- 2.1 Inertial- and Video-based HAR datasets: Compared with CMU-MMAC, ADL, and ActionSense, WEAR is larger and uniquely emphasizes changing outdoor environments rather than non-changing indoor kitchen settings.
- 2.2 Inertial-based HAR: Deep learning has become standard for inertial HAR, with DeepConvLSTM combining convolutional feature extraction and recurrent temporal modeling.
- 2.3 Vision-based HAR: Visual HAR includes recognition, localization, and anticipation, while TAL predicts activity labels together with start and end times in untrimmed video.
- 2.3 Vision-based HAR: ActionFormer and TriDet are single-stage TAL architectures, and WEAR applies them to multimodal fusion by concatenating raw inertial data with vision-based feature embeddings.
3 METHODOLOGY
WEAR uses an ethically approved, reproducible outdoor recording protocol with synchronized egocentric video and accelerometer streams. The dataset covers 22 participants across 11 locations, and synchronization is validated through audio-like inertial visualizations.
- The study protocol was reviewed and approved by the University of Siegen, and participants received information about procedures, risks, data use, anonymization, and withdrawal rights.
- The reproducible recording plan requires only the specified sensors, a yoga mat, and a chair or similar item, allowing sessions at privacy-preserving outdoor locations.
- Data were collected from 22 participants at 11 locations under varying weather conditions, with more than 15 hours from the first 18 participants plus around 4 additional hours from six test participants.
- WEAR provides participant-wise raw and processed 3D acceleration and egocentric video, using four wrist- and ankle-worn Bangle.js accelerometers sampled nominally at 50 Hz.
- Synchronization was checked by converting accelerometer axes and their combined magnitude into WAV files, importing them with video into Final Cut Pro, and visually validating alignment during annotation.
4 BENCHMARKS AND BASELINE RESULTS
The benchmark compares inertial, camera, and multimodal activity recognition using LOSO validation and repeated runs, showing complementary modality strengths and strong performance from TAL models on inertial data.
- 4.4 Test Set Results: The evaluation used LOSO cross-validation with three random seeds, and the additional test set included unseen participants, a new location, different weather, and a new camera sensor.The test-set trends were similar to LOSO, with increased average mAP and classification metrics for both modalities and their combination.
- 4.2 Single-modality Experiments: Vision models struggled with occluded or out-of-view activities, whereas inertial models were reliable for limb-orientation tasks and vision better separated activities from the NULL class.These differences motivate combining the modalities rather than treating one as universally superior.
- 4.2 Single-modality Experiments: TAL models performed best with 1-second clips, while inertial architectures achieved higher classification results with 2-second windows.The comparison used multiple clip lengths and retained the same general benchmark framework across modalities.
- 4.2 Single-modality Experiments: Plain ActionFormer and TriDet produced competitive inertial classification with less activity confusion than classic inertial architectures, although their errors largely remained within activity categories.Inertial baselines showed more confusion between activities and the NULL class, partly because they classify sliding windows individually.
- 4.3 Multimodal Experiments: Early fusion with TriDet and ActionFormer achieved the highest average mAP and near-best F1-scores across experiments, while oracle late fusion remained significantly stronger.The multimodal representation concatenates vectorized inertial windows with corresponding two-stream I3D video embeddings.
5 ABLATION EXPERIMENTS
The ablations examine sensor subsets, repeated participant sessions, and predictive-stream behavior to probe robustness and the value of multimodal learning.
- 5.1 Sensor Placement: Using only wrist-worn inertial data caused a clear overall decrease across F1-score and mAP compared with using all sensors.The comparison evaluated right-wrist data and right-wrist-plus-right-ankle data against the full sensor setup.
- 5.2 Second execution of workout sessions: The predictive streams of early fusion resembled oracle late fusion, while oracle combinations still exposed additional room for improvement beyond learned fusion.The visualization compares inertial, camera, fusion, and oracle streams using 1-second windows with 50% overlap.
- 5.2 Second execution of workout sessions: Repeated-session results improved for sbj_0 but not sbj_14, with changes remaining within the expected 15%–20% participant-level standard deviation.The two recordings also differed in temperature and sunlight, so the small two-participant analysis does not isolate familiarity effects.
6 LIMITATIONS
WEAR provides a multimodal outdoor HAR benchmark, but its current participant population and recording coverage limit representativeness and leave clear directions for extension.
- 6 LIMITATIONS: The test set broadens conditions with unseen participants, an unseen location, different weather, and a new camera sensor, while preserving the LOSO trends.This provides a limited robustness check rather than comprehensive coverage of environments and populations.
- 6 LIMITATIONS: Using only wrist-worn sensors reduced performance across evaluation metrics compared with the full sensor configuration, underscoring the scope of the reported sensor placement.The ablation compared right-wrist and right-wrist-plus-ankle subsets with all sensors.
- 6 LIMITATIONS: WEAR is biased toward young, healthy participants, limiting the demographic scope of conclusions drawn from the dataset.Proposed extensions include older or younger participants and people with known physical impairments.
- 6 LIMITATIONS: Future recordings could add locations outside Germany, different seasons, higher-end wearables, additional sensing modalities, and more repeated sessions.Suggested additions include gyroscopes, magnetometers, PPG, and earables.
7 DISCUSSION & CONCLUSION
WEAR establishes a challenging multimodal HAR benchmark spanning inertial and egocentric-video data, with outdoor conditions and limited information overlap between modalities. Its benchmarks show complementary modality strengths and demonstrate that single-stage TAL models can process inertial data and fuse both modalities through simple concatenation.
- 7 DISCUSSION & CONCLUSION: WEAR provides a common benchmark for inertial- and vision-based HAR in changing outdoor environments with limited information overlap between modalities.The dataset is intended to assess methods that combine the strengths of both approaches.
- 7 DISCUSSION & CONCLUSION: Separate-modality benchmarks reveal complementary strengths and weaknesses in inertial and vision prediction performance.
- 7 DISCUSSION & CONCLUSION: Single-stage TAL models can be trained on raw inertial data and can fuse inertial and vision features through simple concatenation.The paper applies vision-derived TAL architectures without pretraining to raw inertial data and combined modalities.