IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition

• Labeled sensor-based HAR datasets remain scarce because collection is expensive and annotation can be time-consuming or impossible, limiting dataset scale and activity coverage.The cited examples include Daphnet with 5 hours from 10 subjects and PAMAP2 with 7.5 hours from 9 subjects.Source: §1 · p. 2
• Using large video repositories offers a potential source of labeled activity data, but making them usable requires activity curation, sensing-domain preprocessing, and handling scene and camera variation.The text also identifies the need for features and recognition models suited to video-sourced motion information.Source: §1 · p. 2
• Earlier work synthesized sensor data from motion capture and explored video-derived motion for sensor-based HAR; IMUTube targets arbitrary activity videos rather than specific scenes or viewpoints.The video-to-HAR precedent is cited as.Source: §1 · p. 2
• IMUTube estimates 3D human motion from activity videos, translates it into virtual IMUs at selected body positions, matches their distribution to the target domain, and trains recognizers.The generated data can be combined with small amounts of real sensor data; the pipeline integrates off-the-shelf vision and graphics techniques.Source: §1 · pp. 2–3
• Across three benchmarks, virtual-only models sometimes reached accuracy comparable to real-data models, while adding small portions of real data could outperform real-only training.The experiments covered RealWorld, PAMAP2, and Opportunity.Source: §1 · p. 3
• The authors frame the work as a first step toward broader recognition scenarios and propose that it could support more complex activities and sensor configurations.They describe a research agenda toward next-generation sensor-based HAR.Source: §1 · p. 3

2 EXTRACTING VIRTUAL IMU DATA FROM VIDEOS

• IMUTube replaces conventional sensor-data collection with a pipeline intended to make large video repositories usable for HAR training.The authors envision potentially much larger training datasets than standard practice.Source: §2 · p. 3

2.1 IMUTube Overview

• The overview contrasts laboratory recording and annotation with repository videos converted into virtual IMU data for HAR training.The conventional procedure is labor-intensive and often error-prone, while video search results currently require curation.Source: §2.1 · pp. 3–4, Figure 1 · p. 4

2.2 Motion Estimation for 3D Joints

• The pipeline estimates 2D poses, tracks people across frames, filters unreliable sequences, and lifts the remaining poses to 3D.It assumes people in a scene perform the same activity; sequences shorter than one second and poses missing over half their joints are removed.Source: §2.2 · pp. 4–5

2.3 Global Body Tracking in 3D

• Virtual sensors need both local joint motion and global 3D body movement, so the pipeline tracks pose location and orientation throughout the scene.Global acceleration is extracted alongside local joint acceleration.Source: §2.3 · p. 5

2.3.1 3D Pose Calibration.

• For each frame, PnP calibrates 3D pose rotation and translation from corresponding 2D and 3D keypoints, using estimated camera intrinsics.The intrinsics are averaged across frames under the assumption that a clip was recorded with one camera.Source: §2.3.1 · pp. 5–6

2.3.2 Estimation of Camera Ego-motion.

• The pipeline estimates camera ego-motion from successive point clouds and applies it to calibrated poses to track people in the full scene.It uses background point clouds for ICP because foreground people can confuse estimates of camera and human movement.Source: §2.3.2 · pp. 6–8, Figure 3 · p. 7

2.4 Generating Virtual Sensor Data

• From tracked 3D motion, the pipeline derives virtual sensor streams by tracking joint orientation and transforming acceleration into local sensor coordinates.It also uses IMUSim to model realistic sensor behavior, including device mechanics and electronics, to address differences from real IMU data.Source: §2.4 · p. 8

2.5 Distribution Mapping for Virtual Sensor Data

• Before HAR training, rank-based distribution mapping transfers virtual sensor data toward the target real-sensor distribution.The authors report that a few seconds to minutes of real sensor data can suffice for calibration.Source: §2.5 · p. 8

3 TRAINING ACTIVITY RECOGNITION CLASSIFIERS WITH VIRTUAL IMU DATA

• Experiments compare real-only, virtual-only, and mixed training for models tested on real IMUs, primarily using Random Forests with leave-one-subject-out evaluation.Source: §3 · p. 9

3.1 Feasibility Experiment under Controlled Conditions

• The controlled RealWorld experiment uses video and IMU recordings from 15 subjects performing eight activities, extracting 12 hours of virtual data from 20 hours of real IMU data.Virtual IMUs are generated at seven body locations; the resulting datasets contain 86k virtual and 221k real one-second windows.Source: §3.1 · p. 9, Data · p. 9
• RealWorld provides both video and real IMU data, enabling a controlled comparison of virtual- and real-trained classifiers despite potential sources of noise.Source: §3.1 · p. 9

Data.

• Random Forest evaluations use ECDF features and leave-one-subject-out folds, with 13 training subjects, one validation subject, and one test subject per fold.Source: Method · p. 9

Method.

• DeepConvLSTM is evaluated with a single-subject holdout, trained on raw data with Adam, early stopping, and a grid search over learning rate and weight decay.Source: Method · p. 9
• Both model evaluations report the highest test F1-score achieved using any training-data amount, with 95% Wilson score intervals.Source: Method · p. 10
• Mix2R surpasses R2R by 12% for Random Forests, while V2R reaches 98% of R2R F1-score on RealWorld's eight classes.The reported V2R F1-score is 0.57, within 2% of R2R.Source: Results · p. 10, Table 1 · p. 10

Results.

• Mix2R reaches an F1-score of 0.64, a 12% relative performance gain over R2R.Source: Results · p. 10
• For DeepConvLSTM, Mix2R outperforms R2R by 6.6%, while V2R achieves 75% of R2R; the authors suggest raw-input feature learning may contribute to the gap.Source: Results · p. 10
• The results indicate that virtual IMU data alone can train capable classifiers, although the authors note that the data-quality assessment remains to be examined.Source: Results · p. 10

3.2 Performance on Common Activity Recognition Datasets

• The motivation is to use existing activity videos rather than require synchronized video and real-IMU collection for the target actions.Source: §3.2 · p. 10
• Across Opportunity and PAMAP2, Random Forest V2R achieves 95% of R2R F1-scores on average, while Mix2R exceeds R2R by 5% on average.Source: Table 2 · p. 11
• The transfer setting reuses RealWorld videos to train models tested on Opportunity and PAMAP2, whose activity labels roughly overlap with RealWorld.Source: §3.2 · p. 11
• The comparison uses four overlapping classes in Opportunity and eight locomotion classes in PAMAP2, with one-second windows at 50% overlap.Source: Data · p. 11

Data.

• PAMAP2 virtual data uses four possible arm-and-chest location combinations, yielding 152k virtual windows versus 24k real windows.Opportunity virtual data was re-extracted at eleven body positions, yielding 46k virtual windows versus 40k real windows.Source: Data · p. 11

Results.

• Random Forest trained on virtual IMU alone achieves 94% and 96% of R2R performance on Opportunity and PAMAP2, respectively.The evaluations transfer from RealWorld videos to settings with different collection conditions, including laboratory activities.Source: Results · p. 11
• Mixing real and virtual IMU data exceeds R2R F1-scores by 5% on average, with the authors suggesting added motion variety may improve generalization.The virtual data comes from a related but different scenario.Source: Results · p. 12
• For DeepConvLSTM, V2R recovers 84% of R2R F1-scores, while Mix2R and R2R scores are statistically comparable.The accompanying result is identified as Table 2b.Source: Results · p. 12

3.3 Virtual IMU Data for Complex Activity Recognition

• The dataset includes about 10 hours of virtual data from 7,255 videos covering vacuum cleaning, ironing, rope jumping, and cycling.Videos came from activity datasets spanning experiment scenarios and in-the-wild sources, including YouTube.Source: Data · p. 12
• Adding seven RealWorld videos expands the task to 11 PAMAP2 classes, producing 390k virtual and 38k real IMU windows.Four possible sensor-location combinations yielded four times as much virtual data per video; windows were one second with 50% overlap.Source: Data · p. 12
• The four video-sourced complex activities were selected from annotated segments in computer-vision activity datasets.The sources include ActivityNet, Kinetics700, HMDB51, MPIIHPD, UCF101, Charades, AVA, MSRdailyactivity3D, and NTU RGB-D.Source: Data · p. 12

Data.

• 0.58 F1-score under V2R is 80% of the 0.72 R2R F1-score for complex PAMAP2 activities.The authors cite differences in data sources and label interpretation, possible virtual-data quality limits, and greater domain shift as likely factors in the weaker result.Source: Results · pp. 12–13

Results.

• 23% gain yields a 0.71 F1-score in Mix2R versus 0.58 in V2R, although Mix2R remains 2% below R2R.The passage says the DeepConvLSTM evaluation aligns with the Random Forest results.Source: Results · p. 13
• The experiments support the feasibility of training capable classifiers from virtual IMU alone under in-the-wild and complex-activity conditions, with mixing also a potential source of gains.The reported experiments span locomotion and more complex activities.Source: Results · p. 13

4 UNDERSTANDING VIRTUAL IMU DATA

• V2R performance ranges from 80% to 90% of R2R performance and matches or exceeds R2R only when trained alongside real IMU data.On PAMAP2 11-class, V2R does not outperform R2R even with a larger virtual dataset extracted from multiple video sources.Source: §4 · p. 13

4.1 Comparing Virtual and Real IMU

• Virtual signals capture large x-axis movement changes and z-axis acceleration spikes, but differ most from real signals along the y-axis.The authors suggest the y-axis discrepancy reflects the difficulty of reconstructing motion perpendicular to the visual plane from 2D video.Source: Analysis · p. 14

4.2 Coping with Domain Shift

• Without mapping, cross-dataset training incurs 66% and 64% F1-score drops for real and virtual data, respectively; mapping resolves these drops.The comparison uses the four-class Opportunity task, including training data from PAMAP2, Realworld, or Realworld videos.Source: Table 4 · p. 15, Analysis · pp. 15–16
• With 10 minutes of real IMU per class for mapping, FID scores are within 6% of the scores obtained using all real data.All four evaluated cases show an abrupt FID drop with under 100 seconds of real samples.Source: Figure 5 · p. 16, Analysis · p. 16

4.3 Varying the Mixture and Size of Training Data

• Mix2R outperforms R2R on 3 of 4 datasets across inspected training amounts, with at least 9% higher F1-score on Realworld.The greatest Mix2R–R2R difference occurs with under 100 seconds of training data per class.Source: Analysis · pp. 16–17, Figure 6 · p. 17
• Adding virtual data gives comparable or better F1-scores at all tested ratios on Realworld, Opportunity, and PAMAP2 8-class, but hurts PAMAP2 11-class.Realworld gains most at a 1:5 real-to-virtual ratio; Opportunity and PAMAP2 8-class each improve by 6% at 1:1, and performance does not rise monotonically with more virtual data.Source: Analysis · p. 17, Table 5 · p. 18

5 TRANSFER LEARNING WITH VIRTUAL IMU DATA FOR HAR CLASSIFIERS

• The explored transfer-learning settings address incomplete or absent virtual-data labels, but the work does not yet incorporate automated video labeling or search.The authors identify ambiguous video descriptions as a potential curation challenge.Source: §5 · p. 18

5.1 Supervised Transfer Learning

• Supervised transfer pre-trains on labeled virtual IMU and fine-tunes on labeled real IMU, even when the activity labels differ.For the label-mismatch case, the final layer is replaced to match the target number of classes, and all model weights are updated.Source: §5.1 · p. 18, Method · p. 18
• Supervised pre-training yields statistically significant F1-score gains of 14% on Realworld and 3% on Opportunity over training from scratch.With limited real data for fine-tuning, Realworld surpasses R2R; for PAMAP2 11-class, pre-training on virtual 8-class data yields an F1-score of 0.71, statistically comparable to R2R.Source: Table 6 · p. 19, Results · p. 19

5.2 Unsupervised Transfer Learning

• Unsupervised transfer using virtual IMU reaches 95%–100% of R2R F1-scores on Realworld, Opportunity, and PAMAP2 11-class, and exceeds R2R by 5% on PAMAP2 8-class.This setup uses unlabeled virtual data to pre-train a convolutional autoencoder, then trains a random forest on representations extracted from real IMU data.Source: Results · p. 20, Method · p. 20

6.1 Demonstrating Feasibility

• Virtual IMU alone recovered up to 90% of real-IMU results on Realworld, Opportunity, and PAMAP2 8-class; mixing virtual and real data improved recognition by 4%-12%.On PAMAP2 11-class, V2R reached at least 80% of R2R, but mixing decreased performance by 2%.Source: §6.1 · p. 20
• PAMAP2 11-class covers complex non-locomotion activities; virtual data still reached at least 80% of R2R, but mixing decreased results by 2%.The authors say modeling complex activities and the value of additional accelerometry warrant further investigation.Source: §6.1 · pp. 20–21

6.2.1 From Vision To Pose.

• Pose-estimation and scene-understanding errors can accumulate into distorted 3D motion, especially with occlusion, motion blur, or changing light in videos in the wild.The pipeline uses off-the-shelf techniques, supporting reproducibility but limiting labeled-data quality; more robust pose tracking and camera-motion methods are proposed.Source: §6.2.1 · p. 21, §6.2 · p. 21
• The virtual sensor assumes joint acceleration equals acceleration at the nearest body-worn device, overlooking body mass, device movement, and skin friction.Body-mesh modeling could improve realism; the pipeline could also be extended beyond accelerometry to gyroscope and magnetometer signals.Source: §6.2.2 · p. 21

6.2.4 Learning from Virtual Data.

• Virtual-to-real deployment faces domain shift; the authors propose input adaptation or learning domain-invariant features as possible responses.They also suggest adapting virtual data to reproduce real sensor noise such as drift, hysteresis, and calibration effects.Source: §6.2.3 · p. 21, §6.2.4 · p. 22

6.2.5 Which Videos are Currently Suitable for IMUTube?

• Large camera ego-motion, multiple moving objects, and occlusion may hurt recognition; videos with fewer motion artifacts and activities with distinctive motions tend to classify well.Similar-pose activities can be harder to distinguish, and objects may occlude arm movement in vacuum-cleaning and ironing videos.Source: §6.2.5 · p. 22

6.3.1 Large-scale Data Collection.

• IMUTube aims to make large-scale labeled-video datasets possible, but the authors have not yet produced a large-scale HAR dataset, leaving deep-learning benefits prospective.The validation videos were curated, and reducing or eliminating that selection effort remains a goal.Source: §6.3.1 · p. 22, §6.3.2 · p. 22

6.3.3 Extending the Field of HAR.

• Virtual sensors can be placed at many body locations to test which placement best recognizes an activity set; real IMU data can serve as a seed for more sophisticated models.The authors also point to generating virtual readings for loosely connected clothing and to focusing real-data collection on high-quality labels across varied subjects.Source: §6.3.3 · p. 23, §6.3.4 · p. 23

7.1 Sensor Data Collection in HAR

• Lab studies require participants, synchronized video and sensors, and costly manual annotation, making large-scale datasets difficult to collect.EMA can support larger-scale collection but may lose sample-precise labels; both approaches still involve participants wearing sensors.Source: §7.1 · pp. 23–24
• Simulation struggles to represent complex activities, while mocap datasets often omit everyday behavior; online-video methods offer access to real-world activities.The closest prior video approach used a fixed viewpoint and synced wearable recordings, unlike IMUTube’s stated scope and adaptation approach.Source: §7.1 · p. 24

7.2 Tackling the Sparse Data Problem

• Small, imbalanced HAR datasets encourage overfitting and constrain complex models and transfer learning; augmentation and generative methods have their own limitations.Perturbation-based augmentation does not add the movement diversity of many subjects, while GANs require substantial data and may be unstable or fail to converge.Source: §7.2 · pp. 24–25
• IMUTube addresses sparse labels by generating large quantities of virtual IMU data from videos containing many individuals and potentially diverse, realistic movements.The approach contrasts with methods that chiefly perturb or resample existing sensor data.Source: §7.2 · p. 25

8 CONCLUSION

• IMUTube is a validated framework for automatically extracting virtual IMU data from video to support large-scale labeled datasets for HAR.The conclusion characterizes its initial findings as promising at least for simple activities whose main IMU characteristics are confined to 2D.Source: §8 · p. 25
• The authors identify collaboration across computer vision, signal processing, and activity recognition as necessary to advance the work’s broader agenda.They describe opportunities to improve automation, condition virtual data, and apply modern learning techniques.Source: §8 · pp. 25–26