PaperlayerView exact revision on arXiv

Source-linked AI summary

COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition

Baiyu Chen, Wilson Wongso, Zechen Li, Yonchanok Khaokaew, Hao Xue, Flora Salim

arXiv:2503.07259v2cs.CVcs.AIcs.LGcs.MM

TL;DR

Wearable HAR must balance video’s semantic richness against IMUs’ efficient, privacy-preserving deployment, while IMU learning is constrained by scarce annotations. COMODO uses label-free self-supervised distillation from a frozen video encoder and dynamic queue to transfer semantic structure to IMUs, matching or surpassing supervised models with strong cross-dataset generalization.

  • Problem

    Video-based HAR is semantically strong but costly, privacy-sensitive, and lighting-dependent, while IMUs are efficient yet constrained by scarce annotated data and weaker generalization.

  • Method

    COMODO distills video knowledge into IMU representations by aligning cross-modal similarity distributions with a frozen video encoder and dynamic instance queue.

  • Results

    COMODO matches or surpasses fully supervised time-series models across three egocentric datasets and shows strong cross-dataset generalization.

  • Takeaways & Limitations

    COMODO supports efficient, privacy-preserving IMU deployment while transferring semantic structure from data-rich video without manual sensor-data annotation.

  • Takeaways & Limitations

    Training requires synchronized video–IMU pairs, and precise temporal alignment between heterogeneous modalities remains challenging.

Abstract

from arXiv · show

The goal of creating intelligent, human-centered wearable systems for continuous activity understanding faces a fundamental trade-off: Egocentric video-based models capture rich semantic information and have demonstrated strong performance in human activity recognition (HAR), but their high power consumption, privacy concerns, and dependence on lighting limit their feasibility for continuous on-device recognition. In contrast, inertial measurement unit (IMU) sensors offer an energy-efficient, privacy-preserving alternative, yet lack large-scale annotated datasets, leading to weaker generalization. To bridge this gap, we propose COMODO, a cross-modal self-supervised distillation framework that transfers semantic knowledge from video to IMU without requiring labels. COMODO leverages a pretrained and frozen video encoder to construct a dynamic instance queue to align the feature distributions of video and IMU embeddings. This enables the IMU encoder to inherit rich semantic structure from video while maintaining its efficiency for real-world applications. Experiments on multiple egocentric HAR datasets show that COMODO consistently improves downstream performance, matching or surpassing fully supervised models, and demonstrating strong cross-dataset generalization. Benefiting from its simplicity and flexibility, COMODO is compatible with diverse pretrained video and time-series models, offering the potential to leverage more powerful teacher and student foundation models in future ubiquitous computing research. The code is available at this repository: https://github.com/cruiseresearchgroup/COMODO.

1 Introduction

Egocentric video offers rich semantics but is costly and impractical for continuous wearable inference, while efficient IMUs face data scarcity and cross-modal alignment challenges. COMODO transfers structured video knowledge to IMUs through label-free self-supervised distillation, achieving strong performance and generalization while using IMUs at deployment.

  • Motivation: Video models provide strong semantic information but impose high power, privacy, and lighting costs, whereas IMUs are efficient and robust for always-on recognition.IMUs operate independently of lighting conditions and visual occlusions while consuming significantly less power.
  • Challenges: The framework addresses knowledge transfer, heterogeneous signal alignment, and generalization across encoder architectures, activity classes, and sensing devices.These challenges arise from the modality and performance gap between video and IMU representations.
  • Method: COMODO aligns video and IMU similarity distributions with a dynamic instance queue that supplies a stable, diverse reference and preserves teacher semantic structure.The queue also introduces a temporal prior that stabilizes cross-modal optimization.
  • Results: COMODO matches or surpasses fully supervised fine-tuned time-series models across three egocentric datasets spanning over 70 activity categories.The evaluation includes fine-grained datasets with more than 30 distinct daily activities.
  • Deployment: Videos are used during training, while lightweight IMUs support efficient, privacy-preserving inference in real-world wearable applications.This separates rich semantic supervision from deployment-time sensing requirements.
  • Contributions: COMODO combines label-free cross-modal transfer, a cross-modal FIFO queue, model-agnostic teacher–student configurations, and cross-dataset evaluation.The framework is designed to leverage pretrained video and time-series models.

2 Related Works

Prior work uses self-supervision, knowledge distillation, multimodal learning, and synthetic data to address limited labeled sensor data. COMODO differs by distilling structured video knowledge into IMU representations without labeled data or multimodal fusion at inference.

  • Self-supervised HAR: Self-supervised HAR research has used contrastive learning, teacher–student learning, and synchronized multi-device transformations to exploit unlabeled sensor data [29] [51].These approaches address the dependence of sensor-based HAR on large labeled datasets.
  • Knowledge distillation: Knowledge distillation commonly transfers representations from large teachers to smaller students, including video-to-IMU and multimodal supervision approaches [42] [46] [50] [52].Prior work includes contrastive representation distillation and transfer from RGB, optical flow, and audio.
  • Cross-modal distillation: COMODO performs self-supervised cross-modal distillation from video to IMU without labeled data, unlike prior approaches requiring video at inference or target-dataset fine-tuning [42] [50].It explicitly transfers knowledge across heterogeneous modalities rather than focusing on single-modal representation learning.
  • Synthetic and multimodal learning: Other multimodal studies synthesize IMU from video or language and align motion time-series with language models to reduce real-world sensor-data collection demands [30] [32] [34] [35].These works reflect broader efforts to address data scarcity and domain shifts through multimodal inputs.
  • Cross-modal distillation: Unlike instance-wise contrastive methods, COMODO transfers structured distributional knowledge from video embeddings to improve robust IMU-based HAR.The method avoids requiring large-scale labeled sensor datasets.

3 Methodology

COMODO distills semantic structure from a frozen video encoder into a trainable IMU encoder without labels. It uses a FIFO queue of video embeddings and cross-entropy alignment between video and IMU similarity distributions.

  • Cross-modal self-supervised distillation: COMODO trains a pretrained IMU encoder to inherit semantic information from a frozen pretrained video encoder without labeled data.The learned IMU representation is intended to capture video-derived semantics while retaining IMU-only inference.
  • Cross-modal queuing mechanism: A continuously updated FIFO queue stores video embeddings as a large, diverse reference set for cross-modal similarity comparisons.Current-batch embeddings are enqueued and the oldest batch is removed to maintain a fixed queue size.
  • Similarity distribution alignment: COMODO minimizes cross-entropy between video and IMU similarity distributions computed over the shared video-embedding queue.The distributions use inner products and temperature scaling before alignment.
  • Similarity distribution alignment: Because the video teacher is frozen, its distribution entropy is constant, so minimizing cross-entropy is equivalent to minimizing KL(P_v∥P_x).This distills the structure of the teacher’s similarity space rather than enforcing only instance-wise alignment.
  • Relation to InfoNCE: Soft distribution targets preserve semantically close teacher neighbors that hard InfoNCE treats as uniformly dissimilar negatives.The paper motivates this choice for heterogeneous video and IMU modalities and reports empirical comparison against InfoNCE and L2 alignment.
  • Inference: After distillation, only the IMU encoder is used for inference, while the video encoder, projectors, and queue are discarded.Representation quality is evaluated with an RBF-kernel SVM, using labels only for downstream classifier training.

4 Experiments

Experiments use three egocentric HAR datasets with different activity granularities, sensor configurations, and sampling rates. The setup includes dataset-specific preprocessing, standardized splits, and analysis of MMEA’s variable-length sequences.

  • Datasets: The evaluation spans Ego4D, EgoExo4D, and MMEA, which together provide distinct sensor configurations and sampling rates.Dataset statistics are summarized in Table 2.
  • Dataset activity labels: The datasets cover 31 filtered Ego4D activities, 8 coarse EgoExo4D categories, and 32 fine-grained MMEA daily activities.MMEA includes activities such as drinking, cooking, writing, walking, and watch-TV.
  • Preprocessing: Ego4D and EgoExo4D remove samples without IMU recordings, filter classes with fewer than 10 samples, and use 7:2:1 train-test-validation splits.Their IMU signals are standardized to 200 Hz.
  • Preprocessing: MMEA uses raw accelerometer and gyroscope readings sampled at 25 Hz and follows its official train-validation-test splits.The preprocessing normalizes accelerometer readings by R_acc = 16384 and gyroscope readings by R_gyro = 16.4.
  • Preprocessing: MMEA’s sequence lengths vary substantially, with an average of 418.57 samples and a median of 422.This variability is used to assess adaptability and robustness to variable-length inputs.

4.2 Experimental Settings

The experiments assess IMU representations on downstream activity classification across three egocentric datasets. COMODO is compared with supervised and self-supervised baselines using lightweight time-series students and pretrained video teachers.

  • Results: The reported comparison finds COMODO comparable or superior to fully supervised models and consistently better than previous state-of-the-art self-supervised methods.Best and second-best results are marked in Table 3; virtual-only and virtual-plus-real IMU settings are distinguished.
  • Training configuration: COMODO is trained for 20 epochs with batch size 128 and learning rate 3e-4, using τ_v = 0.1 and τ_x = 0.05.These settings define the cross-modal self-supervised distillation stage.
  • Video teacher: COMODO uses pretrained TimeSformer-base as the default frozen video teacher, with VideoMAE-Base-k400 also evaluated as a teacher.Video embeddings are projected to 128 dimensions and precomputed for efficient training.
  • IMU student: The IMU student models are Mantis, with 8.1M parameters, and MOMENT-small, with 37.9M parameters.Mantis is the default student choice.

4.3 IMU-based Human Activity Recognition

COMODO is evaluated for IMU-based activity classification across three egocentric datasets, comparing self-supervised and supervised baselines. It achieves state-of-the-art performance across all three datasets and ranks first on most metrics.

  • 4.3 IMU-based Human Activity Recognition: COMODO achieves state-of-the-art performance across all three datasets, ranking first on most metrics.The evaluation covers three egocentric datasets and compares supervised and self-supervised approaches.
  • 4.3 IMU-based Human Activity Recognition: On Ego4D, COMODO exceeds IMU2CLIP by 4.3% in A@1 and surpasses Mantis and Attend and Discriminate by 0.77% and 1.01%.
  • 4.3 IMU-based Human Activity Recognition: On EgoExo4D, COMODO leads in A@1 and A@3, outperforming IMU2CLIP by 5.46% in A@1 and Mantis by 0.7% in A@3.It achieves second-best A@5, narrowly behind Attend and Discriminate.
  • 4.3 IMU-based Human Activity Recognition: On MMEA, COMODO exceeds IMU2CLIP by 1.07% in A@1, trails supervised Mantis by only 0.53%, and achieves the best A@3 and A@5 scores.
  • 4.3 IMU-based Human Activity Recognition: COMODO consistently excels in A@3 and A@5 across datasets, indicating richer semantic representations for a more accurate prediction distribution.

4.4 Cross-Dataset Generalization

The cross-dataset evaluation measures transfer to unseen environments with differing activity classes and devices. COMODO improves over IMU2CLIP, CrossHAR, and supervised fine-tuned Mantis in all tested transfer settings.

  • 4.4 Cross-Dataset Generalization: COMODO improves over IMU2CLIP, CrossHAR, and supervised fine-tuned Mantis in every cross-dataset transfer case.The tested directions include EgoExo4D → Ego4D, Ego4D → EgoExo4D, and transfers to MMEA.
  • 4.4 Cross-Dataset Generalization: In EgoExo4D → Ego4D, COMODO surpasses IMU2CLIP by 4.87% in A@1, Mantis by 4.04%, and CrossHAR by 26.07%.
  • 4.4 Cross-Dataset Generalization: When transferring from Ego4D to EgoExo4D, COMODO achieves the highest A@1 of 82.54%, improving over Mantis, IMU2CLIP, and CrossHAR by 1.75%, 2.84%, and 14.32%.
  • 4.4 Cross-Dataset Generalization: On MMEA, COMODO surpasses Mantis by 2.5% and 2.58% when trained on Ego4D and EgoExo4D, respectively.

4.5 Efficiency Analysis

COMODO is assessed for inference and training efficiency in resource-constrained settings. It offers competitive accuracy with substantially lower inference cost than video teachers and uses offline teacher features for scalable training.

  • 4.5 Efficiency Analysis: COMODO reaches 78.8% accuracy with 1.70G FLOPs and 75.7ms latency, versus TimeSformer’s 89.8% with 196.05G FLOPs and 3590.3ms latency.This corresponds to a 47× speedup and 115× FLOPs reduction, with an 8.1M-parameter model versus 121.4M parameters.
  • 4.5 Efficiency Analysis: COMODO provides up to +18.2% mean A@1 over DeepConvLSTM and Attend and Discriminate while maintaining millisecond-level inference latency.
  • 4.5 Efficiency Analysis: With the same architecture and hardware footprint as IMU2CLIP, COMODO achieves +3.6% mean A@1 without additional inference overhead.
  • 4.5 Efficiency Analysis: COMODO caches video embeddings offline and reuses them during IMU training, decoupling training cost from the video teacher’s complexity.The frozen video teacher is not invoked during IMU encoder training.

4.6 Ablation Study

The ablation study examines teacher–student model combinations, teacher performance, and queue size. COMODO consistently outperforms IMU2CLIP across configurations, while its best queue size depends on the dataset.

  • 4.6.1 Different Teacher & Student Networks.: COMODO consistently and substantially outperforms IMU2CLIP across every teacher–student configuration on all three datasets.The combinations use Mantis or MOMENT-small students with VideoMAE-Base-k400 or TimeSformer-Base-k400 teachers.
  • 4.6.1 Different Teacher & Student Networks.: Mantis is consistently a stronger student than MOMENT-small, while neither VideoMAE-Base-k400 nor TimeSformer-Base-k400 is universally superior.The stronger Mantis student paired with either teacher yields the best and second-best results overall.
  • 4.6.2 Video Teacher Performance: Video teachers achieve near-perfect accuracy on EgoExo4D and MMEA, whereas lower scores on Ego4D indicate greater visual-task complexity.COMODO closes a significant portion of the modality gap, especially on Ego4D.
  • 4.6.3 Queue Size: The optimal queue size is dataset-specific, and increasing queue size does not always improve performance.

92.8 MMEA

COMODO’s ablations show that performance depends on dataset-appropriate queue sizing, similarity-distribution distillation, concat pooling, and FIFO sampling that stabilizes optimization.

  • Queue Size: The best queue size is dataset-dependent: 65,536 is not optimal for Ego4D, sizes above 1,024 reduce EgoExo4D accuracy, and 2,048 performs best on MMEA.Queue size must balance negative diversity with dataset scale and class concentration.
  • Pooling Ablation: Concat pooling outperforms mean pooling across datasets, improving A@1 by 1.82% on Ego4D, 0.98% on EgoExo4D, and 6.84% on MMEA.The results suggest that retaining channel-wise information avoids the information loss caused by averaging sensor channels.
  • Distillation Methods: COMODO consistently outperforms L2 and InfoNCE across datasets by distilling the teacher’s similarity distribution rather than relying only on instance-wise alignment.This preserves structure among semantically similar negatives and produces more robust, generalizable representations.
  • Sampling Methods: FIFO sampling yields superior performance across all datasets, motivating analysis of its effect on optimization dynamics.The comparison uses identical numbers of distillation samples for FIFO and random sampling.
  • Sampling Methods: FIFO maintains higher gradient continuity than random sampling across Ego4D and EgoExo4D for every tested moving-average window K.Larger K values smooth the moving-average curves, while FIFO’s advantage persists across window sizes.

5 Discussion and Implications

COMODO is presented as a scalable way to transfer semantic knowledge from pretrained video models to efficient IMU-based wearable HAR without manual IMU annotation. Its practical value lies in separating video-based training from privacy- and power-constrained IMU deployment, while remaining compatible with evolving model architectures.

  • 5.1 Implications for Egocentric HAR: A Path Beyond Manual Annotation: COMODO offers a scalable response to the annotation bottleneck by using naturally paired video–IMU data instead of manually labeling extensive IMU recordings.This is especially relevant for nuanced, context-specific activities that are difficult to cover comprehensively with labeled IMU data.
  • 5.1 Implications for Egocentric HAR: A Path Beyond Manual Annotation: COMODO transfers semantic structure from large-scale pretrained vision models to IMU models without requiring a single IMU label.Its cross-dataset results are presented as evidence that the transferred knowledge captures transferable activity semantics rather than only dataset-specific correlations.
  • 5.2 Implications for Ubiquitous and Wearable Systems: Designing for Practicality: COMODO decouples training-time video requirements from deployment-time sensing by learning offline from video and operating online with IMU.This addresses the power and privacy barriers associated with continuous wearable video while retaining access to video-derived semantics.
  • 5.2 Implications for Ubiquitous and Wearable Systems: Designing for Practicality: The model-agnostic framework allows newer video or time-series foundation models to replace current teachers or students without redesigning the entire system.This flexibility is demonstrated through the framework’s compatibility with different architectures.
  • 5.3 Limitation and Future Work: COMODO still requires synchronized video–IMU pairs, remains sensitive to temporal misalignment and teacher capacity, and may need stronger robustness to severe asynchronicity.The authors identify mutual distillation, personalization, and contextual adaptation as future directions.

6 Conclusion

COMODO distills knowledge from a frozen video encoder to a lightweight IMU encoder through a dynamic instance queue, avoiding costly manual sensor annotation. Experiments show supervised-level performance, strong cross-dataset generalization, and a flexible design for future wearable systems.

  • COMODO distills knowledge from a powerful, frozen video encoder to a lightweight IMU encoder through a dynamic instance queue.The framework targets the trade-off between video semantics and IMU deployability.
  • The approach avoids costly manual annotation of sensor data while learning IMU representations from video knowledge.
  • COMODO matches or surpasses supervised baselines and learns representations with strong cross-dataset generalization.The paper identifies cross-dataset generalization as important for real-world utility.
  • COMODO’s model-agnostic and flexible design provides a scalable path for wearable systems to leverage progress in visual foundation models.

A Baseline Implementation Detail

The evaluation uses carefully configured supervised and self-supervised baselines, including foundation models and cross-modal methods, with adjustments for fair architectural and sequence comparisons. COMODO substantially outperforms the baseline models across the tested configurations.

  • The supervised baseline suite includes time-series, HAR-specific, foundation, IMUGPT 2.0, and CrossHAR models.
  • Baseline comparisons use model-specific sequence lengths or fixed 100-length segmentation to accommodate differences in sequence handling.DLinear and TimesNet use length 931, Informer uses 512, and HAR-specific models use non-overlapping length-100 segments.
  • COMODO substantially outperforms all baseline models regardless of their configuration in the MMEA sequence-length comparison.The authors conclude that baseline-setting choices do not affect the overall superiority of COMODO.
  • Self-supervised baselines include MOMENT-small, Mantis, CrossHAR, and IMU2CLIP, evaluated through a standard two-stage representation-learning protocol.
  • The IMU2CLIP reimplementation preserves contrastive InfoNCE training while replacing teacher and student components to match the evaluation architecture.
  • A linear projection aligns mismatched IMU and video feature dimensions in the reimplementation.
Loading 2503.07259v2…