Source-linked AI summary
Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognition
Jianyuan Ni, Raunak Sarbajna, Yang Liu, Anne H. H. Ngu, Yan Yan
TL;DR
Multimodal HAR can improve recognition, but wearable devices have limited computational resources and sensor-only approaches may be less reliable. This paper proposes video-to-sensor distillation using GAF-encoded accelerometer data and DASK; the authors report effectiveness across three datasets and higher accuracy than a student-only model in a reported comparison.
Problem
Multimodal HAR combines complementary modalities, but wearable devices have limited computational resources and sensor-based approaches can be less reliable than video-based approaches.
Source: §1 · p. 1Method
VSKD distills knowledge from a video teacher to an accelerometer-based student, using GAF encoding and DASK to address the modality gap.
Source: §1 · pp. 1–2, §3.1 · p. 2, §3.2 · p. 3Results
VSKD is reported effective and competitive on UTD-MHAD, MMAct, and Berkeley-MHAD; on Table 1, it achieves 2.1% higher accuracy performance than the student model alone.
Source: §5 · p. 4, §4.3 · p. 4Takeaways & Limitations
The proposed model is intended to improve sensor-based HAR while reducing computational resource demand during testing.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Human activity recognition (HAR) based on multi-modal approach has been recently shown to improve the accuracy performance of HAR. However, restricted computational resources associated with wearable devices, i.e., smartwatch, failed to directly support such advanced methods. To tackle this issue, this study introduces an end-to-end Vision-to-Sensor Knowledge Distillation (VSKD) framework. In this VSKD framework, only time-series data, i.e., accelerometer data, is needed from wearable devices during the testing phase. Therefore, this framework will not only reduce the computational demands on edge devices, but also produce a learning model that closely matches the performance of the computational expensive multi-modal approach. In order to retain the local temporal relationship and facilitate visual deep learning models, we first convert time-series data to two-dimensional images by applying the Gramian Angular Field ( GAF) based encoding method. We adopted ResNet18 and multi-scale TRN with BN-Inception as teacher and student network in this study, respectively. A novel loss function, named Distance and Angle-wised Semantic Knowledge loss (DASK), is proposed to mitigate the modality variations between the vision and the sensor domain. Extensive experimental results on UTD-MHAD, MMAct, and Berkeley-MHAD datasets demonstrate the effectiveness and competitiveness of the proposed VSKD model which can deployed on wearable sensors.
1. INTRODUCTION
Multimodal HAR can draw on complementary video and sensor information, but wearable devices have limited resources and sensor-only approaches can be less reliable. VSKD transfers knowledge from video to an accelerometer-based student, using GAF encoding and DASK to address the modality gap.
1. INTRODUCTION
- Wearable devices’ limited computational resources constrain deployment of multimodal HAR, while sensor-based HAR can be less reliable than video-based methods.The introduction attributes the sensor difficulty to reliance on single-context information.Source: §1 · p. 1
- VSKD uses video as the teacher modality and accelerometer data as the student input, with GAF encoding to represent the time series as images while preserving temporal information.The proposed DASK loss is used in the distillation process.Source: §1 · p. 1
- The paper presents video-to-sensor knowledge distillation as a new direction, with DASK proposed to alleviate the modality gap between teacher and student.The authors describe this as the first study to conduct KD from video to sensor.Source: §1 · p. 2
- The authors report effectiveness and robustness for VSKD on three public datasets.The datasets are UTD-MHAD, MMAct, and Berkeley-MHAD.Source: §1 · p. 2
2. RELATED WORK
Prior HAR research uses video, wearable time-series data, or their combination, while resource limits constrain multimodal deployment on wearable devices. The related work identifies the use of time-series data as the student modality in multimodal KD as an unaddressed direction.
2. RELATED WORK
- Video offers rich RGB information but is affected by viewpoint and illumination, while wearable sensor approaches face noisy data or sensor variation and limited context.The section describes multimodal methods as combining advantages of different modalities, but low-cost wearables cannot directly support their computational complexity.Source: §2 · p. 2
- Earlier HAR distillation transferred sensor information to video or used inertial sensors as side information, rather than using time-series data as the student modality.The paper identifies this reversed teacher-student direction as a gap in prior multimodal KD work.Source: §2 · p. 2
3. METHODOLOGY
The method encodes tri-axial accelerometer signals as images, then trains a sensor student using knowledge distillation augmented with distance, angle, and semantic losses.
3.1. Virtual Image Generation
- Tri-axial accelerometer signals are assembled as n × n × 3 images after encoding normalized signal values as polar coordinates with time as the radius.The transformation supports calculating correlation coefficients between vectors through trigonometric sums.Source: §3.1 · p. 2
3.2. DASK Loss
- The student knowledge-distillation loss combines cross-entropy on student predictions with KL divergence between teacher and student class probabilities.The temperature controlling the probability distribution is set to 4.Source: §3.2 · p. 3
- Distance-wise and angle-wise losses transfer structural relations by penalizing distance differences and minimizing angles between teacher and student examples.The distance-wise function uses a normalization factor and Huber loss.Source: §3.2 · p. 3
- The semantic loss compares teacher and student features from the layer before the final fully connected layer.The method uses shared semantic content across the modalities to define this loss.Source: §3.2 · p. 3
- The final student objective augments the original distillation loss with distance, angle, and semantic losses, balanced by tunable hyperparameters.The hyperparameters α, β, and γ balance the loss terms.Source: §3.2 · p. 3
4. EXPERIMENTS
The experiments evaluate video-to-sensor distillation across three multimodal HAR datasets, using video as the teacher modality and accelerometer data as the student modality. VSKD outperforms the reported comparable sensor-based methods, while ablations favor DASK and its semantic component.
4.1. Dataset
- The experiments use MMAct, UTD-MHAD, and Berkeley-MHAD, with video streams as teacher inputs and accelerometer data as student inputs.Source: §4.1 · p. 3
4.2. Experimental Settings
- The teacher is multi-scale TRN with BN-Inception; the student baseline is ResNet18, and performance is evaluated by accuracy and F-measure.Teacher segment counts are eight for Berkeley-MHAD and UTD-MHAD, and three for MMAct.Source: §4.2 · pp. 3–4
4.3. Experimental Results
- On MMAct, Table 3 reports F-measure performance and parenthetical increases over the student baseline.Source: Table 3 · p. 4
- Across UTD-MHAD, Berkeley-MHAD, and MMAct, VSKD outperforms the reported comparable methods, including methods using accelerometer and gyroscope data at test time.Reported accuracy improvements range from 0.37% to 6.77%; on Berkeley-MHAD, improvements range from 0.55% to 1.46%.Source: §4.3 · p. 4
4.4. Ablation Study
- On UTD-MHAD, DASK outperforms previous comparable KD losses; semantic loss contributes 0.70% accuracy improvement versus 0.22% for angle-wise loss.The authors also report that semantic loss contributes more accuracy improvement than structural-relation information.Source: §4.4 · p. 4, Table 4 · p. 4
- The proposed VSKD model, evaluated with ResNet18 as its student baseline, achieved 96.97% accuracy, versus 95.34% for the proposed VSKD model evaluated with VGG16 as its student baseline.Source: §4.4 · p. 4
5. CONCLUSION
VSKD improves sensor-based HAR performance while reducing test-time computational demand. DASK emphasizes structural relations and semantic information to bridge vision–sensor modality differences, and experiments on three public datasets demonstrate the model's effectiveness and competitiveness.
- VSKD improves sensor-based HAR performance and reduces computational-resource demand during testing.Source: §5 · p. 4