PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition

Yang Liu, Keze Wang, Guanbin Li, Liang Lin

arXiv:2009.00210v5cs.CV

TL;DR

Vision-based action recognition faces viewpoint, occlusion, and appearance challenges, while wearable sensors provide complementary motion information across a substantial modality gap. SAKDN transfers knowledge from multiple wearable-sensor teachers to an RGB-video student using semantic guidance and adaptive fusion. Experiments on three benchmark datasets validate the framework, and a reported comparison improves video action recognition from 94.87% to 98.60%.

  • Problem

    Vision and wearable action data differ in dimension, distribution, and information content, while existing approaches often fail to account for their complementary relationship and domain divergence.

    Source: §I · pp. 1–2

  • Method

    SAKDN adaptively distills knowledge from multiple wearable-sensor teacher networks into an RGB-video student, with semantic guidance for fusion and transfer.

    Source: §I · pp. 2–3

  • Results

    Experiments on three benchmark datasets validate SAKDN, and video action recognition improves from 94.87% (baseline) to 98.60%.

    Source: §I · p. 3, C. Ablation Study · p. 11

  • Takeaways & Limitations

    The framework combines complementary wearable-sensor knowledge to improve action recognition in the vision-sensor modality.

    Source: §V · p. 14

Abstract

from arXiv · show

Existing vision-based action recognition is susceptible to occlusion and appearance variations, while wearable sensors can alleviate these challenges by capturing human motion with one-dimensional time-series signal. For the same action, the knowledge learned from vision sensors and wearable sensors, may be related and complementary. However, there exists significantly large modality difference between action data captured by wearable-sensor and vision-sensor in data dimension, data distribution and inherent information content. In this paper, we propose a novel framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) by adaptively transferring and distilling the knowledge from multiple wearable sensors. The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality. To preserve local temporal relationship and facilitate employing visual deep learning model, we transform one-dimensional time-series signals of wearable sensors to two-dimensional images by designing a gramian angular field based virtual image generation model. Then, we build a novel Similarity-Preserving Adaptive Multi-modal Fusion Module to adaptively fuse intermediate representation knowledge from different teacher networks. Finally, to fully exploit and transfer the knowledge of multiple well-trained teacher networks to the student network, we propose a novel Graph-guided Semantically Discriminative Mapping loss, which utilizes graph-guided ablation analysis to produce a good visual explanation highlighting the important regions across modalities and concurrently preserving the interrelations of original data. Experimental results on Berkeley-MHAD, UTD-MHAD and MMAct datasets well demonstrate the effectiveness of our proposed SAKDN.

I. INTRODUCTION

Vision-based action recognition faces viewpoint, occlusion, appearance, and privacy constraints, while wearable signals offer complementary motion information. SAKDN transfers knowledge from multiple wearable-sensor teachers to an RGB-video student, using semantic guidance to address the modalities’ substantial differences.

Motivation

  • Vision methods are vulnerable to viewpoint changes, clutter, occlusion, and appearance variation, while wearable signals offer privacy protection and robustness to illumination and viewpoint changes.The paper identifies the lack of methods that account for complementary information and domain divergence between vision and wearable action data.Source: §I · p. 1

Challenges

  • Wearable data are one-dimensional signals, whereas vision data contain richer image or video information, creating challenges in representation, fusion, and knowledge transfer across modalities.The paper highlights the need to handle multiple wearable modalities as well as differences in data dimension, distribution, and information content.Source: §I · p. 2

Proposed framework

  • SAKDN distills complementary knowledge from multiple wearable-sensor teacher modalities into an RGB-video student, guided by shared action semantics.The framework targets sensor-to-vision heterogeneous action recognition.Source: §I · p. 2
  • GAF-based virtual images convert one-dimensional wearable signals into two-dimensional representations while preserving local temporal relationships and supporting visual models.The teacher modalities include acceleration, gyroscope, and orientation signals.Source: §I · p. 2

Proposed contributions

  • SPAMFM combines intra-modality similarity, semantic embeddings, and relational knowledge to adaptively recalibrate teacher features; GSDM transfers graph-guided semantic attention while preserving modality-specific data interrelations.The paper presents these modules as components for fusing teacher knowledge and guiding the student network.Source: §I · p. 3

II. RELATED WORK

Prior work covers unimodal recognition, cross-domain multimodal recognition, and teacher–student knowledge distillation. The paper positions sensor-to-vision recognition as heterogeneous transfer and introduces semantic guidance and relational fusion to address gaps in those approaches.

A. Uni-modal Action Recognition

  • Unimodal vision methods face viewpoint, clutter, occlusion, illumination, privacy, and computational constraints, while wearable approaches often use raw signals and simple fusion.The paper proposes GAF virtual images and SPAMFM to represent wearable signals and adaptively fuse complementary sensor knowledge.Source: A. Uni-modal Action Recognition · p. 3

B. Multi-modal Action Recognition

  • Cross-view, cross-spectral, and cross-media methods address other modality shifts, but sensor-to-vision recognition remains limited and involves heterogeneous data and ignored semantic relationships.Prior sensor-to-vision approaches often use raw one-dimensional signals, which lack local temporal relationships, color, and texture.Source: B. Multi-modal Action Recognition · pp. 3–4

C. Knowledge Distillation

  • Knowledge distillation transfers knowledge from trained teachers to students, while prior approaches include softened labels, attention, relational losses, and modality transfer.SAKDN's GSDM transfers semantics-aware attention from multiple teachers and uses graph-guided ablation analysis to highlight regions and preserve intrinsic structures.Source: C. Knowledge Distillation · p. 4

A. Framework Overview

SAKDN distills knowledge from acceleration, gyroscope, and orientation teacher networks into an RGB-video student through an end-to-end framework. Its GAF virtual images encode wearable time series as two-dimensional representations designed to preserve temporal relationships.

A. Framework Overview

  • SAKDN combines wearable-sensor virtual-image generation, training of multiple teacher networks, and distillation from those teachers into an RGB-video student.Teacher inputs are acceleration, gyroscope, and orientation signals; the student input is RGB video.Source: A. Framework Overview · p. 4
  • Teacher networks are trained with semantic-preserving loss, while student training also uses GSDM, soft-target distillation, and cross-entropy losses.Source: A. Framework Overview · pp. 4–5

B. Virtual Image Generation

  • GAF transforms normalized one-dimensional sensor signals into polar coordinates and represents correlations between time intervals with a matrix of cosine sums of their angles.The normalized signal is mapped to [−1, 1]; each matrix element is cos(θ_i + θ_j).Source: B. Virtual Image Generation · p. 5
  • The GAF encoding is described as bijective with a unique inverse, and polar coordinates preserve absolute temporal relations as correlations.Each tri-axial sensor signal becomes three n × n matrices assembled as a three-channel GAF virtual image for teacher-network input.Source: B. Virtual Image Generation · p. 5

C. Similarity-Preserving Adaptive Multi-modal Fusion

SPAMFM fuses intermediate features from multiple wearable-sensor teacher networks. It combines within-modality similarity with cross-modal context to adaptively recalibrate teacher features.

C. Similarity-Preserving Adaptive Multi-modal Fusion

  • SPAMFM extends attention-based fusion by integrating intra-modality similarity, semantic embeddings, and multiple relational knowledge to exploit complementary teacher features.Source: C. Similarity-Preserving Adaptive… · p. 5

1) Intra-modality Similarity Matrix Generation:

  • For each teacher layer, rowwise L2-normalized outer products of reshaped activations produce mini-batch similarity matrices that encode within-modality activation relationships.These similarities guide fusion of different relation functions in global context modeling.Source: 1) Intra-modality Similarity Matrix… · pp. 5–6

2) Global Context Modeling and Feature Recalibration:

  • Global context modeling combines similarity matrices with globally averaged feature vectors using concatenation, summation, and Hadamard-product relations.Source: 2) Global Context Modeling and Feature… · p. 6
  • The resulting context supports adaptive channel-wise recalibration across teacher modalities while preserving intra-modality similarities and complementary information.The excitation signals gate the input features through a channel-wise product.Source: 2) Global Context Modeling and Feature… · p. 6

D. Graph-guided Semantically Discriminative Mapping

GSDM transfers semantics-aware attention knowledge from teacher networks to a student at convolutional layers. It uses graph-guided ablation analysis to highlight class-relevant regions while preserving data relationships.

D. Graph-guided Semantically Discriminative Mapping

  • GSDM addresses heterogeneous sensor-to-vision distillation by transferring semantics-aware attention knowledge from multiple teachers at convolutional layers.The method is motivated by the complementary information in intermediate layers, which traditional last-layer distillation overlooks.Source: D. Graph-guided Semantically… · p. 6
  • GSDM uses ablation drops and semantic embeddings to form visual explanations, rather than relying on globally pooled gradients and class scores.The paper describes class scores as too strict for heterogeneous sensor-to-vision action recognition.Source: D. Graph-guided Semantically… · p. 7
  • The method constructs separate similarity graphs for original and ablated data, with embedded features as vertices and Gaussian similarities as edge weights.Normalized graph Laplacians represent the data manifold structure.Source: D. Graph-guided Semantically… · p. 7
  • Graph-guided slopes measure changes in transformed semantic embeddings after ablation, using the original and ablated graph matrices to preserve data structure.Source: D. Graph-guided Semantically… · p. 7
  • The distillation loss uses mean squared error between normalized teacher and student GSDMs across selected network layers.Source: D. Graph-guided Semantically… · p. 7

E. Semantics-aware Adaptive Knowledge Distillation

SAKDN integrates virtual-image generation, SPAMFM, and GSDM into an adaptive distillation framework. Wearable-sensor GAF images feed teacher modalities, while RGB video feeds the student.

E. Semantics-aware Adaptive Knowledge Distillation

1) Training of teacher networks:

The teacher networks process Gramian Angular Field images from wearable-sensor modalities, with SPAMFM inserted at selected VGG16 layers. A semantic-preserving loss aligns their fc2 features with action-class representations while the networks are trained.

1) Training of teacher networks:

  • For m wearable-sensor modalities, the method builds m VGG16-backbone teacher networks, feeds each modality’s GAF images to its corresponding network for training, and adds SPAMFM at selected VGG16 layers.Source: 1) Training of teacher networks · p. 8
  • At fc2, the teacher networks use an MSE semantic-preserving loss between raw features and their corresponding action-class semantic representations.For teacher k, the raw fc2 feature is H_k and its corresponding semantic representation is F_k.Source: 1) Training of teacher networks · p. 8
  • The teachers are trained simultaneously using the semantic-preserving loss and the summed cross-entropy loss across teacher networks.The passage identifies Y_k and Z_k as teacher k's predicted labels and class probabilities.Source: 1) Training of teacher networks · p. 8

2) Training of student network:

The student is an RGB-video network trained with fixed wearable-sensor teachers. Its objective combines classification and distillation losses, including intermediate-feature mapping and semantic preservation.

2) Training of student network:

  • The student is a TRN with BN-Inception that takes RGB videos, while the trained teacher networks remain fixed during student training.GSDM is placed at selected convolutional layers between BN-Inception and VGG16 for teacher–student pairs.Source: 2) Training of student network · p. 8
  • The student uses GSDM intermediate-feature distillation, a complementary last-layer knowledge-distillation loss, and semantic preservation between student fc1 and teacher fc2 features.The semantic-preserving loss aligns the student fc1 features with teacher fc2 features, whose semantic knowledge was learned during teacher training.Source: 2) Training of student network · p. 8
  • Student training combines cross-entropy with GSDM, soft-target, and semantic-preserving losses, weighted by α, β, and γ.The soft-target loss uses teacher and student class-probability predictions with temperature T = 4.Source: 2) Training of student network · p. 8

IV. EXPERIMENTS

The experiments evaluate sensor-to-vision action recognition on three benchmarks, using dataset-specific implementation settings. Berkeley-MHAD uses RGB video and accelerometer data with a subject-based train/test split.

A. Experimental Setup

  • The experimental program compares SAKDN with knowledge-distillation, vision-based, and multimodal action-recognition methods, then studies ablations, network choices, hyperparameters, training curves, and CAM visualizations.The implementation details for the three benchmark datasets are summarized in Table I.Source: A. Experimental Setup · p. 8, Table I · p. 9

1) Berkeley-MHAD [76]:

  • On Berkeley-MHAD, the experiments use 7,900 RGB videos and six accelerator modalities, with the first seven subjects for training and the last five for testing.The dataset has 11 action classes, 12 subjects, five repetitions per action, and 12 camera views; the paper uses RGB video and accelerometers.Source: 1) Berkeley-MHAD [76] · p. 9

2) UTD-MHAD [77]:

Across the reported benchmarks, SAKDN outperforms all compared methods on Berkeley-MHAD and UTD-MHAD and most on MMAct, where TSM is comparable. Ablations report complementary contributions from the framework's components and show performance patterns that vary with the evaluation setting.

2) UTD-MHAD [77]:

  • UTD-MHAD experiments use RGB video with accelerometer and gyroscope modalities; the first two samples per action form training data and the remaining samples form testing data.The dataset has 27 actions, eight subjects, and four repetitions per action.Source: 2) UTD-MHAD [77] · p. 9

3) MMAct [5]:

  • MMAct evaluation uses RGB video and four wearable-sensor modalities across cross-subject, cross-view, cross-scene, and cross-session settings.MMAct contains more than 36,000 trimmed clips, seven modalities, and 37 action classes; its data include four camera views, random walks, and occlusion scenes.Source: 3) MMAct [5] · p. 9

B. Comparison with State-of-the-Art Methods

  • SAKDN outperforms all compared action-recognition, multimodal, and knowledge-distillation methods on Berkeley-MHAD and UTD-MHAD; on MMAct it beats most, with TSM comparable.The authors attribute the MMAct comparison to a framework integrating GAF-based virtual-image generation, SPAMFM, and GSDM.Source: Table II · p. 9, Table III · p. 9, Table IV · p. 9

C. Ablation Study

  • On Berkeley-MHAD, adding accelerators to the student improves performance from 95.32% to 99.33%; SAKDN also exceeds the listed ablated variants.The authors note that wearable sensors on this dataset lack color and texture information, which may degrade their standalone representation.Source: Table V · p. 10, C. Ablation Study · p. 10
  • On UTD-MHAD, SAKDN improves video action-recognition accuracy from 94.87% for the baseline to 98.60%, and outperforms the SKDN and AKDN teacher variants.The passage attributes the gains to the integrated complementary modules SPAMFM, ST, GSDM, and SP.Source: Table VI · p. 11, C. Ablation Study · p. 11
  • On MMAct, SAKDN outperforms the multi-teacher and baseline models in every setting; in cross-view evaluation it improves by 3.33%.The reported comparison links wearable-sensor robustness to occlusion and appearance variation with cross-view and cross-scene settings, while appearance and texture matter in cross-subject and cross-session settings.Source: Table VII · p. 11, C. Ablation Study · pp. 11–12
  • UTD-MHAD visualizations show GSDM highlighting important regions in video frames for predicting semantic concepts.The paper describes the maps as an intermediate visual pattern for knowledge distillation.Source: C. Ablation Study · p. 12, Figure 6 · p. 12

D. Effect of Different Transfer Layers and Backbones

On UTD-MHAD, SAKDN performs better than the 94.87% baseline across the tested BNInception transfer-layer configurations, with marginal gaps between transfer-layer counts. ResNet18, ResNet50, and BNInception also outperform the baseline, while ResNet50 and BNInception outperform ResNet18.

D. Effect of Different Transfer Layers and Backbones

E. Parameter Sensitivity Analysis

The sensitivity analysis identifies dataset-specific settings for β while α and γ are consistent across datasets. Training curves converge, and visualizations usually highlight dominant movement but also show failures in especially challenging scenes.

E. Parameter Sensitivity Analysis

  • The optimal {α, β, γ} values are {0.1, 0.1, 1} for Berkeley-MHAD and {0.1, 1, 1} for UTD-MHAD and MMAct.The authors link the lower Berkeley-MHAD β to its six teacher modalities, to avoid overfitting to any one teacher; α and γ are 0.1 and 1, respectively, on all three datasets.Source: E. Parameter Sensitivity Analysis · pp. 12–13, Figure 7 · p. 13

F. Visualization Analysis

  • Across Berkeley-MHAD, UTD-MHAD, and MMAct, training loss decreases and converges while training accuracy increases.The authors interpret these curves as evidence of good convergence for the proposed loss function.Source: F. Visualization Analysis · p. 13, Figure 8 · p. 13
  • CAM heat maps usually correlate high activation with dominant movement despite camera motion, clutter, occlusion, and appearance variation, but can mistake similar-looking background objects for people.The examples cover Berkeley-MHAD, UTD-MHAD, and MMAct; Fig. 9 overlays heat maps, with red and blue denoting high and low activation.Source: F. Visualization Analysis · p. 13, Figure 9 · p. 14

V. CONCLUSION

  • SAKDN distills complementary knowledge from multiple wearable-sensor teachers to a video student; SPAMFM fuses relational and semantic information, while GSDM highlights important regions and preserves data interrelations.The conclusion reports effectiveness on three benchmarks.Source: §V · p. 14

Related papers

Loading 2009.00210v5…