PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Temporal Alignment Distillation for Camera-to-mmWave Radar Human Activity Recognition

Jiuqi Yan, Chengbo Zhu, Chengdong Xu, Qisong Wu

arXiv:2610.36637v1eess.SP

TL;DR

Sparse, noisy radar reflections make action-relevant body structure and motion difficult to recover, and fixed temporal matching is unsuitable when teacher states and student segments have different granularities. The paper uses camera-keypoint supervision to train a radar student with segment-conditioned retrieval of teacher evidence. On MM-Fi, it reports higher radar-only accuracy and mean scores under most dropout settings, with improved accuracy after adaptation to mmO-HAR.

  • Problem

    Sparse, noisy radar reflections make action-relevant body configuration and motion difficult to recover, and teacher time steps do not correspond rigidly to the student’s fewer segments.

    Source: §I · p. 1, E. Soft Temporal Alignment Distillation · p. 5

  • Method

    Camera-derived keypoints supervise a radar student, with each student segment retrieving an attention-weighted target from the complete teacher sequence; teacher and alignment branches are removed for inference.

    Source: E. Soft Temporal Alignment Distillation · p. 5, 3) Overall Training Objective · p. 7, §VI · p. 11

  • Results

    On MM-Fi, accuracy/macro-F1 rise from 87.04%/87.10% to 89.17%/89.18%, the highest evaluated means; mean scores improve under most dropout settings, and mmO-HAR accuracy rises from 80.33% to 87.75%.

    Source: §VI · p. 12

  • Takeaways & Limitations

    The distilled representation supports radar-only inference with the same 5.568 M parameters and 16.020 GMACs per sequence as the independently trained student.

    Source: §VI · p. 11

    The MM-Fi improvements are numerical mean differences, not uniform fold-wise or statistically significant superiority; standard-deviation ranges overlap and paired hypothesis tests are absent.

    Source: A. Results Analysis · p. 9, §VI · p. 12

Abstract

from arXiv · show

Millimeter-wave (mmWave) radar provides privacy?preserving human activity recognition (HAR), but sparse point clouds limit its representation of body structure and motion. We propose a cross-modal knowledge distillation framework in which a teacher trained on camera-derived two-dimensional (2D) human keypoints supervises a radar-only student. The stu?dent combines PointNet-based point-set encoding, bidirectional temporal modeling, segment-level aggregation, and lightweight shared latent-channel scaling. The teacher retains contextualized states at every time step, whereas the student summarizes the radar sequence into a small number of segments, making fixed timestamp correspondence unsuitable for feature transfer. Soft Temporal Alignment Distillation lets each student segment adaptively retrieve and aggregate teacher temporal evidence into its distillation target. Alignment denotes student-conditioned representation-space matching rather than chronological or monotonic registration. On the Multi-Modal Non-Intrusive 4D Human Dataset (MM-Fi), the method improves radar-only ac?curacy from 87.04% to 89.17%, achieving the highest mean accuracy among the evaluated established and recent distillation baselines. It also obtains higher mean accuracy and macro?averaged F1 score (macro-F1) than the radar student under most evaluated point- and frame-dropout settings. Adaptation to our independently collected millimeter-wave Office Human Ac?tivity Recognition Dataset (mmO-HAR) improves accuracy from 80.33% to 87.75%, supporting the transferability of the distilled radar representation. The teacher and alignment branch are removed after training, enabling privacy-preserving radar-only inference without additional sensing or computational branches.

I. INTRODUCTION

Sparse radar reflections make body configuration and motion difficult to recover, while camera keypoints offer training-time motion guidance. The paper proposes segment-conditioned temporal distillation for radar-only recognition and reports improved MM-Fi results.

I. INTRODUCTION

  • Camera-derived keypoints supervise training, but the teacher and alignment branch are removed for inference, leaving radar-only operation without a camera feed or teacher computation.The method uses synchronized camera keypoints and radar during model development.Source: §I · p. 1
  • Sparse, noisy radar reflections hinder recovery of action-relevant body configuration and motion, while teacher and student features differ in modality and temporal granularity.The teacher retains states for all T time steps; the student compresses radar into K segments, making fixed frame-to-segment correspondence rigid.Source: §I · p. 1

INFERENCE

  • On MM-Fi, Soft-Align KD raises radar-only accuracy from 87.04% to 89.17%, with higher mean scores under dropout and evidence of transfer to independently collected mmO-HAR.The supplied summary reports transferability but does not give the mmO-HAR accuracy value.Source: INFERENCE · p. 2
  • The radar student combines PointNet-based frame encoding, bidirectional temporal modeling, segment pooling, and shared latent-channel scaling.The design targets sparse mmWave point-cloud sequences.Source: INFERENCE · p. 2
  • Soft Temporal Alignment Distillation constructs a separate teacher target for each student segment by retrieving contextualized keypoint states across the full sequence.The matching is student-conditioned and potentially non-monotonic, rather than chronological frame registration.Source: INFERENCE · p. 2

A. Multimodal Fusion with Radar

The section distinguishes training-only cross-modal guidance from multimodal fusion and reviews prior distillation approaches. It formulates radar-only prediction from synchronized radar and camera-keypoint sequences.

A. Multimodal Fusion with Radar

  • Prior radar–visual fusion combines sensor features or decisions, whereas this work focuses on visual guidance during training for a radar-only student.The reviewed studies use different tasks and protocols and are not directly ranked against this work.Source: A. Multimodal Fusion with Radar · p. 2

B. Cross-Modal Knowledge Distillation

  • Cross-modal distillation transfers features or predictions across speech, pose, and heterogeneous sensors, while several general objectives match global predictions or spatial features.These approaches address transfer settings distinct from unequal temporal granularity in pose-to-radar HAR.Source: B. Cross-Modal Knowledge Distillation · pp. 2–3
  • Shin et al. directly match corresponding graph-network features, whereas this method retrieves a separate full-sequence teacher target for each radar segment.Tac-Mamba also uses pose guidance but supports pose-guided fusion and a radar-only fallback.Source: B. Cross-Modal Knowledge Distillation · p. 3

A. Problem Formulation

  • The task pairs synchronized radar point clouds P and camera-derived keypoints Q during training, but the student predicts action labels from radar alone at inference.The two sequences contain T frames and each paired sample has an action label.Source: A. Problem Formulation · p. 3

B. Radar Point Cloud Representation

C. Student Network for Radar Point Clouds

The radar student encodes unordered point clouds frame by frame, then models temporal dependencies with a bidirectional LSTM. Its design includes shared latent-channel scaling.

C. Student Network for Radar Point Clouds

1) Frame-wise Point Cloud Encoding:

  • Each frame’s unordered radar points pass through a PointNet-based encoder with learned spatial and feature transformations, shared pointwise mappings, and symmetric max pooling.The spatial input transform acts on x, y, and z; Doppler and intensity are concatenated afterward, and pooling makes the frame representation invariant to point order.Source: 1) Frame-wise Point Cloud Encoding · pp. 3–4

2) Temporal Modeling:

  • A bidirectional LSTM processes the frame-feature sequence to model temporal dependencies using forward and backward context.The student uses LS recurrent layers, with HS hidden units per direction; no dropout is used inside the LSTM.Source: 2) Temporal Modeling · p. 4

3) Segment-Level Feature Aggregation:

The student pools radar features into ordered temporal segments, rescales their latent channels, and uses them for action classification and segment-level distillation.

3) Segment-Level Feature Aggregation:

  • Each of K contiguous, non-overlapping temporal segments is summarized by element-wise max pooling, retaining each dimension’s maximum while discarding within-segment order and non-maximum responses.Source: 3) Segment-Level Feature Aggregation · p. 4

4) Shared Latent-Channel Scaling:

  • A learnable channel-scaling vector shared across segments rescales latent contributions; it is static, not sample-dependent, and cannot restore information discarded by max pooling.Source: 4) Shared Latent-Channel Scaling · p. 4

5) Action Classification and Distillation Projection:

D. Teacher Network for Camera-Derived 2D Keypoints

The teacher uses synchronized, camera-derived 2D keypoints to encode pose information and produce frame-level features.

D. Teacher Network for Camera-Derived 2D Keypoints

  • The teacher input is a fixed-order sequence of 2D joint coordinates synchronized with radar frames; keypoints preserve body configuration and joint trajectories while suppressing appearance cues.Source: D. Teacher Network for Camera-Derived… · p. 5

1) Frame-wise Keypoint Encoding:

  • A single-channel ResNet-18 processes each keypoint frame as a coordinate tensor, and global average pooling produces its frame-level feature.The backbone is ImageNet-initialized except for its newly constructed input convolution, then fine-tuned end-to-end on MM-Fi.Source: 1) Frame-wise Keypoint Encoding · p. 5

2) Temporal Modeling:

The teacher retains contextualized states across the sequence, while its classification branch pools them; distillation instead supplies temporal evidence to radar student segments.

2) Temporal Modeling:

  • A bidirectional LSTM produces concatenated forward and backward states at each frame, retaining the teacher’s complete temporal feature sequence.Source: 2) Temporal Modeling · p. 5

3) Temporal Feature Aggregation and Classification:

E. Soft Temporal Alignment Distillation

  • Because the teacher has T frame states and the student K segments, with K ≪ T, each radar segment retrieves its own attention-weighted teacher target rather than using fixed timestamp matching.The overview depicts the teacher processing camera-derived 2D joint coordinates and the student processing radar point clouds.Source: E. Soft Temporal Alignment Distillation · pp. 5–6, Figure 2 · p. 6

1) Student-Conditioned Temporal Evidence Retrieval:

Each student segment retrieves teacher temporal evidence through attention, and normalized feature matching trains the student to align with those targets.

1) Student-Conditioned Temporal Evidence Retrieval:

  • Each segment’s projected embedding queries all teacher states using temperature-scaled dot-product similarities, and attention weights form a target that need not follow temporal order or exclude shared evidence.Independent normalization permits multiple segments to retrieve the same discriminative teacher evidence.Source: 1) Student-Conditioned Temporal… · p. 6

2) Feature Distillation Loss:

  • The distillation loss compares ℓ2-normalized student embeddings with their retrieved teacher targets using mean-squared error, while gradients update the student encoder and projection.Normalization primarily compares representation directions rather than magnitudes.Source: 2) Feature Distillation Loss · p. 6

3) Overall Training Objective:

The training objective combines ground-truth classification with feature distillation from the teacher’s unpooled temporal states. The experiments use subject-disjoint MM-Fi evaluation, with implementation and deployment profiling specified for both modalities.

3) Overall Training Objective:

  • Training combines ground-truth cross-entropy with λ-weighted feature distillation from unpooled teacher temporal states, not teacher logits.λ weights the distillation loss.Source: 3) Overall Training Objective · pp. 6–7

A. Experimental Databases

  • MM-Fi provides 40 subjects, 27 activities, four indoor environments, 297 synchronized frames per sample, 17 image-plane joints, and radar 3D, Doppler, and intensity detections.Source: A. Experimental Databases · p. 7

B. Data Splits and Evaluation Protocol

  • Eight MM-Fi subjects form the fixed test set, while the remaining 32 are split into five subject-disjoint development folds; reported standard deviations reflect runs, not independent test folds.Source: B. Data Splits and Evaluation Protocol · p. 7

C. Implementation Details

  • Implementation uses stacked NumPy keypoints and radar frames adjusted to 112 points, with masking to exclude padded detections from point-wise max pooling.Teacher and student use FP32; deployment profiling uses a batch size of one and removes the training-only alignment projection.Source: C. Implementation Details · pp. 7–8

D. mmO-HAR Dataset and Cross-Dataset Adaptation Protocol

The experiments evaluate MM-Fi recognition, dropout robustness, deployment complexity, and adaptation to mmO-HAR. On MM-Fi, Soft-Align KD improves the radar student’s mean accuracy and macro-F1, while the reported baseline comparisons remain numerical rather than statistically significant.

D. mmO-HAR Dataset and Cross-Dataset Adaptation Protocol

  • mmO-HAR contains 960 sequences from eight subjects performing six actions; six subjects form the development set and two subjects the held-out test set.The dataset is used as the fine-tuning target, with subject-disjoint testing.Source: D. mmO-HAR Dataset and Cross-Dataset… · p. 8

V. EXPERIMENTAL RESULTS AND ANALYSIS

  • The evaluation covers MM-Fi recognition, point- and frame-dropout robustness, deployment complexity, and mmO-HAR adaptation.Source: §V · p. 8

A. Results Analysis

  • 89.17 ± 6.71% accuracy and 89.18 ± 6.68% macro-F1 with Soft-Align KD, versus 87.04 ± 2.97% and 87.10 ± 2.75% for the radar student on MM-Fi.The mean accuracy gain is 2.13 percentage points; the reported dispersion indicates sensitivity to the development protocol.Source: Table III · p. 8, A. Results Analysis · p. 8
  • Conventional objectives and LS-KD do not improve on the radar student; Shin’s adaptation reaches 87.96% accuracy, while the Tac-Mamba-inspired adaptation reaches 79.07%.The latter result characterizes a modified implementation, not the original Mamba-based model.Source: A. Results Analysis · p. 8, Table IV · p. 9
  • Soft-Align KD’s mean accuracy and macro-F1 exceed strongest competing WKD-L by 1.02 and 0.87 points, respectively, but overlapping standard-deviation ranges and absent paired tests preclude claims of statistical significance.Source: A. Results Analysis · p. 9

B. Deployment Complexity

Soft-Align KD yields the strongest reported distillation result by retrieving teacher evidence for each student segment rather than imposing fixed intervals. Ablations examine segmentation, shared scaling, segment count, and alignment constraints, while deployment analysis compares the radar student’s cost with the teacher’s.

B. Deployment Complexity

  • The deployed student has 5.568 M parameters and occupies 21.24 MiB in FP32, both 74.3% below the teacher, while its 16.020 GMAC count is higher.The independently trained and Soft-Align KD students have identical deployment costs; server measurements do not establish embedded latency or energy use.Source: B. Deployment Complexity · p. 9, Table V · p. 10

1) Role of Temporal Segment Pooling:

  • Segment pooling alone is algebraically equivalent to global max pooling for classification, so the small difference between the first two Table VI rows is treated as run-to-run variation.Segmentation still supplies K local distillation units and enables the segment-count analysis.Source: 1) Role of Temporal Segment Pooling · p. 9, Table VI · p. 10

2) Effect of Shared Latent-Channel Scaling:

  • At K = 4, shared latent-channel scaling has higher mean accuracy and macro-F1 than the segment-only variant: 87.04% and 87.10% versus 86.30% and 86.13%.The scale is shared across samples and segments and rescales aggregated hidden channels; it is not sample-dependent gating or recovery of responses discarded by max pooling.Source: 2) Effect of Shared Latent-Channel… · p. 9

3) Sensitivity to the Number of Temporal Segments:

  • K = 4 yields the highest evaluated accuracy and macro-F1 among tested segment counts, supporting it as the default rather than a universal optimum.Three segments may merge distinct phases, while five or six create shorter windows potentially more sensitive to sparse or transient responses.Source: 3) Sensitivity to the Number of… · p. 9

4) Effect of Soft-Align Knowledge Distillation:

  • Hard-Align KD raises accuracy from 87.04% to 87.78% but prespecifies which teacher interval supervises each student segment.It divides the teacher sequence using the student’s fixed segment boundaries and max-pools within each interval.Source: 4) Effect of Soft-Align Knowledge… · p. 9
  • 89.17% accuracy and 89.18% macro-F1 make Soft-Align KD higher in mean than the radar-only student and fixed segment alignment.Each student segment retrieves a weighted target from the complete teacher sequence; Table VII reports the comparison.Source: 4) Effect of Soft-Align Knowledge… · p. 9, Table VII · p. 10

5) Effect of the Order-and-Diversity Constraint:

  • The constrained variant reaches 88.52% accuracy and 88.40% macro-F1, 0.65 percentage points below unconstrained Soft-Align KD in accuracy.Its auxiliary objective reduces duplication, order violations, and attention overlap while broadening attention, but these properties do not coincide with the classification objective.Source: 5) Effect of the Order-and-Diversity… · p. 10, Table VIII · p. 10

D. Robustness Experiment

With model parameters fixed, both distilled models have higher mean scores than the radar-only student under most tested point- and frame-dropout settings. The unconstrained method leads on clean data and several corruption levels, but the constrained variant leads at some intermediate dropout levels.

D. Robustness Experiment

  • Point dropout is tested at ratios 0.0, 0.2, 0.4, and 0.6, and frame dropout at 0.0, 0.1, 0.2, and 0.3, with parameters fixed during inference.The comparison includes the radar-only student and both unconstrained and constrained Soft-Align KD models.Source: D. Robustness Experiment · pp. 10–11
  • At 0.6 point dropout, unconstrained Soft-Align KD raises mean accuracy from 70.19% to 80.46% and mean macro-F1 from 69.44% to 80.33%.At 0.3 frame dropout, mean accuracy changes from 78.33% to 84.91% and mean macro-F1 from 77.80% to 85.05%. Figures 4 and 5 visualize these trends.Source: Table IX · p. 11, D. Robustness Experiment · p. 11, Figure 4 · p. 11
  • The constrained variant has the highest means at point-dropout ratios 0.2 and 0.4 and is marginally higher at frame-dropout ratio 0.3, while unconstrained Soft-Align KD leads on clean data, at 0.6 point dropout, and at frame-dropout ratios 0.1 and 0.2.Run-to-run standard-deviation ranges substantially overlap, and the constraint provides no consistent robustness advantage.Source: D. Robustness Experiment · p. 11

E. Cross-Dataset Adaptation Experiment

On the independently collected mmO-HAR target, Soft-Align KD pretraining improves accuracy over training from scratch under a subject-disjoint test protocol. The conclusion also reports MM-Fi gains, radar-only deployment costs, and boundaries on the scope of transfer and hardware evaluation.

E. Cross-Dataset Adaptation Experiment

  • On mmO-HAR, Soft-Align KD pretraining achieves 87.75% accuracy and 86.74% macro-F1, gains of 7.42 and 6.26 percentage points over training from scratch.The target differs from MM-Fi in sensor type, environment, activity categories, and sequence length; the result supports adaptation, not universal cross-dataset generalization.Source: Table X · p. 11, E. Cross-Dataset Adaptation Experiment · p. 11

VI. CONCLUSION

  • Removing the teacher and alignment projection leaves radar-only inference at 5.568 M parameters and 16.020 GMACs per sequence, identical to the independently trained student.The training-time alignment projection is not part of deployment.Source: §VI · p. 11
  • On MM-Fi, Soft-Align KD raises accuracy/macro-F1 from 87.04%/87.10% to 89.17%/89.18%, the highest evaluated means; WKD-L reaches 88.15%/88.31%.The five development-fold runs share a fixed test set, so the mean improvements are not uniform fold-wise or statistically significant superiority.Source: §VI · p. 12
  • A100 profiling does not establish embedded-device latency or energy consumption; on-device evaluation remains future work.The conclusion also identifies broader cross-domain evaluation and deterministic repeated trials as future directions.Source: §VI · p. 12
Loading 2610.36637v1…