PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Continuous Human Activity Recognition Using Deep Learning and Distributed Radar Sensors

Sohani Munteha Hiam, Mohammad Soleymani

arXiv:2609.10419v1eess.SP

TL;DR

Continuous radar HAR must handle aspect-angle-dependent signatures and ambiguous transitions between activities. The paper combines spectrogram encoding, adaptive fusion of five radar views, and bidirectional temporal modeling; it achieves 87.56% mean L1PO accuracy and improves on the published CNN–RNN reference.

  • Problem

    Continuous radar HAR must label uninterrupted sequences despite ambiguous transitions and Doppler signatures that vary with radar aspect angle.

    Source: §I · p. 1, A. Literature Review · p. 2

  • Method

    A shared ConvNeXt-inspired encoder, radar-wise MHSA fusion, radar and spectrogram regularization, and stacked BiGRUs process Doppler–time spectrograms from five radar nodes.

    Source: §I · p. 1, §V · p. 12

  • Results

    87.56% mean L1PO accuracy, improving the published CNN–RNN reference model by 2.45 percentage points.

    Source: §V · p. 12

  • Takeaways & Limitations

    Increasing the radar configuration from one to five nodes improved mean validation accuracy from 79.9% to 90.8%.

    Source: §V · p. 12

Abstract

from arXiv · show

Continuous human activity recognition (HAR) with distributed radar sensor networks is challenging because Doppler signatures depend strongly on aspect angle, the informativeness of individual radar views varies with motion direction, and activity transitions in uninterrupted sequences are often ambiguous. This paper proposes a ConvNeXt-MHSA-BiGRU framework for frame-wise continuous HAR from Doppler-time spectrograms acquired by five spatially distributed radar nodes. A shared ConvNeXt-inspired encoder extracts time-frequency representations from each radar stream, while radar-wise multi-head self attention (MHSA) models inter-node dependencies and adaptively fuses complementary views. RadarDropout and SpecAugment-style time-frequency masking regularize the network against unreliable radar views and local spectrogram perturbations, whereas stacked bidirectional gated recurrent units (BiGRUs) exploit temporal context for frame-wise classification across activity transitions. The framework is evaluated on nine activities performed by 14 participants using leave-one-person-out (L1PO) subject-independent testing with fivefold cross-validation. It achieves a mean L1PO test accuracy of 87.56% across the 14 held-out subjects and improves recognition performance over the published CNN-RNN baseline. The results demonstrate that combining modern spectrogram encoding, adaptive radar view fusion, and bidirectional temporal modeling provides an effective framework for continuous and distributed radar-based HAR.

I. Introduction

Continuous radar HAR must resolve ambiguous activity transitions despite viewpoint-dependent Doppler signatures. The paper motivates a framework combining spectrogram encoding, adaptive multi-radar fusion, and temporal modeling, then evaluates it against a published baseline.

I. Introduction

  • Continuous HAR must assign labels to every temporal bin despite variable durations and ambiguous transitions, compounded by radar signatures that depend on aspect angle.Source: §I · p. 1, A. Literature Review · p. 2

A. Literature Review

  • Prior work spans deep radar representations, temporal sequence models, distributed sensing, and attention-based fusion, while radar HAR remains sensitive to subject, sensor-position, and deployment variation.Source: A. Literature Review · pp. 1–2

B. Motivation

  • Jointly learning broad time-frequency representations, dependencies among radar views, view regularization, and frame-wise temporal predictions remains comparatively unexplored.Source: B. Motivation · p. 3

C. Contributions and Paper Organization

  • The proposed five-node framework combines a shared ConvNeXt-inspired encoder, radar-wise MHSA with RadarDropout and time-frequency masking, and stacked BiGRUs for frame-wise recognition.Source: C. Contributions and Paper Organization · p. 3
  • The framework is evaluated on nine classes from 14 participants using subject-independent L1PO testing with fivefold cross-validation and compared with the published CNN–RNN benchmark.Source: C. Contributions and Paper Organization · p. 3

II. Dataset and Preprocessing

The study uses continuous sequences from a five-node UWB radar dataset with natural activity transitions. It organizes aligned Doppler–time spectrograms into five radar-specific inputs for the proposed network.

II. Dataset and Preprocessing

A. Distributed UWB Radar Setup

B. Activity Classes and Continuous Sequences

C. Radar Data Representation and Input Organization

A. Problem Formulation

The formulation treats each normalized sample as five radar streams and assigns one of nine activity labels to every temporal bin. The network maps these inputs to frame-wise class probabilities.

A. Problem Formulation

  • The normalized input tensor has dimensions N × 30 × 32 × 5, representing samples, temporal bins, Doppler bins, and radar nodes, and each batch is separated into radar-specific streams.Source: A. Problem Formulation · p. 6
  • Each sample has a ground-truth label for every temporal bin, with labels drawn from the nine activity classes.Source: A. Problem Formulation · p. 6
  • The proposed network maps each input to a probability vector over activity classes for each temporal bin.Source: A. Problem Formulation · p. 6

B. Shared ConvNeXt-Style Time–Frequency Encoder

A shared ConvNeXt-style encoder processes each radar stream to extract time–frequency features. Its design combines broad depthwise convolution, channel expansion, normalization, residual learning, and dropout.

B. Shared ConvNeXt-Style Time–Frequency Encoder

  • The encoder follows ConvNeXt principles, including large-kernel depthwise convolution, pointwise channel expansion, GELU activation, normalization, and residual learning.Its 1 × 1 convolutions expand the 64-channel representation to 256 channels before projecting it back.Source: B. Shared ConvNeXt-Style… · p. 6
  • The same ConvNeXt-style encoder processes all five radar streams because their sensing modality and input format are the same.The encoder uses a convolutional stem, frequency-axis pooling, ConvNeXt-style blocks, and spatial dropout.Source: B. Shared ConvNeXt-Style… · p. 6
  • Frequency pooling reduces the Doppler dimension from 32 bins to 16 before two ConvNeXt-style blocks process the pooled features.Spatial dropout is applied after the two blocks during training.Source: B. Shared ConvNeXt-Style… · p. 6

C. Intermediate Radar-Wise MHSA Fusion

The intermediate fusion module treats the five radar streams as views, uses radar-wise MHSA to adaptively combine them, and then forms a temporal sequence for frame-wise prediction.

C. Intermediate Radar-Wise MHSA Fusion

  • Radar-wise MHSA models dependencies among radar views and adaptively combines complementary information because signature quality varies with subject position, movement direction, and radar aspect angle.Prior distributed radar studies have reported benefits from feature-level fusion of complementary viewpoint-dependent motion signatures,,.Source: C. Intermediate Radar-Wise MHSA Fusion · p. 6
  • RadarDropout randomly removes complete radar feature streams during training to improve robustness to weak or unreliable views.The radar-specific feature maps are stacked along an explicit radar-view axis before this operation.Source: C. Intermediate Radar-Wise MHSA Fusion · p. 6
  • For each sample, temporal bin, and Doppler index, the five radar nodes serve as view tokens for attention across the radar axis.The implementation uses four attention heads with key dimension 32 and reduces the attended representations across the radar dimension.Source: C. Intermediate Radar-Wise MHSA Fusion · pp. 6–7
  • The resulting sequence is processed by stacked BiGRU layers and classified at each temporal bin into one of nine activity classes.This sequence-to-label flow is shown in the proposed framework.Source: Figure 4 · p. 7
  • SpecAugment-style time–frequency masking is applied to the fused feature map during training.Frequency-axis averaging then converts that map into a temporal sequence.Source: C. Intermediate Radar-Wise MHSA Fusion · p. 7

D. BiGRU Temporal Modeling Module

The temporal module projects the fused sequence into an embedding and processes it with stacked bidirectional GRUs. Each temporal prediction can use both preceding and following context within the input window.

D. BiGRU Temporal Modeling Module

  • The fused temporal sequence is projected into a 128-dimensional embedding and processed by three stacked BiGRU layers.Each layer has 64 hidden units in each direction, producing a concatenated 128-dimensional feature at every temporal bin.Source: D. BiGRU Temporal Modeling Module · p. 7
  • Each BiGRU layer runs forward and backward GRUs, with the two directional outputs concatenated along the feature dimension.The layers use input dropout of 0.10 during training, with no recurrent dropout.Source: D. BiGRU Temporal Modeling Module · p. 7
  • Bidirectional processing lets each temporal prediction use both preceding and following context within the input window.The final BiGRU layer supplies the temporal representation to the classifier.Source: D. BiGRU Temporal Modeling Module · p. 7

E. Time Distributed Classification Head

The classification head produces a separate activity prediction for every temporal bin. It applies a time-distributed dense layer and computes class probabilities for the mini-batch.

E. Time Distributed Classification Head

  • The classification head predicts one activity label for each temporal bin using a time-distributed dense layer with 64 hidden units.The class probability vector is computed for each sample and temporal bin.Source: E. Time Distributed Classification Head · p. 7
  • The model outputs a mini-batch tensor of frame-wise class probabilities for nine activity classes.The notation defines P_B as the frame-wise class probability tensor and the predicted label as the activity at a given sample and temporal bin.Source: E. Time Distributed Classification Head · pp. 7–8

F. Training Objective

The proposed ConvNeXt–MHSA–BiGRU model improves mean subject-balanced L1PO accuracy over the CNN–RNN reference. Evaluation averages fivefold results for each held-out subject, while the training objective uses label smoothing for ambiguous transition frames.

F. Training Objective

  • Label smoothing with ε_ls = 0.03 regularizes categorical cross-entropy and can reduce overconfidence on ambiguous transition frames.Source: F. Training Objective · p. 8

B. Comparison with the CNN–RNN Reference Model

  • 87.56% mean fivefold L1PO test accuracy versus 85.11% for CNN–RNN (BiGRU), a 2.45-percentage-point improvement.The proposed model's maximum subject-wise accuracy is 93.06%, compared with 90.80% for the reference; improvement is not uniform across subjects.Source: Figure 5 · p. 9, B. Comparison with the CNN–RNN… · p. 9

IV. Experimental Results and Discussion

  • For each held-out subject, five models use the same fivefold train–validation split strategy, and test accuracy is averaged across folds and subjects equally.Pooled frame-level predictions are used only for class-wise analysis through the row-normalized confusion matrix.Source: §IV · p. 9

A. Experimental Setup and Implementation Details

  • All L1PO fivefold experiments use the same workstation hardware, software environment, and training settings for consistent comparison.The workstation includes an Intel i9-14900HX CPU, 32 GB RAM, and an NVIDIA GeForce RTX 4070 Laptop GPU.Source: A. Experimental Setup and… · p. 9

C. Subject-wise Performance of the Proposed Model

The proposed model averages 87.56% accuracy across held-out subjects, with performance varying by subject and activity class. In temporal ablation, BiGRU has the highest mean accuracy among the tested modules; remaining confusions cluster around related motion phases and transitions.

D. Class-wise Confusion Matrix Analysis

  • Fig. 6's row-normalized diagonal values represent class-wise recall; class 5 reaches 94.6%, while class 3→4 and 4→3 confusions are 12.0% and 14.3%.Other confusions include class 7→1 at 12.9% and class 9→8 at 8.6%; remaining errors mainly involve related motion phases or adjacent transitions.Source: Figure 6 · p. 10, D. Class-wise Confusion Matrix Analysis · p. 10

C. Subject-wise Performance of the Proposed Model

  • 87.56% mean accuracy, 84.88% mean macro precision, 83.99% mean macro recall, and 83.95% mean macro F1 summarize subject-balanced performance.Subject I reaches 93.06% accuracy, while subject D has the lowest accuracy at 77.52%.Source: C. Subject-wise Performance of the… · pp. 9–10

E. Temporal Module Ablation

  • 87.56% mean accuracy for ConvNeXt–MHSA–BiGRU exceeds the RNN variant's 82.37% by 5.19 percentage points; Conformer reaches 86.91%.The fixed encoder and radar fusion are paired with different temporal modules; the results suggest bidirectional context suits this dataset, while Conformer's higher complexity does not raise mean accuracy.Source: Figure 7 · p. 10, E. Temporal Module Ablation · pp. 10–11

F. Model Complexity

The BiGRU offers the strongest accuracy–complexity trade-off among the tested temporal modules. Temporal errors cluster near activity transitions, while adding radar nodes improves validation accuracy with diminishing gains; the conclusion reports 87.56% mean L1PO accuracy and a 2.45-point improvement over the published CNN–RNN reference.

F. Model Complexity

  • 347,593 total parameters yielded the best mean accuracy for ConvNeXt–MHSA–BiGRU, which used 569,088 fewer parameters than the larger Conformer without sacrificing accuracy.The RNN variant had the fewest parameters (222,793) but the lowest mean L1PO accuracy; the Conformer had 916,681 parameters and did not outperform the BiGRU.Source: F. Model Complexity · p. 11

G. Temporal Prediction Behavior

  • Selected transition-enriched regions scored 83.75%, 80.63%, and 83.44% for Subjects E, G, and B, below their respective full-sequence accuracies of 89.78%, 88.23%, and 92.27%.The timelines generally follow the continuous activity sequence, with most mismatches near activity changes or short transitions; the selected regions are illustrative, not replacements for full-sequence or subject-wise accuracy.Source: G. Temporal Prediction Behavior · p. 11, Figure 8 · p. 11

H. Effect of Radar Configuration

  • Validation accuracy rose from 79.9% with one radar to 90.8% with five, with progressively smaller gains after adding more nodes.The five-radar configuration also achieved higher mean validation accuracy than the CNN–RNN (BiGRU) baseline; the single-radar case lacks MHSA, so the initial increase combines additional viewpoints and attention-based fusion.Source: H. Effect of Radar Configuration · p. 12, Figure 9 · p. 12

V. Conclusion

  • 87.56% mean L1PO accuracy, 2.45 percentage points above the published CNN–RNN reference, was achieved with the proposed framework.The conclusion also reports mean macro precision, recall, and F1-score of 84.88%, 83.99%, and 83.95%, respectively.Source: §V · p. 12
  • Ablations reported 87.56% mean L1PO accuracy for BiGRU, compared with 82.37% for RNN and 86.91% for Conformer; transition regions remained a principal source of error.The conclusion describes the combined approach as effective and computationally compact for subject-independent continuous radar HAR, and identifies cross-environment generalization, adaptive node selection, and degraded or unavailable radar observations as future work.Source: §V · p. 12

Appendix A Input Window Duration and Segmentation

Each radar stream is segmented into non-overlapping Doppler–time windows, with 30 temporal bins per input. For a nominal 120-second S2 trial, this produces 15 complete windows, while the figure’s transitions are schematic rather than aligned to physical activity boundaries.

Appendix A Input Window Duration and Segmentation

  • Each input spans 32 Doppler-frequency bins and 30 temporal bins, with consecutive spectrogram timestamps separated by 262.4 ms.The nominal temporal extent of a window is approximately 7.872 seconds.Source: Appendix A · p. 12
  • Fifteen nominal 7.872-second windows cover 118.08 seconds of a 120-second recording, leaving approximately 1.92 seconds.The remaining duration is insufficient to form another complete 30-bin window; small differences from radar-stream alignment or spectrogram generation are not included.Source: Appendix A · pp. 12–13
  • Each 120-second S2 trial is segmented into 15 complete non-overlapping input windows, each containing 30 temporal bins.The full dataset comprises 14 subjects, 7 sequences, and 4 repetitions per sequence, totaling 392 recordings; the illustration shows only S2.Source: Figure 10 · p. 13, Appendix A · p. 13
  • S2 covers sitting-down and standing-up-from-sitting, labeled 3 and 4; illustrated transition locations do not imply physical boundaries at window edges.The temporal label sequence for each extracted input has length 30.Source: Appendix A · p. 13
  • The five-radar input is represented as a 5 × 30 × 32 tensor, paired with a frame-wise activity-label vector of length 30.For each radar node, the spectrogram is arranged in temporal-major form as X_r in R^(30×32).Source: Appendix A · p. 13

Appendix B Subject-Wise and Attention-Based Model Analysis

Representative held-out subjects show that class-wise errors vary across people, with several activities especially difficult in lower-performing cases. The average radar-wise attention matrix indicates modest self-preference alongside substantial cross-radar information exchange, but is not a measure of causal radar importance.

A. Representative Subject-Wise Confusion Matrices

  • Subject I achieved 93.06% mean accuracy, with recalls of 99.7% and 99.6% for classes 5 and 6, while classes 7 and 9 had the largest remaining errors.The row-normalized confusion matrices pool all five test folds; values are class-wise percentages.Source: A. Representative Subject-Wise… · p. 13, Figure 11 · p. 14
  • For median-range Subject G, classes 3 and 4 had recalls of 65.2% and 59.7%; lowest-performing Subject D had 77.52% mean accuracy, with class 4 confused with class 3 in 23.5%.Subject D also had reduced recalls for classes 8 and 9 (65.5% and 52.8%), while classes 1 and 7 retained recalls of 88.5% and 92.0%; degradation was concentrated in a subset of confusable activities.Source: A. Representative Subject-Wise… · pp. 13–14, Figure 11 · p. 14
  • The subject-wise analysis identifies classes 3, 4, 8, and 9 as principal sources of subject-dependent degradation, while many classes remain well recognized across unseen subjects.The authors suggest subject adaptation and improved activity-transition modeling for further investigation.Source: A. Representative Subject-Wise… · p. 14

B. Radar-Wise MHSA Attention Analysis

  • Average attention was approximately 0.236 on diagonal entries and 0.191 off diagonal, showing modest self-preference while retaining contributions from other radar views.Uniform weighting across five radar nodes would be 0.20; R1 and R5 had self-attention weights of 0.25 and 0.26, respectively.Source: B. Radar-Wise MHSA Attention Analysis · p. 14, Figure 12 · p. 14
  • The average attention matrix diagnoses learned fusion behavior, not causal radar importance, and averaging suppresses sample-dependent attention variation.Substantial off-diagonal attention is consistent with information exchange among spatially distributed radar views.Source: B. Radar-Wise MHSA Attention Analysis · p. 15

Related papers

Loading 2609.10419v1…