Source-linked AI summary
Single-Stream Multi-Feature Fusion with Temporal Robustness for Gait Emotion Recognition
Shirong Lyu, Silu Quan, Yixuan Ding, Chengpeng Wang
TL;DR
Gait emotion recognition suffers from scarce annotations, heterogeneous data, and poor generalization across sequence conditions. The paper proposes SV-GCN, which combines joint-level early fusion and frame-rate-insensitive motion features with global valid-frame masking. On E-Gait, it achieves competitive performance and shows robustness across heterogeneous subsets, supporting transfer from large-scale skeleton-action datasets.
Problem
Gait emotion recognition faces high annotation costs, data scarcity, inefficient feature fusion, and inconsistent sequence lengths and frame rates that limit generalization.
Method
SV-GCN uses intra-frame relative motion features, joint-level shallow multi-feature embedding, and a global mask-guided valid-frame spatio-temporal graph convolution module.
Results
On E-Gait, SV-GCN achieves competitive performance and demonstrates robustness across heterogeneous subsets with varying lengths and frame rates.
Takeaways & Limitations
The results suggest a viable pathway for transfer learning from large-scale skeleton-based action recognition datasets to gait emotion recognition.
Abstract
from arXiv · showhide
3D skeleton-based gait emotion recognition faces high annotation costs, data scarcity, and poor generalization on heterogeneous data. This paper proposes SV-GCN, a single-stream multi-feature fusion framework with temporal invariance. We introduce intra-frame relative motion features to eliminate frame-rate sensitivity and embed heterogeneous cues at shallow layers, enabling early fusion without multi-stream complexity. For variable-length sequences, we design a global mask-guided valid-frame spatio-temporal graph convolution module, introducing frame-rate insensitivity for the first time in this domain. On the E-Gait dataset, our method achieves performance comparable to state-of-the-art while demonstrating strong generalization across varying sequence lengths and frame rates, offering a viable pathway for pre-training on large-scale skeleton-based action recognition datasets.
1 Introduction
Gait emotion recognition is hindered by heterogeneous features, scarce and inconsistent data, and temporal sensitivity. SV-GCN addresses these issues through joint-level early fusion, frame-rate-insensitive motion features, and global masking for variable-length sequences.
- Existing approaches either concatenate affective and learned features or use affective features as global constraints, preventing fine-grained local-joint fusion.These strategies can also increase complexity when extended to multiple streams.
- High annotation costs, scarce data, and inconsistent sequence lengths, frame rates, and gait cycles limit generalization.Existing models may pad sequences to fixed lengths, causing temporal distortion, feature contamination, and inaccurate batch-normalization statistics.
- SV-GCN combines joint-level multi-feature fusion with temporal robustness to address heterogeneous data in gait emotion recognition.Its design targets inefficient feature fusion, frame-rate sensitivity, and padding interference.
- Frame-rate-insensitive intra-frame relative motion features describe coordinated body movements without direct dependence on temporal variables.The relative motion state of each joint is calculated with respect to the root joint within the same frame.
- SMEF embeds affective and other heterogeneous features into a unified per-joint space, enabling shallow-layer early fusion in a single-stream ST-GCN.The approach is intended to avoid parameter explosion and overfitting while learning richer local patterns.
- VF-STGCN applies global masking so temporal aggregation uses only valid frames, while mask-aware normalization prevents padding-related statistical drift.The mechanism is introduced as frame-rate insensitivity for gait emotion recognition and operates only on valid frames.
2 Related Work
Skeleton-based action recognition has progressed from recurrent models to graph convolutions and Transformers, supported by benchmarks such as NTU RGB+D and Kinetics. Related gait-emotion work evolved from handcrafted features and classical classifiers toward LSTM, ST-GCN, and GCN-based methods.
- GCNs became dominant after RNNs and LSTMs because they explicitly model human body structure, while Transformers capture global dependencies.ST-GCN established the spatio-temporal graph-convolution foundation for subsequent methods.
- NTU RGB+D and Kinetics are widely used skeleton-action benchmarks for evaluating model generalization.
- Gait-emotion recognition progressed from handcrafted features with SVMs to LSTM fusion and later GCN-based methods such as STEP and semi-supervised extensions.
- The proposed framework is presented as an SV-GCN overview with joint-level multi-feature fusion and valid-frame processing for variable-length sequences.
3 Method
SV-GCN fuses pose, motion, and affective features at the joint level in a single stream, then uses valid-frame masking to handle variable-length sequences. Its design combines adaptive feature weighting, temporal robustness, and normalization and aggregation restricted to valid frames.
- Single-stream Multi-feature Embedding and Fusion: Affective features are assigned to their dominant joints, enabling local geometric cues such as limb angles to participate in shallow-layer fusion.For limb connections, the angle feature is assigned to the middle joint.
- Single-stream Multi-feature Embedding and Fusion: Relative motion compares each joint with the root within the same frame, removing direct dependence on temporal intervals and providing a time-scale-invariant movement description.The motion feature includes linear velocity, linear acceleration, angular velocity, and angular acceleration components; the relative representation has 10 dimensions.
- Single-stream Multi-feature Embedding and Fusion: SV-GCN combines joint coordinates, motion, and affective features into an initial fused representation for single-stream processing.The fused representation contains 14 feature dimensions and is arranged over batches, channels, time steps, and joints.
- Single-stream Multi-feature Embedding and Fusion: Channel attention adaptively weights embedded features after LayerNorm and projection, using valid-frame masking during temporal and joint pooling.The projection includes a linear transformation with ReLU, while attention weights are generated channel-wise.
- Adaptive Alignment Valid Frame ST-GCN: VF-STGCN restricts temporal reductions and graph processing to valid frames, while masked GroupNorm avoids statistics from invalid frames and varying sequence lengths.Valid frames are extracted into a compact tensor before graph convolution, and the temporal mask is updated after pooling and downsampling.
- Adaptive Alignment Valid Frame ST-GCN: The backbone uses three VF-STGCN groups with channels 64, 128, and 256, followed by masked global average pooling and a fully connected classifier.Temporal downsampling uses stride 2 with successive kernel sizes 11, 7, and 3.
4 Experiments
Experiments evaluate feature embeddings, variable-length handling, and comparison with existing methods on the heterogeneous E-Gait dataset.
- Datasets: EG-1835 contains fixed-length samples, whereas EG-342 contains variable-length sequences with different class distributions and frame-rate conditions.EG-1835 has 1,835 samples; EG-342 has 342 samples with sequence lengths varying between 18 and 75 frames before padding.
- Experimental setup: The experiments report accuracy averaged over three runs using the authors’ code as the STEP baseline.Training uses the stated default optimization settings, including Adam and 200 epochs.
- Feature embedding ablation: Table 1 progressively adds joint coordinates, affective features, movement features, and channel-wise attention, with the complete configuration performing best.The evaluation uses the uniform EG-1835 subset.
- Sequence-length and frame-rate consistency: Table 2 compares fixed-length and variable-length configurations while progressively introducing VF-STGCN, intra-frame velocity, and smaller temporal kernels.The baseline uses inter-frame absolute velocity, BatchNorm, and a fixed kernel size of 9.
- Comparison with SOTAs: Competitive accuracy against state-of-the-art methods is achieved after pretraining on 43,515 NTU RGB+D samples, with significant improvement over STEP.The comparison uses the configuration from Table 2 row 8 and a 16-joint conversion of NTU RGB+D.
5 Conclusion
The paper concludes that SV-GCN addresses inefficient feature fusion, frame-rate sensitivity, and padding interference through joint-level fusion and valid-frame processing. On E-Gait, it achieves competitive performance and shows robustness across heterogeneous subsets, supporting transfer-learning potential from large-scale skeleton action data.
- Conclusion: SV-GCN combines SMEF for joint-level affective integration with VF-STGCN for globally masked variable-length sequence handling.The framework targets inefficient multi-feature fusion, frame-rate sensitivity, and padding interference.
- Conclusion: On E-Gait, SV-GCN achieves competitive performance and robust behavior across heterogeneous subsets.The conclusion links this robustness to potential transfer learning from large-scale skeleton-based action recognition datasets.
6 Supplementary Material
Supplementary attention analysis examines how the model weights pose, affective, and motion features.
- Attention analysis: Attention visualization assigns strong importance to joint-coordinate and affective channels, while linear motion channels retain moderate weights around 0.4.Channels 0–2 represent coordinates, channels 11–13 affective and angular features, and channels 3–10 linear motion features.
- Attention analysis: Affect-related joint rotation receives prominent attention, while per-joint channel attention is omitted to avoid noise from insufficient valid samples under GroupNorm.The normalization choice reduces the number of valid samples available for computing statistics.