Toward Postural State Classification in Immersive VR with Multimodal Data and Explainability Analysis

Nipa Anjum; Md Irfan Pavel; Robert Gonzalez; Kevin Desai; Alberto Cordova; M. Rasel Mahmud; John Quarles

Source-linked AI summary

Toward Postural State Classification in Immersive VR with Multimodal Data and Explainability Analysis

Nipa Anjum, Md Irfan Pavel, Robert Gonzalez, Kevin Desai, Alberto Cordova, M. Rasel Mahmud, John Quarles

arXiv:2608.28844v1Aug 28, 2026cs.HCcs.AIcs.LG

TL;DR

Postural state classification in VR must address imbalance, participant variability, class imbalance, and limited interpretability for safety-sensitive applications. This study compares ML and DL models using multimodal sensor data under participant-independent evaluation, with MI-CNN achieving the strongest overall performance and SHAP identifying dominant kinematic features. The findings support interpretable classification of VR-related instability, while generalization remains limited by the small, healthy, controlled-scenario dataset.

Problem
Postural state classification in VR faces class imbalance, diverse balance strategies, limited generalizability to new individuals, and insufficient interpretability for clinical and rehabilitation use.
Method
The study compares classical ML and DL architectures for binary balanced-versus-imbalanced classification using multimodal sensor data, participant-wise preprocessing, LOPO evaluation, and SHAP-based interpretation.
Results
MI-CNN achieved the best overall performance under LOPO evaluation, while SHAP analysis identified trunk, lumbar, and upper-body kinematic features as the most important classification factors.
Takeaways & Limitations
SHAP-guided feature prioritization can preserve classification performance while identifying biomechanically meaningful motion features for future sensor and feature prioritization.
Takeaways & Limitations
The dataset included only eleven healthy young adults and controlled visual perturbations, limiting direct generalization to older adults, clinical populations, and spontaneous real-world imbalance.

Abstract

from arXiv · show

Ensuring a safe virtual reality (VR) experience requires systems that can predict and respond when users lose their balance. Although prior work has examined fall prediction and motion sickness, many approaches are regression-based and postural state classification remains less explored. This study compares machine learning (ML) and deep learning (DL) models for classifying postural states in VR under visual perturbations. We used a multimodal dataset containing kinematic, electromyographic (EMG), and electrodermal activity (EDA) signals. The data were prepared for a binary task to distinguish balanced from imbalanced postural states, and participant-wise downsampling addressed class imbalance. All models were evaluated with Leave-One-Participant-Out (LOPO) cross-validation to test generalization to unseen participants. Among the models, the Mamba-inspired CNN (MI-CNN) achieved the highest accuracy of 96.76%. SHapley Additive exPlanations (SHAP) analysis improved interpretability and identified the most influential classification factors. The SHAP results showed that kinematic features were dominant, indicating that body-motion patterns are informative for detecting imbalance in VR. We also evaluated MI-CNN using only the top two-thirds of features ranked by SHAP importance. Despite a 33% reduction in input dimensionality, the model maintained performance, achieving 0.957 accuracy and 0.957 F1-score, with about a 1% decrease compared with the full-feature model. These findings suggest that multimodal sensing, temporal deep learning, and explainable AI can support reliable classification of balance-related instability in VR. Accurate recognition of imbalanced postural states may raise awareness of fall risk and guide safer, adaptive VR systems that respond to instability while improving user safety and experience. Code is available at: https://github.com/NipaAnjum/MI-CNN.

1 INTRODUCTION

The study addresses the underexplored problem of classifying balanced versus imbalanced postural states in immersive VR, where visual perturbations can threaten stability. It compares ML and DL models under participant-independent evaluation and uses SHAP to identify influential features and assess feature reduction.

Visual and vestibular conflicts during VR perturbations can induce imbalance or falls, making early detection relevant to user safety.
Postural-state classification can support rehabilitation assessment and adaptive VR systems for users facing increased balance demands.
Class imbalance, diverse balance strategies, and limited interpretability complicate reliable generalization to unseen users.
The study compares classical ML and DL models for VR perturbation-induced postural-state classification using LOPO evaluation.
Participant-wise performance analysis quantifies inter-individual variability across unseen users.
SHAP identifies biomechanically meaningful motion features and evaluates whether performance can be preserved after feature reduction.

2 RELATED WORK

Prior research has used VR for balance assessment, intervention, and related safety tasks, while postural-state classification in immersive VR remains comparatively limited. Explainability methods motivate identifying which sensor features drive imbalance classification and can inform future data collection and adaptive systems.

Research on VR balance assessment, predictive modeling, and explainability has largely progressed separately, leaving their integration limited.
VR can reproduce balance challenges in controlled settings through immersive perturbations, tracking, feedback, and adaptable training tasks.
Earlier balance-prediction studies used sensor data but faced limitations including class imbalance, overfitting, larger errors for lower scores, or narrow datasets and conditions.
Prior work has mainly studied clinical or home fall risk and VR safety factors rather than balanced-versus-imbalanced postural-state classification in immersive VR.
Explainability research has used SHAP to identify important inertial, gait, and health features for fall detection and risk prediction.
Applying SHAP to VR postural classification can reveal influential sensor data and balance measures for experimental design and dataset development.

3 METHODOLOGY

The methodology combines multimodal VR recordings, participant-wise preprocessing and balancing, sliding-window inputs, classical ML and temporal DL models, and LOPO evaluation. MI-CNN is a Mamba-inspired architecture designed for multivariate time-series classification, while SHAP supports feature interpretation.

The study uses fixed-length sliding windows and evaluates all models under the same windowing configuration for fair comparison.Each window contains 100 consecutive samples at 60 Hz, representing approximately 1.67s, with a 50-sample stride.
The dataset records multimodal responses to unpredictable visual perturbations including scene motion, directional translations, object motion, and height-related scenarios.
After removing duplicate or inconsistent data and resolving participant duplication, the dataset contains 11 participants and 1001 synchronized features per time sample.The features comprise 982 Xsens, 12 Shimmer, and 7 EMG channels.
Participant-wise downsampling addresses the original imbalance by grouping perturbed trials as imbalanced and non-perturbed trials as balanced.
The comparison includes Logistic Regression, Random Forest, XGBoost, LightGBM, ExtraTrees, LSTM, TCN, CNN-LSTM, and MI-CNN models.
MI-CNN combines Mamba’s split-gate-project structure with depthwise causal convolutions and gating for lightweight multivariate time-series classification.Its input window has shape 100 × 1001, is projected to d_model = 64, and passes through four stacked Mamba blocks.

4 RESULTS

LOPO evaluation showed that temporal deep models outperformed classical baselines, with MI-CNN achieving the strongest overall performance and robustness across participants and evaluation settings. SHAP analyses identified motion-related features as dominant, while reducing the feature set preserved strong classification performance.

LOPO Cross-Validation Performance: MI-CNN achieved the highest performance, with F1 = 0.968±0.015 and ROC-AUC = 0.996±0.003.Temporal models outperformed classical baselines, including Logistic Regression at F1-score 0.627 and ExtraTrees at 0.860.
Robustness Analyses: 0.921 ± 0.038 accuracy was achieved under leave-one-scenario-out evaluation, while non-overlapping windows produced 0.954 ± 0.028 accuracy.The lowest scenario-specific result was 0.855 accuracy for the fall-from-height/vertigo condition.
Global SHAP Explanation: Global SHAP identified trunk motion, lumbar orientation, upper-body kinematics, and gyroscope measures as primary classification drivers.Balanced-state predictions emphasized upper-body kinematics and gyroscope signals, whereas imbalanced-state predictions emphasized lumbar orientation, trunk bending, and gyroscope-related measures.
Local SHAP Explanation: Local SHAP explanations showed different influential features for balanced and imbalanced predictions.Balanced examples emphasized knee rotation, shoulder, and foot positioning; imbalanced examples emphasized lower-leg orientation, lumbar flexion, neck orientation, and skin conductance.
Sensor-Level Contribution: Full-body kinematic signals contributed the largest share of modality-level predictive importance.The modality comparison matched feature counts to the smallest sensor group, EMG, when aggregating SHAP importance.
SHAP-Guided Feature Reduction: 95.68% F1-score and 99.24% AUC were achieved using the top two-thirds of SHAP-ranked features.The bottom one-third alone also achieved 94.16% F1-score and 99% AUC, indicating that reduced feature sets retained strong performance.

5 DISCUSSION

Temporal deep models, especially MI-CNN, performed strongly for multimodal postural-state classification under LOPO evaluation. SHAP analysis linked classification primarily to trunk and upper-body kinematics, while feature reduction preserved strong performance.

Model comparison: Temporal architectures, particularly MI-CNN, outperformed classical baselines under LOPO evaluation by modeling sequential dependencies in multimodal sensor data.MI-CNN also maintained strong performance with non-overlapping windows and unseen perturbation scenarios.
Generalization: Fall-from-heights/vertigo scenarios produced the lowest performance, indicating more heterogeneous or scenario-specific postural responses.
Feature reduction: Using only the top two-thirds of SHAP-ranked features maintained comparable performance, suggesting redundancy in the full feature space and supporting more efficient implementations.
SHAP interpretation: Trunk, lumbar, and upper-body kinematic features dominated SHAP importance, while Shimmer gyroscope signals contributed complementary information and EMG features had comparatively lower influence.Individual predictions could instead emphasize shoulder or hip orientation.
Computational feasibility: MI-CNN used 256,066 trainable parameters and a 3.07 MB model size, with warm-up inference times of 3.72 ± 0.46 ms on GPU and 9.45 ± 1.21 ms on CPU.End-to-end online performance would additionally depend on sensing, synchronization, preprocessing, and VR-engine communication.

6 LIMITATIONS AND FUTURE WORK

The study is an initial exploration based on a small, controlled sample and offline evaluation. Broader validation is needed before extending the findings to real-world VR instability or diverse populations.

Scope: The dataset included only eleven healthy young adults exposed to controlled visual perturbations, limiting representation of spontaneous real-world VR imbalance.
Future validation: Generalization to older adults, clinical populations, and slips, trips, or falls requires further validation.
Deployment: The offline evaluation did not assess end-to-end sensor streaming or sustained real-time operation.
Participant factors: The limited participant count prevents determining whether age, height, or weight systematically affects model performance.

7 CONCLUSION

The study classifies balanced versus imbalanced postural states in immersive VR using multimodal sensing, comparative ML/DL evaluation, and SHAP interpretability. MI-CNN performed best, while SHAP-guided reduction retained strong performance with fewer features.

Framework: The framework distinguishes balanced and imbalanced postural states in immersive VR using multimodal sensor data.
Evaluation: Several ML and DL models were evaluated under LOPO cross-validation to assess robustness across individuals.
Performance: MI-CNN achieved the best overall performance among the comparison models, with statistically significant improvements over the alternatives.
Interpretability: SHAP identified trunk, lumbar, and upper-body kinematic features as the most important classification factors.
Feature reduction: Using only the top two-thirds of SHAP-ranked features retained strong performance with a marginal decrease, supporting more efficient sensor setups.