PaperlayerView exact revision on arXiv

Source-linked AI summary

FedMHAR: Federated Multimodal Human Activity Recognition using Multi-Agent Reinforcement Learning

Debasmita Dey, Tanmay Sen, Himel Mallick

arXiv:2609.33492v2cs.AI

TL;DR

Fixed fusion weights overlook that sensor usefulness, quality, and acquisition cost can vary across samples. The paper uses reinforcement-learning policies for adaptive, cost-aware multimodal fusion and federated client optimization, reporting improved accuracy over centralized and federated baselines across two datasets.

  • Problem

    Fixed fusion weights do not account for differences in modality importance, acquisition cost, and per-sample reliability.

    Source: §10 · p. 25

  • Method

    PPO agents learn per-sample modality weights for cost-aware fusion, while BiFL-PPO learns client trust weights that guide federated aggregation and local optimization.

    Source: §1 · p. 2, §10 · p. 25

  • Results

    87.30% and 94.98% centralized accuracy and 79.74% and 77.49% federated accuracy, with the frameworks outperforming the reported centralized and federated baselines.

    Source: §1 · p. 3, §10 · p. 25

  • Takeaways & Limitations

    Learned, context-dependent sensor and client trust is associated with consistent accuracy and training-stability improvements across two datasets and five federated baselines, while reducing reliance on costlier sensors when cheaper alternatives suffice.

    Source: §10 · p. 25

    The clearest benefit from learned weighting occurs when exploitable heterogeneity exists, and testing under deliberately induced sensor degradation remains future work.

    Source: §10 · p. 25

Abstract

from arXiv · show

Human Activity Recognition (HAR) from heterogeneous wearable sensors is fundamental to the Internet of Health Things (IoHT), supporting rehabilitation, elderly care, and smart healthcare. Existing multimodal fusion methods often assign fixed equal weights to sensor streams, overlooking differences in modality importance, acquisition cost, and sensor quality, which can vary due to movement, incorrect placement, or temporary blockage. We propose an adaptive and cost-aware multimodal HAR framework based on multi-agent reinforcement learning for centralized HAR and extend it to federated learning as FedMHAR. In the centralized setting, multimodal fusion is formulated as a cooperative Multi-Agent Reinforcement Learning (MARL) problem, where each sensing modality is assigned a PPO-based agent that learns per-sample fusion weights, enabling the model to emphasize informative modalities while down-weighting costly sensors when cheaper alternatives provide sufficient information. In the federated setting, we introduce BiFL-PPO, a bidirectional federated optimization strategy in which a server-side PPO policy learns client-specific trust weights and feeds them back to adapt local learning rates and proximal regularization. Unlike round-level optimization, BiFL-PPO uses dense batch-level rewards for more frequent feedback and stable training under heterogeneous client data. Evaluation on the MEx Rehabilitation and UTD Multimodal Human Action datasets shows that the centralized framework achieves 87.30% and 94.98% accuracy, respectively, outperforming conventional fusion methods and state-of-the-art HAR models. FedMHAR achieves 79.74% and 77.49% in the federated setting, consistently surpassing FedAvg, FedProx, FedBN, FedNova, and AdaFedProx, while providing more stable performance and reducing sensor acquisition cost.

1 Introduction

The paper replaces fixed multimodal fusion weights with adaptive, cost-aware weighting and extends the approach to federated HAR through client-specific trust signals. Experiments report improvements over centralized and federated baselines, with more stable federated performance.

1 Introduction

  • Each modality has a PPO agent that learns per-sample fusion weights for modality-specific experts, while acquisition-cost rewards down-weight expensive sensors when cheaper ones suffice.The policy adapts to the current activity instance and sensing conditions.Source: §1 · pp. 2–3
  • Reinforcement learning frames fusion as sequential decision-making, allowing rewards to balance recognition accuracy against costly sensing and adapt weights to current conditions.Source: §1 · p. 2
  • FedMHAR introduces BiFL-PPO, whose server-side policy learns client-specific trust weights for aggregation and sends them to clients to adapt local learning rates and proximal regularization.The shared trust signal is intended to support stable optimization under heterogeneous data while preserving privacy.Source: §1 · pp. 2–3
  • The cost-aware reward incorporates normalized sensor acquisition cost and encourages lower-cost modalities without requiring supervision on sensor quality.Source: §1 · p. 3
  • The framework outperforms equal-weight fusion and centralized HAR methods on MEx Rehabilitation and UTD-MHAD, and improves on federated baselines with more stable performance across seeds.Source: §1 · p. 3

2 Related Work

Related work reviews HAR, deep reinforcement learning, federated learning, and Mixture-of-Experts methods. The paper positions its approach as adaptive multimodal fusion that combines per-sample weighting with sensor-cost awareness and federated optimization.

2.1 Human Activity Recognition

  • Recent HAR work uses deep architectures and multimodal fusion, while challenges including sensor noise, data imbalance, computational overhead, and cross-domain generalization remain active.Source: §2.1 · p. 3

2.2 Deep Reinforcement Learning in Human Activity Recognition

  • Prior DRL-based HAR methods focus on single-modality frame selection; MAPPO-MoE instead applies cooperative policies across wearable streams with cost penalization and a federated extension.Source: §2.2 · p. 3

2.3 Federated Learning

  • FedProx, FedNova, FedBN, and AdaFedProx address client drift, update-count bias, feature shifts, or adaptive proximal regularization; this work learns aggregation weights with a server PPO policy.The policy uses per-client validation signals and a Beta policy to generate adaptive weights.Source: §2.3 · p. 4

2.4 Mixture of Experts (MoE)

  • MoE architectures support dynamic expert specialization; this work uses PPO to weight modality experts per sample and encourages cheaper sensors when sufficient for classification.Source: §2.4 · p. 4

3 Background Concepts

MAPPO coordinates agents through a shared team reward and centralized training critic, while MoE combines specialized experts using input-dependent weights. Federated learning trains a shared model across clients without sharing raw data, and aggregation algorithms address heterogeneity in different ways.

3.1 Multi-Agent Proximal Policy Optimization

  • MAPPO uses a centralized critic with global state during training, while agents act from local observations during execution; its cooperative objective maximizes expected discounted team reward.Clipped policy updates limit excessively large changes, and entropy regularization encourages exploration.Source: §3.1 · pp. 4–5

3.2 Mixture of Experts

  • MoE combines outputs from specialized expert networks using input-dependent gating weights, allowing different experts to contribute differently to each sample.The gating network converts scores to normalized weights with softmax.Source: §3.2 · pp. 5–6

3.3 Federated Learning

  • Federated learning lets clients train a global model without sharing raw data; clients optimize locally and send model updates for server aggregation.The global model is broadcast again for the next communication round.Source: §3.3 · p. 6

3.4 Federated Aggregation Algorithms

  • Federated aggregation methods address heterogeneity differently: FedAvg uses data-size-weighted averaging, FedProx adds proximal regularization, FedNova normalizes updates, and FedBN keeps batch-normalization parameters local.AdaFedProx also uses proximal regularization, with client models aggregated by fixed or uniform averaging.Source: §3.4 · pp. 6–7

4 Proposed Methodology

The methodology combines adaptive, cost-aware multimodal recognition in a centralized setting with a federated extension for privacy-preserving collaborative learning. BiFL-PPO connects server aggregation and client-side optimization through learned client trust weights.

4 Proposed Methodology

  • The framework has two components: centralized adaptive, cost-aware multimodal HAR and a federated extension using BiFL-PPO.The centralized and federated architectures are shown in Figs. 1 and 2, respectively.Source: §4 · p. 7
  • In centralized fusion, modality-specific experts and MAPPO agents learn adaptive trust weights; in the federated setting, a server PPO agent learns client trust weights for aggregation and feedback-guided local optimization.This creates bidirectional interaction between the server and participating clients.Source: §4 · p. 7

4.1 Centralised Architecture

The centralized architecture extracts modality-specific features and predictions, then uses MAPPO agents to produce sample-specific weights for fusing expert outputs. Its reward balances improvement over equal-weight prediction with sensor acquisition cost.

4.1 Centralised Architecture

  • Independent 1D CNN encoders process each modality, and modality-specific MLP experts classify the resulting embeddings.The embeddings preserve modality-specific temporal characteristics and feed the corresponding experts.Source: §4.1 · p. 7
  • Experts are pretrained under equal-weight fusion, whose ground-truth-class probability serves as the reference confidence for the adaptive-fusion reward.The initial fused prediction uniformly averages expert logits.Source: §4.1 · pp. 7–8
  • Each modality has an independent MAPPO agent that learns a sample-specific fusion weight from the global multimodal representation.The representation concatenates embeddings from all modality-specific encoders, and every agent observes the complete state.Source: §4.1 · pp. 8–9
  • The agents' modality actions are normalized with softmax into fusion weights, which weight expert logits before a final softmax produces activity probabilities.The fused prediction uses the weighted combination of modality-specific expert logits.Source: §4.1 · p. 9
  • The dense reward measures prediction-confidence improvement over equal-weight fusion for every training sample, providing feedback that alleviates sparse rewards.A positive improvement is rewarded, while an inferior decision is penalized.Source: §4.1 · p. 9
  • A cost-aware reward penalizes larger weights on higher-cost modalities, with λ controlling the accuracy-cost trade-off and favoring cheaper sensors when comparable performance is possible.The normalized cost penalty lies in [0, 1].Source: §4.1 · p. 9

4.2 Proposed Federated Learning

BiFL-PPO assigns client trust from recent training signals and feeds those weights back into both aggregation and local optimization. A shared server policy learns trust through PPO, with smoothed signals and training-stability measures supporting the federated process.

4.2.1 Server State Representation

  • The server state summarizes each client’s smoothed accuracy and loss, gradient norm, and fixed data-share; accuracy and loss are smoothed to limit reactions to noisy rounds.The data-share is the fraction of total training data held by that client.Source: §4.2.1 · p. 9

4.2.2 Formulating Trust Assignment as Reinforcement Learning

  • The server formulates trust assignment as reinforcement learning: its state summarizes client behavior, its action samples a trust weight per client, and a reward evaluates the sampled assignment.A critic estimates expected discounted return and supports the GAE-based advantage used for PPO.Source: §4.2.2 · p. 10

4.2.3 Policy and Value Networks

  • A shared trunk and client-specific heads produce client trust distributions; the policy uses Beta distributions, which remain within a fixed range and can express confidence or uncertainty.The shared network builds a round-wide representation before each head specializes the trust belief for one client.Source: §4.2.3 · p. 11

4.2.4 Trust Computation and Bidirectional Feedback

  • The final trust blends the policy estimate with an accuracy-favoring prior, is smoothed across rounds, and has a minimum floor so no client is completely excluded.That trust is used for aggregation and fed back to shape local training.Source: §4.2.4 · p. 12
  • Trust weights jointly determine server aggregation and client-side learning rates and proximal penalties, while blending aggregation trust with each client’s data-share.Higher-trust clients receive a higher effective learning rate; lower-trust clients receive a stronger proximal penalty.Source: §4.2.4 · pp. 12–13

4.2.5 PPO Update

  • PPO updates follow three buffered rounds: GAE estimates advantages, the policy uses a clipped-surrogate objective with an entropy bonus, and the critic is trained by MSE regression.The reported settings are γ = 0.99 and λ = 0.95.Source: §4.2.5 · p. 13

4.2.6 Training Stability

  • Training stability measures preserve optimizer momentum across rounds, decay client learning rates to 30% of their initial value, and report a global-model exponential moving average with decay 0.7.The PPO-guided aggregation also blends learned trust with data-share while retaining each baseline’s local objective and aggregation structure.Source: §4.2.6 · p. 13

5 Experimental Setup

The experiments use MEx and UTD-MHAD to evaluate centralized and federated learning under a shared implementation and evaluation protocol. UTD-MHAD uses skeletal positions, derived joint velocity, and inertial signals, excluding RGB and depth.

5 Experimental Setup

  • The proposed centralized and federated models and baselines were implemented in Python 3.12 and PyTorch 2.12 on one NVIDIA Tesla T4 GPU; federated clients represented sensor modalities, with only expert classifier heads globally aggregated.Baselines were implemented under the same architecture and evaluation protocol.Source: §5 · p. 13

5.1 Dataset Description

  • The experiments use two datasets, MEx and UTD-MHAD.The section describes the datasets used in FedMHAR.Source: §5.1 · p. 13

5.1.1 MEx Dataset

  • MEx contains 30 subjects performing 7 physiotherapy exercises, with wearable accelerometers, a pressure mat, and a depth camera.The dataset is designed for exercise recognition and rehabilitation monitoring.Source: §5.1.1 · p. 13

5.1.2 UTD MHAD Dataset

  • UTD-MHAD contains 27 actions performed by 8 subjects; this work uses skeletal joint positions and inertial signals, excluding RGB and depth.Joint velocity is derived from normalized skeleton sequences and adds a motion-dynamics modality without requiring additional sensors.Source: §5.1.2 · pp. 13–14

6 Results & Discussion

The results compare adaptive MAPPO-based fusion in centralized HAR and PPO-based aggregation in federated learning across MEx and UTD-MHAD. Adaptive fusion improves centralized accuracy, while federated gains vary by dataset and aggregation baseline.

6.2.1 MEx Dataset

  • By round 50, PPO-enhanced variants improve over all five vanilla baseline aggregation methods; FedNova shows the largest gain (+8%, from 64% to 72%).FedAvg gains 5%, FedBN and AdaFedProx each gain 4%, and FedProx gains 1%.Source: §6.2.1 · p. 18
  • 79.74% mean test accuracy and 0.98% standard deviation: BiFL-PPO exceeds FedAvg’s 78.09% and 4.83% under a non-IID MEx split.BiFL-PPO outperforms FedAvg on four of five seeds; its server-learned client trust weights feed back as adaptive learning-rate and proximal-penalty signals.Source: §6.2.1 · pp. 18–19, Table 6 · p. 18

6.2.2 UTD MHAD Dataset

  • 96.05% peak accuracy on UTD-MHAD: the proposed federated method exceeds FedAvg (95.81%), AdaFedProx (92.79%), FedNova (91.63%), and FedDPO (91.16%).It reaches approximately 82% by round 2 and exceeds 91% by round 5, maintaining 93–96% throughout training.Source: §6.2.2 · pp. 20–21
  • On UTD-MHAD, Server PPO is non-deteriorating in 18 of 25 seed-method comparisons; FedNova improves from 90.19% to 91.02% while its standard deviation falls from ±1.82% to ±1.30%.FedAvg, FedProx, FedBN, and AdaFedProx change only marginally, within ±0.05%.Source: §6.2.2 · p. 21, Table 9 · p. 21

7 Baseline Results

The baseline comparisons report higher accuracy for MAPPO-MoE than the evaluated HAR models on both datasets. The gains are reported for MEx and the more heterogeneous UTD-MHAD dataset.

7 Baseline Results

  • The section compares FedMHAR’s performance with state-of-the-art federated methods.The supplied MEx and UTD-MHAD results report centralized MAPPO-MoE comparisons.Source: §7 · p. 22

7.1 MEx Dataset

  • 87.30% ± 2.53%: MAPPO-MoE achieves the highest accuracy among LSTM-CNN, MhaGNN, and TCN-Inception on MEx.The passage also reports lower standard deviation across random seeds.Source: §7.1 · p. 22, Table 10 · p. 22

7.2 UTD MHAD

  • 94.98% ± 0.88%: MAPPO-MoE outperforms all baselines on UTD-MHAD and has the lowest variance across seeds.The passage describes UTD-MHAD as the more heterogeneous dataset and attributes the gain to adaptive modality weighting of complementary sensor information.Source: §7.2 · p. 22, Table 11 · p. 22

8 Ablation Study

The modality ablations examine when adaptive expert weighting improves centralized fusion. Across the two datasets, the reported gains are strongest when multiple complementary modalities can be coordinated.

8 Ablation Study

  • With all four MEx modalities, MAPPO improves accuracy from 0.84 to 0.91 and reduces standard deviation from ±0.04 to ±0.01.Its benefit increases with the number of available modalities; single-modality settings show little benefit, while multi-modality settings show consistent improvements, including +0.06 for act+pm and +0.09 for a challenging three-modality subset without dc.Source: §8 · pp. 22–23
  • The reported ablations indicate that MAPPO is most effective when multimodal expert coordination can exploit complementary information.The MEx results describe little benefit when only one modality is available.Source: §8 · p. 23
  • On UTD-MHAD, inertial alone scores 86.28% ± 0.71, while all three modalities achieve the highest Baseline MoE accuracy, 93.40% ± 0.74.MAPPO improves six of seven settings, including inertial+skeleton from 91.91% to 94.19% (+2.28%) and the full-modality configuration from 93.40% to 95.35% (+1.95%); joint velocity alone is the sole drop.Source: §8 · pp. 23–24

9 Error Analysis

The confusion analysis shows slight reductions in residual errors for similar actions, with most misclassifications concentrated in a few kinematically similar pairs.

9 Error Analysis

  • Accuracy rose from 95.58% to 96.05% and weighted F1-score from 95.49% to 95.88% for MAPPO-MoE versus Baseline MoE on UTD-MHAD.Fig. 12 shows the normalized confusion matrices; remaining errors cluster among similar actions, including catch, knock, throw, bowling, and tennis swing.Source: §9 · p. 24
  • Catch accounts for 7 of 18 errors, while Draw triangle and Knock each account for 4, with errors concentrated among similar action pairs.Draw triangle errors map entirely to Draw circle (CCW); Knock is confused with Clap in 3 of 4 errors.Source: §9 · pp. 24–25
  • Throw contributes 2 errors and Tennis-swing 1, while the concentrated error pattern accompanies accuracy of 95.81% and weighted F1-score of 95.88%.The authors describe the Draw triangle–Draw circle and Knock–Clap confusions as plausibly related to kinematic similarity.Source: §9 · p. 25

10 Conclusion and Future Work

The conclusion presents adaptive, cost-aware fusion for centralized and federated HAR, reporting improvements across two datasets and federated baselines. It identifies evaluation under degraded sensors and broader federated and clinical settings as future work.

10 Conclusion and Future Work

  • Centralized PPO agents learn per-modality, per-sample fusion weights, while BiFL-PPO uses server-learned client trust weights to guide aggregation and client-side optimization.The framework aims to emphasize informative sensors and down-weight costlier ones when cheaper modalities suffice.Source: §10 · p. 25
  • Centralized MARL achieved 87.30% and 94.98% accuracy on MEx Rehabilitation and UTD-MHAD, while FedMHAR achieved 79.74% and 77.49%, surpassing five federated baselines.The federated baselines are FedAvg, FedProx, FedBN, FedNova, and AdaFedProx.Source: §10 · p. 25
  • The authors report consistent accuracy and training-stability gains across two datasets, centralized and federated learning, and five federated baselines, alongside reduced reliance on costlier sensors.The stated conclusion concerns learned, context-dependent sensor and client trust rather than fixed assumptions.Source: §10 · p. 25
  • Future evaluation should test deliberate sensor degradation or corruption, newer RL-based federated baselines, more clients and rounds, and longitudinal clinical populations.The current comparison covers five representative federated baselines, and the authors identify clean, well-controlled inputs as a setting needing further testing.Source: §10 · p. 25

Funding

The funding statement identifies support for Debasmita Dey’s postdoctoral fellowship project.

Funding

  • Debasmita Dey’s ANRF India National Post-Doctoral Fellowship project grant was administered by the Indian Statistical Institute, Kolkata.The project number is PDF/2026/002233.Source: Funding · p. 26

Related papers

Loading 2609.33492v2…