Source-linked AI summary
Meta-HAR: Federated Representation Learning for Human Activity Recognition
Chenglin Li, Di Niu, Bei Jiang, Xiao Zuo, Jianming Yang
TL;DR
Privacy concerns make centralized collection of labeled sensor data problematic, while federated HAR faces accuracy challenges with decentralized users. Meta-HAR meta-learns shared signal representations and adapts personalized classifiers locally; experiments report that it outperforms baselines, including on held-out users.
Problem
Central collection of labeled mobile-sensor data raises privacy concerns, and FedAvg shows activity-prediction accuracy degradation relative to Central models.
Source: §1 · p. 1, §4.4 · p. 7Method
Meta-HAR federatively meta-trains a shared embedding network and adapts a user-specific classification model on top of its representations.
Source: §1 · p. 2, §3.1 · p. 4Results
24.7%: Meta-HAR outperformed FedReptile on merged-dataset meta-train users; across the evaluated datasets, it significantly outperformed all baselines.
Source: §6 · p. 10Takeaways & Limitations
The newly collected dataset is publicly available to facilitate future development in sensor-based human activity recognition.
Source: §6 · p. 10Without collecting user data, identifying the global activity set and aligning labels across users can be difficult and time-consuming.
Source: §2.1 · pp. 3–4
Abstract
from arXiv · showhide
Human activity recognition (HAR) based on mobile sensors plays an important role in ubiquitous computing. However, the rise of data regulatory constraints precludes collecting private and labeled signal data from personal devices at scale. Federated learning has emerged as a decentralized alternative solution to model training, which iteratively aggregates locally updated models into a shared global model, therefore being able to leverage decentralized, private data without central collection. However, the effectiveness of federated learning for HAR is affected by the fact that each user has different activity types and even a different signal distribution for the same activity type. Furthermore, it is uncertain if a single global model trained can generalize well to individual users or new users with heterogeneous data. In this paper, we propose Meta-HAR, a federated representation learning framework, in which a signal embedding network is meta-learned in a federated manner, while the learned signal representations are further fed into a personalized classification network at each user for activity prediction. In order to boost the representation ability of the embedding network, we treat the HAR problem at each user as a different task and train the shared embedding network through a Model-Agnostic Meta-learning framework, such that the embedding network can generalize to any individual user. Personalization is further achieved on top of the robustly learned representations in an adaptation procedure. We conducted extensive experiments based on two publicly available HAR datasets as well as a newly created HAR dataset. Results verify that Meta-HAR is effective at maintaining high test accuracies for individual users, including new users, and significantly outperforms several baselines, including Federated Averaging, Reptile and even centralized learning in certain cases.
1 INTRODUCTION
Privacy constraints make centralized collection of labeled sensor data difficult, while federated HAR must contend with users’ differing activity labels and signal distributions. Meta-HAR meta-trains a shared representation across users, then adapts a personalized classifier locally; experiments report strong results for existing and new users.
1 INTRODUCTION
- Each user’s activity labels may differ, and signal distributions can also vary across users even for the same activity.These label and signal heterogeneities are identified as obstacles for federated HAR.Source: §1 · p. 2
- A single federated global model may not adapt or generalize well to individual users, especially new users or users with newly introduced activities.The paper argues that personalized models should combine insights from users with predictors fitted to local data.Source: §1 · p. 2
- Meta-HAR combines a shared, federatively meta-trained embedding network with a device-specific classification network for personalized activity prediction.The framework uses heterogeneous data across devices for representation learning and local prediction for personalization.Source: §1 · p. 2
- The shared embedding is trained by treating each device as a task and using a federated version of Reptile with pairwise similarity loss.The loss encourages same-class samples to cluster in the embedded space and different-class samples to separate.Source: §1 · p. 2
- Local adaptation fine-tunes the embedding with pairwise loss, then jointly fine-tunes it with a client-specific output layer for classification.The authors report that this adaptation can work with small local datasets and can sometimes outperform a centrally trained model, especially for new users.Source: §1 · p. 2
- Meta-HAR achieved 92.5%/98.39% test accuracy for HHAR meta-test/meta-train users and 91.07%/93.79% for USC-HAD equivalents, and significantly outperformed FedReptile on merged datasets.The merged stress test combined the public datasets into 23 users and seven activities, with five overlapping activities.Source: §1 · p. 2
2 PROBLEM AND MOTIVATION
The section frames federated HAR as per-user activity recognition without central sensor-data collection, but users can have sparse activity sets and distinct signal distributions. Its motivating experiments and discussion highlight accuracy loss under FedAvg and practical difficulty establishing a global activity set and aligned labels.
2 PROBLEM AND MOTIVATION
- Without collecting other users’ data, local supervised learning cannot leverage their samples and may be unable to predict activities absent from a user’s sparse local dataset.The problem is framed as activity recognition for each individual user, whose observed activity set may differ from others'.Source: §2 · p. 3
2.1 Challenges to Federated HAR
- FedAvg may perform poorly when users have different activity labels and different input-signal distributions, including for the same activity.The section notes that signal heterogeneity in HAR had not been reported in prior literature.Source: §2.1 · p. 3
- FedAvg-User accuracy falls from over 97% for Central to below 80%, while FedAvg-Shuffle performs close to Central; Figure 1 reports the comparison.The HHAR setup has IID activity labels, so the user-versus-shuffle comparison isolates heterogeneity in signal distributions.Source: Figure 1 · p. 3, §2.1 · p. 3
- Figure 2’s HHAR visualization shows samples of the same “Stair-down” activity clustering by user, despite activities clustering within one user.The comparison illustrates distinct signal distributions across users for a shared activity.Source: Figure 2 · p. 3, §2.1 · p. 3
- Without data collection, identifying the global activity set and aligning labels across users can be difficult and time-consuming.The global activity set determines the output dimension of a global classifier, and a large class set makes classification harder.Source: §2.1 · pp. 3–4
3 METHODS
Meta-HAR learns a shared signal embedding across users while leaving activity classification to user-specific output layers. Its method combines federated meta-training, pairwise embedding learning, and local personalization.
3 METHODS
- Meta-HAR learns a common signal representation across users and feeds it to separately learned classifiers suited to each user's activity set.Source: §3 · p. 4
3.1 Federated Representation Learning
- Federated Reptile treats each user as a task and meta-trains the shared embedding network across decentralized users.Users pull the current embedding parameters, train locally, and send updated parameters to the server for aggregation.Source: §3.1 · p. 4
3.2 Embedding Network Architecture and Local Training
- A CNN-RNN embedding network processes sensor signals, while pairwise loss trains it to cluster same-class embeddings and separate different-class embeddings.The architecture uses convolutional layers to model relationships within and across sensors, then LSTM layers to extract temporal relevance and produce a fixed-length embedding.Source: §3.2 · pp. 5–6
3.3 Personalized Classification
- Each user adapts the global embedding with pairwise loss, then fine-tunes it jointly with a user-specific output layer using cross-entropy.The output layer conforms to the user's local activity set and output dimension.Source: §3.3 · p. 6
4 EXPERIMENTS
The experiments evaluate Meta-HAR on public, merged, and collected HAR datasets against centralized and federated baselines. The results report strong personalized performance, particularly for meta-test users, alongside a limitation when local data are scarce.
4.1 Datasets
- The public HHAR and USC-HAD datasets were down-sampled and segmented into fixed-length samples; removing up to two activities per user introduced label and signal heterogeneity before the datasets were merged.The merged dataset contains 23 users and seven activities.Source: §4.1 · p. 6
- The newly collected dataset includes 48 users and six activities, with uncontrolled activity performance and greater heterogeneity in devices, phone positions, signal distributions, and sampling frequencies.The authors describe it as noisier and more heterogeneous than datasets collected in controlled environments.Source: §4.1 · pp. 6–7
4.2 Experiment Setups
- Experiments held out users for meta-testing, split each user’s data 80/20 for training and testing, compared Central, FedAvg, FedReptile, Meta-HAR, and Meta-HAR-CE, and repeated each dataset evaluation five times.The compared personalization strategies included Separated, Merged, and the proposed Two-stage method.Source: §4.2 · p. 7
4.3 Implementation Details
- The model combines 1-D and 2-D convolutional layers with two LSTM layers and a 100-dimensional latent vector; local training uses batches of 64.The implementation used Adam and dropout at a rate of 0.3.Source: §4.3 · p. 7
4.4 Experimental results
- Personalized Meta-HAR and FedReptile models were comparable to Central on meta-train users and outperformed it on meta-test users, while FedAvg degraded versus Central across datasets.On USC-HAD meta-train users, the best personalized models were 5.33% below Central; the authors attribute this to users’ small local datasets.Source: §4.4 · pp. 7–8, Table 1 · p. 8
- Meta-HAR generally outperformed Meta-HAR-CE on meta-test users, though it was slightly worse on meta-train users; the authors also report scalability to new activity types for Meta-HAR but not Meta-HAR-CE.The paper attributes these advantages to using pairwise loss rather than cross-entropy to train the embedding network.Source: §4.4 · p. 8
- On the merged dataset, Meta-HAR reached 95.35% versus FedReptile’s 70.65% on meta-train users, and 75.83% versus 69.4% on HHAR meta-test users and 90.23% versus 74.83% on USC-HAD meta-test users.The authors report close performance between Meta-HAR and FedReptile on simpler public data, but a larger Meta-HAR advantage on more heterogeneous data and meta-test users.Source: Table 2 · p. 8, §4.4 · p. 8
- The proposed Two-stage strategy significantly outperformed Merged and Separated fine-tuning on meta-test users, while Merged performed better than Separated.The comparison used the merged dataset; the authors report that the other strategies’ improvement was insufficient for fast adaptation to new users.Source: §4.4 · pp. 8–9, Table 3 · p. 9
5 RELATED WORK
Prior work spans deep-learning methods for HAR, federated optimization, and personalization. Meta-HAR is positioned as addressing federated HAR’s signal heterogeneity and personalization needs, including users with new activity types.
Deep HAR and federated optimization
Federated HAR and personalization
- Meta-HAR combines a global shared representation with personalization, addressing signal heterogeneity that prior federated HAR work did not tackle and enabling different output dimensions across clients.The authors contrast their approach with Sozinov et al.’s application of FedAvg to HAR and with prior personalization work.Source: §5 · pp. 9–10
Federated multi-task and device-selection approaches
- MOCHA addresses general federated multi-task learning but does not discuss heterogeneous input signals or adaptation to new users; FAVOR targets participant data heterogeneity but does not provide personalized models for all users.MOCHA lacks a global shared representation model, while FAVOR uses reinforcement learning for device selection.Source: §5 · p. 9
Meta-learning
Personalized learning
- Peer-to-peer personalization relies on client relationships that the authors consider unrealistic at federated scale, where relationships may be complicated and dynamic.The authors note that newly generated mobile data are often consumed locally.Source: §5 · p. 9
6 CONCLUSIONS
Meta-HAR meta-learns a shared embedding network across distributed users and adapts user-specific output layers for personalized activity classification. Experiments report gains over all baselines, including FedReptile, across the merged dataset’s user groups.
Framework
- Meta-HAR meta-learns a federated embedding network from heterogeneous user datasets, then obtains personalized models with user-specific output layers through adaptation.The paper’s stated goal is accurate personalized activity classification without centrally collecting users’ sensor data.Source: §6 · p. 10
Results
- On the merged dataset, Meta-HAR outperformed FedReptile by 24.7% on meta-train users, 6.43% on HHAR meta-test users, and 15.4% on USC meta-test users.The paper reports that its approach significantly outperformed all baseline methods across its experiments.Source: §6 · p. 10
Evaluation and dataset
- Experiments cover one newly collected dataset, two public datasets, and their merged dataset; the newly collected dataset is made publicly available.Source: §6 · p. 10
A COLLECTED DATASET
The authors collected mobile sensor data with an Android logging app under users’ freely performed activities, retaining a heterogeneous set of 48 users. They filtered and standardized samples, trained on naturally unbalanced local data, and evaluated with balanced test data.
Data collection
- The app offered nine activity choices, but enough samples were collected for only six: walking, biking, upstairs, downstairs, running, and taking bus.Figure 5 shows the logging app; the number after each action type is its collected and uploaded sample count.Source: A COLLECTED DATASET · p. 11, Figure 5 · p. 11
- Participants started and stopped each activity recording in the app, which then uploaded the collected sensor signals to a pre-assigned server.Source: A COLLECTED DATASET · p. 11
Preprocessing
- Each sample uses seven seconds of consecutive signals; the first five and last seven seconds are discarded, and only sequences of 150–200 points are retained before truncating them to 150.The sampling rate was set to 25 Hz, but device and operating-system differences produced variable rates.Source: A COLLECTED DATASET · p. 11
Data collection
- Users performed activities freely rather than in a controlled setting, so the dataset includes heterogeneity in activity types, devices, class balance, phone position, signal distributions, and sampling rates.Participants submitted short videos of their surroundings and actions to help verify the selected activity.Source: A COLLECTED DATASET · p. 11
Dataset composition
- The dataset retains 48 users, each with at least three activity types, and no user provides all six selected activities.Walking and taking bus/taxi are common, while upstairs, downstairs, and running have fewer samples.Source: A COLLECTED DATASET · p. 11
- Figure 6 depicts the Stair-down sample distribution across HHAR users and the activity distributions for an individual user.Source: Figure 6 · p. 11
Evaluation
- Local training preserves the collected class imbalance, while model evaluation uses a balanced test set across activity types.The authors retain the imbalance because user-side data distributions cannot be controlled in real-world federated learning.Source: A COLLECTED DATASET · p. 11