PaperlayerView exact revision on arXiv

Source-linked AI summary

HALO: A Heterogeneity-Aware Language-Aligned IMU Foundation Model for Open-Set Human Activity Recognition

Zihan Ding, Liyu Zhang, Xiaomin Ouyang

arXiv:2608.27233v1cs.LG

TL;DR

HAR lacks a unified model for heterogeneous sensors and unseen activities or label vocabularies. HALO addresses this with heterogeneity-aware IMU pretraining and synonym-aware text alignment, outperforming five baselines across eight aggregate metrics while remaining vulnerable to severe distribution shift.

  • Problem

    HAR still lacks unified models that generalize across diverse sensors, devices, activities, and unseen label vocabularies.

  • Method

    HALO uses two-stage heterogeneity-aware IMU pretraining and synonym-aware soft contrastive alignment with text for cosine-similarity open-set recognition.

  • Results

    HALO outperforms five state-of-the-art baselines across all eight aggregate metrics on held-out datasets.

  • Takeaways & Limitations

    HALO provides one pretrained model for diverse IMU configurations and open-vocabulary activity recognition without per-dataset adaptation.

  • Takeaways & Limitations

    Robustness to severe distribution shift remains an open challenge, particularly for HARTH and VTT-ConIoT under zero-shot transfer.

Abstract

from arXiv · show

Human Activity Recognition (HAR) using inertial measurement units (IMUs) enables a wide range of applications, yet the field still lacks a unified model that can generalize across diverse subjects, devices, and activities. Training such a model is difficult due to two key challenges: sensing heterogeneity -- differences in sampling rates, channel configurations, and sensor placements -- and poor generalization to unseen activities and label vocabularies. We introduce HALO (Heterogeneity-Aware Language-aligned Open-set model), a domain-specific IMU foundation model that addresses both challenges through a two-stage training framework. Stage 1 pretrains the IMU encoder with heterogeneity-aware self-supervised learning, including adaptive-pooling tokenization, channel-independent feature extraction, and contextualized sensor conditioning that injects natural-language sensor descriptions into each channel embedding. Stage 2 aligns this IMU encoder with text embeddings via synonym-aware soft contrastive learning, enabling open-set recognition via cosine-similarity retrieval without per-dataset classifiers. Trained on 10 public HAR datasets and evaluated on 7 held-out datasets, HALO outperforms five state-of-the-art baselines on all 8 aggregate metrics, and still leads on 3 of 4 settings under baseline-matched inputs. Despite using only ~35M trainable parameters -- 10x fewer than the latest foundation model MOMENT (341.2M) -- HALO improves zero-shot open-set accuracy, measured over all 87 training labels, by 13.7 percentage points. On two further datasets with severe distribution shift, every model including HALO collapses zero-shot. A video demonstration of HALO's performance in real world is available at https://youtu.be/rooVKragtFU

1 INTRODUCTION

HALO targets unified HAR across heterogeneous sensors and unseen activities by combining heterogeneity-aware IMU pretraining with language alignment. It is evaluated across multiple datasets and settings against existing baselines.

  • HALO addresses sensing heterogeneity and limited generalization across subjects, devices, activities, sampling rates, sensor suites, and label vocabularies.
  • HALO accepts diverse IMU configurations and recognizes activities through open-vocabulary cosine-similarity retrieval without per-dataset classifiers or architectural modification.
  • Stage 1 combines adaptive-pooling tokenization, channel-independent processing, sensor conditioning, masked autoencoding, and contrastive learning for heterogeneity-aware IMU representations.
  • Stage 2 aligns IMU features with text semantics using synonym-aware soft contrastive learning.
  • HALO is trained on 10 public HAR datasets and evaluated on seven held-out datasets across zero-shot and supervised fine-tuning settings against five state-of-the-art baselines.
  • HALO adapts effectively with as little as 1% labeled data, recovering to the strongest supervised performance on the harder of two datasets.

2 RELATED WORK

Related HAR work improves representation learning, heterogeneity handling, or language alignment, but existing approaches generally retain fixed sensing assumptions or closed-world classifiers. HALO occupies a domain-specific middle ground between narrow HAR models and broad general-purpose time-series foundation models.

  • Existing HAR systems address cross-user, cross-device, multimodal, or federated learning challenges but remain tied to fixed channel counts, sampling rates, and closed-world labels.
  • Self-supervised methods reduce dependence on labeled IMU data through contrastive learning and masked reconstruction, while CrossHAR combines both for cross-dataset HAR.
  • MOMENT provides broad time-series coverage with 341.2M parameters, whereas HALO uses HAR-specific inductive biases for wearable sensing and language-based label interfaces.
  • Language-grounded models enable open-vocabulary classification, but ImageBind is not designed for cross-dataset HAR or variable sensor configurations, and LanHAR aligns sensor representations with generated activity descriptions.

3 MOTIVATION

HAR datasets vary substantially in sampling rates, channel configurations, placements, and activity vocabularies, while fixed closed-world classifiers limit transfer across these conditions. These observations motivate rate-agnostic processing, variable-channel handling, and semantic label alignment.

  • The benchmark spans 20–100 Hz, 3–45 channels, and 4–19 activity labels across 14 HAR datasets.
  • The benchmark includes configurations from a 3-axis smartwatch or phone sensor to multi-sensor body suits, producing substantially different raw data for the same activity.
  • Sampling rates from 20 Hz to 100 Hz make fixed timestep-indexed positional encodings conflate physical time with sequence length.
  • Channel counts from 3-axis wrist accelerometers to 51-channel body suits make fixed-width input layers impractical without padding artifacts or dataset-specific heads.
  • Closed-world classifiers fail when label vocabularies differ or test activities are genuinely novel and absent from training labels.
  • Synonym variation such as “jogging” and “running” can produce contradictory gradients under one-hot supervision despite similar motions.
  • The resulting requirements are arbitrary-rate, variable-channel, and variable-placement processing; open-vocabulary recognition; and semantic alignment of equivalent labels.

4 SYSTEM OVERVIEW

HALO uses a two-stage pipeline: heterogeneity-aware IMU pretraining followed by synonym-aware alignment with text. At inference, it retrieves labels by cosine similarity from a selectable label inventory without a per-dataset classifier.

  • HALO is an IMU-to-text semantic alignment model trained across heterogeneous HAR datasets for open-set, zero-shot activity recognition.
  • Stage 1: Heterogeneity-aware pretraining: Stage 1 maps variable-rate, variable-channel sessions to fixed embeddings using adaptive pooling, channel-independent processing, sensor conditioning, masked autoencoding, and patch-level contrastive learning.
  • Stage 2: Synonym-aware IMU-to-text alignment: Stage 2 aligns the pretrained encoder with frozen Sentence-BERT using synonym-aware soft contrastive learning and a shared 384-dimensional space.
  • Inference: Inference selects the label embedding with maximum cosine similarity, allowing arbitrary candidate inventories without a per-dataset classifier.
  • Inference: The model contains approximately 35M trainable parameters and caches sensor-description and candidate-label embeddings for deployment.

5 DESIGN OF HALO

HALO’s design uses two stages: heterogeneity-aware self-supervised encoder pretraining followed by synonym-aware contrastive text alignment.

  • HALO’s two stages are heterogeneity-aware encoder pretraining and synonym-aware contrastive text alignment.

5.1 Heterogeneity-aware Pretraining

Heterogeneity-aware pretraining adapts HALO to variable sampling rates, sensor suites, and placements through specialized tokenization, shared channel processing, and sensor conditioning. These components produce a fused IMU embedding while supporting transfer across configurations.

  • Adaptive-Pooling Tokenization: Adaptive pooling lets one tokenizer process 20–100 Hz inputs without resampling or architectural changes.The fixed 1.0 s default stays within 1.3 pp of per-dataset optima across the full 20–100 Hz range.
  • Channel-Independent Feature Extraction: Channel-independent shared weights handle sensor suites ranging from 3-channel smartphone data to 45-channel full-body configurations.Channel masks and cross-channel fusion reduce variable channel dimensions to fixed-width representations.
  • Contextualized Sensor Conditioning: Natural-language sensor descriptions condition each channel token with modality, axis, and placement information before Transformer processing.Descriptions may fall back to generic metadata when placement information is unavailable.
  • Contextualized Sensor Conditioning: Sensor conditioning is the single largest contributor to zero-shot transfer, with removal reducing ZS-O by 19.6 pp.With generic descriptions, HALO still outperforms baselines on 3 of 4 settings.
  • Self-Supervised Pretraining: Masked autoencoding and contrastive learning jointly capture local signal fidelity and global motion discriminability.Signal augmentation closes a 9.5 pp held-out generalization gap that is invisible on in-distribution validation.

5.2 Synonym-aware IMU-to-text Alignment

HALO aligns IMU and text embeddings while addressing synonym conflicts across datasets. Synonym augmentation, soft semantic targets, and cosine retrieval support open-vocabulary recognition without retraining.

  • Synonym-Aware Label Augmentation: Synonym-aware augmentation diversifies activity labels and reduces contradictory treatment of synonymous labels during alignment.Across the 87-label vocabulary, datasets average 2.3 synonyms per activity.
  • Soft Contrastive Learning: Soft contrastive targets give semantically related labels partial credit instead of forcing synonymous labels apart.This converts residual synonym conflict into a signal reinforcing semantic structure.
  • IMU-to-Text Alignment: A frozen Sentence-BERT text encoder aligns activity descriptions with IMU embeddings through a bidirectional CLIP-style contrastive objective.FIFO queues increase effective negatives per step from 32 to 288.
  • Zero-Shot Inference: Zero-shot inference selects the label with maximum cosine similarity from an arbitrary candidate library.Changing the label library at runtime enables recognition without retraining or dataset-specific classifiers.

6 EXPERIMENTS

HALO is evaluated against five baselines across held-out datasets and four zero-shot or supervised transfer settings, using protocols designed to expose cross-dataset generalization and label inconsistency.

  • Evaluation setup: HALO is evaluated on seven held-out datasets after joint training on ten public HAR datasets.The held-out evaluation includes five main datasets and two severe out-of-domain datasets.
  • Baselines: HALO is compared with five baselines spanning reconstruction-based, text-aligned, generative, and general time-series foundation-model paradigms.The comparison includes fixed-classifier and text-retrieval architectures with differing input and parameter policies.
  • Baselines: Classifier-based baselines use fixed global-label classifiers, whereas HALO and LanHAR retrieve predictions by cosine similarity between sensor and text embeddings.Text-aligned models support arbitrary label strings at test time, while classifier-based models cannot predict unseen label strings.
  • Evaluation setup: The study measures zero-shot open-set, zero-shot closed-set, 1% supervised fine-tuning, and 10% supervised fine-tuning.Zero-shot open-set prediction uses the full 87-label training vocabulary, while closed-set prediction is restricted to test-dataset labels.
  • Evaluation protocol: The protocol reports accuracy and macro F1 under exact-match scoring for text-aligned models and synonym-group scoring for classifier-based models.The input policy also preserves each architecture’s supported preprocessing, including native-rate HALO inputs and 20 Hz inputs for several baselines.

6.2 Overall Performance

HALO leads aggregate performance on the main held-out datasets and remains competitive under standardized inputs, but severe distribution shifts cause near-universal zero-shot failure.

  • Overall performance: HALO leads on all eight aggregate metrics, including zero-shot open-set accuracy of 42.0% versus MOMENT’s 28.3%.The zero-shot open-set margin is +13.7 percentage points, the largest reported aggregate gain.
  • Overall performance: HALO leads zero-shot on all five main datasets except Opportunity closed-set, where MOMENT reaches 53.9% versus HALO’s 49.3%.At 10% supervision HALO leads on all five datasets, while at 1% it leads on four of five.
  • Severe out-of-domain transfer performance: On HARTH, all embedding-based models achieve below 3% zero-shot accuracy under severe sensor distribution shift.HARTH uses back- and thigh-mounted accelerometers without gyroscopes, unlike the training configurations.
  • Severe out-of-domain transfer performance: On VTT-ConIoT, zero-shot accuracy remains below 9% for all models because only 50% of activities have training equivalents.With supervised data, MOMENT leads at 10% with 34.8%, while HALO provides stronger in-domain than fundamentally different-domain transfer.
  • Ablation study: Removing channel-text fusion reduces zero-shot open-set accuracy by 19.6 percentage points, while removing text augmentation reduces 1% supervised accuracy by 7.0 points.Signal augmentation closes a 9.5-point held-out generalization gap and has a −6.8-point effect at 10% supervision on severe out-of-domain data.
  • Native-rate and fairness analysis: With 20 Hz inputs and generic channel names, HALO still leads on three of four evaluation settings.Under these conservative inputs, its margins over MOMENT are +2.3 pp for ZS-O, +4.7 pp for ZS-C, and +3.4 pp at 10%.
  • Embedding-space analysis: The embedding space clusters semantically related activities and aligns IMU and text centroids, with 76.7% nearest-neighbor accuracy and 0.857 positive similarity.Stationary activities form a distinct cluster from locomotion in the UMAP analysis.
  • Model scaling: The 35M Small model outperforms the Medium model on average accuracy across three main test datasets at 20 Hz.The Medium model regresses on all zero-shot metrics despite having twice as many parameters.

6.4 Patch Size Sensitivity

HALO uses a fixed 1.0-second patch size that remains close to dataset-specific optima while avoiding per-dataset tuning. Model scaling reveals that the 35M-parameter Small variant generalizes best, whereas larger or smaller models perform worse in key settings.

  • Patch Size Sensitivity: 1.0-second patches remain within 1.3 pp of the per-dataset optimum across five datasets.Patches between 1.0 and 1.25 seconds perform consistently best.
  • Patch Size Sensitivity: Fixed 1.0-second patches capture approximately one full motion cycle and avoid splitting complex activities across patch boundaries.The selected duration aligns with typical gait cycles of 0.8–1.2 seconds.
  • Scaling of Model Sizes: 35M parameters define HALO’s Small model, which is identified as the capacity sweet spot.Tiny, Small, and Medium use identical training recipes, hyperparameters, and data.
  • Scaling of Model Sizes: 8.2 pp separates Small from Medium on ZS-O, with Medium regressing despite doubling the parameter count.Medium achieves the highest validation accuracy but the worst held-out generalization.
  • Scaling of Model Sizes: 13.2 pp separates Tiny from Small on ZS-O, while Tiny reaches 86.4% versus Small’s 85.3% at 10% supervision.The result suggests smaller models can benefit more from supervised regularization.

6.6 Inference Overhead on Smartphones

HALO supports real-time smartphone deployment and continuous inference across placement and label shifts. Its isolated model latency is substantially below MOMENT’s, while the full application adds app-level overhead.

  • Inference Overhead on Smartphones: 7.16 ms average latency and 7.37 ms p95 latency keep HALO within real-time smartphone inference requirements.The benchmark uses an iPhone 16 Pro and identical 6-channel Core ML inputs.
  • Inference Overhead on Smartphones: 5.26× faster than MOMENT, HALO runs at 7.16 ms versus 37.69 ms under the same deployment setting.HALO jointly models all six IMU channels in one forward pass.
  • Real-World Case Study: 97.5% accuracy across 40 inference windows includes only one transient error near the pocket-to-wrist transition.The model recovers immediately after the placement shift.
  • Real-World Case Study: The unseen-label phase remains fully correct, distinguishing Marching from Walking and Jogging from Running after SFT.These inference-time labels differ from the SFT classes.
  • Real-World Case Study: 30.35 ms mean end-to-end latency exceeds the isolated 7.16 ms benchmark because of application-level overhead.The deployed model uses 21.8M parameters and approximately 34 MB on disk, with memory at 73–78 MB.

7 CONCLUSION AND DISCUSSIONS

HALO concludes that a compact IMU foundation model can support heterogeneous configurations and open-vocabulary activity recognition without per-dataset adaptation. The discussion identifies richer conditioning, severe distribution shift, and semantic coverage as remaining boundaries.

  • Conclusion: 35M parameters suffice for HALO to outperform five baselines across all eight aggregate metrics.The conclusion presents this as evidence for a compact IMU foundation model.
  • Conclusion: HALO supports diverse wearable configurations and open-vocabulary recognition without per-dataset adaptation.This capability is the central deployment outcome stated in the conclusion.
  • Future Directions and Limitations: 9.9 pp zero-shot performance drops on RealWorld with richer global descriptions, motivating per-sample channel–text conditioning.The authors identify this as a direction for better handling heterogeneous body placements.
  • Future Directions and Limitations: Severe distribution shift remains an open challenge, particularly for HARTH and VTT-ConIoT under zero-shot transfer.The limitation concerns datasets with distributions substantially outside the training conditions.
  • Future Directions and Limitations: Expanding beyond the current 87-label vocabulary and 384-dimensional embedding space may improve semantic coverage and generalization.The conclusion frames both expansions as future work.
Loading 2608.27233v1…