PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyMo: Geometry-Aware Setup-Agnostic Modeling of Human Motion in the Wild

Baiyu Chen, Zechen Li, Wilson Wongso, Lihuan Li, Xiachong Lin, Hao Xue, Benjamin Tag, Flora Salim

arXiv:2605.22715v2cs.CVcs.AIcs.CLcs.HC

TL;DR

Wearable IMU signals vary across sensing setups, limiting transferable motion understanding. AnyMo combines geometry-aware simulation, setup-agnostic representation learning, full-body motion tokens, and language alignment, achieving consistent gains across unseen recognition, retrieval, and captioning tasks.

  • Problem

    Wearable IMU signals depend on sensing setup, making transferable motion representations across devices and datasets difficult to learn.

  • Method

    AnyMo combines physics-grounded geometry-aware IMU simulation, setup-agnostic pre-training, full-body tokenization, and motion-language alignment.

  • Results

    AnyMo consistently outperforms baselines across recognition, retrieval, and captioning, improving held-out IMU→Text MRR from 10.0 to 44.6.

  • Takeaways & Limitations

    AnyMo supports generalist wearable motion understanding across variable setups and unseen datasets.

Abstract

from arXiv · show

As wearable and mobile devices become increasingly embedded in daily life, they offer a practical way to continuously sense human motion in the wild. But inertial signals are highly dependent on the sensing setup, including body location, mounting position, sensor orientation, device hardware, and sampling protocol. This setup dependence makes it difficult to learn motion representations that transfer across devices and datasets, and limits the broader use of wearable IMUs beyond closed-set recognition. We introduce AnyMo, a geometry-aware framework for setup-agnostic human motion modeling. AnyMo uses physics-grounded IMU simulation over dense body-surface placements to generate diverse and plausible synthetic signals, pre-trains a graph encoder from paired synthetic placement views and masked partial observations, tokenizes multi-position IMU into full-body motion tokens, and aligns these tokens with an LLM for motion-language understanding. We evaluate AnyMo on three complementary tasks: zero-shot activity recognition across 14 unseen downstream datasets, cross-modal retrieval, and wearable IMU motion captioning, where it improves average Accuracy/F1/R@2 by 11.7\%/11.6\%/22.6\% on HAR, increases zero-shot IMU-to-text and text-to-IMU retrieval MRR by 15.9\% and 28.6\%, respectively, and improves zero-shot captioning BERT-F1 by 18.8\%. These results support AnyMo as a generalist model for wearable motion understanding in the wild. Project page: https://baiyuchen.com/project/AnyMo.

1 Introduction

AnyMo addresses the difficulty of understanding wearable motion across sensing setups by combining geometry- and physics-based modeling with language-grounded motion representations. It evaluates this approach through zero-shot recognition, cross-modal retrieval, and motion captioning under unseen and sim-to-real conditions.

  • Human motion reflects everyday context, making its understanding important for proactive AI and human-centered computing in real environments.
  • IMU signals are semantically ambiguous and tightly coupled to body location, mounting position, orientation, hardware, and sampling protocol.Similar inertial patterns can correspond to different activities, while different wearable placements can measure distinct dynamics for the same activity.
  • AnyMo exploits structured setup variation through physics-grounded, geometry-aware IMU simulation over dense body-surface placements.The framework treats body motion, surface geometry, sensor orientation, and device response as inductive biases for setup-robust representations.
  • Its pre-training pairs synthetic placement views with masked wearable observations, while tokenization maps multi-position IMU into full-body motion tokens aligned with an LLM.This design supports setup-agnostic representations and open-vocabulary motion understanding.
  • AnyMo is benchmarked on zero-shot activity recognition across 14 unseen datasets, unseen cross-modal retrieval, and wearable IMU motion captioning.Retrieval and captioning include sim-to-real transfer on unseen Nymeria subjects and out-of-domain zero-shot transfer to EgoExo4D.

2 Related Works

Prior wearable motion research improves generalization through synthetic pretraining, self-supervision or cross-dataset adaptation, and tokenization. Setup-variation studies examine cross-location transfer, simulated body-surface placements, and coordinate-conditioned flexible placement, but remain mainly recognition- or pose-centered and do not combine dense loc…

  • Wearable Motion Representations and Setup Transfer: Synthetic pretraining, large-scale self-supervision, cross-dataset adaptation, and tokenization are established approaches for improving wearable motion generalization.
  • Wearable Motion Representations and Setup Transfer: Setup variation has been studied through cross-location transfer, simulated body-surface placement analysis, and coordinate-conditioned flexible placement.
  • Wearable Motion Representations and Setup Transfer: Existing setup-transfer methods remain mainly recognition- or pose-centered and do not combine dense loc…

3 Methodology

AnyMo models wearable motion across variable sensing setups by combining geometry-aware physics-based IMU simulation with masked cross-view graph pretraining. It then converts setup-stable full-body motion representations into compact discrete tokens for motion-language modeling.

  • Setup-Agnostic Motion Representation: AnyMo represents motion over 23 Nymeria anatomical segments and treats each IMU window as three-axis acceleration plus three-axis angular velocity.The same motion can produce different IMU windows under different wearable setups.
  • Geometry-Aware IMU Simulation: Physics-grounded simulation generates dense wearable placements by mapping synchronized body motion to surface vertices, local sensor frames, mounting rotations, device noise, acceleration, and angular velocity.Noise priors estimated from quiet windows of two real Nymeria IMU streams are randomly assigned to synthetic placements.
  • Masked Cross-View Predictive Contrastive Learning: The pretraining objective creates paired full graph views and sparse masked views, then predicts opposite full-view temporal latents with a shared spatio-temporal graph encoder and Transformer predictor.Masked views retain one to five visible segment nodes, encouraging inference of full-body motion from sparse wearable inputs while preserving temporal structure.
  • Masked Cross-View Predictive Contrastive Learning: A bidirectional cross-view predictive InfoNCE loss aligns predicted and target sequence latents using mean cosine similarity over time.The final objective sums the A → B and B → A losses: LMCVPCL = LA→B + LB→A.
  • Full-Body IMU Tokenization: A frozen graph encoder feeds a product-quantized VAE tokenizer that discretizes continuous masked-observation latents into compact IMU tokens for motion-language models.The tokenizer projects each timestep latent, splits it across product codebooks, quantizes chunks to nearest code vectors, and reconstructs the sequence with a temporal convolutional decoder.

4 Experiments

AnyMo is evaluated zero-shot across unseen activity-recognition, cross-modal retrieval, and motion-captioning benchmarks, without using downstream datasets for training. It consistently outperforms baselines, while ablations show that geometry-aware simulation and masked cross-view pre-training are central to synthetic-to-real transfer.

  • Experimental Setup: AnyMo uses Nymeria for training and held-out sim-to-real evaluation, while no downstream benchmark dataset is used for training.Mesh and skeleton motion generate dense geometry-aware IMU candidates; real IMU streams estimate device-noise priors and provide held-out evaluation.
  • Zero-shot Human Activity Recognition: 35.7 Accuracy, 29.5 macro-F1, and 57.5 Recall@2 are AnyMo’s best average results across 14 unseen wearable datasets.These improve over the strongest average baseline by 11.7%, 11.6%, and 22.6%, respectively.
  • Cross-Modal Retrieval: 44.6 IMU→Text MRR and 46.7 Text→IMU MRR on 100-sample Nymeria retrieval substantially exceed baseline values of 10.0 and 6.7.AnyMo remains clearly ahead in the harder all-sample setting and achieves the best or second-best performance on most EgoExo4D metrics.
  • Wearable IMU Motion Captioning: 31.1 ROUGE-L and 69.7 BERT-F1 on Nymeria held-out captioning improve over prompting baselines at 15.7 and 57.3.AnyMo also outperforms GPT-5.4 Mini and Gemma 4 26B across captioning metrics in the held-out and zero-shot settings.
  • Ablation: Removing geometry-aware simulation reduces Acc/F1/R@2 from 35.7/29.5/57.5 to 8.4/3.8/16.3.Replacing masked cross-view predictive contrastive learning also severely hurts performance, indicating both components support transfer from synthetic full-body candidates to sparse real inputs.
  • Ablation: Removing geometry-aware simulation separates real and synthetic embeddings, while removing masked cross-view learning reduces cluster coherence across 10 activity categories.Removing label or narration contrastive tuning also degrades Acc, F1, and R@2.

5 Conclusion · A Experiments and Implementation Details

AnyMo is presented as a geometry-aware, setup-agnostic framework for wearable IMU motion understanding under variable sensing setups. It models setup variation through body-surface-aware simulation, learns full-body motion representations, and connects compact IMU tokens to language models.

  • A Experiments and Implementation Details: The experiments include an ablation study of the proposed framework.Table 4 is identified as an ablation study, but the supplied passage does not report its individual results.
  • 5 Conclusion: AnyMo treats wearable setup variation as structured variation over the body surface.This reframes differences in body location and sensing configuration as a geometric modeling problem.
  • 5 Conclusion: The framework synthesizes dense, physics-grounded IMU candidates from human mesh motion.Synthetic candidates are generated across body-surface placements to represent diverse sensing setups.
  • 5 Conclusion: AnyMo learns setup-agnostic representations of full-body human motion.These representations are designed for wearable IMU understanding across variable sensing setups.
  • 5 Conclusion: The model compresses multi-position inertial observations into compact full-body IMU tokens.These tokens provide the interface between wearable motion signals and language modeling.
  • 5 Conclusion: AnyMo connects full-body IMU tokens to language models for motion-language understanding.The framework extends wearable IMU modeling beyond isolated signal representations toward language-linked understanding.
  • 5 Conclusion: The paper evaluates AnyMo on zero-shot activity recognition, bidirectional IMU-text retrieval, and wearable motion captioning.These tasks assess recognition, cross-modal retrieval, and language generation from wearable motion signals.

A.1 Training Data and Window Construction · A.2 Synthetic IMU Generation

AnyMo builds its training corpus from synchronized Nymeria motion, mesh, annotations, and IMU data, using mostly 5 s windows. Its synthetic pre-training generates geometry-aware IMU signals from dense body-surface placements and augments paired full-body graph views.

  • A.1 Training Data and Window Construction: 90.1% of instruction-tuning rows contain 150 IMU code tokens, corresponding to T = 300 frames in mostly 5 s windows.The original frame count is twice the IMU code-token count.
  • A.1 Training Data and Window Construction: Nymeria is the only source of motion-text supervision across all AnyMo training stages.Motion, mesh vertices, atomic-action annotations, and real IMU streams are synchronized to a common 60 Hz timeline.
  • A.1 Training Data and Window Construction: 828 recording samples and 168,295 windows comprise the text-aligned Nymeria export before held-out-subject splitting.The held-out protocol reserves five subjects while maintaining subject-disjointness and covering all 20 original Nymeria scenarios.
  • A.2 Synthetic IMU Generation: Synthetic signals use Nymeria’s anatomically grounded human model, mesh, and skeleton motion to define surface templates for every body segment.Candidate vertices receive local right-handed sensor frames estimated from surface normals, anatomical tangents, and corresponding binormals.
  • A.2 Synthetic IMU Generation: During representation pre-training, each window is sampled twice to create two full-body graph views with independently selected candidate placements per segment.Surface rotation augmentation spans ±180◦ in-plane rotation and ±10◦ tilt.
  • A.2 Synthetic IMU Generation: Acceleration and angular velocity are computed from simulated rigid-body trajectories in each candidate sensor frame.The procedure combines geometry-aware placement with surface rotation augmentation during representation pre-training.

A.3 Representation Pre-Training and Tokenization

AnyMo pre-trains an ST-GCN over a 23-node body graph and then freezes it to train a product-quantized VAE tokenizer on masked sparse-view latent sequences. The tokenizer converts learned codebook vectors into discrete IMU tokens for LLM integration, with diagnostics indicating broad codebook usage rather than collapse.

  • Encoder pre-training: The IMU encoder is an ST-GCN over a 23-node body graph with ten spatio-temporal blocks, temporal kernel size 9, channels increasing from 64 to 256, and two stride-2 stages.A 300-frame input becomes a latent sequence of about 75 steps with dimension 256.
  • Encoder pre-training: Pre-training samples two independent full-body graph views from the synthetic candidate set and corresponding sparse observations.
  • Tokenizer: After pre-training, the encoder is frozen and a product-quantized VAE is trained on masked sparse-view latent sequences using two 2,048-entry codebooks and a 128-dimensional bottleneck.Each codebook uses 64-dimensional code vectors, with EMA updates at decay 0.99.
  • Tokenizer: The tokenizer adds 4,096 IMU code tokens and boundary tokens to the LLM tokenizer while retaining learned codebook vectors as the continuous representations behind discrete IDs.
  • Tokenizer diagnostics: The final diagnostics report perplexities of 1,285.7 and 1,098.1 out of 2,048 entries, with near-complete usage rates and low dead-code ratios.These statistics assess codebook usage and whether the product quantizer collapses to a small subset of entries.

A.4 Motion-Language Training · A.5 Zero-Shot Recognition Evaluation

AnyMo adapts Qwen2.5-0.5B for motion-language modeling through IMU-token language modeling, paraphrase augmentation, label normalization, and joint instruction-tuning objectives. Zero-shot recognition evaluates frozen representations by ranking dataset activity labels using cosine similarity and reports Accuracy, macro-F1, and Recall@2.

  • A.4 Motion-Language Training: AnyMo uses Qwen2.5-0.5B as its language backbone and replaces IMU-token embeddings with projected codebook vectors at runtime.Motion-language training runs on two NVIDIA L40S GPUs using ms-swift and a custom AnyMo model registration.
  • A.4 Motion-Language Training: The initial adaptation stage trains the language model to predict IMU-token sequences exported from the frozen ST-GCN and tokenizer.
  • A.4 Motion-Language Training: Each Nymeria atomic-action narration is expanded with five GPT-OSS-120B paraphrases that preserve action meaning while varying wording and emphasis.A GPT-OSS-120B self-verifier checks semantic preservation, action order, and related constraints.
  • A.4 Motion-Language Training: Activity labels are normalized by lowercasing, removing punctuation and extra spaces, canonicalizing leading articles, and exact deduplication.The normalized labels are embedded with Qwen3-Embedding-8B for the label-contrastive branch.
  • A.4 Motion-Language Training: Instruction tuning combines language-modeling and contrastive losses for IMU-to-narration generation, IMU-to-activity multiple choice, and aligned IMU-text prompts.Multiple-choice examples use a fixed 35-choice candidate set sampled from the training activity pool, while contrastive prompts use branch-specific latent-attention poolers.
  • A.5 Zero-Shot Recognition Evaluation: Downstream HAR datasets are reserved for evaluation, with adapters standardizing sensor units, resampling windows to 60 Hz, and mapping devices onto the 23-node body graph.Head-mounted video datasets map to Head, while other sensor locations map to corresponding body-graph regions.
  • A.5 Zero-Shot Recognition Evaluation: Zero-shot recognition keeps AnyMo frozen, compares IMU prompts with candidate activity-label prompts using cosine similarity, and ranks labels from each dataset vocabulary.Accuracy uses the top-ranked label, macro-F1 evaluates predicted labels, and Recall@2 counts ground-truth labels appearing among the top two.

A.6 Retrieval and Captioning Evaluation · A.7 Capability Radar Details · A.8 Ablation Implementation Details

The evaluation uses held-out Nymeria and balanced EgoExo4D windows for retrieval and captioning, while the capability radar aggregates recognition, retrieval, and captioning metrics. Ablations preserve the full protocol and isolate geometry-aware simulation, masked cross-view pre-training, contrastive losses, and MCQ instruction tuning.

  • A.6 Retrieval and Captioning Evaluation: A.6 Retrieval and Captioning Evaluation: Nymeria sim-to-real evaluation uses 20 held-out recording samples and 3,908 text-aligned windows from five reserved subjects.EgoExo4D OOD evaluation uses a balanced 4,000-window atomic-action subset, with 500 examples from each of eight parent tasks.
  • A.6 Retrieval and Captioning Evaluation: A.6 Retrieval and Captioning Evaluation: Retrieval reports Recall@1, Recall@5, Recall@10, and MRR in both IMU-to-text and text-to-IMU directions.Captioning uses greedy decoding with a maximum of 64 generated tokens and compares generated captions against primary references.
  • A.7 Capability Radar Details: A.7 Capability Radar Details: Nine radar axes summarize average zero-shot HAR recognition, EgoExo4D retrieval rank quality, and captioning quality across the three evaluation groups.The recognition axes are Exact Activity Match, Balanced Recognition, and Top-2 Activity Recall; retrieval axes are IMU-to-Text Rank Quality and Text-to-IMU Rank Quality.
  • A.8 Ablation Implementation Details: A.8 Ablation Implementation Details: All ablations use the full model’s downstream zero-shot recognition protocol, candidate label vocabularies, and metrics.When an upstream stage changes, dependent artifacts—including tokenizer exports, pre-training rows, instruction-tuning rows, and evaluation tokens—are regenerated.
  • A.8 Ablation Implementation Details: A.8 Ablation Implementation Details: Geometry-aware simulation uses mesh-surface placements with local frames from surface normals and anatomical tangents, whereas its ablation uses one virtual sensor per Xsens segment frame.The ablation therefore removes mesh-vertex candidate placements in favor of a joint-mounted setup across 23 segment frames.
  • A.8 Ablation Implementation Details: A.8 Ablation Implementation Details: Masked cross-view predictive contrastive pre-training maps sparse-view latent sequences to opposite full-view latent sequences using symmetric InfoNCE and stop-gradient targets.The ablation retains synthetic graph-view sampling but replaces the sparse-to-full predictive objective with a standard full-view objective.
  • A.8 Ablation Implementation Details: A.8 Ablation Implementation Details: Motion-language instruction tuning combines language modeling, narration-level symmetric IMU-text contrastive learning, and supervised label-level contrastive learning.The label-contrastive ablation sets the label-level loss weight to zero while retaining narration-level contrastive learning; the MCQ ablation replaces MCQ rows with narration rows without changing total examples.

B AnyMo Bench

AnyMo Bench is a controlled, real-world HAR benchmark built from Nymeria IMU streams to test fine-grained recognition under unseen-subject and cross-device shifts. Its scale, curated label space, and baseline results establish a challenging testbed for robust wearable sensing.

  • Benchmark construction: AnyMo Bench derives an in-the-wild HAR benchmark from real Nymeria IMU streams, targeting fine-grained daily-activity recognition on unseen subjects and cross-device transfer.Nymeria provides synchronized streams from three body positions: Head, Left Wrist, and Right Wrist.
  • Benchmark construction: Cross-device evaluation trains on the first IMU at each position and tests on the second, keeping body placement fixed while changing the IMU unit.The benchmark therefore isolates device variation from body-position variation.
  • Dataset scale and splits: The subject-disjoint 8:2 split yields 157 training and 39 test subjects, covering 154,695 activity windows and 211.6 hours of real in-the-wild IMU data.For Fine150, the split contains 123,874 training windows and 30,965 test windows, with no class missing from either split.
  • Label spaces: Fine150 contains 150 fine-grained classes filtered from AnyMo-180 for subject-split stability and IMU-observable distinctions, while retaining a long-tailed distribution.The construction removes labels with insufficient test-subject or test-window support and highly context-dependent distinctions.
  • Results and implications: 38.5% Acc@1 and 65.2% Acc@5 are achieved by MantisV2 on Fine150 / Unseen Subject, while COMODO reaches 37.8% Acc@1 and 65.2% Acc@5.DeepConvLSTM reaches 35.3% Acc@1 and 63.0% Acc@5; the benchmark remains difficult, especially for tail and fine-grained classes.
  • Results and implications: Realistic subject and device shifts, together with expert-reviewed label consolidation and class-wise auditing, make AnyMo Bench a challenging and carefully curated testbed.Its construction combines automatic label proposal, semantic consolidation, enum-label assignment, relabel/drop decisions, and IMU-pose-aware label-space aggregation.

C Downstream Evaluation Dataset Details · D Baseline Details

The downstream benchmark comprises 14 unseen datasets with dataset-specific activity classes and sensor placements mapped onto AnyMo’s 23-node body graph. Baselines use matched candidate vocabularies and controlled pretraining and temporal settings for fair zero-shot recognition comparisons.

  • C Downstream Evaluation Dataset Details: The zero-shot recognition benchmark uses 14 downstream datasets that are never used to train AnyMo.Evaluation sensor placements are mapped to the closest nodes in AnyMo’s 23-node body graph.
  • C Downstream Evaluation Dataset Details: EgoExo4D, Ego4D, and MMEA follow the train/test splits from COMODO [10]3.This split choice is specified for these three datasets in the benchmark construction.
  • C Downstream Evaluation Dataset Details: Table 9 organizes each benchmark dataset by activity classes and sensor placements.The supplied table caption identifies these as downstream zero-shot HAR datasets.
  • C Downstream Evaluation Dataset Details: The listed datasets span diverse activities, including sports, household tasks, locomotion, gestures, and workplace actions.Examples in the supplied passages include basketball, cooking, walking, typing, and assembling or labeling boxes.
  • C Downstream Evaluation Dataset Details: Sensor configurations include head, forearms, upper arms, hand, lower legs, and task-dependent right-hand or right-upper-leg placements.These placements are reported using readable AnyMo graph segment names or label-specific mappings.
  • D Baseline Details: All baselines use the same dataset-specific candidate activity vocabularies as AnyMo.This keeps recognition candidates consistent across methods.
  • D Baseline Details: ImageBind and IMU2CLIP use Ego4D-pretrained checkpoints for non-Ego4D datasets, while Ego4D evaluation uses checkpoints pretrained on MMEA.The protocol avoids evaluating an Ego4D-pretrained checkpoint on the Ego4D target dataset.
  • D Baseline Details: ImageBind and IMU2CLIP use the same 60 Hz temporal setting as AnyMo.The passage explicitly states this shared temporal configuration.

E Prompt Analysis

AnyMo analyzes how prompt design affects zero-shot HAR recognition and adopts a learnable textual context instead of fixed manual templates. It also specifies distinct prompt templates for motion-language pre-training, instruction tuning, retrieval, recognition, and evaluation.

  • E.1 Prompt Sensitivity: Zero-shot recognition is sensitive to prompted activity labels, so four fixed formats are evaluated under the same protocol: bare, person, IMU, and activity prompts.The compared formats are bare label; “a person is {label}”; “wearable IMU motion of {label}”; and “the activity is {label}”.
  • E.1 Prompt Sensitivity: AnyMo replaces manual templates with an 8-token learnable soft prompt on the text/label branch.The soft prompt learns task-specific textual context during contrastive instruction tuning.
  • E.1 Prompt Sensitivity: The fixed prompts produce different recognition rankings despite unchanged label vocabulary, IMU embeddings, and evaluation protocol.This isolates prompt wording as the source of the observed ranking changes.
  • E.1 Prompt Sensitivity: The IMU-centric template is weaker than person- and activity-centric templates on the benchmark, showing that manual contexts are not consistently beneficial.Prompt wording can shift text embeddings in ways that do not reliably improve recognition.
  • E.1 Prompt Sensitivity: The learnable prompt gives the best average Acc, F1, and Recall@2 among the compared single-prompt settings and is used for final zero-shot recognition.It also removes dataset-specific manual prompt selection at evaluation time.
  • E.2 Prompt Templates: AnyMo uses IMU-token, narration, MCQ instruction, and contrastive templates across motion-language pre-training, instruction tuning, retrieval, recognition, and evaluation.Captioning uses the narration template, MCQ-style evaluation uses the MCQ template, and retrieval and zero-shot recognition use contrastive IMU and text templates.
Loading 2605.22715v2…