Source-linked AI summary
MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment
Duc Duy Nguyen, Tat-Jun Chin, Minh Hoai
TL;DR
IMU–video learning must handle irrelevant visual background, structured multi-sensor configurations, and sub-second temporal correspondence. MoBind addresses these issues with skeletal-motion alignment, body-part-to-IMU pairing, and hierarchical contrastive learning augmented by masked token prediction. Across mRi, TotalCapture, and EgoHumans, it consistently outperforms competing methods across retrieval, synchronization, localization, and action recognition while retaining class semantics.
Problem
Existing IMU–vision methods often use global embeddings that support coarse semantics but miss fine-grained temporal structure, while video and IMU provide complementary yet limited information.
Method
MoBind aligns IMU streams with skeletal motion, pairs sensors with local body-part trajectories, aggregates token- and local-level features globally, and adds MTP for semantic preservation.
Results
MoBind consistently outperforms competing approaches across cross-modal retrieval, temporal synchronization, subject/body-part localization, and action recognition on mRi, TotalCapture, and EgoHumans.
Takeaways & Limitations
MoBind provides robust fine-grained IMU–video alignment while preserving coarse action-level semantic consistency across modalities.
Abstract
from arXiv · showhide
We aim to learn a joint representation between inertial measurement unit (IMU) signals and 2D pose sequences extracted from video, enabling accurate cross-modal retrieval, temporal synchronization, subject and body-part localization, and action recognition. To this end, we introduce MoBind, a hierarchical contrastive learning framework designed to address three challenges: (1) filtering out irrelevant visual background, (2) modeling structured multi-sensor IMU configurations, and (3) achieving fine-grained, sub-second temporal alignment. To isolate motion-relevant cues, MoBind aligns IMU signals with skeletal motion sequences rather than raw pixels. We further decompose full-body motion into local body-part trajectories, pairing each with its corresponding IMU to enable semantically grounded multi-sensor alignment. To capture detailed temporal correspondence, MoBind employs a hierarchical contrastive strategy that first aligns token-level temporal segments, then fuses local (body-part) alignment with global (body-wide) motion aggregation. Evaluated on mRi, TotalCapture, and EgoHumans, MoBind consistently outperforms strong baselines across all four tasks, demonstrating robust fine-grained temporal alignment while preserving coarse semantic consistency across modalities. Code is available at https://github.com/bbvisual/ MoBind.
1. Introduction
IMU–video representations must combine complementary sensing strengths while capturing both coarse action semantics and fine-grained temporal correspondence. MoBind addresses this gap by aligning IMUs with skeletal motion through hierarchical local and global contrastive objectives.
- Motivation: Video provides spatial and semantic information, whereas IMUs provide temporally dense motion signals but lack visual context.Video remains sensitive to occlusion, viewpoint changes, and limited frame rates.
- Research Gap: Joint IMU–video representations are needed for coarse action correspondence and fine-grained, sub-second temporal alignment.Such representations support synchronization and other multimodal analysis capabilities.
- Research Gap: Global clip embeddings can encode coarse semantics but overlook phase shifts, short lags, and repetition boundaries within actions.Sequences differing in temporal phase may therefore collapse to nearby representations.
- Research Gap: Prior sub-second audio–video methods transfer poorly because IMUs are localized, motion-centric, and commonly used in multi-sensor configurations.These properties make much visual background irrelevant to IMU–video alignment.
- MoBind: MoBind aligns IMUs with video-derived skeletal motion, pairs each IMU with its corresponding body part, and aggregates local features globally.This design targets motion-relevant cues and structured multi-sensor association rather than raw-pixel alignment.
- MoBind: A hierarchical objective aligns short IMU–part segments locally and full-body multi-IMU representations globally, while MTP preserves action-level semantics.The method was evaluated on mRi, TotalCapture, and EgoHumans across retrieval, synchronization, localization, and action recognition.
2. Related Work
Prior work largely learns global IMU–vision embeddings for activity recognition and retrieval, while temporal synchronization draws heavily on audio–visual methods. MoBind instead combines structured body-part alignment, multi-level representations, and semantic preservation for IMU–video correspondence.
- IMU–Vision Representation Learning: IMU representation learning has primarily targeted human activity recognition, with newer work learning semantically meaningful cross-modal embeddings for retrieval.IMU2CLIP and ImageBind align IMU features with the CLIP space, often using egocentric video.
- Temporal Synchronization: Audio–visual synchronization research uses statistical correlation, clip- and frame-level contrastive alignment, transformer models, and temporal offset classification.These paradigms motivate synchronization baselines for IMU–video alignment.
- MoBind: MoBind encodes each IMU stream with its corresponding body-part motion, produces token- and local-level representations, and aggregates them into global embeddings.Its contrastive objective operates across all three representation levels, while MTP preserves coarse semantic structure during training.
- Subject Localization: IMU-to-person association has been framed as graph labeling or learned visual–inertial matching, paralleling active-speaker association in audio–visual research.Prior methods use orientation consistency, contrastive feature spaces, or recurrent multi-sensor modeling.
3. MoBind
MoBind learns structured joint representations between wearable IMU signals and video-based skeletal motion. Its hierarchical contrastive alignment combines token-, body-part-, and global-level correspondence, with masked token prediction preserving action-level semantics.
- Modality-Specific Modules: MoBind extracts skeletal joint sequences from video and processes synchronized raw streams from N body-mounted IMU sensors.Modality-specific modules produce representations trained with a contrastive objective for matched and mismatched IMU–video pairs.
- Modality-Specific Modules: Each IMU stream is encoded by 1D convolutions and a Transformer into temporal tokens, then mean-pooled into a local representation.The convolutional feature map is divided into T non-overlapping patches before Transformer encoding.
- Modality-Specific Modules: Per-sensor local IMU representations are concatenated and passed through normalization and an MLP to form a global representation.The pose module uses an analogous body-part encoder and aggregates local body-part representations into a global pose representation.
- Hierarchical Contrastive Alignment: The hierarchical objective aligns corresponding temporal tokens, each IMU sensor with its body part, and aggregated IMU and skeletal representations.Before contrastive learning, both modalities are projected into a shared embedding space; bidirectional objectives use InfoNCE with cosine similarity and a learnable temperature.
- Masked Token Prediction (MTP): Masked Token Prediction complements alignment by masking IMU tokens and using unmasked context to reconstruct them with mean-squared error.The auxiliary loss is jointly optimized with the alignment objective using λ_mtp.
- Results: MoBind consistently outperforms prior contrastive baselines across retrieval directions, datasets, and ranks in Table 1.The reported datasets are mRi, TotalCapture, and EgoHumans, with evaluation in both IMU→Video and Video→IMU directions.
4. Experiments
MoBind is evaluated for retrieval, temporal synchronization, localization, action recognition, ablations, and robustness across multimodal datasets. Results show strong cross-modal alignment, sub-second synchronization, spatial grounding, and resilience to sensor dropout.
- Experimental scope: Experiments cover cross-modal retrieval, temporal synchronization, subject and body-part localization, action recognition, ablations, and simulated sensor failures.Evaluations use mRi, TotalCapture, and EgoHumans with fixed 5 s windows and estimated 2D keypoints.
- Cross-modal retrieval: MoBind consistently outperforms baselines in both retrieval directions, with especially large gains on mRi and TotalCapture.Baselines include IMU2CLIP, DeSPITE, and SyncNet; on TotalCapture, MoBind’s median hard-negative margin is +0.10 versus -0.14, -0.11, and -0.14 for the baselines.
- Temporal synchronization: MoBind maintains accurate synchronization across datasets, with error below 50ms for every EgoHumans action and under one second for every mRi category.The mRi setting is more difficult because repetitive exercises create near-duplicate segments and hard negatives.
- Temporal synchronization: Longer clips improve synchronization on mRi, reaching 97% accuracy with 2-minute clips and 100% with 3-minute clips.The framework supports synchronization for sequences of arbitrary lengths.
- Subject and body-part localization: MoBind achieves body-part localization accuracies of 0.81 on mRi, 0.57 on TotalCapture, and 0.63 on EgoHumans.Local contrastive learning identifies both the person wearing an IMU and the specific attached body part.
- Ablation and robustness: Ablations show that token-, single-sensor-, and multi-sensor contrastive losses provide complementary benefits, while MTP improves action recognition and sensor masking preserves strong retrieval.MTP yields nearly 20% improvement on TotalCapture for both classifiers; performance degrades with fewer sensors but remains competitive with full-sensor baselines.
5. Conclusion
MoBind is a hierarchical contrastive framework aligning IMU signals with video-based skeletal motion through motion-focused, multi-sensor, and multi-level alignment. It supports downstream tasks and achieves state-of-the-art IMU–video alignment while retaining class semantics via the MTP auxiliary task.
- MoBind aligns IMU signals with video-based skeletal motion rather than raw pixels, focusing representation learning on motion-relevant pose cues.
- The framework models multi-sensor structure by aligning each body part with its corresponding IMU.
- MoBind enforces alignment at token, local, and global levels through a hierarchical contrastive framework.
- The learned representation supports downstream tasks and achieves state-of-the-art IMU–video alignment on mRi, TotalCapture, and EgoHumans.
- The MTP auxiliary task retains class semantics alongside fine-grained alignment.