Source-linked AI summary
Sound-based Multi-Person 3D Pose Estimation
Yusuke Oumi, Yuto Shibata, Go Irie, Akisato Kimura, Yoshimitsu Aoki, Mariko Isogawa
TL;DR
Estimating multiple people’s 3D poses from sound is difficult because overlapping motion signatures and inter-person reflections obscure pose-specific acoustic information. SoundMHPE combines multi-scale acoustic encoding with temporal pose decoding and is evaluated on a new AMP dataset, where it outperforms baseline methods across all evaluation metrics.
Problem
Multi-person acoustic pose estimation is unexplored and difficult because superposed motion features and multi-body reflection delays obscure the mapping between individual poses and acoustic signals.
Method
SoundMHPE combines an Acoustic Multi-scale Encoder with a Temporal Pose Decoder to represent diverse acoustic scales and model individual pose evolution across multiple people.
Results
SoundMHPE outperforms baseline methods across all evaluation metrics on the AMP dataset.
Takeaways & Limitations
The paper establishes a first approach and dataset for multi-person pose estimation using active acoustic sensing.
Takeaways & Limitations
Generalization to environments with unseen reflection characteristics remains challenging, although SoundMHPE can estimate coarse poses under different reflection conditions.
Abstract
from arXiv · showhide
Can we recover the 3D poses of multiple people using only sound? This paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals. Estimating the poses of multiple individuals using acoustic signals is inherently challenging due to the superposition of motion-dependent signal variations. Unlike single-person scenarios, the presence of multiple subjects leads to overlapping acoustic signatures, making it difficult to attribute specific signal changes to an individual's pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that obscure the temporal motion-acoustic relationship. To address these issues, we propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework consisting of two key components. First, the Acoustic Multi-scale Encoder captures diverse temporal and fine-grained frequency features to isolate subtle acoustic signatures from complex, overlapping signals. Second, the Temporal Pose Decoder employs an attention mechanism to disentangle multi-person information across successive frames. By jointly accounting for temporal dynamics and inter-person dependencies, this component precisely reconstructs frame-wise individual poses. To validate our approach, we constructed the 6-hour Acoustic Multi-person Pose (AMP) dataset consisting of 432K synchronized frames of multi-person pose and acoustic data, and demonstrated that our SoundMHPE outperforms baseline models. Project page: https://oumi03.github.io/sound-mhpe/
1 Introduction
SoundMHPE addresses the previously unexplored problem of estimating multiple people’s 3D poses from acoustic signals. It combines multi-scale acoustic encoding with temporal pose decoding and introduces the AMP dataset for evaluation.
- Multi-person pose estimation is useful in real-world settings where people coexist and interact, including monitoring, sports analysis, and disaster relief.
- RGB methods are vulnerable to occlusion and low light, while acoustic sensing can operate without lighting and may estimate poses behind obstacles such as metal.
- Multiple people create superposed motion features and reflection-induced propagation delays that obscure the correspondence between poses and acoustic signals.
- SoundMHPE uses an Acoustic Multi-scale Encoder to disentangle motion-dependent information from overlapping acoustic features and reflections across temporal and frequency scales.
- The framework introduces a Temporal Pose Decoder that assigns dedicated queries to individuals across frames, integrating pose evolution with multi-body context.
- The authors construct the 6-hour AMP dataset and report superior performance over baseline methods, with ablations supporting the proposed components.
2 Related Work
Prior multi-person pose-estimation methods use visual, wireless, and acoustic signals, each with distinct environmental constraints. Existing acoustic approaches remain limited to single-person settings, motivating spatio-temporal modeling for multiple people.
- Existing multi-person pose methods use RGB images, videos, RF/WiFi, mmWave, and UWB radar signals.
- RGB methods can suffer from occlusion, low-light degradation, and privacy concerns, while wireless methods face obstacles involving water, metal, or restricted communication environments.
- Active acoustic sensing estimates human states from emitted and received sound and has been applied to action recognition, mesh reconstruction, and pose estimation.
- Prior acoustic pose-estimation methods use chirps, inaudible tones, or music but remain limited to single-person settings.
- Multi-person estimation requires joint modeling of temporal dynamics and inter-person dependencies, motivating temporal and query-based architectures.
3 Methodology
SoundMHPE combines a multi-scale acoustic encoder with a temporal pose decoder to estimate multiple subjects’ 3D pose sequences from acoustic signals. Its attention mechanisms separate temporal, frequency, motion, and inter-person relationships across multi-scale spectrograms and pose queries.
- SoundMHPE estimates multiple subjects’ 3D pose sequences from active acoustic signals using an Acoustic Multi-scale Encoder and Temporal Pose Decoder.The encoder produces acoustic features, while the decoder models pose-related temporal structure.
- Acoustic Multi-scale Encoder: The encoder converts segmented microphone signals into log-Mel spectrograms and produces three resolutions using STFT windows L, 2L, and 4L.The corresponding outputs represent high-temporal, intermediate, and high-frequency resolutions.
- Temporal Pose Decoder: The Temporal Pose Decoder assigns multiple learnable queries to each individual so different queries can attend to acoustic features for different frames.This design replaces a single query that aggregates pose information from multiple frames.
- Acoustic Multi-scale Encoder: The encoder uses temporal self-attention within each STFT resolution and frequency self-attention across spectrograms generated from different window sizes.These mechanisms organize temporal dependencies and cross-resolution relationships in the multi-scale acoustic features.
- Experimental Setup: The AMP dataset uses speakers, an ambisonics microphone, and motion-capture cameras to collect acoustic signals and ground-truth poses from single-, double-, and triple-person groups.The dataset includes 15 participants divided into three groups with 72 minutes of single-person and 24 minutes each of double- and triple-person data per group.
- Temporal Pose Decoder: The decoder separates intra-person temporal dynamics from inter-person interactions rather than applying one global self-attention operation across all queries.Motion self-attention models the same individual across frames, while interaction self-attention models relationships between individuals.
4 Experimental Settings
The experiments use a newly constructed synchronized acoustic–pose dataset, controlled indoor capture, adapted multi-person baselines, and standard pose metrics. Evaluation also includes qualitative results and ablation comparisons.
- Dataset: The AMP dataset synchronizes actively sensed acoustic data with 3D coordinates of multiple individuals.It was constructed because no prior dataset existed for acoustic multi-person pose estimation.
- Data collection: Acoustic data were collected indoors with loudspeakers and an ambisonics microphone, while a 16-camera OptiTrack system provided ground-truth poses.The environment included background noise and reverberation.
- Dataset: Fifteen subjects performed randomized multi-person activities using a 21-joint skeleton, with pairings and positions randomized during collection.The subjects were divided into three groups and performed walking, twisting, and raising both hands at random speeds.
- Baselines: The study adapted a single-person acoustic model and a multi-person WiFi model as baselines for acoustic multi-person pose estimation.The acoustic baseline received a multi-person regression head, while the WiFi model was repurposed because its signal representation is functionally analogous.
- Evaluation: Performance was evaluated with MPJPE, PA-MPJPE, and PCK, while the experiments also included baseline comparisons, ablations, and qualitative frame samples.PCK used the PCKh@0.5 threshold, defined as half the head–neck distance; implementation settings included Nout = 8 and Nprev = 16.
5 Experimental Results
Experiments show that SoundMHPE consistently outperforms baselines, with its multi-scale acoustic features and temporal pose decoding supporting multi-person estimation, cross-modal transfer, and some generalization to unseen reflections.
- 5.1 Comparison with Baseline Methods: SoundMHPE outperforms all baselines across every evaluation metric under cross-subject three-fold validation.Training used ten subjects from two groups, while evaluation used the remaining unseen group of five subjects.
- 5.1 Comparison with Baseline Methods: SoundMHPE reconstructs twisting and subtle arm-raising motions more faithfully than baselines in double- and triple-person scenarios.The authors associate this performance with fine-grained frequency features and explicit inter- and intra-person dependency modeling.
- 5.2 Ablation Study: The complete model achieves the highest accuracy across all ablation metrics, with TPD contributing most significantly to performance improvement.The ablation findings emphasize separate queries for predicted pose frames and temporal information extracted through cross-attention.
- 5.3 Comparison by the Number of Subjects: SoundMHPE maintains competitive accuracy from single-person to three-person settings without significant performance degradation.The comparison uses the easiest single-person and most challenging three-person configurations.
- 5.4 Effect of self-attention: Proposed attention mechanisms outperform standard self-attention in both the encoder and decoder, while the (L, 2L, 4L) STFT configuration achieves the highest accuracy.The window-selection study further indicates that high frequency resolution is critical for resolving subtle acoustic variations from fine-grained multi-person movements.
- 5.6 Generalization to unseen environments: SoundMHPE estimates coarse poses in environments with unseen reflection conditions and transfers its temporal modeling effectively to the WiFi-based PiW benchmark.On PiW, the method estimates dynamic poses more accurately than both the original single-frame model and its sequence-extension variant.
6 Conclusion
The paper presents the first active-acoustic approach to multi-person 3D pose estimation, combining multi-scale spectrogram features with temporal pose decoding. SoundMHPE outperforms baselines across all evaluation metrics, while real-world deployment and broader generalization remain challenges.
- Conclusion: The paper presents the first multi-person pose estimation approach using active acoustic sensing with a pair of speakers and a microphone.Its Acoustic Multi-scale Encoder uses diverse temporal and frequency characteristics, while the Temporal Pose Decoder models frame-level pose–acoustic relationships.
- Conclusion: SoundMHPE outperforms baselines across all evaluation metrics, with the AMP dataset supporting evaluation of the proposed method.The paper identifies dataset construction and multi-person acoustic pose estimation as foundational contributions to this early research area.
- Conclusion: Real-world deployment remains challenging because sound-based human pose estimation is still at an early research stage and lacks large-scale comprehensive datasets.Future work targets evaluation across multiple environments and improved robustness and generalization.