Sound-based Multi-Person 3D Pose Estimation
Abstract
Can we recover the 3D poses of multiple people using only sound? This paper presents the first attempt to estimate multi-person 3D poses solely from acoustic signals. Estimating the poses of multiple individuals using acoustic signals is inherently challenging due to the superposition of motion-dependent signal variations. Unlike single-person scenarios, the presence of multiple subjects leads to overlapping acoustic signatures, making it difficult to attribute specific signal changes to an individual's pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that obscure the temporal motion-acoustic relationship. To address these issues, we propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework consisting of two key components. First, the Acoustic Multi-scale Encoder captures diverse temporal and fine-grained frequency features to isolate subtle acoustic signatures from complex, overlapping signals. Second, the Temporal Pose Decoder employs an attention mechanism to disentangle multi-person information across successive frames. By jointly accounting for temporal dynamics and inter-person dependencies, this component precisely reconstructs frame-wise individual poses. To validate our approach, we constructed the 6-hour Acoustic Multi-person Pose (AMP) dataset consisting of 432K synchronized frames of multi-person pose and acoustic data, and demonstrated that our SoundMHPE outperforms baseline models. Project page: https://oumi03.github.io/sound-mhpe/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents the first comprehensive framework for multi-person 3D pose estimation using active acoustic sensing, introducing a novel dataset and a DETR-inspired temporal decoder that effectively disentangles overlapping acoustic signatures. The technical contributions are significant, particularly the multi-scale encoder and the decoupled self-attention mechanisms, which address the unique challenges of multi-person acoustic superposition. The rigorous experimental validation, including cross-modal generalization to WiFi, strengthens the paper's impact, although the reliance on controlled environments remains a key limitation for real-world deployment.
Methodology
The paper proposes SoundMHPE, a novel encoder-decoder framework for multi-person 3D pose estimation from acoustic signals. The methodology is well-structured, addressing the core challenge of signal superposition in multi-person scenarios. The Acoustic Multi-scale Encoder (AME) effectively utilizes multi-resolution STFT to capture both temporal dynamics and fine-grained frequency features, which is a sound technical choice for handling overlapping acoustic signatures. The Temporal Pose Decoder (TPD) introduces a DETR-like query mechanism adapted for temporal sequences, where multiple queries per person allow for frame-specific attention. The decoupling of self-attention into Motion Self-Attention (intra-person) and Interaction Self-Attention (inter-person) is a thoughtful architectural decision that aligns with the physical nature of the problem. The use of Hungarian matching for loss calculation is standard but appropriate for this detection-like task.
Results & evaluation
The authors constructed a new 6-hour dataset (AMP) with 432K frames, which is a significant contribution given the lack of existing benchmarks for this specific task. The experimental setup includes cross-subject evaluation with 3-fold cross-validation, which is rigorous. The baselines are adapted from single-person acoustic and multi-person WiFi models, which is reasonable given the novelty of the task. The ablation studies are comprehensive, covering the encoder, decoder, attention mechanisms, and window sizes. The cross-modal evaluation on WiFi data (PiW) is a strong point, demonstrating the generalizability of the architectural components. However, the performance metrics (MPJPE, PCK) should be compared against state-of-the-art single-person acoustic methods to provide a clearer benchmark of the "multi-person penalty."
Reproducibility
The paper provides detailed implementation details, including hyperparameters (learning rate, weight decay, number of blocks, query counts) and dataset statistics. The use of standard hardware (OptiTrack, Zoom H3-VR) and software libraries (implied by the architecture) suggests high reproducibility. The project page is provided, which likely contains code and dataset access. The clear definition of the dataset collection protocol (15 subjects, specific poses, randomization) enhances reproducibility.
Limitations
The primary limitation is the reliance on a controlled indoor environment with specific acoustic properties (soundproof room mentioned in Table 1). Generalization to noisy, reverberant, or outdoor environments is not fully explored, though a brief test with partitions is shown. The dataset size, while large for this niche, is still limited compared to RGB datasets, which may constrain the model's robustness to diverse body types and motions. The method assumes a fixed number of maximum people (M=15), which may not scale efficiently to very crowded scenes.
Broader impact
This work opens a new avenue for non-invasive, privacy-preserving pose estimation in scenarios where RGB cameras are unsuitable (darkness, privacy concerns) and wireless signals are obstructed. Applications in healthcare monitoring, sports analysis, and disaster relief are promising. The cross-modal applicability to WiFi suggests that the architectural insights could benefit other signal-based sensing tasks. The construction of the AMP dataset will likely serve as a foundational benchmark for future research in acoustic pose estimation. The paper presents the first comprehensive framework for multi-person 3D pose estimation using active acoustic sensing, introducing a novel dataset and a DETR-inspired temporal decoder that effectively disentangles overlapping acoustic signatures. The technical contributions are significant, particularly the multi-scale encoder and the decoupled self-attention mechanisms, which address the unique challenges of multi-person acoustic superposition. The rigorous experimental validation, including cross-modal generalization to WiFi, strengthens the paper's impact, although the reliance on controlled environments remains a key limitation for real-world deployment.
Author of this paper? Send a correction.