Speaker head orientation estimation with a single microphone array using phase spectrogram features
Abstract
Estimating a speaker's head orientation from audio can provide valuable information in smart environments, meetings, and driver monitoring. We propose a novel approach that leverages the phase component of the short-time Fourier transform from a single microphone array as input to a deep neural network combining convolutional, recurrent, and self-attention layers. Unlike prior methods that use physics-informed handcrafted features or raw waveform inputs, our approach enables robust learning from simulated and real data. Trained on a large-scale dataset generated with voice directivity patterns and fine-tuned on real recordings, our model achieves state-of-the-art accuracy, outperforming baselines under both clean and noisy conditions. Personalization experiments further demonstrate significant gains, reaching a mean angular error of 11.3 degrees when adapting to individual users and environments.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a robust deep learning approach for speaker head orientation estimation using STFT phase features, demonstrating superior performance over raw audio and handcrafted features in noisy and reverberant environments, with significant gains achieved through user-specific personalization.
Methodology
The paper proposes a deep learning framework for speaker head orientation estimation using a single microphone array. The core methodological contribution is the use of the phase component of the Short-Time Fourier Transform (STFT) as the primary input feature, rather than raw waveforms or handcrafted features like GCC-PHAT. The architecture combines 2D Convolutional Neural Networks (CNNs) for spatial feature extraction, Bidirectional Gated Recurrent Units (GRUs) for temporal modeling, and Multi-Head Self-Attention mechanisms. The input representation involves stacking the sine and cosine of the phase from all microphone channels. The model predicts orientation in a continuous regression format using sine/cosine representation to handle circular continuity. The approach is technically sound, leveraging established deep learning components in a novel configuration for this specific acoustic task. The use of phase-only features is the key differentiator, justified by the hypothesis that phase carries robust directional cues, particularly in reverberant environments.
Results & evaluation
The evaluation is comprehensive, covering both simulated and real-world data. The simulated dataset is generated using Voice Directivity Patterns (VDP) and the VCTK corpus, with room acoustics simulated via the image source method. This allows for controlled testing across various noise levels (clean, moderate, high SNR) and reverberation conditions. The paper compares the proposed method against three baselines: Soundr (CNN+LSTM on raw audio), a method based on ITD/ILD features, and a physics-informed feature-based method. Results indicate that the phase-based model outperforms baselines in noisy and reverberant conditions, which is a significant finding given the sensitivity of phase to environment. Personalization experiments (fine-tuning on user/room data) show significant performance gains, reducing Mean Angular Error (MAE) to 11.3 degrees. The evaluation on a real-world dataset further validates the generalization capability, although the absolute performance metrics on real data are not explicitly detailed in the text provided (referenced as Table REF). The ablation of reverberation effects (anechoic vs. reverberant) provides valuable insight into the model's behavior.
Reproducibility
The paper provides sufficient detail for reproduction. The STFT parameters (4ms window, 2ms stride), network architecture (3 conv layers, 2 GRU layers, 2 attention blocks), and training details (Adam optimizer, MSE loss, 200k iterations) are specified. The dataset generation process is clearly described, including the use of VCTK, VDPs, and Pyroomacoustics for simulation. The noise augmentation strategy (phase-randomized monophonic noise) is also detailed. However, the specific hyperparameters for the personalization fine-tuning (learning rate, number of epochs) are not explicitly stated, which might require some trial and error for exact reproduction. The code is not explicitly linked, but the methodology is clear enough to implement.
Limitations
The paper acknowledges several limitations. First, the model's generalization to unseen users and environments is limited, necessitating personalization/fine-tuning for optimal performance. This suggests the model relies heavily on specific acoustic characteristics of the training data. Second, the reliance on simulated data for pre-training, while effective, introduces a domain gap that requires real-world fine-tuning. Third, the phase-only input might discard magnitude information that could be beneficial in certain conditions, although the authors claim no benefit was found. Finally, the evaluation is limited to azimuthal orientation; elevation estimation is not addressed. The performance in highly non-stationary noise or with significant speaker movement is not evaluated.
Broader impact
Accurate and privacy-preserving head orientation estimation has significant implications for human-computer interaction, smart homes, meeting transcription systems, and driver monitoring. By using audio-only sensors, this technology avoids the privacy concerns associated with cameras. The ability to work with compact microphone arrays makes it deployable in consumer devices. The focus on robustness to noise and reverberation enhances its practical utility in real-world environments. The personalization aspect highlights the need for adaptive systems in user-centric AI. This paper presents a robust deep learning approach for speaker head orientation estimation using STFT phase features, demonstrating superior performance over raw audio and handcrafted features in noisy and reverberant environments, with significant gains achieved through user-specific personalization.
Author of this paper? Send a correction.