VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables
Abstract
Earables, such as True Wireless Stereo earphones and VR/AR headsets, are increasingly popular, yet their compact design poses challenges for robust voice-related applications like telecommunication and voice assistant interactions in noisy environments. Existing speech enhancement systems, reliant solely on omnidirectional microphones, struggle with ambient noise like competing speakers. To address these issues, we propose VibOmni, a lightweight, end-to-end multi-modal speech enhancement system for earables that leverages bone-conducted vibrations captured by widely available Inertial Measurement Units (IMUs). VibOmni integrates a two-branch encoder-decoder deep neural network to fuse audio and vibration features. To overcome the scarcity of paired audio-vibration datasets, we introduce a novel data augmentation technique that models Bone Conduction Functions (BCFs) from limited recordings, enabling synthetic vibration data generation with only 4.5% spectrogram similarity error. Additionally, a multi-modal SNR estimator facilitates continual learning and adaptive inference, optimizing performance in dynamic, noisy settings without on-device back-propagation. Evaluated on real-world datasets from 32 volunteers with different devices, VibOmni achieves up to 21% improvement in Perceptual Evaluation of Speech Quality (PESQ), 26% in Signal-to-Noise Ratio (SNR), and about 40% WER reduction with much less latency on mobile devices. A user study with 35 participants showed 87% preferred VibOmni over baselines, demonstrating its effectiveness for deployment in diverse acoustic environments.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of VibOmni, a novel multi-modal speech enhancement system that effectively utilizes bone-conducted vibrations to improve speech quality in noisy environments. This work represents a meaningful advancement in the field of audio processing for wearable technology, offering a unique approach to overcoming the limitations of traditional speech enhancement systems.
Methodology
The paper introduces VibOmni, a multi-modal speech enhancement system that innovatively combines audio and bone-conducted vibration data using a two-branch encoder-decoder deep neural network. The methodology is well-structured, addressing the challenge of limited paired audio-vibration datasets through a novel data augmentation technique that models Bone Conduction Functions (BCFs). This approach is significant as it enables the generation of synthetic vibration data, which is crucial for training the model effectively. The integration of a multi-modal SNR estimator for continual learning and adaptive inference further enhances the robustness of the system in dynamic environments.
Results & evaluation
The experiments conducted on real-world datasets from 32 volunteers provide a solid foundation for evaluating the effectiveness of VibOmni. The reported improvements in PESQ, SNR, and WER demonstrate the system's capability to outperform existing baselines significantly. The user study with 35 participants adds a valuable subjective dimension to the evaluation, indicating a strong preference for VibOmni over traditional methods. However, details on the specific datasets used and the statistical significance of the results could enhance the evaluation's rigor.
Reproducibility
The paper lacks detailed information on the implementation specifics, such as the architecture of the neural network, hyperparameter settings, and training procedures, which are essential for reproducibility. Additionally, the absence of a publicly available code repository or demo limits the ability of other researchers to replicate the results.
Limitations
One notable limitation is the reliance on a relatively small dataset for training and evaluation, which may affect the generalizability of the model across diverse acoustic environments. Furthermore, while the paper addresses the challenge of noise in telecommunication, it does not explore the potential impact of varying user conditions (e.g., different head shapes or ear canal geometries) on the performance of the bone-conduction approach.
Broader impact
The development of VibOmni has significant implications for the usability of earables in noisy environments, potentially enhancing communication for users in various settings, such as crowded public spaces or during physical activities. The integration of bone-conduction technology could also pave the way for future innovations in wearable devices, improving user experience and interaction with voice assistants. The main contribution of this paper is the introduction of VibOmni, a novel multi-modal speech enhancement system that effectively utilizes bone-conducted vibrations to improve speech quality in noisy environments. This work represents a meaningful advancement in the field of audio processing for wearable technology, offering a unique approach to overcoming the limitations of traditional speech enhancement systems.
Author of this paper? Send a correction.