AISHELL6-whisper: A Chinese Mandarin Audio-visual Whisper Speech Dataset with Speech Recognition Baselines
Abstract
Whisper speech recognition is crucial not only for ensuring privacy in sensitive communications but also for providing a critical communication bridge for patients under vocal restraint and enabling discrete interaction in noise-sensitive environments. The development of Chinese mandarin audio-visual whisper speech recognition is hindered by the lack of large-scale datasets. We present AISHELL6-Whisper, a large-scale open-source audio-visual whisper speech dataset, featuring 30 hours each of whisper speech and parallel normal speech, with synchronized frontal facial videos. Moreover, we propose an audio-visual speech recognition (AVSR) baseline based on the Whisper-Flamingo framework, which integrates a parallel training strategy to align embeddings across speech types, and employs a projection layer to adapt to whisper speech's spectral properties. The model achieves a Character Error Rate (CER) of 4.13% for whisper speech and 1.11% for normal speech in the test set of our dataset, and establishes new state-of-the-art results on the wTIMIT benchmark. The dataset and the AVSR baseline codes are open-sourced at https://zutm.github.io/AISHELL6-Whisper.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This work contributes significantly to the field of whisper speech recognition by providing a comprehensive dataset and a robust model that leverages audio-visual information to improve recognition accuracy. The innovative methodologies and rigorous experimental validation position this research as a valuable resource for future studies in speech recognition and multimodal learning.
Methodology
The paper introduces a novel dataset, AISHELL6-Whisper, which is the largest audio-visual whisper speech dataset for Chinese Mandarin, addressing a significant gap in the existing literature. The methodology employs a parallel training strategy and a projection layer to enhance the performance of the Whisper-Flamingo framework for whisper speech recognition. The integration of visual features through gated cross-attention is a notable advancement, as it leverages multimodal data to improve recognition accuracy. The detailed description of data collection, processing, and the model architecture showcases a well-thought-out approach to tackling the challenges of whisper speech recognition.
Results & evaluation
The experimental results are robust, demonstrating a significant reduction in Character Error Rate (CER) for both whisper and normal speech. The performance metrics provided, including comparisons with existing benchmarks like wTIMIT, highlight the effectiveness of the proposed methods. The experiments also account for various noise conditions, showcasing the model's resilience and adaptability. The use of a large dataset with diverse speakers adds to the credibility of the results.
Reproducibility
The paper mentions that the dataset and the AVSR baseline codes are open-sourced, which is a positive aspect for reproducibility. However, the details regarding the specific configurations and hyperparameters used during training could be more explicitly stated to enhance reproducibility further. The reliance on pre-trained models like Whisper may also introduce variability in results depending on the training conditions of those models.
Limitations
One limitation is the potential overfitting due to the relatively small number of unique sentences per speaker in the wTIMIT dataset, which could affect generalizability. Additionally, while the dataset is large, the diversity of accents and speaking styles in the Mandarin language could be further explored to enhance the model's robustness. The paper could also address the limitations of using a single type of microphone and camera setup during data collection.
Broader impact
The development of this dataset and the associated recognition model has significant implications for applications in healthcare, privacy-sensitive communications, and environments where discreet interaction is necessary. The ability to recognize whisper speech accurately could improve communication for individuals with vocal impairments and enhance user experience in various technology applications, such as virtual assistants and smart devices. This work contributes significantly to the field of whisper speech recognition by providing a comprehensive dataset and a robust model that leverages audio-visual information to improve recognition accuracy. The innovative methodologies and rigorous experimental validation position this research as a valuable resource for future studies in speech recognition and multimodal learning.
Author of this paper? Send a correction.