Noise-Robust AV-ASR Using Visual Features Both in the Whisper Encoder and Decoder
Abstract
In audiovisual automatic speech recognition (AV-ASR) systems, information fusion of visual features in a pre-trained ASR has been proven as a promising method to improve noise robustness. In this work, based on the prominent Whisper ASR, first, we propose a simple and effective visual fusion method -- use of visual features both in encoder and decoder (dual-use) -- to learn the audiovisual interactions in the encoder and to weigh modalities in the decoder. Second, we compare visual fusion methods in Whisper models of various sizes. Our proposed dual-use method shows consistent noise robustness improvement, e.g., a 35% relative improvement (WER: 4.41% vs. 6.83%) based on Whisper small, and a 57% relative improvement (WER: 4.07% vs. 9.53%) based on Whisper medium, compared to typical reference middle fusion in babble noise with a signal-to-noise ratio (SNR) of 0dB. Third, we conduct ablation studies examining the impact of various module designs and fusion options. Fine-tuned on 1929 hours of audiovisual data, our dual-use method using Whisper medium achieves 4.08% (MUSAN babble noise) and 4.43% (NoiseX babble noise) average WER across various SNRs, thereby establishing a new state-of-the-art in noisy conditions on the LRS3 AV-ASR benchmark. Our code is at https://github.com/ifnspaml/Dual-Use-AVASR
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel dual-use method for integrating visual features in AV-ASR systems, significantly improving noise robustness and establishing new benchmarks in the field. The comprehensive evaluation of the methodology and results demonstrates its potential impact on future developments in automatic speech recognition and multimodal systems.
Methodology
The paper introduces a dual-use method for visual feature integration in AV-ASR systems, leveraging both the encoder and decoder of the Whisper ASR model. This approach is innovative as it aims to enhance noise robustness by modeling audiovisual interactions more effectively than traditional fusion methods. The methodology is well-structured, with clear explanations of the dual-use mechanism and comparisons with existing methods, although further details on the implementation could enhance clarity.
Results & evaluation
The experimental setup is robust, utilizing a significant amount of audiovisual data for fine-tuning and evaluating the proposed method against various baselines. The results demonstrate substantial improvements in word error rates (WER) under noisy conditions, establishing a new state-of-the-art performance. However, the paper could benefit from additional comparative analyses with more diverse models and noise types to strengthen the findings.
Reproducibility
The authors provide a GitHub repository with code, which is a positive aspect for reproducibility. However, the paper lacks detailed descriptions of the training processes and hyperparameters, which could hinder full replication of the experiments by other researchers.
Limitations
One limitation is the reliance on specific datasets (LRS3, LRS2, Voxceleb2) that may not generalize to all AV-ASR applications. Additionally, the paper does not address potential scalability issues when applying the dual-use method to larger models or different languages.
Broader impact
The proposed method has significant implications for real-world applications, particularly in environments with high background noise, such as automotive and smart devices. By improving noise robustness in AV-ASR systems, this research could enhance accessibility and usability in various consumer technologies. The paper presents a novel dual-use method for integrating visual features in AV-ASR systems, significantly improving noise robustness and establishing new benchmarks in the field. The comprehensive evaluation of the methodology and results demonstrates its potential impact on future developments in automatic speech recognition and multimodal systems.
Author of this paper? Send a correction.