ReHear: Iterative Pseudo-Label Refinement for Semi-Supervised Speech Recognition via Audio Large Language Models
Abstract
Semi-supervised learning in automatic speech recognition (ASR) typically relies on pseudo-labeling, which often suffers from confirmation bias and error accumulation due to noisy supervision. To address this limitation, we propose ReHear, a framework for iterative pseudo-label refinement that integrates an instruction-tuned, audio-aware large language model (LLM) into the self-training loop. Unlike conventional text-based correctors, our approach conditions the LLM on both the ASR hypothesis and the source audio, allowing it to recover phonetically accurate transcripts even from severe recognition errors. These refined pseudo-labels serve as high-fidelity targets for fine-tuning the ASR model in an iterative cycle. Experimental results across diverse benchmarks demonstrate that ReHear effectively mitigates error propagation, consistently outperforming both supervised and pseudo-labeling baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the ReHear framework, which utilizes an audio-aware large language model for iterative pseudo-label refinement in semi-supervised speech recognition. This innovative approach addresses the limitations of traditional pseudo-labeling methods and demonstrates significant improvements in ASR performance across diverse datasets, highlighting its potential for practical applications in real-world scenarios.
Methodology
The proposed ReHear framework introduces an innovative approach to semi-supervised learning in ASR by integrating an audio-aware large language model (LLM) into the pseudo-labeling process. This methodology diverges from traditional filtering techniques by actively refining noisy labels, which is a significant advancement in addressing error propagation in ASR systems. The iterative self-training loop, where the ASR model and LLM mutually enhance each other's performance, is a novel concept that shows promise in improving the quality of pseudo-labels. The use of multimodal inputs (both audio and text) for conditioning the LLM is particularly noteworthy, as it allows for more accurate corrections compared to text-only methods.
Results & evaluation
The experiments conducted across diverse datasets (Earnings-21, Earnings-22, SPGISpeech, and AMI) demonstrate the effectiveness of the ReHear framework. The results indicate consistent improvements in word error rates (WER) compared to both supervised and traditional pseudo-labeling methods. The ablation studies provide valuable insights into the contributions of different components of the framework, such as the impact of audio conditioning and the effectiveness of various decoding strategies. However, the paper could benefit from a more detailed statistical analysis of the results to strengthen its claims.
Reproducibility
The paper outlines the experimental setup, including datasets and model configurations, which is crucial for reproducibility. However, it lacks specific implementation details, such as hyperparameters and code availability, which would enhance the ability of other researchers to replicate the results. Providing a link to a code repository or supplementary materials would significantly improve reproducibility.
Limitations
One limitation of the proposed method is its reliance on the quality of the initial ASR model, as any inherent biases or inaccuracies in the ASR predictions could still affect the final output. Additionally, while the framework shows promise, its performance in highly noisy or diverse acoustic environments remains to be fully evaluated. The paper also does not address the computational costs associated with training the LLM, which may limit its practical application in resource-constrained settings.
Broader impact
The ReHear framework has the potential to significantly impact the field of automatic speech recognition, particularly in low-resource scenarios where labeled data is scarce. By leveraging unlabeled audio data effectively, it could enable more widespread deployment of ASR systems across various domains, including finance, healthcare, and customer service. The approach may also inspire further research into multimodal learning and the integration of large language models in other areas of machine learning. The main contribution of this paper is the introduction of the ReHear framework, which utilizes an audio-aware large language model for iterative pseudo-label refinement in semi-supervised speech recognition. This innovative approach addresses the limitations of traditional pseudo-labeling methods and demonstrates significant improvements in ASR performance across diverse datasets, highlighting its potential for practical applications in real-world scenarios.
Author of this paper? Send a correction.