Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR
Abstract
Despite advances in large-scale Automatic Speech Recognition (ASR), disfluent speech remains challenging, as state-of-the-art systems are often optimized to omit disfluencies, leading to information loss and hallucinations. Prior work has focused on verbatim transcription and the integration of disfluency markers, but adapting models on limited datasets can lead to catastrophic forgetting of general-domain knowledge. We address this gap by leveraging continual learning (CL) with explicit disfluency tokens. We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms, and then continue training on additional datasets with varying disfluency distributions. Through a detailed analysis of model dynamics during training, we identify a trade-off between marker learning and ASR performance, and a consistent cross-attention head mechanism shared across CL methods.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper makes a substantial contribution to the field of ASR by introducing a continual learning framework that effectively incorporates disfluency markers, thereby enhancing the model's performance on disfluent speech. The rigorous methodology and comprehensive experimental evaluation underscore its significance in advancing ASR technology.
Methodology
The paper presents a novel approach to integrating disfluency markers into a pretrained ASR model through continual learning (CL). The methodology is well-structured, utilizing various CL techniques (EWC, ER, A-GEM, WA) to balance performance on disfluent and non-disfluent speech. The introduction of explicit disfluency tokens is a significant contribution, as it addresses the challenge of catastrophic forgetting while enhancing the model's ability to handle disfluent speech. The use of attention head probing to analyze model dynamics is a strong methodological aspect, providing insights into the internal workings of the model during training.
Results & evaluation
The experiments are comprehensive, employing multiple datasets from the TalkBank repository, which ensures a diverse representation of disfluent speech. The results demonstrate a clear trade-off between ASR performance and marker learning, with detailed metrics reported for both. The paper effectively compares different CL methods, providing a thorough analysis of their performance in retaining knowledge and adapting to new tasks. The results are well-presented, with clear tables and figures that support the findings.
Reproducibility
While the paper provides a detailed description of the methodology and experiments, it lacks specific implementation details that would aid in reproducibility, such as hyperparameter settings, training durations, and the exact configuration of the pretrained model. Including a supplementary material or a dedicated section with this information would enhance reproducibility.
Limitations
One limitation is the reliance on a single backbone model (whisper-small.en), which may restrict the generalizability of the findings. Additionally, the study does not explore the impact of varying dataset sizes or the potential influence of different speaker characteristics on model performance. The focus on specific datasets may also limit the applicability of the results to broader ASR contexts.
Broader impact
The findings have significant implications for improving ASR systems, particularly in clinical settings where understanding disfluent speech is crucial. The approach could lead to better transcription accuracy in applications involving individuals with speech impairments or cognitive challenges. Furthermore, the insights gained from the attention mechanisms could inform future research on model interpretability and robustness in ASR. The paper makes a substantial contribution to the field of ASR by introducing a continual learning framework that effectively incorporates disfluency markers, thereby enhancing the model's performance on disfluent speech. The rigorous methodology and comprehensive experimental evaluation underscore its significance in advancing ASR technology.
Author of this paper? Send a correction.