End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
Abstract
Dysarthric speech reconstruction (DSR) typically employs a cascaded system that combines automatic speech recognition (ASR) and sentence-level text-to-speech (TTS) to convert dysarthric speech into normally-prosodied speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicit-implicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge distillation. Our system has an average response time of 1.03 seconds on Tesla A100, with an average real-time factor (RTF) of 0.71. On the UASpeech dataset, it attains a mean opinion score (MOS) of 4.67 and demonstrates a 54.25% relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: https://wflrz123.github.io/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel end-to-end simultaneous dysarthric speech reconstruction system that effectively addresses the challenges of intelligibility and latency through innovative methodologies. The technical contributions are significant, with promising experimental results that indicate a meaningful advancement in the field of speech processing for individuals with speech impairments.
Methodology
The proposed end-to-end simultaneous dysarthric speech reconstruction (E2E-SDSR) system introduces innovative components such as a frame-level adaptor module and a multiple wait-k autoregressive TTS module. The frame-level adaptor effectively bridges the gap between ASR and TTS, enhancing the robustness of the system against ASR errors through explicit-implicit semantic information fusion. The multiple wait-k strategy in the TTS module allows for flexibility in processing, balancing latency and prosody quality. The methodology is well-structured, with a clear focus on addressing the unique challenges posed by dysarthric speech, particularly in terms of intelligibility and naturalness.
Results & evaluation
The experiments are comprehensive, utilizing both a commercial dysarthric speech dataset and the UASpeech dataset. The reported results, including a mean opinion score (MOS) of 4.67 and a 54.25% reduction in word error rate (WER), demonstrate significant improvements over existing methods. The ablation studies provide valuable insights into the contributions of each component of the proposed system, reinforcing the effectiveness of the adaptor and wait-k strategies.
Reproducibility
While the paper provides a detailed description of the methodology and experimental setup, it lacks specific implementation details such as hyperparameter settings, training duration, and the exact architecture configurations used. The absence of a publicly available code repository limits the reproducibility of the results.
Limitations
The study primarily focuses on dysarthric speech and may not generalize well to other speech disorders or languages. Additionally, the reliance on a limited dataset for training and testing could affect the robustness of the model in real-world applications. The paper does not address potential biases in the dataset or the implications of using commercial data.
Broader impact
The proposed system has the potential to significantly improve communication for individuals with dysarthria, enhancing their quality of life and social interactions. By providing a more efficient and intelligible speech reconstruction method, it could be applied in various assistive technologies and communication devices. The paper presents a novel end-to-end simultaneous dysarthric speech reconstruction system that effectively addresses the challenges of intelligibility and latency through innovative methodologies. The technical contributions are significant, with promising experimental results that indicate a meaningful advancement in the field of speech processing for individuals with speech impairments.
Author of this paper? Send a correction.