Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition
Abstract
Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition, achieving state-of-the-art results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The advancements in semi-supervised learning for speech recognition have significant implications for real-world applications, particularly in environments with limited labeled data. The proposed method could enhance accessibility in speech technology, making it more robust to diverse inputs and conditions. Furthermore, the insights gained from this research could be applied to other sequence-to-sequence tasks, broadening its impact across various domains in machine learning. The main contribution of this paper is the introduction of USR 2.0, which significantly enhances the efficiency and robustness of unified speech recognition through innovative pseudo-labelling techniques. The comprehensive evaluation and solid methodology underscore its potential to advance the field of speech recognition, particularly in semi-supervised settings.
Methodology
The paper introduces a novel approach to Unified Speech Recognition (USR) by proposing CTC-driven teacher forcing, which allows for simultaneous prediction of CTC and attention-based pseudo-labels. This method effectively addresses the limitations of autoregressive decoding, particularly in terms of computational efficiency and robustness to out-of-distribution inputs. The mixed sampling strategy further enhances the model's performance by balancing the use of CTC and autoregressive inputs, which is a thoughtful addition to mitigate exposure bias. Overall, the methodology is well-structured, leveraging existing frameworks while innovating on the pseudo-labelling process.
Results & evaluation
The experiments are comprehensive, covering a range of datasets including LRS3, LRS2, and WildVSR. The results demonstrate significant improvements in word error rates (WER) across various conditions, particularly in out-of-distribution scenarios. The paper provides detailed comparisons with state-of-the-art methods, showcasing the advantages of USR 2.0 in terms of both robustness and training efficiency. The evaluation metrics are appropriate, and the results are clearly presented, reinforcing the effectiveness of the proposed method.
Reproducibility
The implementation details are thorough, with clear descriptions of the architecture, training procedures, and hyperparameters. However, the absence of a public code repository limits the reproducibility of the results. While the methodology is well-documented, the lack of accessible code may hinder other researchers from validating the findings independently.
Limitations
One of the key limitations identified is the potential for the CTC-driven teacher forcing to produce globally incoherent outputs, which may affect performance in certain contexts. Additionally, while the mixed sampling strategy is beneficial, it introduces complexity that may require careful tuning to optimize performance across different datasets. The paper also does not address the scalability of the proposed method to even larger datasets or more complex models beyond the tested configurations.
Broader impact
The advancements in semi-supervised learning for speech recognition have significant implications for real-world applications, particularly in environments with limited labeled data. The proposed method could enhance accessibility in speech technology, making it more robust to diverse inputs and conditions. Furthermore, the insights gained from this research could be applied to other sequence-to-sequence tasks, broadening its impact across various domains in machine learning. The main contribution of this paper is the introduction of USR 2.0, which significantly enhances the efficiency and robustness of unified speech recognition through innovative pseudo-labelling techniques. The comprehensive evaluation and solid methodology underscore its potential to advance the field of speech recognition, particularly in semi-supervised settings.
Author of this paper? Send a correction.