CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonese
Abstract
Automatic speech recognition (ASR) is critical for language accessibility, yet low-resource Cantonese remains challenging due to limited annotated data, six lexical tones, tone sandhi, and accent variation. Existing ASR models, such as Whisper, often suffer from high word error rates. Large audio-language models (LALMs), in contrast, can leverage broader contextual reasoning but still require explicit tonal and prosodic acoustic cues. We introduce CantoASR, a collaborative ASR-LALM error correction framework that integrates forced alignment for acoustic feature extraction, a LoRA-finetuned Whisper for improved tone discrimination, and an instruction-tuned Qwen-Audio for prosody-aware correction. Evaluations on spontaneous Cantonese data show substantial CER gains over Whisper-Large-V3. These findings suggest that integrating acoustic cues with LALM reasoning provides a scalable strategy for low-resource tonal and dialectal ASR.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
CantoASR presents a novel approach to enhancing ASR for low-resource Cantonese by integrating prosody-aware correction with existing ASR models. The combination of acoustic feature extraction and LALM reasoning represents a meaningful step forward in addressing the unique challenges of tonal languages, though further work is needed to ensure reproducibility and broader applicability.
Methodology
The methodology employed in CantoASR is innovative as it combines forced alignment for acoustic feature extraction with a LoRA-finetuned Whisper model for tone discrimination and an instruction-tuned Qwen-Audio for prosody-aware correction. This multi-faceted approach is particularly well-suited for the challenges posed by low-resource Cantonese, addressing tonal complexities and accent variations that are often overlooked in traditional ASR systems. The integration of LALM reasoning with acoustic cues is a notable advancement, though the paper could benefit from a more detailed explanation of the interaction between these components.
Results & evaluation
The evaluation of CantoASR on spontaneous Cantonese data is a strong point, demonstrating substantial character error rate (CER) improvements over the Whisper-Large-V3 model. However, the paper lacks detailed descriptions of the datasets used, including size, diversity, and how they were annotated. Additionally, comparisons with other state-of-the-art models beyond Whisper would strengthen the claims of superiority. The results suggest a promising direction for ASR in low-resource languages, but further validation across different datasets and conditions would enhance credibility.
Reproducibility
The paper does not provide sufficient implementation details or code availability, which raises concerns about reproducibility. While the methodology is described, the absence of a clear protocol for replicating the experiments or access to the trained models limits the ability of other researchers to validate the findings independently.
Limitations
One significant limitation is the reliance on specific models (Whisper and Qwen-Audio) which may not generalize well to other contexts or languages. Additionally, the paper does not address potential biases in the training data or the performance of the model across different dialects of Cantonese. The complexity of the proposed system may also pose challenges in real-world applications, particularly in resource-constrained environments.
Broader impact
The potential applications of CantoASR are significant, particularly in improving accessibility for Cantonese speakers in various domains such as education, healthcare, and technology. By addressing the challenges of low-resource languages, this research could contribute to greater inclusivity in voice-activated systems and promote the preservation of linguistic diversity. However, the impact will depend on the successful deployment and adaptation of the system in real-world scenarios. CantoASR presents a novel approach to enhancing ASR for low-resource Cantonese by integrating prosody-aware correction with existing ASR models. The combination of acoustic feature extraction and LALM reasoning represents a meaningful step forward in addressing the unique challenges of tonal languages, though further work is needed to ensure reproducibility and broader applicability.
Author of this paper? Send a correction.