Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades
Abstract
We analyze how automatic speech recognition (ASR) errors propagate through ASR-LLM cascades in Korean spoken question answering (SQA), focusing on downstream semantic failures that conventional ASR metrics cannot fully capture. Our analysis shows that the relative downstream degradation caused by ASR errors is consistent across LLMs with different absolute performance, suggesting that cascade degradation largely tracks ASR-stage information loss. We further identify single-character Korean ASR errors as a distinct semantic-failure channel, where the gold answer becomes entirely absent from the downstream prediction despite only a minimal transcription difference. Finally, an auxiliary comparison shows that a large audio language model outperforms an ASR-LLM pipeline with a matched language backbone in noisy Korean SQA, indicating the potential of direct audio input to mitigate transcript-induced information loss.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper makes a significant contribution by analyzing the propagation of ASR errors in Korean spoken question answering and identifying unique error channels that affect downstream performance. The methodology is sound, and the findings have the potential to influence future research and applications in multilingual ASR and QA systems.
Methodology
The paper employs a well-structured methodology to analyze the propagation of ASR errors in Korean spoken question answering (SQA). The authors synthesize speech from text, apply noise to create a controlled error environment, and utilize a range of LLMs to evaluate the impact of ASR errors on downstream QA performance. The focus on single-character errors as a unique semantic failure channel is particularly noteworthy, as it highlights the linguistic nuances of Korean that are often overlooked in similar studies focused on English. However, while the methodology is sound, it could benefit from a more detailed exploration of the implications of the findings across different languages and contexts.
Results & evaluation
The experiments are robust, utilizing a well-defined dataset constructed from KorQuAD with a clear filtering process to ensure quality. The use of multiple SNR levels to induce varying degrees of ASR errors allows for a comprehensive analysis of the degradation in QA performance. The results are presented clearly, showing consistent patterns across different LLMs, which strengthens the validity of the conclusions drawn. However, the reliance on a single ASR model (Whisper-large-v3) and a limited set of LLMs may restrict the generalizability of the findings.
Reproducibility
The paper provides sufficient detail regarding the experimental setup, including the ASR model used, the LLMs evaluated, and the metrics for performance assessment. However, the lack of a publicly available code repository or dataset limits reproducibility. Future work should consider making the experimental framework accessible to facilitate validation and further exploration by other researchers.
Limitations
The study is limited to Korean SQA based on TTS-synthesized speech, which may not fully capture the complexities of real-world spoken language understanding. Additionally, the findings regarding the effectiveness of the disclaimer prompt are somewhat disappointing, as they indicate that awareness of ASR errors does not significantly improve downstream performance, which could lead to misconceptions about the utility of such strategies.
Broader impact
This research has significant implications for the development of more robust ASR systems and LLMs, particularly in languages with unique linguistic structures like Korean. The insights gained from this study could inform the design of future systems that integrate direct audio input, potentially improving user experiences in applications such as voice assistants and chatbots. Furthermore, the identification of specific error types that lead to semantic failures could guide the development of targeted error mitigation strategies. The paper makes a significant contribution by analyzing the propagation of ASR errors in Korean spoken question answering and identifying unique error channels that affect downstream performance. The methodology is sound, and the findings have the potential to influence future research and applications in multilingual ASR and QA systems.
Author of this paper? Send a correction.