HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition
Abstract
Despite advances in multilingual automatic speech recognition (ASR), code-switching (CS), the mixing of languages within an utterance common in daily speech, remains a severely underexplored challenge. In this paper, we introduce HiKE: the Hierarchical Korean-English code-switching benchmark, the first globally accessible evaluation framework for Korean-English CS, aiming to provide a means for the precise evaluation of multilingual ASR models and to foster research in the field. The proposed framework not only consists of high-quality, natural CS data across various topics, but also provides meticulous loanword labels and a hierarchical CS-level labeling scheme (word, phrase, and sentence) that together enable a systematic evaluation of a model's ability to handle each distinct level of code-switching. Through evaluations of diverse multilingual ASR models and fine-tuning experiments, this paper demonstrates that while most multilingual ASR models initially struggle with CS-ASR, this capability can be enabled through fine-tuning with CS data. HiKE will be available at https://github.com/ThetaOne-AI/HiKE.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces HiKE, the first public benchmark for Korean-English code-switching ASR, significantly advancing the evaluation of multilingual ASR models. The comprehensive methodology and experimental results provide critical insights into the challenges and opportunities in code-switching speech recognition, marking a meaningful contribution to the field.
Methodology
The methodology is robust, introducing a hierarchical evaluation framework that addresses the unique challenges of Korean-English code-switching in ASR. The authors meticulously crafted a dataset through a combination of human and AI-generated scripts, ensuring quality and relevance. The hierarchical labeling scheme is innovative, allowing for a nuanced evaluation of ASR models across different levels of code-switching. The approach to fine-tuning models with both intra-sentential and inter-sentential data is well-justified and demonstrates a clear understanding of the complexities involved in CS-ASR.
Results & evaluation
The experiments are comprehensive, evaluating nine multilingual ASR models using established metrics (MER and PIER). The results provide valuable insights into the performance of these models in CS contexts, revealing significant performance gaps between monolingual and code-switching scenarios. The fine-tuning experiments effectively illustrate the potential for improving CS-ASR capabilities, although the limited scale of the dataset may restrict broader applicability.
Reproducibility
The paper provides sufficient detail on the dataset creation process, model evaluation, and fine-tuning procedures, which enhances reproducibility. However, the lack of a public demo or interactive component limits immediate accessibility for other researchers.
Limitations
The primary limitations include the restricted availability of ASR models that support both Korean and English, which limits comparative analysis. Additionally, the dataset's size may hinder the exploration of more extensive fine-tuning strategies and the generalization of findings across different language pairs.
Broader impact
This work has significant implications for improving ASR systems for multilingual users, particularly in regions where code-switching is prevalent. By providing a publicly accessible benchmark, it fosters further research and development in the field, potentially enhancing communication technologies for bilingual speakers. The paper introduces HiKE, the first public benchmark for Korean-English code-switching ASR, significantly advancing the evaluation of multilingual ASR models. The comprehensive methodology and experimental results provide critical insights into the challenges and opportunities in code-switching speech recognition, marking a meaningful contribution to the field.
Author of this paper? Send a correction.