Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages
Abstract
Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the Orthographically Informed Word Error Rate (OIWER) metric, which provides a more accurate evaluation of ASR systems for Indian languages by accounting for orthographic variations. This work is significant as it not only improves the evaluation metrics for ASR but also enhances the understanding of language diversity in speech recognition, paving the way for better performance in under-resourced languages.
Methodology
The paper presents a novel framework for evaluating Automatic Speech Recognition (ASR) systems for Indian languages by introducing the Orthographically Informed Word Error Rate (OIWER). This approach effectively captures permissible orthographic variations, which are crucial for under-resourced languages. The methodology leverages large language models (LLMs) to generate context-aware variations, significantly reducing the reliance on manual curation. The human-in-the-loop approach for refining these variations ensures a balance between automation and accuracy, making the methodology robust and innovative.
Results & evaluation
The experiments are comprehensive, involving a wide range of ASR models across 22 Indian languages. The results demonstrate a consistent improvement in evaluation metrics when using OIWER compared to traditional WER, with significant reductions in error rates and better alignment with human perception. The paper provides clear statistical evidence of the improvements, including specific performance metrics that highlight the advantages of the proposed method.
Reproducibility
The paper includes a GitHub repository link for the implementation, which is a positive aspect for reproducibility. However, the paper could benefit from more detailed descriptions of the datasets used, the specific configurations of the models, and the exact parameters for the experiments to enhance reproducibility further.
Limitations
One limitation noted is the dependency on human input for refining the generated variations, which could introduce variability in the results. Additionally, while the OIWER metric shows improvements, there remains a gap between OIWER and perceived WER for certain languages, indicating that further refinements may be needed to fully capture the nuances of human perception in ASR evaluations.
Broader impact
The proposed framework has significant implications for the development of ASR systems in low-resource languages, particularly in the Indian context. By addressing the challenges of orthographic variations, this work can enhance the accessibility of technology for speakers of these languages, potentially leading to broader applications in education, communication, and technology adoption. The main contribution of this paper is the introduction of the Orthographically Informed Word Error Rate (OIWER) metric, which provides a more accurate evaluation of ASR systems for Indian languages by accounting for orthographic variations. This work is significant as it not only improves the evaluation metrics for ASR but also enhances the understanding of language diversity in speech recognition, paving the way for better performance in under-resourced languages.
Author of this paper? Send a correction.