GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
Abstract
While modern ASR systems achieve low error rates on high-resource benchmarks, such performance often overestimates real-world robustness. Existing evaluations address challenges in isolation, lacking a unified benchmark for domain terminology, age variation, dialects, accents, and low-resource languages, particularly across the Middle East and Southeast Asia, representing over one billion under-evaluated speakers. To address this gap, we introduce GigaSpeechBench, a comprehensive multilingual and multidimensional in-the-wild ASR & AST benchmark comprising 680 hours of human-annotated speech. It features five modules: (1) 12 low-resource Middle Eastern and Southeast Asian languages, plus challenging Japanese and Korean; (2) 6 Chinese dialects; (3) 6 English accents; (4) dense terminology across 12 vertical domains for Chinese and English; and (5) older adult and child speech. We further provide human-annotated Chinese and English translations for 11 languages to support AST evaluation. Extensive evaluations of leading foundation models and commercial APIs reveal significant performance degradation in these challenging settings, exposing critical evaluation blind spots.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GigaSpeechBench addresses critical gaps in ASR evaluation by providing a unified, multidimensional benchmark for underrepresented languages, dialects, and real-world acoustic conditions, revealing significant robustness deficits in current foundation models.
Methodology
The paper introduces GigaSpeechBench, a comprehensive benchmark designed to evaluate Automatic Speech Recognition (ASR) systems on underrepresented and challenging dimensions. The methodology focuses on data curation rather than algorithmic innovation. The authors employ a pipeline involving heuristic screening of YouTube videos, manual transcription by professional annotators, and rigorous quality control to create a dataset of 680 hours of "in-the-wild" speech. The benchmark is structured into five distinct modules: low-resource languages (Middle Eastern/Southeast Asian), Chinese dialects, accented English, vertical domain terminology, and age-variant speech (children/elderly). The technical contribution lies in the systematic construction of this multidimensional testbed and the definition of specific evaluation metrics, such as Biased Word Error Rate (B-WER) for domain terminology. While the curation process is robust, the methodological novelty is primarily in the scope and diversity of the data collection rather than in novel computational techniques.
Results & evaluation
The experimental evaluation is extensive and serves as the core contribution of the paper. The authors benchmark a wide array of state-of-the-art systems, including commercial APIs (Azure, Google Chirp, OpenAI, Gemini, ElevenLabs) and open-source foundation models (Whisper, Qwen3-ASR, FunASR, Dolphin, NeMo, Meta OmniASR). The results consistently demonstrate that high performance on standard benchmarks (like Common Voice or FLEURS) does not transfer to these challenging settings. Key findings include significant performance degradation in low-resource languages, particularly Arabic dialects and Southeast Asian languages; poor robustness to accented English; and substantial errors in recognizing dense domain-specific terminology. The inclusion of human-annotated translations for Speech-to-Text (AST) evaluation adds another layer of rigorous assessment. The use of B-WER provides a more granular view of entity recognition capabilities, revealing that aggregate WER often masks critical failures in specialized domains.
Reproducibility
The paper provides high reproducibility standards. The dataset is released on Hugging Face, and the code/evaluation scripts are available on GitHub. The annotation protocol is detailed, including criteria for video selection, segmentation, and quality control (98%+ transcription accuracy). The temporal hold-out strategy (using data from the past year) is explicitly mentioned to mitigate data contamination, which is a critical factor for reproducible benchmarking in the era of large pre-trained models. The detailed breakdown of metrics and the provision of hotword lists for domain evaluation further support reproducibility.
Limitations
The authors acknowledge several limitations. Text normalization for low-resource languages may lack the refinement of native linguistic experts. Chinese dialects often lack unified standard writing systems, leading to transliteration ambiguities that make Character Error Rate (CER) an imperfect metric for some dialects (e.g., Min). The dataset is sourced from YouTube, which may introduce biases related to the demographics of YouTube users in the target regions. Additionally, the benchmark focuses on spontaneous speech, which, while realistic, may not cover all formal or scripted use cases. The evaluation of older adult and child speech is limited to 10 hours per group, which might not fully capture the variance within these demographic groups.
Broader impact
This benchmark has significant broader impact by highlighting the "evaluation blind spots" in current ASR systems. By exposing the poor performance on low-resource languages and dialects, it underscores the risk of exacerbating digital inequality if models are only optimized for high-resource, standard varieties. The focus on domain terminology is crucial for deploying ASR in professional settings (medicine, law, finance). The release of this benchmark encourages the research community to develop more robust, inclusive, and context-aware ASR systems, potentially leading to better service for over one billion under-evaluated speakers. GigaSpeechBench addresses critical gaps in ASR evaluation by providing a unified, multidimensional benchmark for underrepresented languages, dialects, and real-world acoustic conditions, revealing significant robustness deficits in current foundation models.
Author of this paper? Send a correction.