PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects
Abstract
While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription. Existing benchmarks suffer from three critical limitations: a pronounced bias towards high-resource languages, a focus on low-level recognition (ASR) rather than semantic reasoning, and a neglect of regional dialects. To bridge this gap, we introduce PolySpeech-100, a massive-scale benchmark designed to assess `native-level' speech comprehension across 110 linguistic variants. We employ a novel hybrid construction pipeline that augments gold-standard human recordings with instruction-driven synthetic speech, allowing us to cover 19 distinct Chinese dialects and over 80 low-resource languages. Extensive evaluation of 22 state-of-the-art models (including Gemini-3, GPT-Audio, and Qwen2.5-Omni) yields pivotal insights. First, we demonstrate that open-source E2E models outperform Cascade (ASR+LLM) systems on heavy dialects, proving that direct audio processing preserves critical paralinguistic cues and prosodic features (e.g., intonation, stress) that are often lost in standard transcription. Second, we reveal a significant performance gap: while commercial models maintain robustness, open-source models suffer catastrophic degradation on low-resource languages. Finally, counter-intuitively, we observe that under standard zero-shot settings, Chain-of-Thought prompting frequently degrades speech understanding performance for most evaluated models, revealing a potential modality alignment gap in current architectures. PolySpeech-100 establishes a rigorous standard for the next generation of inclusive, omni-capable Speech-LLMs. The data, demo, and code are publicly available at https://github.com/YoungSeng/PolySpeech-100.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces PolySpeech-100, a comprehensive benchmark for evaluating Speech-Large Language Models across 110 languages and dialects, addressing critical gaps in existing evaluation methodologies and providing a robust framework for future research in speech understanding.
Methodology
The paper presents a robust hybrid data construction methodology that integrates human recordings with synthetic speech to create a comprehensive benchmark for speech understanding across diverse languages and dialects. The three-stage framework for data aggregation, synthesis, and quality assurance is well-structured, addressing critical gaps in existing benchmarks. The use of a Rewrite-then-Synthesize strategy for dialect adaptation is particularly innovative, allowing for nuanced representation of regional variations.
Results & evaluation
The evaluation of 22 state-of-the-art models across high-resource, dialectal, and low-resource languages is thorough, providing significant insights into model performance. The results highlight the superiority of End-to-End models over traditional ASR systems, particularly in handling dialects. The performance metrics are well-documented, and the analysis of model robustness under various conditions adds depth to the findings.
Reproducibility
The paper includes a project URL with publicly available data, demo, and code, which supports reproducibility. However, the specifics of the experimental setup, including the exact configurations of the models evaluated, could be elaborated for better clarity.
Limitations
The benchmark relies on a multiple-choice question-answering format, which may not fully capture the complexities of real-world conversational dynamics. Additionally, while the benchmark covers a broad range of languages, there are still gaps in representation for certain dialects and languages.
Broader impact
The development of PolySpeech-100 has the potential to significantly advance the field of speech understanding by providing a more inclusive evaluation framework that emphasizes dialectal and low-resource language performance. This could lead to improved models that better serve diverse linguistic communities and enhance accessibility in speech technology. The paper introduces PolySpeech-100, a comprehensive benchmark for evaluating Speech-Large Language Models across 110 languages and dialects, addressing critical gaps in existing evaluation methodologies and providing a robust framework for future research in speech understanding.
Author of this paper? Send a correction.