DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units
Abstract
We introduce DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery from discrete speech units. DiscoPhon covers 6 dev and 6 test languages, chosen to span a wide range of phonemic contrasts. Given only 10 hours of speech in a previously unseen language, systems must produce discrete units that are mapped to a predefined phoneme inventory, through either a many-to-one or a one-to-one assignment. The resulting sequences are evaluated for unit quality, recognition and segmentation. We provide four pretrained multilingual HuBERT and SpidR baselines, and show that phonemic information is available enough in current models for derived units to correlate well with phonemes, though with variations across languages.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery, which significantly advances the methodologies available for automatic language processing. The structured approach to evaluation, combined with the provision of robust baseline models and datasets, positions this work as a critical resource for future research in the field.
Methodology
The paper introduces DiscoPhon, a benchmark for unsupervised phoneme discovery, which is a significant advancement in the field of speech processing. The methodology is well-structured, featuring two distinct evaluation tracks (many-to-one and one-to-one mappings) that allow for comprehensive assessment of phoneme discovery systems. The use of self-supervised learning models (HuBERT and SpidR) as baselines is appropriate, and the decision to exclude pretrained models on the benchmark languages ensures the novelty of the evaluation. The integration of various evaluation metrics (PNMI, PER, F1, R-value) provides a robust framework for assessing the quality of discovered phonemes.
Results & evaluation
The experiments are thorough, covering a diverse set of languages with varying phonemic inventories. The results demonstrate clear performance differences between the baseline models, with detailed breakdowns of recognition errors and unit quality metrics. The inclusion of both zero-shot and finetuned conditions allows for a nuanced understanding of model performance. However, the paper could benefit from additional comparative analyses with existing benchmarks to contextualize the results further.
Reproducibility
The authors provide a clear description of the datasets, evaluation metrics, and baseline models, which enhances reproducibility. The open-source availability of datasets and pretrained models is a significant strength, enabling other researchers to build upon this work. However, details on hyperparameter settings and training procedures could be more explicit to facilitate complete replication of the experiments.
Limitations
One limitation noted in the paper is the potential for over-segmentation in HuBERT, which may affect the accuracy of phoneme mapping. Additionally, the benchmark currently relies on pseudo-phonemic annotations that may not capture all phonological distinctions, particularly in languages with complex phonetic systems. The authors also acknowledge the need for broader linguistic diversity in future iterations of the benchmark.
Broader impact
This work has significant implications for the field of linguistics and language documentation, particularly for endangered languages. By providing a robust benchmark for phoneme discovery, it can accelerate research in automatic language processing and contribute to the preservation of linguistic diversity. The ability to discover phoneme inventories without prior exposure to a language can facilitate the documentation of underrepresented languages. The main contribution of this paper is the introduction of DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery, which significantly advances the methodologies available for automatic language processing. The structured approach to evaluation, combined with the provision of robust baseline models and datasets, positions this work as a critical resource for future research in the field.
Author of this paper? Send a correction.