voice2mode: Phonation Mode Classification in Singing using Self-Supervised Speech Models
Abstract
We present voice2mode, a method for classification of four singing phonation modes (breathy, neutral (modal), flow, and pressed) using embeddings extracted from large self-supervised speech models. Prior work on singing phonation has relied on handcrafted signal features or task-specific neural nets; this work evaluates the transferability of speech foundation models to singing phonation classification. voice2mode extracts layer-wise representations from HuBERT and two wav2vec2 variants, applies global temporal pooling, and classifies the pooled embeddings with lightweight classifiers (SVM, XGBoost). Experiments on a publicly available soprano dataset (763 sustained vowel recordings, four labels) show that foundation-model features substantially outperform conventional spectral baselines (spectrogram, mel-spectrogram, MFCC). HuBERT embeddings obtained from early layers yield the best result (~95.7% accuracy with SVM), an absolute improvement of ~12-15% over the best traditional baseline. We also show layer-wise behaviour: lower layers, which retain acoustic/phonetic detail, are more effective than top layers specialized for Automatic Speech Recognition (ASR).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces voice2mode, a novel framework for singing voice phonation-mode classification that utilizes self-supervised speech models, demonstrating their applicability beyond traditional speech tasks. The comprehensive analysis of the methodology, experiments, and results highlights the significant advancements made in the field of audio processing and vocal analysis.
Methodology
The methodology presented in this paper is robust and innovative, leveraging self-supervised learning models (HuBERT and wav2vec2) for phonation mode classification in singing. The authors effectively extract layer-wise representations from these models and apply global temporal pooling, which is a thoughtful approach to harness the strengths of deep learning architectures. The choice of classifiers (SVM and XGBoost) is appropriate given the dataset size and complexity, and the experiments are well-structured, employing a 5-fold cross-validation strategy that enhances the reliability of the results.
Results & evaluation
The experiments are comprehensive, utilizing a publicly available soprano dataset with a clear definition of phonation modes. The results demonstrate a significant improvement over traditional spectral features, with HuBERT embeddings achieving the highest accuracy. The comparative analysis with baseline features is well-presented, and the layer-wise evaluation provides valuable insights into the model's performance. However, the dataset's size (763 recordings) could limit the generalizability of the findings.
Reproducibility
The authors have made their code publicly available, which is a strong point for reproducibility. The detailed description of the experimental setup, including data preprocessing and classifier training, further supports the ability of other researchers to replicate the study. However, the paper could benefit from more explicit details on hyperparameter tuning and the specific configurations used for the classifiers.
Limitations
One limitation is the reliance on a single dataset from a single soprano singer, which may not capture the diversity of singing voices and styles. Additionally, the study focuses on a simplified set of phonation labels, which may not encompass the full range of vocal qualities present in singing. Future work should aim to include a broader dataset with varied voice types and more complex phonation categories.
Broader impact
The potential applications of this research are significant, particularly in the fields of vocal training and music analysis. The ability to classify phonation modes accurately could lead to the development of intelligent tools for vocal pedagogy, providing real-time feedback to singers. Furthermore, this work bridges the gap between speech and singing research, suggesting that self-supervised speech models can be effectively utilized in music information retrieval and expressive voice analysis. This paper introduces voice2mode, a novel framework for singing voice phonation-mode classification that utilizes self-supervised speech models, demonstrating their applicability beyond traditional speech tasks. The comprehensive analysis of the methodology, experiments, and results highlights the significant advancements made in the field of audio processing and vocal analysis.
Author of this paper? Send a correction.