NVMOS: Non-Verbal Vocalization Quality Assessment in Speech
Abstract
Non-verbal vocalizations (NVs), such as laughter, sighs, and coughs, are important acoustic cues for emotion and intent. Existing speech quality assessment methods typically focus on overall naturalness, while non-verbal TTS evaluations mainly examine whether a target NV appears with the correct type and position. However, the perceptual quality of NV events themselves remains underexplored. To address this gap, we construct an NV-MOS dataset containing outputs from multiple NV-TTS systems and naturally occurring NV samples, with ratings collected from three acoustic experts on a perceptual quality scale. We further analyze audio-capable multimodal large language models such as Gemini and find clear inconsistencies between their scores and expert ratings. These results suggest that general-purpose multimodal models cannot reliably replace human judgments for NV quality assessment. We then propose NVMOS, to our knowledge the first model that can reliably predict the perceptual quality of NV events in speech. Experimental results show that, with a local NV-event focusing module, NVMOS reaches expert-level or stronger agreement with human MOS.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces NV-MOS, an expert-rated dataset and modeling framework for perceptual quality assessment of non-verbal vocalizations in speech. This work significantly advances the understanding and evaluation of NVs, offering a dedicated approach that outperforms existing general-purpose models and highlighting the need for specialized tools in this domain.
Methodology
The methodology presented in this paper is robust and well-structured. The authors construct the NV-MOS dataset, which is a significant contribution to the field, as it provides a dedicated resource for evaluating non-verbal vocalizations in speech. The proposed NVMOS model employs a novel local NV-event focusing module that effectively utilizes cross-attention mechanisms to assess the perceptual quality of NVs in a way that traditional models do not. This approach is innovative as it combines audio signal processing with textual context, allowing for a more nuanced evaluation of NVs.
Results & evaluation
The experimental evaluation is thorough, utilizing a well-defined dataset with expert ratings to validate the performance of the NVMOS model. The results demonstrate that NVMOS achieves a high level of agreement with human expert ratings, outperforming general-purpose multimodal models. The correlation metrics reported (Pearson, Spearman, and Kendall) provide a strong basis for assessing the model's effectiveness. Additionally, the ablation study effectively highlights the importance of the tag-centered query in improving model performance.
Reproducibility
The paper provides sufficient details regarding the experimental setup, including data splitting, model architecture, and training procedures. However, the lack of a publicly available project URL or demo limits the reproducibility of the results. Future work should consider releasing the code and dataset to facilitate further research in this area.
Limitations
One limitation of the study is the reliance on expert ratings, which, while valuable, may introduce subjectivity into the evaluation process. Additionally, the dataset may not cover all possible NV scenarios, potentially limiting the generalizability of the findings. The performance of the NVMOS model may also vary with different NV types or in more complex acoustic environments.
Broader impact
The implications of this research are significant, particularly in the fields of speech synthesis and human-computer interaction. By improving the quality assessment of non-verbal vocalizations, this work can enhance the expressiveness and naturalness of generated speech in applications such as virtual assistants, gaming, and emotional AI. The findings also highlight the limitations of current multimodal models, paving the way for more specialized approaches in audio quality assessment. The paper introduces NV-MOS, an expert-rated dataset and modeling framework for perceptual quality assessment of non-verbal vocalizations in speech. This work significantly advances the understanding and evaluation of NVs, offering a dedicated approach that outperforms existing general-purpose models and highlighting the need for specialized tools in this domain.
Author of this paper? Send a correction.