Decoding the Ear: A Framework for Objectifying Expressiveness from Human Preference Through Efficient Alignment
Abstract
Recent speech-to-speech (S2S) models generate intelligible speech but still lack natural expressiveness, largely due to the absence of a reliable evaluation metric. Existing approaches, such as subjective MOS ratings, low-level acoustic features, and emotion recognition are costly, limited, or incomplete. To address this, we present DeEAR (Decoding the Expressive Preference of eAR), a framework that converts human preference for speech expressiveness into an objective score. Grounded in phonetics and psychology, DeEAR evaluates speech across three dimensions: Emotion, Prosody, and Spontaneity, achieving strong alignment with human perception (Spearman's Rank Correlation Coefficient, SRCC = 0.86) using fewer than 500 annotated samples. Beyond reliable scoring, DeEAR enables fair benchmarking and targeted data curation. It not only distinguishes expressiveness gaps across S2S models but also selects 14K expressive utterances to form ExpressiveSpeech, which improves the expressive score (from 2.0 to 23.4 on a 100-point scale) of S2S models. Demos and codes are available at https://github.com/FreedomIntelligence/ExpressiveSpeech
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DeEAR, a novel framework for objectively measuring speech expressiveness, which significantly enhances the evaluation and development of speech synthesis models. This work represents a meaningful advancement in the field, addressing a critical gap in the evaluation of speech quality and expressiveness, and offers a scalable solution for improving conversational AI systems.
Methodology
The methodology presented in this paper is robust and innovative, leveraging principles from phonetics and psychology to create a comprehensive framework for evaluating speech expressiveness. The decomposition of expressiveness into three dimensions (Emotion, Prosody, and Spontaneity) is well-justified and allows for a nuanced understanding of speech quality. The use of specialized models for each dimension, along with a fusion function to combine scores, demonstrates a thoughtful approach to addressing the complexity of human expressiveness. The framework's reliance on a limited dataset for training while achieving high correlation with human perception is particularly commendable, showcasing efficiency in data usage.
Results & evaluation
The experiments conducted validate the effectiveness of the DeEAR framework. The strong alignment with human perception, evidenced by high Spearman's and Pearson correlation coefficients, supports the reliability of the proposed metric. Additionally, the application of DeEAR for automated benchmarking of S2S models and data curation is well-executed, with clear improvements in expressiveness scores after fine-tuning with the curated dataset. The use of diverse audio samples for testing enhances the generalizability of the results, although more extensive testing across varied domains could strengthen the findings.
Reproducibility
The paper provides sufficient detail regarding the methodology and experiments, including the datasets used and the model architectures. The availability of code and demos on GitHub enhances reproducibility, allowing other researchers to validate and build upon the work. However, the paper could benefit from clearer descriptions of hyperparameters and training protocols to facilitate easier replication of results.
Limitations
One limitation is the reliance on a relatively small annotated dataset for training, which may affect the generalizability of the model across different languages and dialects. Additionally, while the framework is designed to be efficient, the complexity of the model may pose challenges in real-time applications. The subjective nature of expressiveness also means that the metric may not capture all nuances of human perception, particularly in diverse cultural contexts.
Broader impact
The implications of this research are significant for the fields of speech synthesis and conversational AI. By providing a reliable metric for expressiveness, DeEAR can enhance the development of more engaging and human-like speech systems, with applications in voice assistants, gaming, and mental health support. The creation of the ExpressiveSpeech dataset also contributes to the availability of high-quality resources for training expressive models, potentially advancing research in this area. The main contribution of this paper is the introduction of DeEAR, a novel framework for objectively measuring speech expressiveness, which significantly enhances the evaluation and development of speech synthesis models. This work represents a meaningful advancement in the field, addressing a critical gap in the evaluation of speech quality and expressiveness, and offers a scalable solution for improving conversational AI systems.
Author of this paper? Send a correction.