GSRM: Generative Speech Reward Model for Speech RLHF
Abstract
Recent advances in speech language models, such as GPT-4o Voice Mode and Gemini Live, have demonstrated promising speech generation capabilities. Nevertheless, the aesthetic naturalness of the synthesized audio still lags behind that of human speech. Enhancing generation quality requires a reliable evaluator of speech naturalness. However, existing naturalness evaluators typically regress raw audio to scalar scores, offering limited interpretability of the evaluation and moreover fail to generalize to speech across different taxonomies. Inspired by recent advances in generative reward modeling, we propose the Generative Speech Reward Model (GSRM), a reasoning-centric reward model tailored for speech. The GSRM is trained to decompose speech naturalness evaluation into an interpretable acoustic feature extraction stage followed by feature-grounded chain-of-thought reasoning, enabling explainable judgments. To achieve this, we curated a large-scale human feedback dataset comprising 31k expert ratings and an out-of-domain benchmark of real-world user-assistant speech interactions. Experiments show that GSRM substantially outperforms existing speech naturalness predictors, achieving model-human correlation of naturalness score prediction that approaches human inter-rater consistency. We further show how GSRM can improve the naturalness of speech LLM generations by serving as an effective verifier for online RLHF.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of GSRM, a novel model that enhances speech naturalness evaluation through a combination of acoustic feature extraction and reasoning-based assessments. This approach not only improves the accuracy of naturalness predictions but also provides interpretability, which is crucial for advancing the field of speech synthesis and reinforcement learning from human feedback.
Methodology
The proposed Generative Speech Reward Model (GSRM) introduces a novel approach by integrating acoustic feature extraction with reasoning-based evaluations, which enhances interpretability in assessing speech naturalness. The methodology is well-structured, utilizing a large-scale dataset of expert ratings that strengthens the model's training process. However, the paper could benefit from a more detailed description of the feature extraction techniques and the reasoning framework employed.
Results & evaluation
The experiments are robust, demonstrating GSRM's performance against existing models. The use of a substantial dataset (31k expert ratings) and an out-of-domain benchmark provides a solid foundation for the results. The reported model-human correlation metrics are promising, suggesting that GSRM effectively captures human-like evaluations of speech naturalness. However, more diverse testing scenarios could further validate the model's generalizability.
Reproducibility
The paper lacks sufficient implementation details, such as specific algorithms used for feature extraction and the training process. Without sharing code or a clear methodology, reproducibility may be challenging for other researchers. Providing a GitHub repository or supplementary materials would significantly enhance this aspect.
Limitations
One limitation is the reliance on expert ratings, which may not fully represent the broader population's perception of speech naturalness. Additionally, the model's performance in real-world applications remains to be thoroughly tested, as the current benchmarks are limited to specific datasets.
Broader impact
The GSRM has the potential to significantly improve the quality of synthesized speech in various applications, including virtual assistants, audiobooks, and accessibility tools. By enhancing the naturalness of generated speech, it could lead to more engaging user experiences and broader acceptance of AI-generated audio content. The main contribution of this paper is the introduction of GSRM, a novel model that enhances speech naturalness evaluation through a combination of acoustic feature extraction and reasoning-based assessments. This approach not only improves the accuracy of naturalness predictions but also provides interpretability, which is crucial for advancing the field of speech synthesis and reinforcement learning from human feedback.
Author of this paper? Send a correction.