AudioRole: An Audio Dataset for Character Role-Playing in Large Language Models
Abstract
The creation of high-quality multimodal datasets remains fundamental for advancing role-playing capabilities in large language models (LLMs). While existing works predominantly focus on text-based persona simulation, Audio Role-Playing (ARP) presents unique challenges due to the need for synchronized alignment of semantic content and vocal characteristics. To address this gap, we propose AudioRole, a meticulously curated dataset from 13 TV series spanning 1K+ hours with 1M+ character-grounded dialogues, providing synchronized audio-text pairs annotated with speaker identities and contextual metadata. In addition, to demonstrate the effectiveness of the dataset, we introduced ARP-Eval, a dual-aspect evaluation framework that assesses both response quality and role fidelity. Empirical validation showing GLM-4-Voice trained on AudioRole (which we called ARP-Model) achieve an average Acoustic Personalization score of 0.31, significantly outperforming the original GLM-4-voice and the more powerful model MiniCPM-O-2.6, which specifically supports role-playing in one-shot scenarios. The ARP-Model also achieves a Content Personalization score of 0.36, surpassing the untrained original model by about 38% and maintaining the same level as MiniCPM-O-2.6. AudioRole features dialogues from over 115 main characters, 6 trained ARP-Models that role-play different characters, and evaluation protocols. Together, they provide an essential resource for advancing audio-grounded role-playing research.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents AudioRole, a novel dataset and evaluation framework for audio-grounded character role-playing, significantly advancing the capabilities of large language models in simulating character personas. The comprehensive methodology and rigorous evaluation contribute to the field by addressing the critical interplay between semantic content and vocal characteristics, paving the way for more immersive AI interactions.
Methodology
The methodology presented in this paper is robust and systematic, showcasing a well-defined process for creating the AudioRole dataset. The authors articulate a clear pipeline for speaker diarization, dialogue extraction, and postprocessing, which is essential for ensuring high-quality audio-text pairs. The introduction of ARP-Eval as a dual-aspect evaluation framework is particularly noteworthy, as it addresses both acoustic fidelity and semantic content, which are critical for audio role-playing tasks. However, while the methodology is thorough, it could benefit from additional details on the training procedures for the ARP-Models and the specific metrics used in the evaluation framework.
Results & evaluation
The experimental evaluation is comprehensive, comparing the ARP-Model against state-of-the-art models like GPT-4o Audio and MiniCPM-O-2.6. The results demonstrate significant improvements in Acoustic Personalization and Content Personalization, validating the effectiveness of the AudioRole dataset. The inclusion of both quantitative metrics and qualitative observations strengthens the findings, although the paper could enhance clarity by providing more detailed statistical analysis and visualizations of the results.
Reproducibility
The paper provides a detailed description of the dataset construction and evaluation framework, which aids in reproducibility. However, the lack of a publicly accessible demo or project URL limits the ability of other researchers to replicate the experiments fully. Including code or a demo would significantly enhance reproducibility.
Limitations
The paper acknowledges limitations, such as speaker attribution errors and the evaluation framework's focus on single-turn dialogues. These limitations highlight areas for future research, particularly in improving dataset quality and expanding evaluation metrics to include multi-turn interactions, which are crucial for realistic role-playing scenarios.
Broader impact
The potential applications of this research are significant, particularly in the fields of entertainment, gaming, and human-computer interaction. By enabling more authentic audio-grounded role-playing, this work could enhance user experiences in virtual environments and AI companions. The open-sourcing of the dataset and models could foster further research and innovation in audio role-playing systems. The paper presents AudioRole, a novel dataset and evaluation framework for audio-grounded character role-playing, significantly advancing the capabilities of large language models in simulating character personas. The comprehensive methodology and rigorous evaluation contribute to the field by addressing the critical interplay between semantic content and vocal characteristics, paving the way for more immersive AI interactions.
Author of this paper? Send a correction.