Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization
Abstract
SpeechLLMs typically combine ASR-trained encoders with text-based LLM backbones, leading them to inherit written-style output patterns unsuitable for text-to-speech synthesis. This mismatch is particularly pronounced in Japanese, where spoken and written registers differ substantially in politeness markers, sentence-final particles, and syntactic complexity. We propose a preference-based alignment approach to adapt Japanese SpeechLLMs for speech-worthy outputs: text that is concise, conversational, and readily synthesized as natural speech. To rigorously evaluate this task, we introduce SpokenElyza, a benchmark for Japanese speech-worthiness derived from ELYZA-tasks-100 with auditory verification by native experts. Experiments show that our approach achieves substantial improvement on SpokenElyza while largely preserving performance on the original written-style evaluation. We will release SpokenElyza to support future research on Japanese spoken dialog systems.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel approach to adapting Japanese SpeechLLMs for speech-worthy outputs, introducing the SpokenElyza benchmark and demonstrating the effectiveness of preference-based alignment. This contribution is significant for advancing the field of audio processing and spoken dialog systems, particularly in languages with distinct spoken and written forms.
Methodology
The paper introduces a novel preference-based alignment approach for adapting Japanese SpeechLLMs to produce speech-worthy outputs, addressing a significant gap in the existing literature. The methodology is well-structured, utilizing Direct Preference Optimization (DPO) combined with supervised fine-tuning (SFT) to balance the generation of conversationally natural outputs while maintaining instruction-following capabilities. The construction of the SpokenElyza benchmark is a notable strength, as it incorporates human verification to ensure the quality of the generated speech-worthy text.
Results & evaluation
The experiments are comprehensive, comparing the performance of the proposed method against both the original written-style outputs and the newly created SpokenElyza dataset. The results demonstrate a clear improvement in speech-worthiness while preserving performance on traditional benchmarks, indicating the effectiveness of the proposed approach. The use of LLM-as-Judge for evaluation is a sound choice, although the paper could benefit from more detailed statistical analysis of the results.
Reproducibility
The paper provides a clear description of the model architecture and training procedures, which aids in reproducibility. However, the lack of publicly available code or a demo limits the ability for other researchers to replicate the experiments fully. The paper mentions the use of in-house datasets and models, which may not be accessible to the broader community.
Limitations
One limitation is the focus on the Japanese language, which may restrict the generalizability of the findings to other languages with different spoken and written divergences. Additionally, while the paper addresses the speech-worthiness of responses, it does not explore the potential impact of cultural nuances in conversational styles across different regions in Japan.
Broader impact
The proposed methods and the SpokenElyza benchmark have significant implications for the development of more effective Japanese spoken dialog systems, which can enhance user interactions in various applications, such as virtual assistants and customer service bots. The approach could also inspire similar methodologies in other languages, potentially leading to advancements in multilingual speech synthesis systems. The paper presents a novel approach to adapting Japanese SpeechLLMs for speech-worthy outputs, introducing the SpokenElyza benchmark and demonstrating the effectiveness of preference-based alignment. This contribution is significant for advancing the field of audio processing and spoken dialog systems, particularly in languages with distinct spoken and written forms.
Author of this paper? Send a correction.