DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion
Abstract
Current speech language models generate responses directly without explicit reasoning, leading to errors that cannot be corrected once audio is produced. We introduce \textbf{``Silent Thought, Spoken Answer''} -- a paradigm where speech LLMs generate internal text reasoning alongside spoken responses, with thinking traces informing speech quality. To realize this, we present \method{}, the first diffusion-based speech-text language model supporting both understanding and generation, unifying discrete text and tokenized speech under a single masked diffusion framework. Unlike autoregressive approaches, \method{} jointly generates reasoning traces and speech tokens through iterative denoising, with modality-specific masking schedules. We also construct \dataset{}, the first speech QA dataset with paired text reasoning traces, containing 26K samples totaling 319 hours. Experiments show \method{} achieves state-of-the-art speech-to-speech QA accuracy, outperforming the best baseline by up to 9 points, while attaining the best TTS quality among generative models (6.2\% WER) and preserving language understanding (66.2\% MMLU). Ablations confirm that both the diffusion architecture and thinking traces contribute to these gains.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a groundbreaking approach to integrating reasoning into speech generation through a diffusion-based model, marking a significant advancement in multimodal machine learning. The methodology is innovative, and the results indicate a promising direction for future research in speech language models.
Methodology
The paper introduces a novel diffusion-based speech-text model that integrates reasoning traces into the speech generation process. This approach diverges from traditional autoregressive models by employing a masked diffusion framework, allowing for joint generation of reasoning and speech tokens. The methodology is well-structured, with a clear description of the unified representation of speech and text, and the iterative denoising process is innovative in its application to multimodal tasks. The two-stage training process is also a significant contribution, ensuring effective learning from both speech and text data.
Results & evaluation
The experiments are comprehensive, evaluating the model across multiple tasks including speech-to-speech QA, ASR, and TTS. The results demonstrate state-of-the-art performance, with significant improvements over existing models, particularly in speech QA accuracy and TTS quality. The inclusion of ablation studies strengthens the findings, providing insights into the contributions of the diffusion architecture and reasoning traces. However, the lack of external validation datasets may limit the generalizability of the results.
Reproducibility
The paper provides detailed implementation specifics, including the architecture, training configurations, and dataset construction methods. However, the absence of a publicly available code repository or demo limits reproducibility. Future work should consider releasing code and models to facilitate further research.
Limitations
One limitation is the reliance on a single dataset for training and evaluation, which may introduce biases and limit the model's applicability to diverse real-world scenarios. Additionally, while the reasoning traces improve performance, the model's complexity may hinder real-time applications.
Broader impact
The proposed model has the potential to significantly enhance conversational AI systems by improving the quality and reliability of spoken responses. This could lead to more effective human-computer interactions in various applications, including virtual assistants, educational tools, and accessibility technologies. The ability to reason while speaking could also foster trust in AI systems, as users can better understand the rationale behind responses. This paper presents a groundbreaking approach to integrating reasoning into speech generation through a diffusion-based model, marking a significant advancement in multimodal machine learning. The methodology is innovative, and the results indicate a promising direction for future research in speech language models.
Author of this paper? Send a correction.