Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
Abstract
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of TextPro-SLM, a speech large language model that effectively minimizes the modality gap by integrating prosody-aware input representations, leading to improved performance in both semantic and paralinguistic tasks. This work represents a significant advancement in the field of speech processing and multimodal AI, addressing a critical challenge in the integration of speech and text modalities.
Methodology
The paper presents a novel approach to bridging the modality gap in speech large language models (SLMs) by focusing on the input side rather than the output. The proposed method, TextPro-SLM, integrates a speech encoder that produces synchronized text tokens and prosody embeddings with a language model backbone that retains the semantic capabilities of text large language models (TLMs). This dual-output mechanism allows the model to better capture both linguistic and paralinguistic information, which is a significant advancement over previous methods that primarily addressed the output modality. The architecture is well-justified, and the multi-task training approach is innovative, leveraging knowledge distillation and paralinguistic understanding tasks to enhance model performance.
Results & evaluation
The experimental results are robust, demonstrating that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales. The authors provide comprehensive evaluations across various benchmarks, including both modality gap assessments and paralinguistic understanding tasks. The results indicate that the model not only minimizes the modality gap effectively but also performs competitively on tasks requiring understanding of prosody. The use of diverse datasets, including LibriSpeech and CommonsenseQA, strengthens the validity of the findings.
Reproducibility
The paper includes detailed descriptions of the architecture, training procedures, and datasets used, which facilitates reproducibility. However, the absence of a publicly available code repository or demo limits the ability for others to fully replicate the study. The authors do provide a thorough explanation of their experimental setup, which is a positive aspect for reproducibility.
Limitations
The study does not address the training or evaluation of a speech synthesis module, which is a critical component of SLMs. This omission means that the results are based solely on text outputs, potentially overlooking important aspects of speech generation quality. Additionally, while the paralinguistic evaluation covers several key attributes, it does not encompass the full range of paralinguistic phenomena, which could limit the generalizability of the findings. The sequential processing of speech input may also introduce latency issues in real-time applications.
Broader impact
The proposed methodology has significant implications for the development of more effective and efficient speech-based AI systems. By reducing the modality gap from the input side, this work could enhance the usability of SLMs in various applications, including virtual assistants, automated customer service, and accessibility technologies. The focus on paralinguistic understanding could also lead to more nuanced and human-like interactions in speech-based systems. The main contribution of this paper is the introduction of TextPro-SLM, a speech large language model that effectively minimizes the modality gap by integrating prosody-aware input representations, leading to improved performance in both semantic and paralinguistic tasks. This work represents a significant advancement in the field of speech processing and multimodal AI, addressing a critical challenge in the integration of speech and text modalities.
Author of this paper? Send a correction.