Preserving Speech-to-Text LLM Capabilities in Speech-to-Speech Generation
Abstract
Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can degrade the original S2T performance, while attaching a downstream talker reintroduces a serial text-to-speech bottleneck. We present PRIME-Speech, a frozen-backbone S2S conversion framework that trains only speech-generation modules. PRIME-Speech synchronizes a causal audio post-decoder with intermediate hidden states of the frozen backbone, so codec tokens are generated from the model's evolving reasoning trajectory rather than from completed text chunks. The post-decoder uses mixed hidden-state, text, and audio-history conditioning, and a training-time packing strategy with turn-level audio KV-cache and position reset stabilizes multi-turn spoken interaction without additional multi-turn S2S training data. Multi-token prediction further reduces the effective codec prediction rate and improves first-audio latency without modifying the reasoning path. Across speech translation, spoken QA, speech understanding, and multi-turn dialogue, PRIME-Speech preserves the S2T behavior of the frozen backbone while producing accurate, low-WER spoken responses.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant advancement in Speech-to-Speech generation by introducing a frozen-backbone framework that effectively balances the preservation of reasoning capabilities with the generation of low-latency, high-quality speech, addressing key limitations in current S2S architectures.
Methodology
The paper proposes PRIME-Speech, a novel framework for Speech-to-Speech (S2S) generation that addresses the critical trade-off between preserving the reasoning capabilities of a frozen Speech-to-Text (S2T) backbone and generating high-quality speech output. The core innovation lies in "hidden-state synchronization," where a trainable causal audio post-decoder is attached to intermediate hidden states of the frozen backbone, allowing speech tokens to be generated in parallel with text tokens from the evolving reasoning trajectory. This avoids the serial bottleneck of traditional cascade systems and the catastrophic forgetting associated with full fine-tuning. The methodology includes a sophisticated multi-token prediction (MTP) strategy for efficiency and a specific multi-turn cache policy (accumulating text KV-cache while resetting audio KV-cache) to prevent acoustic drift across turns. The approach is technically sound, leveraging the strengths of large language models while introducing a clean interface for audio generation.
Results & evaluation
The experimental evaluation is comprehensive, covering speech translation (FLEURS, CoVoST-2), spoken question answering (UltraEval-Audio), speech understanding (BigBench-Audio), and multi-turn dialogue. The authors demonstrate that PRIME-Speech preserves the S2T performance of the backbone (Phi-4-MM-7B) while achieving competitive S2S performance. Ablation studies effectively isolate the contributions of the frozen backbone, the audio post-decoder, and the MTP module. The results show that freezing the backbone prevents the degradation of reasoning capabilities seen in other S2S models that fine-tune the entire network. The efficiency gains from MTP are also well-documented, showing significant reductions in Time-to-First-Audio (TTFA) and Real-Time Factor (RTF) with minimal impact on task correctness.
Reproducibility
The paper provides detailed descriptions of the model architecture, training curriculum (two-stage training), and dataset composition. The use of standard codecs (CosyVoice2) and clear definitions of the conditioning mechanisms enhance reproducibility. However, the reliance on proprietary backbone models (Phi-4-MM-7B) and specific internal data synthesis procedures may limit exact replication. The cache reset policy and MTP implementation details are sufficiently described for implementation.
Limitations
The primary limitation is the reliance on a strong, pre-existing S2T backbone, which may not be available or open-source for all use cases. The system's performance is inherently bounded by the reasoning capabilities of the frozen backbone. Additionally, while the paper reports UTMOS scores for fluency, it does not provide extensive subjective human evaluation (MOS) for naturalness or speaker consistency, which are crucial for S2S systems. The current evaluation focuses on transcript-level correctness and WER, which may not fully capture the perceptual quality of the generated speech, especially regarding prosody and emotion.
Broader impact
PRIME-Speech contributes to the development of more natural and efficient human-computer interaction systems by enabling direct speech-to-speech interaction without intermediate text bottlenecks. By preserving the reasoning capabilities of S2T models, it offers a pathway to robust, intelligent voice assistants. The approach could influence the design of future multimodal LLMs, encouraging modular designs that separate reasoning from modality-specific generation. However, the potential for misuse in generating realistic voice clones or deceptive audio content remains a concern, necessitating responsible deployment guidelines. This paper presents a significant advancement in Speech-to-Speech generation by introducing a frozen-backbone framework that effectively balances the preservation of reasoning capabilities with the generation of low-latency, high-quality speech, addressing key limitations in current S2S architectures.
Author of this paper? Send a correction.