Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation
Abstract
Autoregressive generation of interleaved text and acoustic tokens is a common approach to spoken-response generation in speech large language models. Although this design enables streaming generation with explicit textual guidance, generated acoustic tokens become part of the context for subsequent text predictions. Given identical speech inputs, we observe markedly lower answer accuracy for the internal text generated in speech-to-text-and-speech (S2TS) mode than for speech-to-text (S2T) responses. We term this discrepancy the \emph{output-mode gap} (OMG). To reduce OMG, we propose \emph{Joint-Output On-Policy Distillation} (JO-OPD), which distills the model's stronger S2T policy into joint generation using student-generated S2TS trajectories. At each text position, the S2T teacher provides soft targets from a text-only projection of the student's preceding outputs, while the student predicts from the corresponding full interleaved history. A preservation objective further regularizes native non-text predictions. Experiments on Step-Audio-2-mini and Baichuan-Audio-Instruct reveal OMG across two interleaved generation architectures. On Step-Audio-2-mini, JO-OPD reduces OMG from 42.87 to 16.26 percentage points on Spoken-MQA and from 29.72 to 13.04 points on speech-rendered GSM8K, with little change in S2T accuracy and substantially larger reductions than matched SFT baselines. ASR-based evaluation further shows a 7.49-point improvement in spoken-answer accuracy on Spoken-MQA.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper identifies and quantifies the "Output-Mode Gap" in interleaved speech LLMs and proposes Joint-Output On-Policy Distillation (JO-OPD) to mitigate it by distilling the model's stronger text-only policy into the joint generation process. The technical contribution is significant as it provides a principled way to improve text reasoning accuracy in speech-output modes without sacrificing speech generation quality, supported by rigorous experiments on two major architectures and ablation studies validating the key components of the method.
Methodology
The paper introduces Joint-Output On-Policy Distillation (JO-OPD) to address the "Output-Mode Gap" (OMG) in interleaved speech-language models. The core insight is that when a model generates both text and audio tokens, the acoustic tokens in the context degrade the quality of subsequent text predictions compared to a pure text-only mode. JO-OPD leverages the model's own stronger Speech-to-Text (S2T) policy as a teacher to supervise the Speech-to-Text-and-Speech (S2TS) student. A key technical contribution is the "text-only projection" of the student's history for the teacher, allowing the teacher to provide soft targets based on the semantic content without being confused by the acoustic noise in the context. Additionally, a preservation objective regularizes the non-text (acoustic) predictions to prevent the distillation process from degrading speech generation quality. The method is theoretically sound, leveraging on-policy distillation to align the student's distribution with the teacher's on the student's own trajectories, which is a robust approach for reducing exposure bias in autoregressive generation.
Results & evaluation
The experiments are conducted on two distinct architectures: Step-Audio-2-mini and Baichuan-Audio-Instruct. The evaluation metrics focus on answer accuracy for Spoken-MQA and GSM8K, comparing S2T accuracy vs. S2TS internal text accuracy. The results show a significant reduction in OMG (e.g., from 42.87 to 16.26 points on Spoken-MQA for Step-Audio-2). The paper also includes ablations on soft vs. hard targets, the importance of non-text preservation, and the effect of training scale. The inclusion of ASR-based evaluation for spoken answer accuracy provides a holistic view of the impact on final user-facing output. The comparison against Self-SFT and Response-SFT baselines is appropriate and demonstrates the superiority of the proposed on-policy distillation approach.
Reproducibility
The paper provides detailed hyperparameters (learning rate, batch size, top-k values) and describes the training setup clearly. However, the specific implementation details of the "text-only projection" and the exact mechanism for the preservation loss in different architectures (Step-Audio vs. Baichuan) could benefit from more pseudocode or explicit mathematical definitions in the main text. The use of synthesized speech (flite) for training is noted, which is a practical detail for reproducibility, though it may limit generalization to diverse speaker characteristics.
Limitations
The primary limitation is the reliance on the model's own S2T capability as the teacher; if the base model has weak S2T performance, the distillation benefit may be limited. The evaluation is primarily on question-answering and reasoning tasks, so the impact on open-ended conversational quality or prosody is less clear, although the preservation loss aims to mitigate this. The training data size (27k prompts) is relatively small, and the long-term stability of the model after such fine-tuning is not extensively explored.
Broader impact
This work addresses a critical issue in the deployment of speech LLMs: the degradation of reasoning capabilities when speech output is enabled. By providing a method to close this gap, the paper contributes to the development of more reliable and capable voice assistants. The technique of using internal modality policies as teachers is a generalizable insight that could be applied to other multimodal interleaved generation tasks. The paper identifies and quantifies the "Output-Mode Gap" in interleaved speech LLMs and proposes Joint-Output On-Policy Distillation (JO-OPD) to mitigate it by distilling the model's stronger text-only policy into the joint generation process. The technical contribution is significant as it provides a principled way to improve text reasoning accuracy in speech-output modes without sacrificing speech generation quality, supported by rigorous experiments on two major architectures and ablation studies validating the key components of the method.
Author of this paper? Send a correction.