CAAD: Contrastive Audio-Aware Distillation for Efficient Speech Language Models
Abstract
Speech Language Models achieve reasoning capabilities, but are often hindered by massive parameter counts and a tendency to prioritize linguistic priors over acoustic features. While contrastive decoding enhances grounding by contrasting audio-aware and text-only logits, it increases inference latency. We propose Contrastive Audio-Aware Distillation (CAAD), a framework that internalizes the teacher's contrastive reasoning into the student model's weights. To overcome the high computational training overhead in the dual-path token-by-token contrastive distillation process, we introduce a synchronized teacher-forcing strategy. Anchored by unified Pseudo-Ground Truths, this mechanism enables simultaneous full-sequence generation of the teacher's contrastive distributions, allowing student to distill the audio-aware signal efficiently. Overall, CAAD yields a ~8% relative gain over standard knowledge distillation on Dynamic-SUPERB and successfully reduces linguistic bias in MCR-BENCH.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel and effective method for distilling contrastive decoding into Speech Language Models, addressing critical efficiency and bias challenges in multimodal AI through a synchronized teacher-forcing strategy anchored by metadata.
Methodology
The paper proposes Contrastive Audio-Aware Distillation (CAAD), a method to compress Speech Language Models (SLMs) by distilling the benefits of Contrastive Decoding (CD) into a student model's weights. The core innovation is a "synchronized teacher-forcing strategy" that uses a "Pseudo-Ground Truth" (Pseudo-GT) generated from text metadata to anchor both the audio-aware (positive) and text-only (negative) teacher passes. This allows for parallel training, avoiding the sequential bottleneck of standard autoregressive contrastive decoding. The approach effectively transforms a test-time inference technique (CD) into a training-time objective. While the concept of distilling decoding strategies is not entirely new, the specific mechanism of using metadata-anchored pseudo-GT to enable parallel contrastive distillation in SLMs is a novel and practical engineering contribution. It addresses a genuine computational bottleneck in applying CD to large models.
Results & evaluation
The experimental evaluation is robust, utilizing the Dynamic-SUPERB benchmark and the MCR-BENCH for conflict resolution. The results demonstrate that the CAAD-distilled 3B student model significantly outperforms standard KD and even the greedy decoding baseline of the 8B teacher on several metrics, particularly in paralinguistic tasks (PAR) and conflict resolution (MCR-BENCH Shift). The ablation studies effectively validate the components of the method, showing that metadata-based Pseudo-GT outperforms audio-based synchronization and that the contrastive weight is crucial for mitigating linguistic bias. The comparison against Contrastive Decoding at inference time highlights the efficiency gain (single-path vs. dual-path) while acknowledging the performance gap, which is a fair and honest assessment.
Reproducibility
The paper provides sufficient detail regarding the model architectures (Llama-3.2-8B teacher, Llama-3.2-3B student), training configurations (learning rate, optimizer, loss weights), and datasets (DeSTA2, Dynamic-SUPERB, MCR-BENCH). The code repository is linked, which significantly aids reproducibility. The description of the Pseudo-GT generation process is clear enough for replication.
Limitations
The primary limitation is the dependency on the quality of the Pseudo-GT. If the metadata-derived text is inaccurate or lacks nuance, the distillation signal may be noisy. Additionally, the method assumes that the teacher model's contrastive reasoning is transferable via KL divergence, which may not capture all nuances of the teacher's decision boundary. The performance of the student, while improved, still lags behind the teacher's contrastive decoding performance, indicating that some information is lost in the distillation process. The paper also notes that the efficacy is bounded by the student's capacity.
Broader impact
This work contributes to the democratization of SLMs by enabling efficient, low-latency inference without sacrificing the robustness offered by contrastive methods. By mitigating linguistic bias, it promotes more reliable multimodal AI systems, which is crucial for applications in accessibility, customer service, and interactive agents where audio cues are critical. The method is generalizable to other multimodal LLMs beyond speech. The paper presents a novel and effective method for distilling contrastive decoding into Speech Language Models, addressing critical efficiency and bias challenges in multimodal AI through a synchronized teacher-forcing strategy anchored by metadata.
Author of this paper? Send a correction.