Acoustic-to-Text KV Compression for Full-Duplex Speech Models
Abstract
Full-duplex speech language models continuously accumulate acoustic key-value (KV) states, making long-running interactions memory-intensive. During listening, the model can finish processing an audio unit before the next arrives; we term the remaining interval listening-time slack. We propose acoustic-to-text KV compression, which introduces a transcription side channel to convert incoming speech into compact textual memory within this interval. When the cache exceeds a target budget during inference, older acoustic states are evicted while transcripts and recent acoustic context remain. We train the side channel with LoRA using cross-entropy on transcription segments. To preserve listening and speaking behavior, we apply knowledge distillation to the original model's token-level output distributions at native prediction positions. On ten-minute LongSpeech sessions, our MiniCPM-o 4.5 implementation reduces peak streaming KV-cache size by 64.6% compared with the same model without eviction. The proposed method also improves transcription, temporal question answering, and summarization over the baseline. Full-Duplex-Bench evaluations further show comparable pause-handling, turn-taking, and interruption performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a practical and effective method for reducing KV cache memory in full-duplex speech models by converting acoustic history to text during listening slack, demonstrating significant memory savings with preserved conversational quality.
Methodology
The paper proposes a novel "acoustic-to-text KV compression" strategy for full-duplex speech models. The core idea is to exploit "listening-time slack"—the computational gap between audio unit arrivals—to run a lightweight transcription side channel (implemented via LoRA) that converts incoming speech into text tokens. These text tokens are retained in the KV cache while older acoustic KV states are evicted. The methodology is technically sound, leveraging the existing language model backbone rather than a separate ASR model. A significant methodological strength is the use of knowledge distillation from the frozen original model to preserve native listening/speaking behaviors (turn-taking, interruption) which would otherwise be disrupted by the new transcription objective. The training setup uses word-level forced alignments and a composite loss function including cross-entropy for ASR and KL divergence for behavior preservation.
Results & evaluation
The experiments are conducted on the MiniCPM-o 4.5 model. The evaluation covers three key areas: 1) Long-form speech understanding (LongSpeech benchmark), showing improved WER and QA accuracy compared to native streaming, and competitive performance with external ASR cascades. 2) Full-duplex interaction (Full-Duplex-Bench), demonstrating that the distillation component is critical for maintaining natural conversational dynamics (pause handling, turn-taking). 3) Efficiency, showing a 64.6% reduction in peak KV cache size with no real-time deadline misses on H100 GPUs. The ablation study effectively isolates the impact of the distillation loss, showing that without it, the model fails at turn-taking. The comparison with external ASR cascades is particularly relevant, showing that the proposed method achieves similar quality with fewer added parameters and integrated memory management.
Reproducibility
The paper provides sufficient detail for reproduction, including the specific model (MiniCPM-o 4.5), training data (LibriSpeech train-clean), hyperparameters (LoRA rank 16, learning rate, batch size), and the specific eviction strategy (5-unit retention window). The use of standard benchmarks (LongSpeech, Full-Duplex-Bench) facilitates comparison. However, the code is not explicitly linked in the text provided, and the specific implementation details of the "listening-time slack" scheduling might require access to the source code for precise replication.
Limitations
The method is evaluated primarily on a single model architecture (MiniCPM-o 4.5), so generalizability to other full-duplex models is not established. The transcription side channel relies on the model's ability to predict the start of the ASR segment; errors in this gating mechanism could lead to missed transcriptions. The method assumes a consistent "listening-time slack" of ~900ms, which may not hold for all hardware configurations or model variants. Additionally, the WER (12.9%) is higher than dedicated streaming ASR models (e.g., FastConformer at 10.3%), suggesting a trade-off between integration and pure transcription accuracy.
Broader impact
This work addresses a critical bottleneck in deploying full-duplex speech agents: memory consumption during long conversations. By converting acoustic history to text, it enables longer context windows without proportional memory increases. This is highly relevant for real-time voice assistants and interactive agents. The approach of using "slack" time for auxiliary tasks (transcription) is a generalizable principle for efficient inference in streaming multimodal models. The paper introduces a practical and effective method for reducing KV cache memory in full-duplex speech models by converting acoustic history to text during listening slack, demonstrating significant memory savings with preserved conversational quality.
Author of this paper? Send a correction.