Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Abstract
Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SpeechKV introduces a novel in-LLM KV cache compression technique for Speech LLMs that achieves competitive performance with significant decoding speedups by leveraging learned pooling on intermediate layer representations.
Methodology
The paper proposes SpeechKV, a novel approach to compressing the Key-Value (KV) cache of speech tokens within the internal layers of a Speech Large Language Model (Speech LLM). Unlike traditional methods that compress speech embeddings before they enter the LLM (adapter-level compression), SpeechKV applies a learned pooling operation to the KV cache at an intermediate layer ($l_0$). This design preserves full-resolution queries and the residual stream, allowing the model to fuse speech and text representations internally while reducing the sequence length for subsequent attention computations. The method is theoretically sound, leveraging the observation that speech representations become redundant in deeper layers. The approach is simple yet effective, avoiding the need for complex architectural changes or external modules.
Results & evaluation
The authors evaluate SpeechKV on a Qwen3-1.7B backbone trained on 71K hours of ASR data. They compare against adapter-level compression (Concat-MLP, Window Q-Former) and an alternative in-LLM hidden-state compression (HS Compression). Results show that SpeechKV maintains or slightly improves upon the uncompressed baseline performance on both in-domain (OpenASR) and out-of-domain (In-house ER, In-house ASR) benchmarks. Notably, it achieves a 6.6% relative gain in entity recognition and a 2.3% gain in OpenASR WER compared to the baseline, while providing a 1.49x to 2.02x decoding speedup. The ablation studies on compression ratios ($R=4, 8$) and starting layers ($l_0$) provide valuable insights into the trade-offs between compression aggressiveness and performance retention. The attention analysis further supports the claim that SpeechKV leads to more focused attention patterns.
Reproducibility
The paper provides detailed descriptions of the model architecture, training setup (optimizer, learning rate, batch size), and evaluation metrics. The use of open-source components (Qwen3-1.7B, Conformer encoder) and standard ASR benchmarks enhances reproducibility. However, the mention of "proprietary test sets" for In-house ASR and ER limits the ability of external researchers to fully replicate the out-of-domain evaluation. The code is not explicitly linked in the text provided, which is a minor hindrance to immediate reproducibility.
Limitations
The primary limitation is the reliance on proprietary data for a significant portion of the evaluation, which prevents independent verification of the out-of-domain generalization claims. Additionally, the speedup, while significant, is modest (1.49x-2.02x) and may not justify the added training complexity for all use cases, especially given that feed-forward layers and text KV cache remain bottlenecks at higher compression ratios. The method is specific to Speech LLMs and may not generalize directly to other multimodal settings without adaptation.
Broader impact
This work contributes to the efficiency and accessibility of Speech LLMs, which are increasingly important for real-time applications like voice assistants and transcription services. By enabling faster decoding without significant performance loss, SpeechKV could facilitate the deployment of more capable speech models on resource-constrained devices. The insights into layer-wise redundancy in speech representations also offer broader implications for understanding how multimodal models process continuous signals. SpeechKV introduces a novel in-LLM KV cache compression technique for Speech LLMs that achieves competitive performance with significant decoding speedups by leveraging learned pooling on intermediate layer representations.
Author of this paper? Send a correction.