Symbiotic Architecture for Post-Hoc Audio Extension of Frozen Language Models
Abstract
This paper proposes an architecture for equipping large language models (LLMs) with audio-understanding capabilities without fine-tuning their weights. The proposed symbiotic architecture employs an injector module that writes audio-conditioned vectors directly into the target LLM's short-term memory, i.e., the key-value (KV) cache, enabling the LLM to behave as an audio language model (ALM). The architectural advantages are twofold. First, it improves the scalability of ALMs: because the proposed method bypasses the LLM during audio injection, the injection cost is governed by the injector width rather than the backbone width, and can therefore scale more slowly than the cost of full-backbone prefilling. Second, since the training scheme does not update the LLM weights, the original capabilities of the LLM are preserved without the risk of degradation from fine-tuning. The effectiveness of the proposed method is evaluated on both audio-understanding tasks (automatic speech recognition, audio question answering, and acoustic scene classification) and text-only tasks. We confirm that, while activating fewer parameters during audio prefilling, our architecture outperforms the conventional method with a frozen LLM and approaches the performance of a fine-tuned ALM, all while preserving the backbone LLM's original text-only task performance by construction.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a symbiotic architecture that decouples audio prefilling from the LLM backbone by directly generating the KV cache via a lightweight injector, thereby reducing computational cost and preventing catastrophic forgetting. The technical contribution is significant in addressing two major bottlenecks in audio-LLM development, though the experimental validation is limited to small-scale models, leaving the full scalability benefits to be verified in future work.
Methodology
The paper proposes a "symbiotic" architecture where an audio injector module generates the Key-Value (KV) cache for a frozen Large Language Model (LLM), bypassing the need to pass audio embeddings through the LLM's self-attention layers during prefilling. The injector is a CNN-based module (using LConv blocks) that maps audio encoder outputs directly to the LLM's KV space. Key technical contributions include a "KV Scale Matching" strategy to align the injector's output distribution with the LLM's internal KV distribution, and "Noisy RoPE" training to improve robustness to sequence lengths unseen during training. The approach is theoretically sound, leveraging the fact that the KV cache is the interface between context and generation, allowing modality-specific processing to be decoupled from the backbone.
Results & evaluation
Experiments are conducted on a compact model (Qwen3-0.6B) with WavLM as the audio encoder. The paper evaluates ASR (LibriSpeech), Audio QA (Clotho), and Acoustic Scene Classification (CochlScene). The proposed method outperforms the "Encoder-only" (SLM-style) baseline significantly on non-ASR tasks and approaches the performance of a fine-tuned "Monolithic" model while using fewer active parameters during audio prefilling. Crucially, it preserves text-only performance (WikiText-2, HellaSwag, GSM8K) by construction, avoiding catastrophic forgetting. However, the evaluation is limited to a single small backbone, and the speedup is modest (157s vs 199s) in the tested regime.
Reproducibility
The paper provides detailed architectural descriptions, including specific hyperparameters for the injector (kernel size, subsampling stride), training configurations (learning rates, batch size, steps), and specific techniques for stabilization (RMSNorm initialization, noisy RoPE parameters). The use of standard open-source components (WavLM, Qwen3) enhances reproducibility. However, code availability is not explicitly confirmed in the text, and the specific implementation of the KV injection into the inference engine is not detailed.
Limitations
The primary limitation is the scale of the experiments; using a 0.6B LLM does not fully validate the scalability claims for larger models where the prefilling bottleneck is more severe. The speedup demonstrated is relatively small in the current setup, and the authors acknowledge that the CNN-centric injector may limit performance on complex acoustic tasks compared to attention-based injectors. The method relies on specific internal details of the backbone (e.g., Qwen3's RMSNorm), which may require adaptation for other LLM families.
Broader impact
This work offers a practical solution to the high computational cost of multimodal prefilling and the risk of catastrophic forgetting in LLM fine-tuning. By decoupling audio processing from the backbone, it enables the use of larger, more capable LLMs for audio tasks without proportional increases in inference latency for the audio portion. This could facilitate the deployment of ALMs in edge devices or real-time applications where prefilling latency is critical. The paper introduces a symbiotic architecture that decouples audio prefilling from the LLM backbone by directly generating the KV cache via a lightweight injector, thereby reducing computational cost and preventing catastrophic forgetting. The technical contribution is significant in addressing two major bottlenecks in audio-LLM development, though the experimental validation is limited to small-scale models, leaving the full scalability benefits to be verified in future work.
Author of this paper? Send a correction.