CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning
Abstract
Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pretrained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +12.18 over an LLM-only distilled model on AudioCaps and by +5.21 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents CARD, a novel encoder-free audio captioning model that utilizes cross-component knowledge distillation to route teacher representations to appropriate student components, achieving competitive performance with encoder-based baselines while eliminating inference-time encoder costs.
Methodology
The paper proposes CARD, an encoder-free audio captioning framework that eliminates the need for a dedicated audio encoder at inference time. The core methodological innovation is a "cross-component distillation" strategy. Instead of distilling teacher knowledge solely into the Large Language Model (LLM) or the projector, CARD routes the teacher's (CLAP-HTSAT) representations based on semantic abstraction: early perceptual stages ($t_0, t_1$) supervise the audio projector, while later semantic stages ($t_2, t_3$) supervise the LLM's LoRA adapters. This design choice is theoretically sound, aligning the level of feature abstraction with the functional role of each student component. The use of a two-phase training procedure (distillation followed by captioning fine-tuning) and the merging of LoRA adapters for efficient inference are well-structured technical decisions. However, the novelty is somewhat incremental, as the concept of distilling multimodal knowledge into LLMs without encoders (e.g., VoRA, AuRA) has been explored in vision and speech domains; the specific application to audio captioning with this particular component-aware routing is the primary differentiator.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors compare CARD against a strong encoder-based baseline (SLAM-AAC with CLAP) and several ablation variants of their own method (No Distill, LLM Distill, Proj_Full, Proj_Early). The results clearly demonstrate that the proposed cross-component distillation (CARD$^*$) significantly outperforms LLM-only distillation (+12.18 CIDEr-D on AudioCaps) and approaches the performance of the encoder-kept baseline (55.4 vs 66.4). The ablation studies effectively isolate the contribution of projector supervision versus LLM supervision, providing strong empirical evidence for the hypothesis that "where a teacher's knowledge is placed matters." The use of standard benchmarks (AudioCaps, Clotho) and metrics (CIDEr-D, SPIDEr, etc.) ensures comparability with existing literature. The inclusion of a "No Distill" baseline highlights the critical role of distillation in encoder-free models.
Reproducibility
The paper provides sufficient implementation details for reproduction. It specifies the backbone model (Qwen3-4B), the teacher model (CLAP-HTSAT), the projector architecture (strided 1D convolutions), and the distillation loss formulation (cosine similarity). Training hyperparameters such as learning rate, batch size, LoRA rank, and optimization steps are clearly listed. The two-phase training protocol is well-defined. The only potential ambiguity lies in the exact interleaving strategy for text-only instruction data during Phase 1, but the general approach is clear. The code is not provided, which is a minor hindrance but the method is described with enough precision to be implemented.
Limitations
A significant limitation is the performance gap between the encoder-free CARD model (55.4 CIDEr-D) and the encoder-based baseline (66.4 CIDEr-D). While the paper argues that removing the encoder is beneficial for efficiency, the ~11-point drop in performance is substantial. The paper does not extensively analyze the specific types of errors or the scenarios where the encoder-free model fails most severely. Additionally, the reliance on a specific teacher (CLAP) means the distillation is bounded by the teacher's capabilities and biases. The evaluation is limited to two standard datasets; performance on more diverse or noisy real-world audio might differ. The "Qwen3-4B" reference might be a typo for Qwen2-1.5B or Qwen2-7B or a very new model, which could cause confusion regarding the exact baseline capabilities.
Broader impact
This work contributes to the trend of efficient, integrated multimodal models that reduce inference latency and memory footprint by removing dedicated encoders. This has practical implications for deploying audio understanding systems on edge devices or in low-latency applications. By demonstrating that knowledge distillation can effectively transfer acoustic understanding into LLMs, it opens avenues for similar encoder-free architectures in other modalities. However, the reduction in performance compared to encoder-based systems suggests that for high-accuracy applications, the trade-off may not yet be favorable. The paper presents CARD, a novel encoder-free audio captioning model that utilizes cross-component knowledge distillation to route teacher representations to appropriate student components, achieving competitive performance with encoder-based baselines while eliminating inference-time encoder costs.
Author of this paper? Send a correction.