Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs
Abstract
Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a modular, interpretable audio reasoning pipeline that decouples perception from reasoning via a hierarchical knowledge tree. By using frozen encoders to map audio to human-readable nodes and a frozen LLM to reason over them, it achieves competitive performance with significantly lower training costs and offers superior interpretability and adaptability to new domains compared to end-to-end audio-language models.
Methodology
The paper proposes "Listen-to-Reason" (L2R), a modular pipeline that decouples audio perception from language reasoning. Instead of fusing audio embeddings into an LLM, it uses frozen expert encoders (e.g., CLAP, emotion2vec) to map audio chunks to nodes in a hierarchical, human-readable knowledge tree. A frozen text-only LLM then answers questions based on these retrieved nodes and an ASR transcript. The methodology is sound and well-motivated by the need for interpretability and modularity. The "Adapt" mechanism, which adds new domains via small heads on frozen encoders without retraining the core model, is a strong architectural choice that avoids catastrophic forgetting. The use of a hybrid annotation strategy (expert classifiers + LALM) to build the tree is practical, though it introduces potential noise from the annotator LLM.
Results & evaluation
The experiments are extensive, evaluating on four major benchmarks (MMAU, MMAR, SAKURA, MMAU-Pro) and two new bioacoustic domains. The results are honest and nuanced: L2R outperforms LALMs on SAKURA but trails by 6-12 points on MMAU/MMAR. The claim of "1,400x fewer parameters" is striking and well-supported by the training details (7.7M trainable params vs. full fine-tuning). The few-shot adaptation results (13-26 points over QLoRA) are particularly compelling for the niche domain expansion claim. The causal intervention test (78% overturn rate) provides strong evidence for the interpretability claim.
Reproducibility
The paper provides detailed descriptions of the tree structure, router training, and inference process. It mentions code and checkpoints are available on a project page, though the specific URL is not in the provided text. The reliance on specific frozen encoders and the Qwen3-Omni annotator makes reproduction dependent on access to these models, but the core logic is clearly defined.
Limitations
The primary limitation is the performance gap on general audio reasoning benchmarks (MMAU/MMAR) compared to state-of-the-art LALMs. The tree construction relies on a large LLM for annotation, which may propagate biases or errors. The method is heavily dependent on the quality of the frozen expert encoders; if an encoder fails to capture a relevant feature, the LLM cannot recover it. The "human-readable" aspect is subjective and may not scale to all types of audio nuances.
Broader impact
This work offers a viable alternative to end-to-end LALMs for applications where interpretability, auditability, and low-cost adaptation are critical (e.g., healthcare, industrial monitoring). It demonstrates that complex audio reasoning can be achieved with significantly less compute and data by leveraging modular components. It bridges the gap between symbolic knowledge representation and neural audio processing. The paper introduces a modular, interpretable audio reasoning pipeline that decouples perception from reasoning via a hierarchical knowledge tree. By using frozen encoders to map audio to human-readable nodes and a frozen LLM to reason over them, it achieves competitive performance with significantly lower training costs and offers superior interpretability and adaptability to new domains compared to end-to-end audio-language models.
Author of this paper? Send a correction.