SALMONN-2: Advancing General-Purpose Hearing Abilities with Self-Supervised Representations
Abstract
Recent audio large language models (ALLMs) are typically built upon audio encoders trained with large amounts of supervised data. Since self-supervised learning (SSL) audio encoder models are known to learn general-purpose and transferable representations, we investigate whether general-purpose SSL audio representations can serve as an effective foundation for ALLMs. We present SALMONN-2, an ALLM built upon a unified SSL encoder. To better exploit the hierarchical representations learned by SSL encoders, we propose a multi-layer feature fusion (MLF) adapter that aggregates information from all encoder layers before projecting them into the language model. Beyond conventional audio understanding tasks, we further explore multimodal in-context learning (MICL) in ALLMs and study how this capability can be acquired through contextual biasing training. Experimental results show that a general-purpose SSL encoder achieves performance comparable to, or better than, specialised supervised audio encoders while providing a more balanced capability across speech, audio, music and paralinguistic tasks. SALMONN-2 further achieves state-of-the-art performance among comparable-scale open-weight models on ALLM understanding benchmarks, obtaining the best results on MMAU-Pro, MMAR and MMSU. We also show that MICL does not emerge naturally in ALLMs, but can be effectively acquired through targeted contextual biasing training.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SALMONN-2 presents a compelling argument for the use of unified self-supervised audio representations in Audio Large Language Models, achieving state-of-the-art performance with greater efficiency and architectural simplicity than previous multi-encoder approaches. The introduction of the Multi-Layer Feature Fusion adapter and the systematic study of Multimodal In-Context Learning provide valuable technical contributions and insights for the broader field of multimodal AI.
Methodology
The paper proposes SALMONN-2, an Audio Large Language Model (ALLM) built upon a unified Self-Supervised Learning (SSL) encoder (SPEAR). The core methodological contributions are: 1) The adoption of a single, general-purpose SSL encoder (SPEAR) as a unified front-end for diverse audio tasks (speech, music, environmental sound), challenging the prevalent dual-encoder or supervised-encoder paradigms. 2) A Multi-Layer Feature Fusion (MLF) adapter that aggregates hidden states from all layers of the SSL encoder before projection, aiming to exploit the hierarchical nature of SSL representations (low-level acoustic to high-level semantic). 3) A timestamp injection mechanism using textual tokens from the LLM vocabulary to provide explicit temporal grounding without extra parameters. 4) Explicit training for Multimodal In-Context Learning (MICL) in the context of Contextual ASR, demonstrating that this capability does not emerge naturally from standard instruction tuning. The approach is technically sound, leveraging recent advances in unified SSL models (SPEAR) and applying them effectively to the ALLM paradigm. The MLF adapter is a relatively simple but effective architectural choice compared to more complex cross-attention or Q-Former styles.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors evaluate SALMONN-2 on a wide range of canonical tasks (ASR, AAC, Music Captioning, Emotion Recognition, Speech Translation, Overlapped Speech Recognition, Speaker Verification) and general audio understanding benchmarks (MMAU-Pro, MMAR, MMSU). They also include extended audio analysis tasks (Sound Event Detection, Spoofing Analysis, Speech Quality Assessment) and a novel MICL evaluation for Contextual ASR. The results show that SALMONN-2 achieves state-of-the-art performance among comparable-scale open-weight models, often outperforming models with larger LLM backbones or more complex encoder setups. The ablation studies effectively isolate the contributions of the SSL encoder, the MLF adapter, and timestamp injection. The MICL experiments provide valuable insights into the limitations of current ALLMs and how targeted training can address them. The use of less than 20K hours of instruction-tuning data compared to competitors is a significant efficiency claim supported by the results.
Reproducibility
The paper provides detailed implementation details, including the architecture of the SPEAR encoder, the MLF adapter, the LLM backbones (Qwen series), and the training stages. The code and model are open-sourced on GitHub, which significantly aids reproducibility. The dataset construction for MICL is described, though the specific processing of some external datasets (e.g., HoliAntiSpoof, AudioSet-strong) might require careful attention to licensing and availability. The training hyperparameters and data mix are summarized in tables, providing a clear recipe for replication.
Limitations
The paper acknowledges that scaling the LLM backbone from 9B to 30B-A3B did not yield substantial improvements on canonical tasks, suggesting these tasks may be perception-heavy rather than reasoning-heavy. However, it does not deeply explore the failure modes of the SSL encoder on out-of-distribution audio or the potential biases introduced by the SSL pre-training data. The MICL capability is demonstrated primarily on Contextual ASR; its generalizability to other multimodal in-context learning scenarios (e.g., audio QA with few-shot examples) is less explored. The reliance on SPEAR, while effective, ties the model to a specific SSL architecture; the generalizability of the MLF adapter to other SSL models (e.g., WavLM, HuBERT) is not extensively tested, although the principles are likely transferable.
Broader impact
SALMONN-2 contributes to the development of more efficient and general-purpose audio intelligence systems. By demonstrating that a single SSL encoder can replace specialized supervised encoders, it reduces the computational and data overhead required to build ALLMs. The explicit handling of multimodal context has implications for robust speech recognition in noisy or domain-specific environments. The open-sourcing of the model and code facilitates further research in audio-language models. Potential negative impacts include the misuse of powerful audio understanding models for surveillance or deepfake detection evasion, though the model's focus on understanding rather than generation mitigates some generation-specific risks. SALMONN-2 presents a compelling argument for the use of unified self-supervised audio representations in Audio Large Language Models, achieving state-of-the-art performance with greater efficiency and architectural simplicity than previous multi-encoder approaches. The introduction of the Multi-Layer Feature Fusion adapter and the systematic study of Multimodal In-Context Learning provide valuable technical contributions and insights for the broader field of multimodal AI.
Author of this paper? Send a correction.