UniAE-MoE: A Unified Audio Encoder via Mixture of Experts
Abstract
Large Audio Language Models (LALMs) rely on effective audio encoders for multi-task performance. We introduce UniAE-MoE, a unified audio encoder designed to model cross-domain audio representations and achieve outstanding downstream understanding performance via a Mixture-of-Experts (MoE) architecture. Specifically, we explore mainstream audio encoders and integrate those from Qwen2-Audio and Audio-Flamingo 3, which demonstrate superior downstream capabilities. To facilitate effective model fusion, we improve our encoder using SwiGLU with shared experts to decouple encoder networks, and we further introduce a two-stage instruction-tuning strategy to better adapt the model to diverse downstream tasks. Moreover, we propose the task-specific data scaling (TSDS) technique to enhance \tool's understanding capabilities. On the XARES-LLM benchmark, UniAE-MoE attains a score of 0.802, achieving state-of-the-art performance. It also delivers top-tier performance in the official Interspeech 2026 Audio Encoder Capability Challenge, further demonstrating robust generalization across diverse audio tasks. Together, these results validate the effectiveness of \tool for unified audio understanding across speech, music, and general audio domains.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a unified audio encoder that integrates complementary encoders from Qwen2-Audio and Audio-Flamingo 3 through MoE-based fusion, achieving state-of-the-art performance on the XARES-LLM benchmark. The technical contribution is significant in demonstrating that sparse expert architectures can effectively reconcile heterogeneous audio representations from different pre-training regimes, offering a scalable and efficient path toward general-purpose audio understanding.
Methodology
The paper proposes UniAE-MoE, a unified audio encoder that fuses features from two distinct Large Audio Language Model (LALM) backbones: Qwen2-Audio and Audio-Flamingo 3. The core methodological contribution is the use of a Mixture-of-Experts (MoE) architecture with SwiGLU activations to decouple and recombine these heterogeneous representations. The authors argue that while both models share a Whisper-based backbone, their distinct pre-training objectives lead to complementary feature spaces. The MoE router performs token-level conditional expert aggregation, allowing the model to specialize in different acoustic modalities (speech, music, general audio) while a shared expert captures invariant features. Additionally, the paper introduces a two-stage instruction-tuning strategy and a Task-Specific Data Scaling (TSDS) technique to address data imbalance across 20 downstream tasks. The approach is sound, leveraging existing strong encoders rather than training from scratch, which is a practical and effective strategy for current LALM development.
Results & evaluation
The experimental evaluation is robust, targeting the XARES-LLM benchmark and the Interspeech 2026 Audio Encoder Capability Challenge. The model achieves a state-of-the-art score of 0.802 on XARES-LLM, outperforming individual backbones (Qwen2-Audio: 0.737, Audio-Flamingo 3: 0.767) and simple fusion baselines like concatenation (0.778). Ablation studies confirm the importance of SwiGLU and the MoE structure. The TSDS technique shows significant gains on low-resource tasks (e.g., Free Music Archive accuracy jumping from 0.848 to 0.979). The inclusion of hidden test sets from the official challenge provides strong evidence of generalization. However, the reliance on a lightweight 135M-parameter LLM for the decoder limits the assessment of the encoder's potential with larger, more capable language models.
Reproducibility
The authors provide a GitHub repository link, which is a positive step. The paper details the training steps (30k and 80k), hardware (3x NVIDIA L20), and specific hyperparameters (top-2 routing, hidden dimension 3,413). The data sources are clearly listed, including specific datasets for TSDS. However, the exact implementation details of the "Dasheng-base" baseline and the specific preprocessing steps for aligning the two different encoder outputs (beyond "temporal and feature-wise alignment") could be more explicit to ensure full reproducibility.
Limitations
A primary limitation is the computational overhead of running two full encoder backbones (Qwen2-Audio and Audio-Flamingo 3) in parallel, which may hinder real-time or edge deployment despite the MoE efficiency. The paper acknowledges that performance on some hidden generative tasks (AISHELL-6, LibriHeavy) is lower, indicating room for improvement in robust generalization. Furthermore, the method is heavily dependent on the quality of the two specific base encoders; if one backbone is weak or biased, the fusion may not fully compensate. The use of a small LLM (SmolLM2-135M) for evaluation might underestimate the encoder's true capability in complex reasoning tasks.
Broader impact
This work contributes to the trend of "General Audio Intelligence" by demonstrating that combining specialized LALM encoders via MoE can yield superior unified representations. It provides a blueprint for integrating multiple audio models without retraining them from scratch, which is valuable for the community. The TSDS technique offers a practical solution for multi-task learning imbalances, applicable to other multimodal domains. The strong performance on the Interspeech 2026 challenge validates the approach in a competitive, standardized setting. The paper presents a unified audio encoder that integrates complementary encoders from Qwen2-Audio and Audio-Flamingo 3 through MoE-based fusion, achieving state-of-the-art performance on the XARES-LLM benchmark. The technical contribution is significant in demonstrating that sparse expert architectures can effectively reconcile heterogeneous audio representations from different pre-training regimes, offering a scalable and efficient path toward general-purpose audio understanding.
Author of this paper? Send a correction.