SkillFormer: Skill-Decomposed Adaptation for Audio Language Models
Abstract
Audio language models must handle dozens of distinct skills, from pitch comparison and speaker counting to musical tempo estimation and emotion recognition. Joint training on all skills at once causes interference: gains on one skill often come at the cost of another. We propose \textbf{SkillFormer}, which decomposes audio understanding into skill-specific low-rank adapters and composes them at inference time through a learned router. The router examines the question to decide which adapters to activate and how much weight each should carry, so that a pitch query engages different parameters than a genre classification query. An alternating training schedule updates each adapter on its own skill cluster before jointly calibrating the router, preventing the gradient conflicts that arise in standard multi-task optimization. SkillFormer adds fewer than 4\% of the base model's parameters and requires no changes to the audio encoder or language backbone. Evaluated on three architecturally distinct models across MMSU, MMAU-Pro, and MMAR, it raises the average accuracy by 2.5 to 4.1 points, with balanced gains across perception, reasoning, and semantic subcategories.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SkillFormer introduces a skill-decomposed adaptation framework using question-conditioned routing of low-rank adapters to mitigate multi-task interference in audio language models. The paper demonstrates that organizing parameter updates by semantic skill clusters and dynamically routing queries to relevant adapters yields consistent and balanced performance gains across diverse audio benchmarks and model architectures, offering a scalable and parameter-efficient alternative to monolithic fine-tuning.
Methodology
The paper proposes SkillFormer, a parameter-efficient adaptation framework for Audio Language Models (LALMs) that decomposes audio understanding into skill-specific low-rank adapters (LoRA). The core innovation is a question-conditioned router that selects a sparse combination of these adapters at inference time, allowing the model to activate specific expertise (e.g., pitch vs. music genre) based on the user's query. The training procedure employs an alternating schedule: first specializing each adapter on its own skill cluster (Phase A), then jointly calibrating the router (Phase B). This approach directly addresses the "catastrophic interference" problem in multi-task audio learning. The methodology is sound, leveraging standard LoRA mechanics but applying them in a structured, routed manner. The use of k-means clustering on question embeddings to define skill clusters is a pragmatic and effective heuristic for organizing the adapter bank.
Results & evaluation
The experiments are conducted on three distinct 7B-parameter LALMs (Qwen2.5-Omni, Kimi-Audio, MiMo-Audio) across three major benchmarks (MMSU, MMAU-Pro, MMAR). The results show consistent improvements of 2.4 to 4.1 points over the base models and significant gains over single-adapter SFT and uniform multi-LoRA baselines. The ablation studies are particularly strong, isolating the contribution of the router (1.7 points) and the alternating training schedule (2.3 points), which validates the design choices. The analysis of routing weights demonstrates clear specialization, with specific question types activating specific adapters, providing qualitative evidence for the method's effectiveness.
Reproducibility
The paper provides sufficient detail for reproduction, including hyperparameters (rank r=16, K=6 adapters, top-2 routing), training schedules (step counts, learning rates), and data sources (EvoAudio tool library, 50k QA pairs). The initialization strategy for LoRA matrices is specified. However, the exact implementation of the router's MLP and the specific clustering algorithm parameters are not fully detailed, which might require some trial-and-error for exact replication.
Limitations
The method relies on a predefined set of skill clusters derived from training data embeddings, which may not generalize perfectly to out-of-distribution skills. The router adds a small computational overhead at inference, though it is claimed to be negligible. The evaluation is limited to 7B models; it is unclear if the benefits scale similarly to larger models or if the overhead becomes more significant. The paper does not discuss failure modes where the router might misclassify the skill, leading to suboptimal adapter selection.
Broader impact
This work offers a practical solution to the multi-task learning challenge in audio LLMs, which is critical for deploying versatile audio assistants. By enabling specialized parameter activation, it improves efficiency and accuracy without requiring full model fine-tuning. This approach could be extended to other multimodal domains (vision, text) where task interference is a known issue. It provides a blueprint for modular, skill-based adaptation in large language models. SkillFormer introduces a skill-decomposed adaptation framework using question-conditioned routing of low-rank adapters to mitigate multi-task interference in audio language models. The paper demonstrates that organizing parameter updates by semantic skill clusters and dynamically routing queries to relevant adapters yields consistent and balanced performance gains across diverse audio benchmarks and model architectures, offering a scalable and parameter-efficient alternative to monolithic fine-tuning.
Author of this paper? Send a correction.