Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs
Abstract
Large Audio Language Models (LALMs) utilize either continuous features or discrete tokens, yet the optimal representation paradigm for general audio understanding remains debated. Existing benchmarks often focus on narrow domains or evaluate encoders outside LALM contexts. To address these gaps, we systematically evaluate continuous and discrete representations across speech, sound and music. Utilizing our UniARC framework with dual evaluation strategies across model scales from SmolLM2-135M to Llama-3-8B, we analyze the dynamic relationships of data volume, model capacity, and computational efficiency. Our results reveal the pivotal role of semantic constraints in tokenization for audio understanding and demonstrate that scaling backbones fail to compensate for information loss in audio representation, especially in data-limited tasks. These findings offer practical guidance for balancing semantic density, fidelity, and efficiency in future LALMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper provides a rigorous and systematic comparison of continuous and discrete audio representations in LALMs, revealing that semantic compatibility is the primary driver of performance and that LLM scaling cannot compensate for poor audio encoding. This comprehensive benchmarking study offers critical guidance for the development of efficient and effective multimodal audio models.
Methodology
The paper proposes the UniARC framework, a systematic evaluation methodology for comparing continuous (SSL-based) and discrete (clustering/codec-based) audio representations within Large Audio Language Models (LALMs). The methodology is sound, utilizing two distinct evaluation strategies: parameter-efficient fine-tuning (LoRA) on smaller backbones (SmolLM2) and frozen-backbone probing on larger backbones (Llama-3). This dual approach effectively isolates the impact of representation quality from the capacity of the language model. The choice of encoders is comprehensive, covering major SSL models (HuBERT, WavLM, Whisper) and discrete tokenizers (K-means, DAC, WavTokenizer, SpeechTokenizer). The standardization of feature processing (projectors, temporal concatenation) ensures fair comparison.
Results & evaluation
The experiments span three domains (speech, sound, music) and multiple tasks (ASR, emotion, captioning, classification). The results provide clear insights: semantic alignment is more critical than raw fidelity for understanding tasks. The finding that scaling the LLM backbone does not compensate for poor audio representation quality is a significant and counter-intuitive result that adds value. The analysis of data diversity and computational efficiency further strengthens the contribution. However, the sample size of encoders is limited, and the specific hyperparameters for the K-means clustering (1000 centroids) might not be optimal for all scales, though the authors justify this choice.
Reproducibility
The paper provides sufficient details for reproduction, including model names, dataset names, and general training strategies (LoRA rank, learning rates). However, specific details on the projector architecture (beyond "two-layer MLP") and the exact prompt templates used for instruction tuning are not fully detailed in the provided text. The reliance on open-source models (SmolLM2, Llama-3, XARES-LLM base) aids reproducibility.
Limitations
The study is limited to specific model scales (up to 8B) and a specific set of encoders. It does not explore hybrid approaches in depth, only comparing pure continuous vs. pure discrete. The "inverse scaling" phenomenon observed might be confounded by the simple MLP projector, as the authors acknowledge. The lack of ablation studies on the projector design is a minor weakness.
Broader impact
The findings have practical implications for the design of future LALMs, suggesting that investment in semantically rich audio encoders is more valuable than simply scaling up the LLM backbone for understanding tasks. This guides resource allocation in the field. The emphasis on multi-domain pre-training for encoders is also a valuable takeaway. The paper provides a rigorous and systematic comparison of continuous and discrete audio representations in LALMs, revealing that semantic compatibility is the primary driver of performance and that LLM scaling cannot compensate for poor audio encoding. This comprehensive benchmarking study offers critical guidance for the development of efficient and effective multimodal audio models.
Author of this paper? Send a correction.