UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition
Abstract
Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing \textbf{UA-Bench}, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0\%--13.1\% leaf accuracy, against a random reference of 7.7\%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces UA-Bench, the first large-scale benchmark for evaluating Large Audio-Language Models on underwater acoustic signal recognition, combining parameterized source synthesis with modeled propagation to enable traceable error analysis. By revealing that current LALMs perform near-randomly on fine-grained underwater classification despite some capability in coarse routing, the work provides a rigorous diagnostic framework and a necessary reference point for advancing audio understanding in specialized, non-speech domains.
Methodology
The methodology is robust and well-designed for a synthetic benchmark. The authors successfully decouple source generation from channel propagation using BELLHOP, allowing for traceable error attribution. The "gated coarse-to-fine" evaluation protocol is a significant methodological contribution, as it distinguishes between failures in source routing (Active vs. Passive) and fine-grained classification, which standard accuracy metrics obscure. The inclusion of a paired silent-audio control is a rigorous diagnostic tool to verify that models are actually utilizing the waveform rather than relying on question priors or class imbalance. The synthesis pipeline, covering detection pulses, communication waveforms, and ship-radiated noise with specific physical parameters (sound-speed profiles, multipath), demonstrates a deep understanding of underwater acoustics.
Results & evaluation
The experiments are comprehensive, evaluating nine state-of-the-art open-source Large Audio-Language Models (LALMs) without task-specific adaptation. The results are sobering and informative: leaf accuracy ranges from 2.0% to 13.1%, barely above the random reference of 7.7%. The analysis correctly identifies that while some models show significant gains in coarse routing (L1) when audio is present, the fine-grained leaf-level gains are minimal (max +4.7 points). The breakdown of errors into invalid-format, cascade, and classification errors provides valuable insight into the specific failure modes of current LALMs on non-speech audio.
Reproducibility
High. The paper provides extensive details on the synthesis parameters, partitioning strategy (source-grouped to prevent leakage), and evaluation protocol. The release of the training partition and the fixed evaluation subset (UA-Bench-Eval) ensures that future work can be compared fairly. The code for generation and evaluation is implied to be available given the nature of the benchmark release, though specific URLs are not in the text provided.
Limitations
The primary limitation is the synthetic nature of the data. While the propagation is modeled, it may not capture the full complexity of real-world underwater noise (e.g., biological noise, surface clutter, non-linear effects). The paper acknowledges this and positions the benchmark as a controlled reference point rather than a field-ready solution. Additionally, the evaluation is limited to zero-shot performance; the impact of fine-tuning on these models using the provided training partition is not explored in this paper.
Broader impact
This paper provides a critical missing piece in the evaluation of LALMs. By demonstrating that current models struggle significantly with engineered underwater signals, it highlights the gap between general audio pretraining and specialized domain recognition. It sets a baseline for future research in domain adaptation for audio-language models and provides a tool for the maritime and ocean sensing communities to evaluate new models. The paper introduces UA-Bench, the first large-scale benchmark for evaluating Large Audio-Language Models on underwater acoustic signal recognition, combining parameterized source synthesis with modeled propagation to enable traceable error analysis. By revealing that current LALMs perform near-randomly on fine-grained underwater classification despite some capability in coarse routing, the work provides a rigorous diagnostic framework and a necessary reference point for advancing audio understanding in specialized, non-speech domains.
Author of this paper? Send a correction.