Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models
Abstract
Puns represent a typical linguistic phenomenon that exploits polysemy and phonetic ambiguity to generate humour, posing unique challenges for natural language understanding. Within pun research, audio plays a central role in human communication except text and images, while datasets and systematic resources for spoken puns remain scarce, leaving this crucial modality largely underexplored. In this paper, we present APUN-Bench, the first benchmark dedicated to evaluating large audio language models (LALMs) on audio pun understanding. Our benchmark contains 4,434 audio samples annotated across three stages: pun recognition, pun word location and pun meaning inference. We conduct a deep analysis of APUN-Bench by systematically evaluating 10 state-of-the-art LALMs, uncovering substantial performance gaps in recognizing, localizing, and interpreting audio puns. This analysis reveals key challenges, such as positional biases in audio pun location and error cases in meaning inference, offering actionable insights for advancing humour-aware audio intelligence.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces APUN-Bench, a pioneering benchmark for evaluating audio pun understanding in large audio language models, significantly advancing the field of multimodal language processing. The comprehensive methodology and rigorous experimental evaluation highlight the challenges faced by current models, providing actionable insights for future research.
Methodology
The paper presents a novel approach to benchmarking audio pun understanding through the creation of APUN-Bench, which includes a comprehensive dataset of 4,434 audio samples annotated across three distinct stages: pun recognition, pun word location, and pun meaning inference. The methodology is robust, utilizing both synthetic and real-world data, and incorporates human verification to ensure data quality. The multi-stage evaluation framework is well-structured and addresses a significant gap in the understanding of audio puns, making it a valuable contribution to the field.
Results & evaluation
The experiments conducted on 10 state-of-the-art large audio language models (LALMs) provide a thorough analysis of their performance across the three evaluation stages. The results reveal substantial performance gaps, particularly in pun word location and meaning inference, highlighting the limitations of current models. The use of statistical tests to validate findings adds rigor to the experimental evaluation.
Reproducibility
The paper provides sufficient detail regarding the dataset construction, evaluation metrics, and model configurations, which facilitates reproducibility. However, the lack of publicly available URLs for the dataset and models limits the ease of access for other researchers.
Limitations
The study acknowledges several limitations, including the restricted scope of pun types examined, the focus on single-sentence instances rather than multi-turn dialogues, and the limited size of the real-world corpus. These factors may constrain the generalizability of the findings.
Broader impact
The research has significant implications for advancing audio understanding in natural language processing, particularly in applications related to humor, education, and voice assistants. By addressing the complexities of audio puns, the work paves the way for more sophisticated models that can better understand and generate humor in spoken language. This paper introduces APUN-Bench, a pioneering benchmark for evaluating audio pun understanding in large audio language models, significantly advancing the field of multimodal language processing. The comprehensive methodology and rigorous experimental evaluation highlight the challenges faced by current models, providing actionable insights for future research.
Author of this paper? Send a correction.