ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Abstract
Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ParaPairAudioBench provides a critical diagnostic framework for evaluating the reliability and calibration of LALM-as-a-Judge systems in paralinguistic speech tasks, revealing systematic biases and modality dependencies that are invisible to aggregate naturalness metrics.
Methodology
The paper proposes a novel diagnostic benchmark, ParaPairAudioBench, designed to evaluate Large Audio-Language Models (LALMs) as judges for paralinguistic speech attributes. The methodology is rigorous in its design: it decomposes evaluation into five distinct dimensions (Style, Rate, Emphasis, Age, Gender) and employs a pairwise comparison format with a "Tie" option to assess calibration. A key methodological strength is the control of lexical content through same-transcript and cross-transcript conditions, allowing for an isolation of acoustic versus textual reliance. The use of position-swapping to quantify order bias is also a robust analytical technique. However, the novelty is somewhat limited by the fact that it is a benchmark/evaluation paper rather than a new model architecture or algorithm. It repurposes existing LALMs and public datasets (Expresso, Sonos, LibriTTS, EARS) into a new evaluation protocol. EXPERIMENTAL_EVALATION: The experimental setup is well-structured, evaluating five representative LALMs (Gemini 2.5 Flash, GPT-4o Audio, Kimi-Audio-7B, Qwen2.5-Omni-7B, SpeechJudge-7B) against human baselines. The results provide valuable insights: significant gaps in human-model alignment (32% average lag), severe calibration failures in Tie cases (models forcing preferences), and asymmetric modality dependence (over-reliance on text for Style, better acoustic reliance for Emphasis in cross-transcript settings). The inclusion of human evaluation on a subset with inter-rater reliability metrics adds credibility. The analysis of position bias (up to 29.4% gap) is a critical finding that highlights a systematic flaw in current judge pipelines. The data presentation is clear, though the paper relies on tables that are not fully rendered in the text provided, the described trends are specific and actionable.
Reproducibility
The paper provides a GitHub repository URL for data and code, which is a strong positive for reproducibility. The dataset construction methodology is described in detail, including the sources of public corpora and the constraints used for pair selection (label constraints, transcript control). The evaluation protocol is clearly defined, including prompt templates (though not explicitly listed, the structure is described) and decoding settings (greedy vs. majority voting). The use of standard public datasets ensures that other researchers can replicate the data generation process.
Limitations
The primary limitation is the scope of the benchmark. It covers only five paralinguistic dimensions and uses a limited number of public datasets, which may not capture the full diversity of speech styles and accents. The Tie condition construction is acknowledged as difficult, particularly for Emphasis, leading to its exclusion in some cases. The human evaluation is limited to a 250-item subset, which may not be fully representative of the entire benchmark's difficulty distribution. Furthermore, the study focuses on existing LALMs; it does not propose a new model or fine-tuning strategy to address the identified failures, limiting its immediate technical impact on model development.
Broader impact
This paper has significant broader impact for the field of audio-language model evaluation. By exposing the systematic weaknesses of current LALM judges in paralinguistic tasks, it provides a necessary diagnostic tool for the community. The findings on calibration failures and position bias are critical for the development of reliable automated evaluation systems, which are increasingly used in training generative models (RLHF). The benchmark encourages the community to move beyond holistic naturalness scores to more fine-grained, robust evaluation metrics. It also highlights the need for better acoustic understanding in LALMs, particularly for localized prosodic features. ParaPairAudioBench provides a critical diagnostic framework for evaluating the reliability and calibration of LALM-as-a-Judge systems in paralinguistic speech tasks, revealing systematic biases and modality dependencies that are invisible to aggregate naturalness metrics.
Author of this paper? Send a correction.