HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding
Abstract
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
Methodology
The paper introduces a hierarchical benchmark (HEAR) grounded in Erber’s Auditory Hierarchy, decomposing speaker-attributed reasoning into Discrimination, Attribution, and Reasoning. This is a methodologically sound approach to diagnosing specific failure modes in Speech Language Models (SLMs). The proposed training strategy, A2R, utilizes Group Relative Policy Optimization (GRPO) with a composite reward function that explicitly rewards speaker-tagged transcription accuracy (cpWER, speaker count, order) alongside final answer correctness. The creation of the CASH dataset, which uses voice cloning to create counterfactual hard negatives where the transcript remains constant but the speaker identity changes, is a clever and effective way to force the model to rely on acoustic cues rather than semantic priors. The "transcription-first" objective is a strong inductive bias for this specific task.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors evaluate 20 leading SLMs, providing a clear baseline comparison. The introduction of the "paired accuracy" metric for the Reasoning dimension is particularly insightful, as it exposes models that rely on semantic shortcuts by requiring them to answer both the original and the counterfactual variant correctly. The zero-shot transfer experiments to unseen benchmarks (WDYL, Gaokao, FTS) and the validation on real-world human voices (vs. synthesized) strongly support the claim that the learned attribution capability is generalizable and not an artifact of the training data distribution. The ablation studies effectively isolate the contribution of hard negatives and the structured reasoning trace.
Reproducibility
The paper provides high reproducibility. It details the source datasets (AMI, ICSI, VoxMM), the specific TTS models used for synthesis (VoxCPM2, IndexTTS 2), and the filtering criteria (WER < 0.1, speaker similarity > 0.7). The reward function components are clearly defined. The project page is provided, and the authors commit to releasing code and gated access to synthetic data, which is appropriate given the ethical concerns of voice cloning.
Limitations
The primary limitation is the reliance on synthesized speech for the training data (CASH) and parts of the evaluation, although the authors mitigate this by testing on real human voices. The "transcription-first" approach introduces latency, which the authors acknowledge as a barrier to real-time applications. Additionally, the benchmark size (2.4K samples) is relatively small compared to large-scale LLM benchmarks, though the complexity of the tasks justifies this. The ethical implications of voice cloning are addressed, but the potential for misuse of the released synthetic data remains a concern, mitigated only by a Data Use Agreement.
Broader impact
This work has significant impact on the development of multi-party speech understanding systems. By identifying and addressing the "semantic hallucination" problem in SLMs, it provides a pathway to more robust voice assistants and meeting analysis tools that can accurately track who said what. The benchmark HEAR will likely become a standard evaluation suite for future SLMs, and the CASH dataset offers a valuable resource for training models that prioritize acoustic evidence. The paper presents a rigorous and novel framework for evaluating and improving speaker-attributed reasoning in Speech Language Models. By introducing the HEAR benchmark and the CASH dataset with counterfactual hard negatives, the authors effectively demonstrate that current models rely on semantic priors rather than acoustic cues, and propose a GRPO-based training method (A2R) that successfully shifts this reliance, resulting in strong performance gains and zero-shot generalization to unseen multi-speaker tasks.
Author of this paper? Send a correction.