Probabilistic Fusion and Calibration of Neural Speaker Diarization Models
Abstract
End-to-End Neural Diarization (EEND) systems produce frame-level probabilistic speaker activity estimates, yet since evaluation focuses primarily on Diarization Error Rate (DER), the reliability and calibration of these confidence scores have been largely neglected. When fusing multiple diarization systems, DOVER-Lap remains the only established approach, operating at the segment level with hard decisions. We propose working with continuous probability outputs, which enables more sophisticated calibration and fusion techniques that can leverage model uncertainty and complementary strengths across different architectures. This paper presents the first comprehensive framework for calibrating and fusing EEND models at the probability level. We investigate two output formulations (multilabel and powerset representations) and their impact on calibration and fusion effectiveness. Through extensive experiments on the CallHome two-speaker benchmark, we demonstrate that proper calibration provides substantial improvements even for individual models (up to 19% relative DER reduction), in some cases mitigating the absence of domain adaptation. We reveal that joint calibration in powerset space consistently outperforms independent per-speaker calibration, and that the Fuse-then-Calibrate ordering generally outperforms calibrating individual models before fusion while requiring calibration of only a single combined model. Our best configuration outperforms DOVER-Lap in terms of DER while providing reliable confidence estimates essential for downstream applications. This work proposes best practices for probability-level fusion of EEND systems and demonstrates the advantages of leveraging soft outputs over hard decisions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper provides a comprehensive framework for calibrating and fusing EEND systems at the probability level, marking a significant contribution to the field of speaker diarization. The innovative methodology and thorough experimental evaluation demonstrate its potential to enhance the reliability and effectiveness of neural diarization systems.
Methodology
The paper presents a novel framework for calibrating and fusing End-to-End Neural Diarization (EEND) models at the probability level, which is a significant advancement over existing methods that primarily operate on hard decisions. The authors explore two output formulations (multilabel and powerset) and their effects on calibration and fusion, providing a systematic approach that leverages model uncertainty. The methodology is well-structured, with clear definitions of calibration strategies and fusion methods, including both unsupervised and supervised techniques.
Results & evaluation
The experimental evaluation is thorough, utilizing the CallHome two-speaker benchmark to demonstrate the effectiveness of the proposed methods. The results show substantial improvements in Diarization Error Rate (DER) and calibration quality, with detailed comparisons against existing methods like DOVER-Lap. The experiments are well-designed, covering various configurations and providing insights into the impact of calibration and fusion strategies.
Reproducibility
The paper includes a link to the publicly available code repository, which enhances reproducibility. The implementation details are sufficiently described, allowing other researchers to replicate the experiments. However, the paper could benefit from more explicit details on hyperparameter settings and training procedures.
Limitations
One limitation is the focus on two-speaker scenarios, which may not generalize to more complex multi-speaker environments. Additionally, while the paper discusses the importance of calibration, it does not explore the potential trade-offs between calibration quality and computational efficiency in depth.
Broader impact
The proposed framework has significant implications for improving speaker diarization systems, particularly in applications where accurate speaker identification is critical, such as in transcription services, meeting analysis, and audio indexing. By enhancing the reliability of confidence scores, this work can lead to better performance in downstream tasks that rely on speaker diarization. This paper provides a comprehensive framework for calibrating and fusing EEND systems at the probability level, marking a significant contribution to the field of speaker diarization. The innovative methodology and thorough experimental evaluation demonstrate its potential to enhance the reliability and effectiveness of neural diarization systems.
Author of this paper? Send a correction.