Slots Are Not Speakers: Training-Free Speaker Linking for Frozen Streaming Diarizers
Abstract
This paper presents a causal, training-free linker that allows a frozen streaming diarizer to track more speakers than it has output slots. Streaming end-to-end diarizers such as Streaming Sortformer emit a fixed number N of concurrent speaker slots. Above N speakers, they keep detecting speech but reuse slots across people, because N bounds how many speakers the model can express at once, not how many it will meet. The linker uses only the host's slot posteriors and the audio. It keeps the frames where one slot is active, splits each slot where the speaker changes, embeds each piece with an off-the-shelf speaker encoder, and matches it against a bank of speakers seen so far. It only relabels, so missed speech and false alarm are identical to the host's and every gain is a reduction in speaker confusion. On 1418 sessions from eight corpora, it reduces the 4-slot host's DER above its cap from 36.8% to 22.6%, and barely moves it below the cap. The same constants transfer to the host's 8-slot successor and to two streaming EEND hosts from a different model family. On courtroom hearings with up to 23 speakers it reduces cpWER from 82.7% to 24.4%.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a training-free, causal linker that enables frozen streaming diarizers to track more speakers than their fixed output slots by using global speaker embeddings to relabel segments. This approach significantly reduces speaker confusion in high-speaker-count scenarios without altering the host model's voice activity detection, offering a practical and effective solution for production-grade streaming diarization systems.
Methodology
The paper proposes a causal, training-free post-processing module ("linker") for frozen streaming diarizers. The core insight is that the fixed number of output slots in models like Streaming Sortformer represents a concurrency limit, not a lifetime speaker limit. The method identifies "pure" frames where only one slot is active, extracts speaker embeddings using an off-the-shelf encoder (TitaNet), and maintains a global speaker bank to relabel segments. By strictly relabeling without altering voice activity boundaries, the method guarantees that Missed Speech and False Alarm rates remain identical to the host model, isolating the improvement to Speaker Confusion reduction. The approach is elegant in its simplicity and its adherence to the "frozen host" constraint, making it highly practical for production systems where retraining the base model is infeasible.
Results & evaluation
The experimental setup is robust, utilizing 1,418 sessions from eight diverse corpora, including Indic languages, earnings calls, and courtroom hearings. The evaluation correctly stratifies results by speaker count relative to the model's capacity cap. The results show a significant reduction in DER (36.8% to 22.6%) for sessions exceeding the 4-slot cap, with negligible impact on below-cap sessions. The comparison against a widened 8-slot host demonstrates that the linker provides greater gains than simply increasing the model's output channels, a strong argument for the proposed method. The inclusion of downstream transcription metrics (cpWER) further validates the practical utility of the diarization improvements.
Reproducibility
The paper provides high reproducibility. It specifies the exact host model version (Sortformer v2.1), the speaker encoder (TitaNet-large), and the specific hyperparameters (link threshold 0.60, purity score threshold 60%). The use of standard scoring tools (pyannote 4.1) and the detailed description of the bootstrap confidence intervals allow for rigorous verification of the claims. The code is not explicitly linked in the text provided, but the methodological details are sufficient for re-implementation.
Limitations
The primary limitation is the reliance on an off-the-shelf speaker encoder, which may not be optimal for all acoustic conditions (e.g., heavy overlap or noisy environments). The method is also limited by the host model's ability to detect speech; if the host misses speech, the linker cannot recover it. Additionally, the paper notes that the linker can sometimes over-count speakers in below-cap scenarios, indicating a potential trade-off in identity stability. The latency of the embedding extraction and matching process is not deeply analyzed, though it is described as running on CPU.
Broader impact
This work has significant impact on the deployment of streaming diarization systems in real-world applications where speaker counts are unpredictable, such as large meetings, courtrooms, and call centers. By decoupling speaker identity tracking from the fixed-capacity diarization model, it offers a scalable solution that can be retrofitted to existing production systems without retraining. This approach could be extended to other fixed-capacity sequence labeling tasks in speech processing. The paper introduces a training-free, causal linker that enables frozen streaming diarizers to track more speakers than their fixed output slots by using global speaker embeddings to relabel segments. This approach significantly reduces speaker confusion in high-speaker-count scenarios without altering the host model's voice activity detection, offering a practical and effective solution for production-grade streaming diarization systems.
Author of this paper? Send a correction.