Nemotron 3 Diarization: Advancing the Accuracy-Efficiency Frontier in Streaming Speaker Diarization
Abstract
Streaming Sortformer introduced an arrival-order speaker cache to maintain speaker identities across chunks in end-to-end streaming speaker diarization, with a single model supporting configurable latency profiles spanning offline operation to ultra-low-latency streaming. Nemotron 3 Diarization addresses the original model's four-speaker limitation by extending support to eight speakers while improving both accuracy and computational efficiency. The previous FastConformer-Transformer stack is streamlined into a pure Transformer encoder. A learned upsampling layer then maps the encoder's coarse 80-ms representations to 10-ms resolution, enabling fine-grained predictions. Training leverages large-scale simulated mixtures and adds activity and phantom-speaker losses to the standard Sortformer objectives. Despite doubling speaker capacity over its predecessor, the new architecture reduces training memory by a factor of 2.8, makes training $2.2\times$ faster, and delivers up to $6.4\times$ higher inference throughput. Nemotron 3 Diarization establishes a new state of the art in DER on DIHARD III and CALLHOME. When used to condition DiCoW, it also achieves state-of-the-art tcpWER on the single-microphone conditions of AMI and NOTSOFAR-1. Training code is open source, and model weights are released under the permissive OpenMDW-1.1 license.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Extends the Streaming Sortformer framework to support eight speakers with improved accuracy and efficiency. The paper presents a robust engineering and methodological contribution to the field of speaker diarization, effectively balancing the trade-offs between speaker capacity, temporal resolution, and computational cost through architectural simplification and targeted auxiliary losses.
Methodology
The paper proposes a streamlined architecture for streaming speaker diarization, replacing the previous FastConformer-Transformer stack with a pure 31-layer Transformer encoder. Key methodological contributions include a learned subpixel convolution layer for upsampling 80-ms representations to 10-ms resolution, which allows for fine-grained boundary detection without increasing encoder computational cost. The authors introduce two auxiliary losses: an activity loss to model overall speech activity (silence, single, overlap) and a phantom loss to suppress false activations in unused speaker channels, addressing the overcounting issue inherent in fixed-capacity EEND models. The training regimen involves a two-stage process: offline pre-training on simulated mixtures initialized from a self-supervised NEST model, followed by streaming fine-tuning with an Arrival-Order Speaker Cache (AOSC) and block-causal tail augmentation to simulate varying latency contexts.
Results & evaluation
The experimental evaluation is extensive, covering both offline and streaming settings across multiple latency profiles (30.4s, 1.04s, 0.32s). The model achieves state-of-the-art DER on DIHARD III and CALLHOME, and SOTA tcpWER on AMI and NOTSOFAR-1 when conditioning DiCoW. Ablation studies rigorously quantify the impact of the backbone change, upsampling layer, auxiliary losses, and data composition. The paper also provides strong evidence of efficiency gains, reporting a 2.8x reduction in training memory and 6.4x higher inference throughput compared to the predecessor, despite doubling the speaker capacity.
Reproducibility
Reproducibility is high. The authors release the model weights under a permissive license and provide the training code via the NVIDIA NeMo Speech Framework. Specific configuration files for both offline and streaming training are linked in the paper. Detailed descriptions of data sources, simulation protocols (FastMSS), and training hyperparameters are provided.
Limitations
The model is still limited to a maximum of 8 speakers, which may be insufficient for very large group conversations. The evaluation relies heavily on simulated mixtures for training, which may introduce domain gaps not fully captured by the augmentation strategies. The paper notes that DER comparisons on some datasets (AMI, AliMeeting, NOTSOFAR-1) are not directly comparable to previous SOTA due to differences in ground-truth annotation styles (forced-aligned vs. segment-level), although they mitigate this by also reporting tcpWER.
Broader impact
This work significantly advances the practicality of streaming speaker diarization by making high-accuracy, multi-speaker models computationally efficient enough for real-time deployment on consumer hardware. The open-sourcing of weights and code under a permissive license facilitates widespread adoption in both research and industry, particularly for meeting transcription and multi-talker ASR systems. The integration with downstream ASR models like DiCoW demonstrates the utility of the diarization component in end-to-end speech processing pipelines. Extends the Streaming Sortformer framework to support eight speakers with improved accuracy and efficiency. The paper presents a robust engineering and methodological contribution to the field of speaker diarization, effectively balancing the trade-offs between speaker capacity, temporal resolution, and computational cost through architectural simplification and targeted auxiliary losses.
Author of this paper? Send a correction.