End-to-End Joint ASR and Speaker Role Diarization with Child-Adult Interactions
Abstract
Accurate transcription and speaker diarization of child-adult spoken interactions are crucial for developmental and clinical research. However, manual annotation is time-consuming and challenging to scale. Existing automated systems typically rely on cascaded speaker diarization and speech recognition pipelines, which can lead to error propagation. This paper presents a unified end-to-end framework that extends the Whisper encoder-decoder architecture to jointly model ASR and child-adult speaker role diarization. The proposed approach integrates: (i) a serialized output training scheme that emits speaker tags and start/end timestamps, (ii) a lightweight frame-level diarization head that enhances speaker-discriminative encoder representations, (iii) diarization-guided silence suppression for improved temporal precision, and (iv) a state-machine-based forced decoding procedure that guarantees structurally valid outputs. Comprehensive evaluations on two datasets demonstrate consistent and substantial improvements over two cascaded baselines, achieving lower multi-talker word error rates and demonstrating competitive diarization accuracy across both Whisper-small and Whisper-large models. These findings highlight the effectiveness and practical utility of the proposed joint modeling framework for generating reliable, speaker-attributed transcripts of child-adult interactions at scale. The code and model weights are publicly available
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a unified framework for joint ASR and speaker role diarization, significantly improving the accuracy and efficiency of transcribing child-adult interactions. The methodology is innovative, addressing key challenges in the field and demonstrating substantial technical contributions through rigorous experimentation.
Methodology
The proposed methodology integrates automatic speech recognition (ASR) and speaker role diarization into a unified end-to-end framework using the Whisper architecture. The approach is innovative in its serialized output training scheme, which allows for the simultaneous prediction of speaker tags, timestamps, and transcriptions, thereby addressing the limitations of traditional cascaded systems that suffer from error propagation. The introduction of a lightweight diarization head and a state-machine-based forced decoding mechanism further enhances the model's robustness and output structure. Overall, the methodology is well-structured and leverages existing technologies effectively while introducing novel components that improve performance in child-adult interaction contexts.
Results & evaluation
The experiments are comprehensive, utilizing two distinct datasets that reflect real-world child-adult interactions. The evaluation metrics, including multi-talker word error rates (mtWER) and diarization error rates (DER), are appropriate for assessing the model's performance. The results indicate significant improvements over baseline methods, demonstrating the effectiveness of the proposed framework. However, the paper could benefit from additional ablation studies to further clarify the contributions of each component.
Reproducibility
The authors have made their code and model weights publicly available, which is a positive step towards reproducibility. However, detailed hyperparameter settings and training configurations could be more thoroughly documented to facilitate easier replication of results by other researchers.
Limitations
One limitation is the potential for overfitting due to the relatively small size of the ADOS dataset, which may affect the generalizability of the model. Additionally, the paper does not extensively discuss the challenges faced during training or the specific conditions under which the model may fail, such as in cases of overlapping speech or extreme background noise.
Broader impact
The proposed framework has significant implications for developmental and clinical research, particularly in the context of assessing child language development and social communication patterns. By automating the transcription and diarization processes, the model can facilitate large-scale studies that were previously hindered by the labor-intensive nature of manual annotation. This could lead to more efficient data collection and analysis in clinical settings, ultimately benefiting research in child development and related fields. The paper presents a unified framework for joint ASR and speaker role diarization, significantly improving the accuracy and efficiency of transcribing child-adult interactions. The methodology is innovative, addressing key challenges in the field and demonstrating substantial technical contributions through rigorous experimentation.
Author of this paper? Send a correction.