When Consistency Becomes Bias: Interviewer Effects in Semi-Structured Clinical Interviews
Abstract
Automatic depression detection from doctor-patient conversations has gained momentum thanks to the availability of public corpora and advances in language modeling. However, interpretability remains limited: strong performance is often reported without revealing what drives predictions. We analyze three datasets: ANDROIDS, DAIC-WOZ, E-DAIC and identify a systematic bias from interviewer prompts in semi-structured interviews. Models trained on interviewer turns exploit fixed prompts and positions to distinguish depressed from control subjects, often achieving high classification scores without using participant language. Restricting models to participant utterances distributes decision evidence more broadly and reflects genuine linguistic cues. While semi-structured protocols ensure consistency, including interviewer prompts inflates performance by leveraging script artifacts. Our results highlight a cross-dataset, architecture-agnostic bias and emphasize the need for analyses that localize decision evidence by time and speaker to ensure models learn from participants' language.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper makes a significant contribution by revealing the systematic bias introduced by interviewer prompts in automatic depression detection models. It underscores the necessity for researchers to critically assess the influence of interview structures on model performance, paving the way for more accurate and interpretable AI applications in mental health.
Methodology
The methodology is robust, employing a comparative analysis of participant-only and interviewer-only models across three datasets. The use of both graph convolutional networks and transformers allows for a comprehensive investigation of the prompt-induced bias phenomenon. The paper effectively demonstrates how models can exploit interviewer prompts to achieve high classification scores, emphasizing the need for careful consideration of these biases in model training and evaluation. The qualitative analysis of decision evidence localization adds depth to the findings, showcasing the models' reliance on specific prompts rather than participant language.
Results & evaluation
The experiments are well-structured, utilizing multiple datasets (ANDROIDS, DAIC-WOZ, E-DAIC) to validate the findings. The results consistently show that interviewer prompts can inflate performance metrics, which is a significant insight into the evaluation of automatic depression detection systems. The use of both development and test splits enhances the credibility of the findings, and the performance metrics are clearly reported, allowing for easy comparison across models.
Reproducibility
The paper provides sufficient detail regarding the data preparation and model training processes, including the use of automatic speech recognition for generating transcripts where necessary. However, the reliance on ASR-generated transcripts for two of the datasets introduces potential variability that could affect reproducibility. Future work should aim to provide ground-truth transcripts to strengthen the reproducibility of the results.
Limitations
The study acknowledges several limitations, including the quality of ASR-generated transcripts and the focus on text-only analysis, which may overlook multimodal aspects of communication that could influence depression detection. The potential for transcription noise to affect model performance is a significant concern that could skew results.
Broader impact
The findings have substantial implications for the field of AI in healthcare, particularly in the development of conversational agents for mental health assessment. By highlighting the biases introduced by interviewer prompts, the work encourages a reevaluation of current methodologies in automatic depression detection and emphasizes the importance of developing bias-aware evaluation protocols. This could lead to more reliable and interpretable AI systems in clinical settings. The paper makes a significant contribution by revealing the systematic bias introduced by interviewer prompts in automatic depression detection models. It underscores the necessity for researchers to critically assess the influence of interview structures on model performance, paving the way for more accurate and interpretable AI applications in mental health.
Author of this paper? Send a correction.