From Speech to Profile: A Protocol-Driven LLM Agent for Psychological Profile Generation
Abstract
The psychological profile that structurally documents the case of a depression patient is essential for psychotherapy. Large language models can be applied to summarize the profiles from counseling speech, however, it may suffer from long-context forgetting and produce unverifiable hallucinations, due to overlong length of speech, multi-party interactions and unstructured chatting. Hereby, we propose a StreamProfile, a streaming framework that processes counseling speech incrementally, extracts evidences grounded from ASR transcriptions by storing it in a Hierarchical Evidence Memory, and then performs a Chain-of-Thought pipeline according to PM+ psychological intervention for clinical reasoning. The final profile is synthesized strictly from those evidences, making every claim traceable. Experiments on real-world teenager counseling speech have shown that the proposed StreamProfile system can accurately generate the profiles and prevent hallucination.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of StreamProfile, a novel framework that integrates streaming processing, CoT reasoning, and evidence memory to generate accurate and verifiable psychological profiles from counseling sessions. This work represents a significant advancement in the application of LLMs in mental health, addressing critical challenges and demonstrating substantial improvements over existing methods.
Methodology
The methodology presented in this paper is innovative, combining a streaming framework with a Chain-of-Thought (CoT) reasoning process and a Hierarchical Evidence Memory (HEM) to generate psychological profiles from counseling sessions. The approach addresses critical issues such as long-context forgetting and hallucinations in LLMs by ensuring that every claim made in the generated profiles is traceable to specific utterances from the counseling session. The use of a structured protocol (PM+) to guide the reasoning process is particularly noteworthy, as it aligns the model's outputs with clinical standards.
Results & evaluation
The experiments conducted on the Psy-Bench dataset demonstrate a rigorous evaluation of the proposed system against various LLM baselines. The results indicate significant improvements in both profile generation performance and hallucination reduction. The use of multiple evaluation metrics, including ROUGE-L, BERTScore, and subjective assessments of hallucination and consistency, provides a comprehensive understanding of the system's capabilities. The ablation studies further validate the effectiveness of the CoT and HEM components.
Reproducibility
The paper provides detailed descriptions of the experimental setup, including the LLMs used, evaluation metrics, and dataset characteristics. However, the lack of a publicly available codebase or demo limits the reproducibility of the results. The authors mention using specific models and configurations, but without access to the implementation, it may be challenging for other researchers to replicate the findings.
Limitations
One limitation is the reliance on a specific dataset (Psy-Bench) that may not generalize to other contexts or languages, as the experiments are conducted on a Chinese dataset. Additionally, while the framework addresses hallucinations effectively, the potential for misinterpretation of nuanced clinical language remains a concern. The paper also does not discuss the computational resources required for real-time processing, which could impact practical deployment.
Broader impact
The proposed framework has significant implications for mental health care, particularly in enhancing the efficiency and accuracy of psychological assessments. By automating the generation of structured profiles from counseling sessions, it could aid clinicians in delivering timely and informed interventions. However, ethical considerations regarding patient data privacy and the potential for over-reliance on AI in sensitive clinical contexts must be addressed. The main contribution of this paper is the introduction of StreamProfile, a novel framework that integrates streaming processing, CoT reasoning, and evidence memory to generate accurate and verifiable psychological profiles from counseling sessions. This work represents a significant advancement in the application of LLMs in mental health, addressing critical challenges and demonstrating substantial improvements over existing methods.
Author of this paper? Send a correction.