Evaluating Diagnostic Accuracy and Treatment Efficacy in Mental Health: A Comparative Analysis of Large Language Model Tools and Mental Health Professionals
Problem
Whether large language models can accurately assess mental health disorders and recommend treatments compared to mental health professionals was underexplored.
Why it matters
Diagnostic accuracy in mental health directly affects treatment quality for conditions like depression, schizophrenia and PTSD.
What the AI did
Four LLMs were given text vignettes of conditions such as depression, suicidal ideation, schizophrenia, social phobia and PTSD, and their diagnostic accuracy, treatment recommendations and predicted outcomes were compared to professional norms.
Results
ChatGPT-4 achieved higher diagnostic accuracy than human professionals for depression and PTSD, but only 55% accuracy on early schizophrenia, where other LLMs and professionals performed better.
Impact
LLMs matched or exceeded professionals on some diagnoses but performed variably on complex cases like early schizophrenia.
Record
| Venue | European Journal of Investigation in Health Psychology and Education |
|---|---|
| Published | 2025 |
| Work type | article |
| Authors | Inbar Levkovich |
| Citations | 29 (OpenAlex, at import time) |
| References | 39 |
| Model or method | Gemini 2.0 Flash Experimental, Claude 3.5 Sonnet, ChatGPT-3.5, ChatGPT-4 |
| Organization | not stated in source |
| Primary source | https://doi.org/10.3390/ejihpe15010009 |
| Open access | gold |
| Keywords | LLM, mental health, diagnosis, ChatGPT-4, treatment recommendations |
Importance rating
significant — thousands to millions affected
Mental health diagnostic accuracy directly affects treatment for depression, PTSD, schizophrenia and suicidal ideation; LLMs matched or exceeded professionals on some diagnoses but varied on complex cases.