Evaluating clinical AI summaries with large language models as judges
Problem
Human review, the gold standard for checking AI-generated EHR summaries, is costly and slow, making reliable evaluation of clinical summaries difficult.
Why it matters
Inaccurate clinical summaries could misinform providers, and manual verification cannot scale.
What the AI did
An LLM-as-a-Judge framework automatically scores real-world multi-document EHR summaries, benchmarked against the validated PDSQI instrument with human evaluators.
Results
GPT-o3-mini achieved intraclass correlation 0.818 (95% CI 0.772-0.854) with human evaluators, median score difference of 0, and 22-second evaluations.
Impact
Demonstrates automated LLM evaluation matching human raters far faster than manual review.
Record
| Venue | npj Digital Medicine |
|---|---|
| Published | 2025 |
| Work type | article |
| Authors | Emma Croxford, Yanjun Gao, Elliot First, Nicholas Pellegrino, Miranda Schnier, John Caskey, Madeline K. Oguss, Graham J Wills et al. |
| Citations | 47 (OpenAlex, at import time) |
| References | 22 |
| Model or method | GPT-o3-mini (LLM-as-a-Judge) |
| Organization | not stated in source |
| Primary source | https://doi.org/10.1038/s41746-025-02005-2 |
| Open access | gold |
| Keywords | LLM-as-a-Judge, EHR summarization, PDSQI, inter-rater reliability, clinical AI |
Importance rating
real but narrower advance
Abstract states human review of EHR summaries is costly and slow and inaccurate summaries risk misinforming providers; a scalable evaluation method benefits clinical AI deployment, but no population-scale figures given.