Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation
Problem
Public benchmarks suffer data contamination and LLM-as-judge self-preference, leaving no reliable way to evaluate and compare clinical large language models.
Why it matters
Unreliable LLM evaluation risks selecting wrong models for use in clinical practice.
What the AI did
Proposes CLEVER, a blind, randomized, preference-based evaluation by practicing medical doctors, demonstrated by comparing GPT-4o against two healthcare-specific LLMs on summarization, information extraction and biomedical question answering.
Results
Medical doctors preferred the John Snow Labs medical model–small LLM over GPT-4o 45% to 92% more often (abstract truncated).
Impact
Framework developed and validated via blind expert comparison across three clinical tasks.
Record
| Venue | JMIR AI |
|---|---|
| Published | 2025 |
| Work type | article |
| Authors | Veysel Kocaman, Mustafa Aytuğ Kaya, Andrei Marian Feier, David Talby |
| Citations | 23 (OpenAlex, at import time) |
| References | 19 |
| Model or method | GPT-4o (OpenAI); healthcare-specific LLMs of 8B and 70B parameters; John Snow Labs medical model |
| Organization | John Snow Labs; OpenAI (named in abstract) |
| Primary source | https://doi.org/10.2196/72153 |
| Open access | diamond |
| Keywords | large language models, clinical evaluation, expert review, GPT-4o, benchmarking, preference study |
Importance rating
real but narrower advance
Addresses unreliable clinical LLM evaluation (contamination, self-preference), which matters for safe model selection in clinical practice; a methodological framework rather than direct patient impact.