Large language models encode clinical knowledge

Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, JASON KAI WEI LEE, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani et al.

Nature 2023 article cited by 3,955

Source Open-access PDF More in Medicine & Health

importance 7/10 partially solved Artificial Intelligence in Healthcare and Education medical AI stage: research proprietary model

Problem

Reliably assessing whether large language models possess real clinical knowledge, beyond limited automated benchmarks.

Why it matters

Clinical applications demand high accuracy; unsafe or wrong medical answers could harm patients.

What the AI did

Built the MultiMedQA benchmark and a human evaluation framework (factuality, reasoning, harm, bias), then evaluated PaLM and Flan-PaLM with prompting strategies.

Results

State-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA)

Impact

Established a benchmark and evaluation framework for measuring LLMs' clinical knowledge.

Record

Venue	Nature
Published	2023
Work type	article
Authors	Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, JASON KAI WEI LEE, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani et al.
Citations	3,955 (OpenAlex, at import time)
References	105
Model or method	PaLM (540B parameter) and Flan-PaLM
Organization	not stated in source
Primary source	https://doi.org/10.1038/s41586-023-06291-2
Open access	hybrid
Keywords	large language models, clinical knowledge, medical question answering, benchmark, PaLM, MultiMedQA

Importance rating

7 / 10

significant — thousands to millions affected

Safety of clinical LLM use matters broadly; abstract stresses high bar for clinical applications and harm/bias evaluation but cites no scale of use.

Basis: inferred. Judged from the source text, which gives no explicit scale. Stated scale: not stated in source. Method and limits.

More in Medicine & Health

#	Paper	Year	Citations	Importance
1	Multimodal deep learning of fundus abnormalities and traditional risk factors for cardiovascular risk prediction Yeong Chan Lee, Jiho Cha, Injeong Shim, Woong‐Yang Park, Se Woong Kang...	2023	107	10
2	Reliable Tuberculosis Detection Using Chest X-Ray With Deep Learning, Segmentation and Visualization Tawsifur Rahman, Amith Abdullah Khandakar, Muhammad Abdul Kadir, Khand...	2020	651	10
3	A comparative analysis of machine learning classifiers for stroke prediction: A predictive analytics approach Nitish Biswas, Khandaker Mohammad Mohi Uddin, Sarreha Tasmin Rikta, Sa...	2022	116	10
4	Cardiovascular Disease Diagnosis from DXA Scan and Retinal Images Using Deep Learning Hamada R. H. Al-Absi, Mohammad Tariqul Islam, Mahmoud Ahmed Refaee, Mu...	2022	89	10
5	Proteomics and Machine Learning Approaches Reveal a Set of Prognostic Markers for COVID-19 Severity With Drug Repurposing Potential Kruthi Suvarna, Deeptarup Biswas, Medha Gayathri J. Pai, Arup Acharjee...	2021	76	10