Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study

Junhyuk Seo, Dasol Choi, Taerim Kim, Won Chul Cha, Minha Kim, Haanju Yoo, Namkee Oh, Yongjin Yi et al.

Journal of Medical Internet Research 2024 article cited by 52

Source Open-access PDF More in Medicine & Health

importance 4/10 ai assisted Artificial Intelligence in Healthcare and Education health IT stage: research

Problem

There was no established framework or quality standards for assessing accuracy and clinical applicability of LLM-generated emergency department records.

Why it matters

Unreliable AI-generated medical records could compromise patient care and block clinical adoption.

What the AI did

A Healthcare Prompt-a-thon had 52 participants generate 33 ED records with HyperCLOVA X, which were then assessed via a dual clinical evaluation framework involving 4 medical professionals.

Results

not stated in source

Impact

A validated evaluation framework was developed, but no accuracy or usability scores are stated in the abstract.

Record

Venue	Journal of Medical Internet Research
Published	2024
Work type	article
Authors	Junhyuk Seo, Dasol Choi, Taerim Kim, Won Chul Cha, Minha Kim, Haanju Yoo, Namkee Oh, Yongjin Yi et al.
Citations	52 (OpenAlex, at import time)
References	18
Model or method	HyperCLOVA X
Organization	not stated in source
Primary source	https://doi.org/10.2196/58329
Open access	gold
Keywords	large language models, medical documentation, evaluation framework, emergency department, usability

Importance rating

4 / 10

real but narrower advance

Evaluation framework for LLM-generated ED records addresses documentation reliability and clinical adoption concerns; real but narrow methodological advance, no population-scale impact stated.

Basis: inferred. Judged from the source text, which gives no explicit scale. Stated scale: not stated in source. Method and limits.

More in Medicine & Health

#	Paper	Year	Citations	Importance
1	Multimodal deep learning of fundus abnormalities and traditional risk factors for cardiovascular risk prediction Yeong Chan Lee, Jiho Cha, Injeong Shim, Woong‐Yang Park, Se Woong Kang...	2023	107	10
2	Reliable Tuberculosis Detection Using Chest X-Ray With Deep Learning, Segmentation and Visualization Tawsifur Rahman, Amith Abdullah Khandakar, Muhammad Abdul Kadir, Khand...	2020	651	10
3	A comparative analysis of machine learning classifiers for stroke prediction: A predictive analytics approach Nitish Biswas, Khandaker Mohammad Mohi Uddin, Sarreha Tasmin Rikta, Sa...	2022	116	10
4	Cardiovascular Disease Diagnosis from DXA Scan and Retinal Images Using Deep Learning Hamada R. H. Al-Absi, Mohammad Tariqul Islam, Mahmoud Ahmed Refaee, Mu...	2022	89	10
5	Proteomics and Machine Learning Approaches Reveal a Set of Prognostic Markers for COVID-19 Severity With Drug Repurposing Potential Kruthi Suvarna, Deeptarup Biswas, Medha Gayathri J. Pai, Arup Acharjee...	2021	76	10