Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Problem
Large language models tend to generate incorrect or harmful statements, limiting their adoption for clinical medical guideline and treatment recommendations.
Why it matters
Incorrect medical guidance can harm patients and blocks clinical adoption of LLMs.
What the AI did
Almanac augments an LLM with retrieval from curated medical resources; a panel of eight board-certified clinicians and two healthcare practitioners evaluated it against ChatGPT-4, Bing, and Bard on 314 clinical questions across nine specialties.
Results
Significant improvement in performance versus standard LLMs (ChatGPT-4, Bing, Bard) on 314 clinical questions; detailed metrics truncated in provided abstract.
Impact
Clinician-panel evaluation indicates retrieval augmentation improves LLM performance for medical guideline and treatment recommendations.
Record
| Venue | NEJM AI |
|---|---|
| Published | 2024 |
| Work type | article |
| Authors | Cyril Zakka, Rohan Shad, Akash R. Chaurasia, Alex R. Dalal, Jennifer L. Kim, Michael Moor, Robyn Fong, Curran Phillips et al. |
| Citations | 401 (OpenAlex, at import time) |
| References | 31 |
| Model or method | Almanac (retrieval-augmented LLM framework) |
| Organization | not stated in source |
| Primary source | https://doi.org/10.1056/aioa2300068 |
| Open access | green |
| Keywords | large language models, retrieval augmentation, clinical medicine, medical guidelines, safety |
Importance rating
significant — thousands to millions affected
LLM-generated incorrect/harmful medical statements block clinical adoption; abstract frames clinical decision-making potential but gives no scale of affected patients.