Method
This page exists so the numbers can be argued with. Everything below is how an entry gets here, and what the ratings do and do not mean.
1 · Finding candidates
Searches run against OpenAlex, arXiv and Crossref for work describing an AI contribution to a concrete problem. Every candidate keeps its real abstract, DOI, publication date, venue and author list — the search only supplies the query, never the content.
2 · Two-stage review
An independent model (glm-5.3, via Ollama Cloud) judges each candidate against its own abstract and returns a verdict — ok, suspect or unknown — plus a confidence score and notes. Entries that are not ok go to a separate file and are never published here. Model agreement is not treated as verification: the check is whether the source text supports the claim, and doubts are recorded rather than smoothed over.
Current status split of the published set: verified 1,138 cases, mean validator confidence 76.3%.
3 · The importance rating
Each published case is rated 0–10 for the importance of the problem — not of the AI method, and not of the paper. The rating is produced by the same model, from the source text only, against the rubric below, and is stored together with a short rationale.
| Score | Means |
|---|---|
| 10 | Existential or global scale — e.g. a leading cause of death worldwide |
| 8–9 | Major: affects millions of people, a whole industry, or an entire field |
| 6–7 | Significant: a real problem at regional or sector scale |
| 4–5 | Real but narrower — useful, limited reach |
| 1–3 | Incremental or niche |
4 · Stated vs inferred
Each rating is flagged for its basis: stated means the source itself quantifies the scale or impact (for example “17.9 million deaths per year”), and that figure is stored verbatim; inferred means the source gives no scale and the rating is an explicit judgement from context.
Currently: 393 stated, 745 inferred, of 1,138 rated cases.
5 · Known limits
- The rating is AI-assisted, not measured, and not peer-reviewed. It is reproducible from the rubric but it is still a judgement.
- Ratings are dominated by what abstracts say. A paper that does not state its scale cannot score “stated”, which understates work whose importance is obvious to specialists.
- Two of the added fields — deployment stage and model openness — are thin, because abstracts rarely mention them. They are shown where present and marked “not stated” where not.
- Coverage is biased toward English-language, indexed work in medicine and biology, which is where the harvest was densest.
- “Verified” means the evidence supported the description. It does not mean reproduced.
Data refreshed hourly from the source sheet. Something wrong here? It should be arguable from the primary source on each case page.