A bilingual benchmark for evaluating large language models
importance 5/10
partially solved
Topic Modeling NLP evaluation stage: research
Problem
Lack of standardized benchmarks to evaluate large language models' Arabic capabilities and compare them to English.
Why it matters
Arabic-speaking users may get poorer LLM performance without rigorous bilingual evaluation.
What the AI did
Built a bilingual evaluation dataset from the General Aptitude Test and used it to measure and compare ChatGPT, GPT-4, and fastText performance in Arabic versus English.
Results
fastText surpassed ChatGPT on Arabic word analogies; GPT-4 Arabic findings truncated in abstract.
Impact
Enables direct English–Arabic LLM comparison and reveals Arabic performance gaps.
Record
| Venue | PeerJ Computer Science |
|---|---|
| Published | 2024 |
| Work type | article |
| Authors | Mohamed Alkaoud |
| Citations | 11 (OpenAlex, at import time) |
| References | 66 |
| Model or method | ChatGPT, GPT-4, fastText |
| Organization | not stated in source |
| Primary source | https://doi.org/10.7717/peerj-cs.1893 |
| Open access | gold |
| Keywords | large language models, Arabic, benchmark, ChatGPT, GPT-4, evaluation |
Importance rating
5
real but narrower advance
First bilingual English-Arabic LLM benchmark enabling direct comparison; addresses limited Arabic evaluation, but abstract states no user or deployment scale.
More in Software & Coding
| # | Paper | Year | Citations | Importance |
|---|---|---|---|---|
| 1 | ImageNet classification with deep convolutional neural networks | 2017 | 106,345 |
8
|
| 2 | Deep Residual Learning for Image Recognition | 2016 | 221,134 |
8
|
| 3 | Faster sorting algorithms discovered using deep reinforcement learning | 2023 | 175 |
8
|
| 4 | A deep learning‐based approach for software vulnerability detection using code metrics | 2022 | 42 |
7
|
| 5 | Deep Learning-Based Attack Detection and Classification in Android Devices | 2023 | 41 |
7
|