BioCoder: a benchmark for bioinformatics code generation with large language models

Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, Mark Gerstein

Bioinformatics 2024 article cited by 30

Source Open-access PDF More in Software & Coding

importance 4/10 ai assisted Software Engineering Research bioinformatics software stage: research mixed

Problem

Large language models lacked a dedicated benchmark for generating bioinformatics-specific code involving domain knowledge, algorithms and data operations.

Why it matters

Bioinformatics code must handle intricate domain-specific tasks correctly as LLMs scale up.

What the AI did

Built BioCoder, a benchmark of 1026 Python functions and 1243 Java methods extracted from GitHub plus 253 Rosalind Project examples, with a fuzz-testing framework to evaluate LLM code generation.

Results

Benchmark spans 1026 Python functions, 1243 Java methods, 253 Rosalind examples; topic modeling shows coverage representative of bioinformatics calculations.

Impact

Provides a standardized benchmark for evaluating LLM bioinformatics code generation.

Record

Venue	Bioinformatics
Published	2024
Work type	article
Authors	Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, Mark Gerstein
Citations	30 (OpenAlex, at import time)
References	6
Model or method	not stated in source
Organization	not stated in source
Primary source	https://doi.org/10.1093/bioinformatics/btae230
Open access	gold
Keywords	BioCoder, LLM benchmark, code generation, bioinformatics, fuzz testing

Importance rating

4 / 10

real but narrower advance

Benchmark fills a gap for evaluating LLM bioinformatics code generation; useful standardized evaluation tool, but abstract states no scale of downstream impact beyond benchmark size.

Basis: inferred. Judged from the source text, which gives no explicit scale. Stated scale: not stated in source. Method and limits.

More in Software & Coding

#	Paper	Year	Citations	Importance
1	ImageNet classification with deep convolutional neural networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton	2017	106,345	8
2	Deep Residual Learning for Image Recognition Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun	2016	221,134	8
3	Faster sorting algorithms discovered using deep reinforcement learning Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco S...	2023	175	8
4	A deep learning‐based approach for software vulnerability detection using code metrics Fazli Subhan, Xiaoxue Wu, Lili Bo, Xiaobing Sun, Muhammad Rahman	2022	42	7
5	Deep Learning-Based Attack Detection and Classification in Android Devices Alfonso Gómez, Antonio Muñoz	2023	41	7