Challenges in Testing Large Language Model-Based Software: A Faceted Taxonomy

Felix Dobslaw, ROBERT H. FELDT, Juyeon Yoon, Shin Yoo

ACM Transactions on Software Engineering and Methodology 2026 article cited by 4

Source Open-access PDF More in Software & Coding

importance 6/10 partially solved Software Testing and Debugging Techniques software testing stage: research

Problem

LLM-based software is non-deterministic, so traditional testing cannot verify correctness within individual test cases.

Why it matters

LLMs are becoming integral to software systems, making undetected output variability a correctness risk.

What the AI did

Provides a four-faceted taxonomy of LLM test case design, an LLM-assisted analysis of six open-source testing frameworks, and a sensitivity study of an agent-based system across model configurations.

Results

Taxonomy with four facets; analysis of six open-source testing frameworks; sensitivity study across model configurations

Impact

Organizes the field and identifies open challenges; does not solve LLM software testing.

Record

Venue	ACM Transactions on Software Engineering and Methodology
Published	2026
Work type	article
Authors	Felix Dobslaw, ROBERT H. FELDT, Juyeon Yoon, Shin Yoo
Citations	4 (OpenAlex, at import time)
References	16
Model or method	not stated in source
Organization	not stated in source
Primary source	https://doi.org/10.1145/3806396
Open access	hybrid
Keywords	LLM testing, non-determinism, test case design, taxonomy, multi-agent LLMs

Importance rating

6 / 10

significant — thousands to millions affected

Correctness verification for LLM-based software is a foundational challenge as LLMs become integral to software systems; abstract frames it as a field-wide open problem but gives no scale.

Basis: inferred. Judged from the source text, which gives no explicit scale. Stated scale: not stated in source. Method and limits.

More in Software & Coding

#	Paper	Year	Citations	Importance
1	ImageNet classification with deep convolutional neural networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton	2017	106,345	8
2	Deep Residual Learning for Image Recognition Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun	2016	221,134	8
3	Faster sorting algorithms discovered using deep reinforcement learning Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco S...	2023	175	8
4	A deep learning‐based approach for software vulnerability detection using code metrics Fazli Subhan, Xiaoxue Wu, Lili Bo, Xiaobing Sun, Muhammad Rahman	2022	42	7
5	Deep Learning-Based Attack Detection and Classification in Android Devices Alfonso Gómez, Antonio Muñoz	2023	41	7