SimLLM: Calculating Semantic Similarity in Code Summaries using a Large Language Model-Based Approach

Xin Jin, Zhiqiang Lin

Proceedings of the ACM on software engineering 2024 article cited by 11

Source Open-access PDF More in Software & Coding

importance 4/10 partially solved Software Engineering Research software engineering stage: research

Problem

Existing metrics like BLEU and BERTScore fail to capture semantic nuances and domain-specific terminology when evaluating code summaries.

Why it matters

Code summaries drive software readability, maintainability, and collaboration, so poor metrics misjudge their quality.

What the AI did

Built SimLLM, an autoregressive LLM with specialized pretraining on permutated inputs and pooling-based pairwise similarity, to measure semantic similarity of code summaries.

Results

Stated to outperform existing metrics; no numeric results in abstract

Impact

Peer-reviewed publication offering an improved summary-similarity metric; adoption is not stated.

Record

Venue	Proceedings of the ACM on software engineering
Published	2024
Work type	article
Authors	Xin Jin, Zhiqiang Lin
Citations	11 (OpenAlex, at import time)
References	67
Model or method	SimLLM (autoregressive LLM)
Organization	not stated in source
Primary source	https://doi.org/10.1145/3660769
Open access	hybrid
Keywords	large language models, code summaries, semantic similarity, evaluation metrics, software engineering

Importance rating

4 / 10

real but narrower advance

Better evaluation of code summaries aids software maintainability and collaboration; a useful methodological advance for software engineering research, with no stated scale of impact.

Basis: inferred. Judged from the source text, which gives no explicit scale. Stated scale: not stated in source. Method and limits.

More in Software & Coding

#	Paper	Year	Citations	Importance
1	ImageNet classification with deep convolutional neural networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton	2017	106,345	8
2	Deep Residual Learning for Image Recognition Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun	2016	221,134	8
3	Faster sorting algorithms discovered using deep reinforcement learning Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco S...	2023	175	8
4	A deep learning‐based approach for software vulnerability detection using code metrics Fazli Subhan, Xiaoxue Wu, Lili Bo, Xiaobing Sun, Muhammad Rahman	2022	42	7
5	Deep Learning-Based Attack Detection and Classification in Android Devices Alfonso Gómez, Antonio Muñoz	2023	41	7