COFFE: A Code Efficiency Benchmark for Code Generation
importance 4/10
partially solved
Software Engineering Research code generation stage: research mixed
Problem
No suitable benchmark existed to evaluate the time efficiency of LLM-generated code, since correctness benchmarks cannot distinguish efficiency.
Why it matters
Time-inefficient AI-generated code undermines real-world software performance even when functionally correct.
What the AI did
Proposed COFFE, a benchmark for measuring the time efficiency of LLM-generated code solutions, addressing unstable execution-time measurement.
Results
not stated in source
Impact
Abstract supports the benchmark proposal only; no measured outcomes are shown.
Record
| Venue | Proceedings of the ACM on software engineering |
|---|---|
| Published | 2025 |
| Work type | article |
| Authors | Yun Peng, Jun Wan, Yichen Li, Xiaoxue Ren |
| Citations | 14 (OpenAlex, at import time) |
| References | 60 |
| Model or method | not stated in source |
| Organization | not stated in source |
| Primary source | https://doi.org/10.1145/3715727 |
| Open access | hybrid |
| Keywords | code generation, LLM, benchmark, time efficiency, software engineering |
Importance rating
4
real but narrower advance
Fills a real gap: correctness benchmarks cannot distinguish time efficiency of LLM-generated code, undermining real-world software performance; a narrower benchmarking advance.
More in Software & Coding
| # | Paper | Year | Citations | Importance |
|---|---|---|---|---|
| 1 | ImageNet classification with deep convolutional neural networks | 2017 | 106,345 |
8
|
| 2 | Deep Residual Learning for Image Recognition | 2016 | 221,134 |
8
|
| 3 | Faster sorting algorithms discovered using deep reinforcement learning | 2023 | 175 |
8
|
| 4 | A deep learning‐based approach for software vulnerability detection using code metrics | 2022 | 42 |
7
|
| 5 | Deep Learning-Based Attack Detection and Classification in Android Devices | 2023 | 41 |
7
|