<scp>GeoAnalystBench</scp> : A <scp>GeoAI</scp> Benchmark for Assessing Large Language Models for Spatial Analysis Workflow and Code Generation
Problem
It was unclear whether large language models can actually automate real-world geospatial analysis and GIS workflows, and rigorous evaluation of this capability was lacking.
Why it matters
Automating GIS workflows could reduce expert effort in real-world spatial analysis, but unverified claims risk unreliable automation.
What the AI did
The authors built GeoAnalystBench, 50 expert-validated Python geoprocessing tasks, and evaluated proprietary and open-source LLMs on workflow validity, structural/semantic alignment, and code quality.
Results
ChatGPT-4o-mini: 95% validity, CodeBLEU 0.39; DeepSeek-R1-7B: 48.5% validity, CodeBLEU 0.272
Impact
The benchmark reveals a clear capability gap and shows full GIS automation claims are premature.
Record
| Venue | Transactions in GIS |
|---|---|
| Published | 2025 |
| Work type | article |
| Authors | Qianheng Zhang, Song Gao, Wei Chen, Yibo Zhao, Ying Nie, Ziru Chen, Shijie Chen, Yu Su et al. |
| Citations | 16 (OpenAlex, at import time) |
| References | 30 |
| Model or method | ChatGPT-4o-mini and DeepSeek-R1-7B (among proprietary and open-source models assessed) |
| Organization | not stated in source |
| Primary source | https://doi.org/10.1111/tgis.70135 |
| Open access | hybrid |
| Keywords | LLM benchmark, GeoAI, GIS, code generation, geospatial analysis |
Importance rating
real but narrower advance
Benchmark rigor for LLM GIS automation claims; abstract frames it as evaluation before automation claims, a useful but narrow research advance; no scale of affected users stated.
More in Software & Coding
| # | Paper | Year | Citations | Importance |
|---|---|---|---|---|
| 1 | ImageNet classification with deep convolutional neural networks | 2017 | 106,345 |
8
|
| 2 | Deep Residual Learning for Image Recognition | 2016 | 221,134 |
8
|
| 3 | Faster sorting algorithms discovered using deep reinforcement learning | 2023 | 175 |
8
|
| 4 | A deep learning‐based approach for software vulnerability detection using code metrics | 2022 | 42 |
7
|
| 5 | Deep Learning-Based Attack Detection and Classification in Android Devices | 2023 | 41 |
7
|