Can Large Language Models Transform Computational Social Science?
Problem
Whether large language models can reliably classify and explain social phenomena (e.g., persuasiveness, ideology) for computational social science.
Why it matters
Reliable LLM annotation could reduce costly human coding in social science research.
What the AI did
Evaluated 13 language models zero-shot on 25 English CSS benchmarks, contributing prompting best practices and an evaluation pipeline.
Results
LLMs fail to outperform best fine-tuned models on classification but reach fair human agreement; free-form explanations often exceed crowdworker references.
Impact
Abstract concludes today's LLMs can augment the CSS pipeline in two ways (list truncated).
Record
| Venue | Computational Linguistics |
|---|---|
| Published | 2023 |
| Work type | article |
| Authors | Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, Diyi Yang |
| Citations | 559 (OpenAlex, at import time) |
| References | 366 |
| Model or method | not stated in source |
| Organization | not stated in source |
| Primary source | https://doi.org/10.1162/coli_a_00502 |
| Open access | diamond |
| Keywords | large language models, computational social science, zero-shot, classification, text annotation |
Importance rating
real but narrower advance
LLM annotation could reduce costly human coding across social science; useful augmentation of a research field, but no scale or impact figures stated.