What's the Story?

Abstract

The objective of this study is to assess the extent to which Large Language Models (LLMs) can accurately automate the qualitative coding of policy narratives compared to a manually coded dataset. Narratives are integral to human communication, providing the structure through which individuals interpret facts and shape practices that evolve into formal policies. Analyzing policy narratives is crucial for understanding stakeholder interests, managing conflicts, and enhancing policy effectiveness. While traditional methods of narrative analysis have been manual, the exponential growth of available data has made such approaches impractical. This study explores the feasibility of using LLMs to automate the systematic analysis of policy narratives within the framework of the Narrative Policy Framework (NPF). The findings reveal that while LLMs exhibit good repeatability and perform better in scenarios with fewer categories in the codebook, their overall performance remains inadequate compared to manual coding. Metrics such as accuracy, precision, recall, F1 scores, and Krippendorff's alpha indicate significant limitations in their current ability to perform this task reliably. The significance of this research lies in its potential impact on policy analysis, as successful automation could allow for the analysis of much larger corpora of data, leading to richer insights. However, the study also identifies several limitations, including the time constraints, the number of available coders, and the limited range of LLM models tested. Future research directions include the development of a robust codebook through collaboration among multiple experienced coders, the use of specialized LLMs, and improvements in prompt design through the incorporation of more examples, chain-of-thought reasoning, and breaking down the task into smaller, more manageable parts.