Source-linked AI summary
Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding
Ziang Xiao, Xingdi Yuan, Q. Vera Liao, Rania Abdelghani, Pierre-Yves Oudeyer
TL;DR
Qualitative coding is labor-intensive and difficult to scale, while existing AI approaches can be hard to steer or resource-intensive. The paper combines GPT-3 with expert-developed codebooks for deductive coding and evaluates the approach on children’s curiosity-driven questions. It achieves fair to substantial agreement with experts, with performance varying by prompt design and example count.
Problem
Qualitative coding is labor-intensive for complex and large datasets, while existing unsupervised and supervised tools have steering, data, or computing limitations.
Method
The study combines GPT-3 with expert-developed codebooks to perform deductive coding of children’s curiosity-driven questions across question complexity and syntactic structure.
Results
Cohen’s κ=0.61 for Question complexity and Cohen’s κ=0.38 for Syntactic Structure indicate substantial to fair agreement with experts.
Takeaways & Limitations
The preliminary findings indicate that combining GPT-3 with a codebook can support deductive coding and may help analyze increasingly large qualitative datasets.
Takeaways & Limitations
The study measures agreement but does not yet provide detailed error analyses or establish extension to different contexts and more nuanced coding schemes.
Abstract
from arXiv · showhide
Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While recent AI-based tools demonstrate utility, researchers may not have readily available AI resources and expertise, let alone be challenged by the limited generalizability of those task-specific models. In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined codebooks to label the data into a fixed set of codes. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results. We lay out challenges and opportunities in using LLMs to support qualitative coding and beyond.
1 INTRODUCTION
The paper addresses labor-intensive qualitative coding by exploring GPT-3 with expert-developed codebooks for deductive coding. It reports fair to substantial agreement with experts and examines how prompt designs affect results.
- Qualitative coding identifies patterns and categories but requires substantial researcher effort, especially for complex, nuanced, and large datasets.
- Existing unsupervised models are difficult to steer, while supervised models often require large high-quality datasets or substantial computing resources.
- LLMs can perform prompted classification with specified labels, instructions, and examples without relying on task-specific model training.
- The study asks how closely an LLM-based deductive coding approach agrees with experts and how prompt designs affect coding results.
- Cohen’s κ=0.61 for Question complexity and Cohen’s κ=0.38 for Syntactic Structure indicate substantial and fair agreement, respectively.
2 DEDUCTIVE CODING TASK
The study uses deductive coding, in which data are labeled according to a pre-developed codebook, and evaluates the approach on children’s curiosity-driven questions.
- Deductive coding labels data using a codebook containing codes, descriptions, and examples developed from the research focus or theory.
- The approach targets the coding stage of qualitative analysis and may also help researchers rapidly iterate on codebooks.
- The case study analyzes children’s curiosity-driven questions across question complexity and syntactic structure.
- Question complexity distinguishes questions requiring simple facts from questions requiring explanations of mechanisms or relationships.
- The dataset contains 668 French children’s questions coded by psychologists across semantic and syntactic dimensions.
3 GPT-3 SETUP AND PROMPT DESIGN
The prompt study varies prompt structure and example count when applying GPT-3 to codebook-based qualitative coding. Codebook-centered prompts with examples achieve the highest reported agreement.
- The experiments used GPT-3 davinci-text-002 with temperature 0.0 to support reproducible prompting.
- Prompt designs varied along codebook-centered versus example-centered structure and zero-shot, one-shot, versus few-shot settings.
- Codebook-centered prompts follow Code/Description/Examples, whereas example-centered prompts explain each example’s rationale.
- All prompt variants included an identity modifier and instructions constraining outputs to the specified code set.
- Codebook-centered prompts with examples achieved the highest agreement between GPT-3 and experts.
4 RESULTS
GPT-3 with an expert-developed codebook achieved fair to substantial agreement with expert coding, though expert–expert agreement was higher. Codebook-centered prompts and examples generally improved performance.
- Cohen’s Kappa measured agreement between GPT-3 and expert coding and between the two experts who originally coded the dataset.
- Cohen’s κ=0.61 for Question complexity and Cohen’s κ=0.38 for Syntactic Structure represented substantial and fair agreement with experts.
- Experts agreed more strongly with each other: Cohen’s κ=0.88 for Question complexity and Cohen’s κ=0.90 for Syntactic Structure.
- Codebook-centered prompts performed better than example-centered prompts, and the largest performance gain occurred when moving from zero-shot to one-shot prompting.
- Performance differed little between one-shot and few-shot settings.
5 OPPORTUNITIES AND CHALLENGES
The authors identify opportunities to extend LLM-supported qualitative analysis while emphasizing challenges around capability, reliance, codebook design, and inductive coding.
- The approach may help researchers analyze increasingly large qualitative datasets because LLMs offer accessibility and flexibility.
- Interrogate Model Capability: The preliminary evaluation measured agreement but requires detailed error analysis to clarify model capability.
- Interrogate Model Capability: Extension to different contexts and coding schemes remains unclear when coding requires more nuanced signals.
- Design for Appropriate Reliance: Because the model produces incorrect labels, deployment should support appropriate reliance and prevent over-trusting and misuse.The authors suggest explainable AI methods to calibrate user trust over time.
- Design Codebook for LLMs: Expert-oriented codebooks provide transparency and explicit control but may limit performance, motivating codebooks designed for LLM task performance and understanding.
- Support Inductive Coding: Supporting inductive coding will require new interaction techniques and human-AI collaboration controls because the demonstrated approach addressed deductive coding.