Source-linked AI summary
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Shih-Chieh Dai, Aiping Xiong, Lun-Wei Ku
TL;DR
Thematic analysis is labor-intensive because reliable coding typically involves multiple human coders and iterative consensus-building. The paper proposes an in-context-learning framework pairing a human coder with an LLM to develop and apply a codebook. Across two case studies, the framework achieved comparable coding quality while reducing labor and time demands.
Problem
Thematic analysis commonly requires at least two human coders and repeated discussion, creating substantial labor and time demands.
Method
The paper uses in-context learning to loop a human coder and LLM through code extraction, refinement, and codebook generation for thematic analysis.
Results
The human–LLM framework achieved almost perfect agreement, with κ = 0.87 for Music Shuffle and κ = 0.81 for Password Manager, while producing quality comparable to two human coders.
Takeaways & Limitations
The framework can perform thematic analysis with one human coder instead of two, reducing labor and time demands.
Takeaways & Limitations
The study evaluated only GPT-3.5, so comparable results with other LLMs are not guaranteed, and computational resources and cost may limit adoption.
Abstract
from arXiv · showhide
Thematic analysis (TA) has been widely used for analyzing qualitative data in many disciplines and fields. To ensure reliable analysis, the same piece of data is typically assigned to at least two human coders. Moreover, to produce meaningful and useful analysis, human coders develop and deepen their data interpretation and coding over multiple iterations, making TA labor-intensive and time-consuming. Recently the emerging field of large language models (LLMs) research has shown that LLMs have the potential replicate human-like behavior in various tasks: in particular, LLMs outperform crowd workers on text-annotation tasks, suggesting an opportunity to leverage LLMs on TA. We propose a human-LLM collaboration framework (i.e., LLM-in-the-loop) to conduct TA with in-context learning (ICL). This framework provides the prompt to frame discussions with a LLM (e.g., GPT-3.5) to generate the final codebook for TA. We demonstrate the utility of this framework using survey datasets on the aspects of the music listening experience and the usage of a password manager. Results of the two case studies show that the proposed framework yields similar coding quality to that of human coders but reduces TA's labor and time demands.
1 Introduction
Thematic analysis is widely used but requires multiple human coders and iterative discussion, making it labor-intensive and time-consuming. The paper proposes an LLM-in-the-loop framework to improve TA efficiency through human–LLM collaboration.
- TA identifies themes in qualitative data but typically relies on at least two expert human coders.
- Multiple discussion rounds are needed to resolve ambiguities and reach consensus, increasing TA’s labor and time demands.
- LLMs have outperformed crowd workers on several text-annotation tasks, motivating their use in thematic analysis.
- The framework loops ChatGPT as a machine coder with a human coder to extract, group, and refine codes through in-context learning.
- The framework was evaluated on Music Shuffle and Password Manager survey datasets, with results suggesting quality comparable to two human coders.
- The paper contributes a human–LLM TA framework, a coder-discussion loop, and a partial-data solution for long qualitative inputs.
2 Background and Related Work
Prior work has used topic modeling, clustering, trained models, and LLMs to support qualitative coding. This paper situates its approach within inductive and deductive coding and human–AI collaboration.
- Existing NLP approaches extract themes using topic modeling or cluster phrases according to sentiment or semantic similarity.
- Models can learn human coding behavior, and prior frameworks have combined human–AI collaboration with thematic analysis.
- Recent LLM studies have explored deductive coding, collaborative qualitative-analysis interfaces, and ChatGPT-supported thematic analysis.
- Inductive coding develops a codebook bottom-up, whereas deductive coding applies a codebook top-down across the data.
- The paper completes both inductive and deductive coding through human–AI collaboration using two existing open-access datasets.
3 Human–LLM Collaboration Framework
The framework uses in-context learning to guide a human coder and machine coder through exemplar creation, fine-grained code generation, refinement, and evaluation. Their discussions produce a final codebook for coding the data.
- Figure 1 presents the framework as a four-step process for human–LLM collaboration in thematic analysis.
- Data Familiarization and Exemplar Generation: The human coder familiarizes themselves with the data and creates four to eight exemplars for in-context learning.
- Data Familiarization and Exemplar Generation: Each exemplar pairs an open-ended response with actions that quote relevant text, define a code, and produce the code.
- Initial Code Generation: The machine coder extracts fine-grained codes capturing semantic and latent meanings from free-text responses using the exemplars and task context.
- Code Refinement and Initial Codebook Generation: The model groups similar initial codes, after which the human coder and machine coder iteratively review, revise, and justify changes.
- Theme Identification and Evaluation: The human coder and machine coder identify themes, generate the final codebook, code the data, and evaluate agreement with Cohen’s κ.
4 Evaluation
The evaluation compares human–LLM and human-only coding across music-listening and password-manager survey cases using agreement and similarity measures. The framework generally matches human-coder quality, while error analysis identifies ambiguity, granularity, and distinction as sources of reduced agreement.
- Evaluation metrics: Coding quality was evaluated with Cohen’s κ, cosine similarity, accuracy, and recall against the datasets’ author-provided codes.Cohen’s κ assessed agreement among coders, while the author codes served as gold labels for comparison.
- Results: In the MS case, HC+MC achieved κ = 0.87 versus Gold κ = 0.66, while Coder 3+MC and Coder 3+HC achieved κ = 0.54 and κ = 0.62.The HC+MC result was classified as almost perfect agreement; the Coder 3 comparisons remained moderate to substantial.
- Results: In the PM case, HC+MC achieved κ = 0.81 versus Gold κ = 0.77, and MC-extracted codes captured the semantic meaning of the authors’ codes.The authors conclude that the framework performs as well as two human coders while using one human coder instead of two.
- Error Analysis: The authors characterize thematic analysis as inherently subjective and note that discrepancy discussion may be needed to improve inter-annotator agreement.They also report that iterative codebook generation can make the codebook less biased.
- Results: Using partial data for codebook development remained feasible, with Unseen HC+MC agreement of κ = 0.82 and cosine similarity of 0.8895 to gold codes.The authors present partial-data development as a response to LLM input-size limitations.
- Error Analysis: Lower Coder 3 agreement was associated with ambiguity, granularity, and distinction differences across coders’ coding decisions.These factors reflect synonymous codes within themes, different coding fineness, and divergent interpretations of responses or the codebook.
5 Conclusion
The paper proposes an LLM-in-the-loop framework for thematic analysis and evaluates it in two case studies. Human–machine coding achieves high agreement, while partial-data codebook development addresses LLM input-size limitations.
- The paper proposes a human–LLM collaboration framework, called LLM-in-the-loop, for thematic analysis.
- Across two case studies, human and machine coders achieve almost perfect agreement: κ = 0.87 in Music Shuffle and κ = 0.81 in Password Manager.Dataset-author agreement was κ = 0.66 for Music Shuffle and κ = 0.77 for Password Manager.
- Using partial data for codebook development is a feasible approach to the LLM input-size limitation.
Limitations
The work identifies limitations concerning prompt optimality, data governance, model generalizability, and possible training-data overlap. These constraints bound how broadly the framework’s results can be interpreted and applied.
- Prompts: The prompts achieved the study’s goals, but the authors cannot claim they are optimal or correct.They suggest that better prompts might produce better outcomes.
- Application: Institutional review boards may require survey or interview data, especially sensitive information, to remain on institutional servers.Users must check applicable IRB documents and regulations before applying the method.
- Model: The study evaluated only GPT-3.5, so comparable results with other available LLMs are not guaranteed.Applying the framework to other models may also involve computational-resource and cost limitations.
- Data: GPT-3.5 may have encountered or been trained on data similar to the Music Shuffle dataset before the study.The dataset was published in 2020, while GPT-3.5 training data extended through September 2021.
Ethics Statement
The study used public datasets without privacy-sensitive information with OpenAI GPT-3.5, but users must confirm compatibility with their IRB requirements.
- Ethics Statement: The datasets were public for reuse and contained no privacy-sensitive information, supporting their use with OpenAI GPT-3.5.The authors distinguish this setting from cases governed by other institutional requirements.
- Ethics Statement: Users need to confirm with their IRB before using the proposed method for thematic analysis.This requirement follows from institutional rules governing data use and storage.
A Qualitative Research and Thematic Analysis
Qualitative research uses interviews and open-ended survey questions to analyze participants’ meanings and identify patterns. Thematic analysis is a common method requiring iterative, multi-coder coding for reliable results.
- A Qualitative Research and Thematic Analysis: Qualitative research collects free-text responses through techniques such as interviews and open-ended survey questions.These data support analysis of participants’ meanings and experiences.
- A Qualitative Research and Thematic Analysis: Thematic analysis is a common method for analyzing qualitative data to identify patterns of meaning.The cited framework positions TA as a widely used qualitative-analysis approach.
- A Qualitative Research and Thematic Analysis: Reliable thematic-analysis coding requires at least two human coders to learn the data, create a codebook, and iteratively code until acceptable agreement is reached.The process combines coder familiarity, codebook generation, and repeated coding.
B Model and Parameters
The study used OpenAI’s GPT-3.5 model with mostly default parameters and temperature set to zero for reproducibility. GPT-4 offered greater capacity but faced access and throughput constraints.
- B Model and Parameters: The experiments used OpenAI’s gpt-3.5-turbo-16k LLM.
- B Model and Parameters: GPT-4 had a longer context window and was described as superior to GPT-3.5, but its access and tokens-per-minute constraints were greater.The passage reports 40,000 TPM for GPT-4 versus 180,000 TPM for gpt-3.5-turbo-16k.
- B Model and Parameters: Temperature was set to 0 while other OpenAI parameters remained at their default settings.The authors used zero temperature to reduce output randomness and support reproducibility.
C Datasets
The study uses two survey datasets: Music Shuffle examines music listening experiences, while Password Manager examines password-manager usage and habits.
- Music Shuffle contains four open-ended questions from 397 participants, with all responses combined for thematic analysis.
- Password Manager draws on eight open-ended questions from 277 GWU participants across faculty, staff, and students.
- The selected Password Manager survey questions contain 280 responses (n = 280).
D Prompts
The prompts guide initial code generation, code grouping, and iterative refinement between machine and human coders. They require structured theme outputs, duplicate-code merging, and explicit agreement or disagreement rationales.
- Initial Codes Generation: Generated codes link quoted response text to code definitions and then identify the resulting code.
- Initial Codes Generation: Initial code generation uses survey-question context, step-by-step examples, and four to eight exemplars.Each exemplar pairs an open-ended response with associated coding actions.
- Code Grouping: Code grouping organizes codes into named themes in JSON, with each code assigned to only one theme.
- Code Grouping: The prompts instruct the system to merge duplicate codes and produce themes with code lists and definitions.
- Code Refinement: Code refinement presents machine themes alongside human revisions and requested actions and reasons for discussion.
- Discussion: The discussion prompt asks for revised themes plus JSON-listed agreements, disagreements, and reasons.