Source-linked AI summary
Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies
Jakub Drápal, Hannes Westermann, Jaromir Savelka
TL;DR
Inductive coding is used in empirical legal studies, but few studies have examined it, and identifying typical-crime categories from court descriptions is time-consuming. The paper applies an expert-supported GPT-4 framework to theft opinions; its findings indicate reasonable, improvable initial codes, promising classification, and viable theme discovery.
Problem
Few studies have focused on inductive coding in empirical legal studies, while identifying shared features of criminal acts from statutes or case facts is difficult because the former are too general and the latter too specific.
Source: §1 · p. 2Method
Results
Initial codes judged reasonable rose from 72.6% to 88.8% after expert feedback (+16.2% improvement); zero-shot theme classification was promising, and end-to-end theme discovery appeared viable.
Source: §6 · pp. 7–8, §7 · p. 9Takeaways & Limitations
The findings suggest the framework can support initial coding, theme prediction, and end-to-end theme discovery in the studied theft analysis.
Source: §6 · pp. 7–8, §7 · p. 9The evaluation concerns Czech theft opinions; the authors call for validation beyond court opinions and criminal law, and note the proprietary LLM's black-box nature.
Source: §6 · p. 9, §7 · p. 10
Abstract
from arXiv · showhide
Thematic analysis and other variants of inductive coding are widely used qualitative analytic methods within empirical legal studies (ELS). We propose a novel framework facilitating effective collaboration of a legal expert with a large language model (LLM) for generating initial codes (phase 2 of thematic analysis), searching for themes (phase 3), and classifying the data in terms of the themes (to kick-start phase 4). We employed the framework for an analysis of a dataset (n=785) of facts descriptions from criminal court opinions regarding thefts. The goal of the analysis was to discover classes of typical thefts. Our results show that the LLM, namely OpenAI's GPT-4, generated reasonable initial codes, and it was capable of improving the quality of the codes based on expert feedback. They also suggest that the model performed well in zero-shot classification of facts descriptions in terms of the themes. Finally, the themes autonomously discovered by the LLM appear to map fairly well to the themes arrived at by legal experts. These findings can be leveraged by legal researchers to guide their decisions in integrating LLMs into their thematic analyses, as well as other inductive coding projects.
1. Introduction
Thematic analysis is widely used in empirical legal studies, but inductive coding has received little research attention. The paper proposes LLM support for phases 2 and 3 and studies its use in identifying typical thefts from Czech court opinions.
1. Introduction
- Thematic analysis is an inductive coding method that discovers codes from data while applying them, and it is widely used in empirical legal studies.The paper notes that few studies have focused on inductive coding in ELS.Source: §1 · p. 1
- The framework supports a subject matter expert using an LLM in phases 2 and 3 of thematic analysis.Figure 1 places those phases within a six-phase process running from raw data to scholarly report.Source: §1 · p. 2, Figure 1 · p. 2
- The study applies the framework to Czech criminal court opinions about theft, seeking behavior-based categories of typical crimes.Statutory offense definitions are too general and case facts too specific to reveal shared features of criminal acts directly; identifying those features is time-consuming.Source: §1 · p. 2
- The study asks whether GPT-4 can generate initial codes, improve them through expert feedback, predict themes, and autonomously discover and associate themes with data.These questions address initial coding, expert revision, theme prediction, and theme discovery.Source: §1 · p. 2
2. Related Work
Prior work examines LLMs in thematic analysis, pattern discovery in criminal justice data, and inductive analysis of legal texts. This paper distinguishes its approach through a thematic-analysis framework and GPT-4 collaboration with a subject matter expert.
2. Related Work
- Earlier LLM studies used GPT-3.5 for full thematic analysis, codebook suggestions, and code generation, with reported concerns about methodological implications and code coherence.One study found generated codes granular but inconsistent across similar clusters.Source: §2 · pp. 2–3
- Criminal justice pattern studies have used content analysis alongside factor, latent-profile, or cluster analyses to identify offense types and patterns.Examples include burglary types, sexual-murder patterns and motives, stranger-rape themes, and frequent cybercrime types.Source: §2 · p. 3
- Legal-text pattern research includes grounded-theory analysis and topic modeling; the cited topic-modeling study found 33% of discovered topics relevant.The authors identify that study as the state-of-the-art attempt on inductive coding of legal texts.Source: §2 · p. 3
- The proposed work differs by using an established thematic-analysis framework and GPT-4 to let experts influence inductive analysis through research questions and instructions.The paper also predicts themes for case facts as deductive coding within an otherwise predominantly inductive analysis.Source: §2 · p. 3
3. Dataset
The dataset contains 785 anonymized, edited facts descriptions from Czech theft cases decided in 2017. Law students manually developed and applied 14 themes focused on modus operandi and theft targets.
3. Dataset
- The dataset contains 785 facts descriptions from Czech court cases decided in 2017, selected from 834 guilty-theft cases after 49 exclusions.The sample slightly over-represented the most serious offenses; theft includes burglary and pickpocketing in Czechia.Source: §3 · p. 3
- Extracted facts descriptions were anonymized and, when necessary, shortened or partially rewritten, with lengths from 73 to 29,695 characters.The median length was 782 characters.Source: §3 · p. 4
- Three law students under an author's supervision developed 14 high-level themes focused on modus operandi and theft targets, then assigned one theme per description.Two students independently selected each theme under specified rules, with disagreements resolved after rereading; shop theft was 29.0% and breaking into another object 17.5%.Source: §3 · p. 4
4. Proposed Framework
The framework uses GPT-4 and general and analysis-specific resources to generate initial codes, iteratively collate them into themes, and classify data points using the resulting theme list. Expert feedback can revise codes, while the final classifications provide a starting point for later theme review.
4. Proposed Framework
- The framework uses GPT-4 in zero-shot settings, with temperature 0.0 and an 8,192-token overall prompt-and-completion limit.Other settings include top_p 1 and zero frequency and presence penalties.Source: §4 · p. 4
- General resources define thematic analysis and its phases, while context-specific resources specify research questions, analysis parameters, what counts as a theme, and custom requirements.The authors did not rigorously adhere to the process described in.Source: §4 · pp. 4–5
- The system batches data to fit the prompt, processes shorter descriptions first, and truncates an overlong description by retaining its beginning and ending.Each batch includes analysis instructions and a random sample of codes from preceding batches; an expert may add instructions and repeat code generation until the codes meet expectations.Source: §4 · pp. 5–6
- Initial codes are collated iteratively into candidate themes and then compacted into high-level themes, with candidate themes retained as sub-themes.The iterative process includes common themes from preceding batches and then submits the full candidate set for high-level collation.Source: §4 · p. 6
- The final stage uses deductive coding to label each data point from the discovered theme list, producing an artifact for experts to review in the next thematic-analysis phase.Figure 3 depicts batching, iterative code and theme generation, possible expert feedback, and predictions for each input data point.Source: Figure 3 · p. 5, §4 · p. 6
5. Experimental Design
The experiments assess GPT-4’s initial coding, the effect of expert feedback, zero-shot theme prediction, and end-to-end theme discovery. The design uses expert judgments and compares predicted themes with manually identified themes.
5. Experimental Design
- Initial codes were manually assessed for whether they addressed how the theft happened and what was stolen, stopping after a how-related issue was confirmed.The evaluation scheme checks “how” before “what.”Source: §5 · p. 6
- Expert feedback directed coding toward target, modus operandi, seriousness, and intent, and away from multiplicity, completion, coresponsibility, and stolen-goods value.The prompt also included three examples of desirable codes.Source: §5 · pp. 6–7
- For zero-shot theme prediction, the pipeline’s system was instructed to assign each data point three human-expert-derived themes, evaluated by recall at 1 (R@1) and recall at 3 (R@3).Descriptions could contain multiple themes, but experts were instructed to assign the most specific and salient one.Source: §5 · p. 7
- The end-to-end evaluation generated initial codes with expert feedback, collated and grouped potential themes, then assigned each description a high-level theme for comparison with expert results.Source: §5 · p. 7
6. Results and Discussion
GPT-4 generated mostly reasonable initial codes, with a higher share judged reasonable after expert feedback, and showed promising but uneven performance in predicting expert themes. Its end-to-end themes mapped to expert themes, though the process could benefit from expert intervention and has an evaluation caveat.
6. Results and Discussion
- 72.6% of initial codes were deemed reasonable, rising to 88.8% after expert feedback, a reported 16.2% improvement.Reasonable codes described both how the theft happened and what was stolen.Source: §6 · p. 7
- The code evaluator knew which round each code came from, a possible limitation of the initial-code experiment.Source: §6 · p. 7
- R@1 was .66 and R@3 was .82 for zero-shot prediction of the 14 manually discovered themes, with performance varying across themes.Shop and energy theft predictions worked remarkably well, while open-access-place theft and cellar robbery predictions were rather low.Source: §6 · pp. 7–8, Table 3 · p. 8
- Some potential themes were near-duplicates or overly specific, and multiplicity or completion stage appeared despite instructions not to focus on them.Source: §6 · p. 8
- The LLM discovered 8 themes versus the experts’ 14, with some model themes mapping to expert themes and all manual themes mappable to potential themes within higher-level groups.The LLM also identified theft from a gym, a behavior the experts had missed, perhaps in error.Source: §6 · p. 8, Figure 5 · p. 9
- Expert intervention at multiple stages may improve theme quality and alignment with research questions; the proprietary model’s black-box nature is also problematic for researcher agency.Source: §6 · p. 9
7. Conclusions and Future Work
The framework produced reasonable initial codes that improved with expert feedback, and its end-to-end performance suggested viability while highlighting the importance of expert supervision. Future work should extend support beyond phases 2 and 3 and validate findings outside court opinions and criminal law.
7. Conclusions and Future Work
- The end-to-end pipeline appeared viable for discovering and predicting themes, while highlighting the importance of subject-matter expert supervision.Source: §7 · p. 9
- Initial coding was reasonable and improved with expert feedback, while zero-shot theme classification was promising but could benefit from expert feedback.Source: §7 · p. 9
- Future work should extend support beyond phases 2 and 3 and validate findings beyond court opinions and criminal law.Source: §7 · pp. 9–10