Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?

1. Introduction

• The study evaluates whether GPT-4 can analyze court-opinion sentences to support interpretation of statutory legal concepts, a task requiring specialized expertise.It compares GPT-4 with human annotators and explores batch prediction and prompt-engineering approaches.Source: §1 · p. 1
• The paper situates this work amid annotation approaches that use language-model prompts to reduce demand for costly human expertise.Human annotation has long supported empirical text analysis, including legal workflows and research.Source: §1 · p. 1
• The study examines batch prediction, explanation prompting, guideline revisions, and robustness to prompt changes unrelated to task definition.These questions cover cost-effective processing, prediction accuracy, guideline shortcomings, and prediction stability.Source: §1 · p. 2
• The authors describe contributions including benchmarking GPT-4 against human annotators using nearly unchanged guidelines and comparing batch with single-item prediction.They also report prompt-engineering efforts and analyze prediction robustness.Source: §1 · pp. 1–2

2. Related Work

• Earlier computational text analysis relied on human-annotated resources, while prompted language models have been explored as a way to reduce demand for annotators.Prior studies also report ChatGPT performance in annotation tasks and raise reliability concerns.Source: §2 · p. 2
• Related research applies GPT models to specialized tasks including legal exams, statutory interpretation, legal reasoning, and legal-document annotation.Other cited examples address implicit hate speech, genre identification, and social-computing labels.Source: §2 · p. 2
• AI & Law research has pursued annotation through Boolean rules, semantic similarity, human-confirmed machine predictions, active learning, and human-aided cognition.This study evaluates GPT-4's zero-shot capabilities for text analysis.Source: §2 · p. 2

3. Data

• The mock-up shows a user selecting a phrase in a statutory provision, receiving useful sentences, and opening an opinion to see a sentence in context.The interface example uses the phrase “common business purpose.”Source: Figure 1 · p. 3
• The task evaluates whether court-decision sentences responsive to a phrase query provide explanatory value for that phrase in a source provision.The analysis of prior treatment of a phrase can support arguments about its meaning when applying statutory rules.Source: §3 · pp. 2–3
• Labels distinguish sentences that explicitly explain a phrase, support modest conclusions, add information without explaining it, or add no useful information.The categories are High value, Certain value, Potential value, and No value.Source: §3 · p. 3
• The original annotations involved 11 law students and two legal scholars, with student-to-consensus agreement of 0.1 < α < 0.6 and scholar agreement of α = 0.79.The dataset uses consensus labels after scholars' second-pass review.Source: §3 · p. 3
• The original dataset contains 26,959 labeled sentences for 42 phrases across 20 legal-regulation areas, while this study selects 256 sentences for five phrases.The authors note the subset is limited but consider it sufficient for their experiments.Source: §3 · p. 4

4. Model

• The study uses GPT-4, described as a dialog-focused model, and outlines the GPT model lineage and zero- and few-shot learning context.The passage notes that GPT-4 technical details were not disclosed.Source: §4 · p. 4
• The model temperature was set to 0.0, with max_tokens varying by output type and top_p set to 1.Single-label classification used a 50-token limit, while batch classification with chain-of-thought prompting used 1,500 tokens; the total prompt and completion limit was 8,192 tokens.Source: §4 · p. 4

5.1. GPT-4 Text Analysis (RQ1)

• For single-sentence prediction, the researchers adapted human annotation guidelines into a system prompt with minimal changes and submitted each data point individually with its source context.They omitted instructions specific to the human annotation environment and translated a visual workflow into questions.Source: §5.1 · pp. 4–5

5.2. Batch Prediction (RQ2)

• Batch prediction reused the system prompt, submitted multiple sentences per message, and dynamically sized batches to fit the prompt token limit.Batches typically contained several tens of sentences, and this approach was significantly cheaper than submitting data points individually.Source: §5.2 · p. 5

5.3. Explanations – CoT (RQ3)

• The explanation experiments requested either a natural-language explanation before each prediction or schematic answers to guideline questions before batch predictions.The batch variant used schematic explanations because natural-language explanations would have drastically reduced batch size; both explanation approaches were costlier than their corresponding prediction-only settings.Source: §5.3 · pp. 5–6

5.4. Prompt (Annotation Guidelines) Modification (RQ4)

• To test guideline refinement, the researchers modified the system prompt to address a weakness and repeated the preceding experiments; few-shot learning and fine-tuning were left for future work.The paper treats prompt refinement as analogous to refining annotation guidelines in a human workflow.Source: §5.4 · p. 6

5.5. Robustness (RQ5)

• The robustness experiment assessed whether prompting form changed predictions across single and batch settings, with and without explanations, while holding annotation instructions constant within each guideline version.The comparison covered labels produced under both the original and updated guidelines.Source: §5.5 · pp. 6–7, Table 2 · p. 7

6.1. GPT-4 Text Analysis (RQ1)

• GPT-4’s F1 was .53, and its agreement with gold labels was on par with well-performing law students.Source: §6.1 · p. 7, Figure 8 · p. 8

6.2. Batch Prediction (RQ2)

• .52 F1 for batch prediction was a slight decrease from single-item prediction, while the authors note its significantly lower cost may justify the difference.Source: §6.2 · p. 7, Figure 7 · p. 8

6.3. Explanations – CoT (RQ3)

• F1 fell from 0.53 to 0.51 and accuracy from 0.46 to 0.40 when single-item predictions included explanations.Source: §6.3 · p. 7
• Batch predictions with explanations completely degraded, with many sentences mislabeled Certain; schematic explanations generally agreed with predictions whether correct or not.Source: §6.3 · pp. 8–9, Figure 7 · p. 8

6.4. Prompt (Annotation Guidelines) Modification (RQ4)

• .57 F1 under updated single-item instructions improved on .53 with original instructions, addressing overprediction of Certain but making the model reluctant to assign that label.Source: Figure 7 · p. 8, §6.4 · p. 9
• With updated guidelines, explanations no longer caused dramatic performance deterioration, and the batch explanation decrease was smaller than under original guidelines.Source: §6.4 · p. 9

6.5. Robustness (RQ5)

• Agreement among predictions under identical guidelines was relatively low, and the authors suggest small output-format changes can dramatically affect predictions.Source: §6.5 · p. 9, Table 3 · p. 10

• Because the study tests one specialized task on a relatively small dataset, its findings should not be generalized to all domain-expertise tasks.Source: §7 · p. 10
• Limited access to proprietary GPT models and model updates can make independent replication difficult and undermine consistent baselines across studies.Source: §7 · p. 10

8. Conclusions and Future Work

• GPT-4 performed at a level comparable to well-trained law student annotators in interpreting legal concepts in court opinions.The authors also describe its ability to follow instructions across multi-page documents and apply them to complex real-world text.Source: §8 · p. 10
• Batch predictions offered significant cost reductions without a major decline in performance, while chain-of-thought prompting yielded no noticeable improvement.Source: §8 · p. 10
• Analyzing GPT-4’s predictions helped identify and address annotation-guideline deficiencies, leading to improved model performance.Source: §8 · p. 10
• Minor prompt-formatting changes had a substantial impact on GPT-4’s predictions, highlighting brittleness.Source: §8 · p. 10
• Future work should evaluate GPT-4 on broader specialized tasks and domains with larger datasets, and investigate ways to improve robustness.The authors also propose exploring alternative prompting techniques or fine-tuning strategies.Source: §8 · p. 10