Source-linked AI summary
Getting in the Door: Streamlining Intake in Civil Legal Services with Large Language Models
Quinten Steenhuis, Hannes Westermann
TL;DR
Civil legal aid intake is time-consuming, and substantive eligibility criteria can be nuanced and frequently changing. The paper presents a rules-and-LLM intake platform and evaluates its recommendations, reporting promising results and identifying several limitations to applicability and safety.
Problem
Substantive legal-aid eligibility criteria are nuanced, open-textured, and frequently changing, complicating intake that already involves long waits and limited program capacity.
Source: §1 · p. 1Method
The paper pilots a housing-intake platform that combines hand-coded rules with an LLM to inform applicants about their likelihood of qualifying before a telephone intake queue.
Source: §1 · p. 2Results
The authors report that LLM errors were minor and similar to those of human intake workers, with some errors helping identify annotation mistakes; qualitative review found thoughtful questions and sound recommendations.
Source: §5.2 · p. 7, §7 · p. 10Takeaways & Limitations
The authors conclude that LLMs can help improve eligibility information and reduce barriers to finding legal assistance.
Source: §7 · p. 10Content censorship may limit applicability to other legal topics, including those involving violence, abuse, or unethical behavior.
Source: §6.4 · p. 10
Abstract
from arXiv · showhide
Legal intake, the process of finding out if an applicant is eligible for help from a free legal aid program, takes significant time and resources. In part this is because eligibility criteria are nuanced, open-textured, and require frequent revision as grants start and end. In this paper, we investigate the use of large language models (LLMs) to reduce this burden. We describe a digital intake platform that combines logical rules with LLMs to offer eligibility recommendations, and we evaluate the ability of 8 different LLMs to perform this task. We find promising results for this approach to help close the access to justice gap, with the best model reaching an F1 score of .82, while minimizing false negatives.
1. Introduction
Civil legal aid intake can involve long waits, while substantive eligibility criteria shift, vary by program, and require nuanced judgments. The study examines whether LLM-assisted intake can address this challenge and asks about rule application, errors, and follow-up questions.
1. Introduction
- Long telephone intake queues add a barrier to reaching legal aid, while programs can assist only a limited number of eligible applicants.Unresolved civil legal problems can harm individuals and burden society.Source: §1 · p. 1
- Substantive eligibility rules change frequently, vary across organizations, and rely on nuanced, open-textured assessments, making them difficult to automate with traditional methods.Online tools can provide round-the-clock intake without staff time, but traditional logical rules suit formal requirements better than these changing substantive criteria.Source: §1 · p. 2
- The study pilots a Missouri housing-intake platform with four legal aid programs, combining hand-coded rules and an LLM to inform applicants about their likelihood of qualifying before they wait in a telephone queue.The authors argue that the diversity of tested rules makes the results broadly relevant to legal intake.Source: §1 · p. 2
1.1. Research Questions
- The research questions ask how accurately LLMs apply rules to lay-described legal scenarios, what errors they make relative to staff, and whether they can elicit missing information through high-quality follow-up.Source: §1.1 · p. 2
2. Prior Work
Prior work spans AI applications in access to justice, legal reasoning, and fact elicitation. This study positions its contribution as testing whether LLMs can apply textual intake rules to eligibility scenarios and elicit relevant facts.
2.1. AI for access to justice
- Prior access-to-justice work includes eligibility systems based on decision trees and formal methods; this study tests whether LLMs can read verbatim intake guidelines, which the authors describe as easier and faster to deploy.The field also includes legal information, form-filling, and dispute-resolution applications [8, 31,49] [43,48].Source: §2.1 · p. 2
2.2. LLMs for Legal Reasoning
- LLM legal-reasoning research covers tasks from bar exams to statutory reasoning, while this paper examines applying complex textual intake rules to factual scenarios.The LegalBench study measures 20 models on 161 legal tasks.Source: §2.2 · p. 3
2.3. Spotting legal issues and eliciting facts
- Earlier work used machine learning to spot legal issues and direct users to decision-support tools; this approach instead uses zero-shot LLMs to assess eligibility from textual rules.Source: §2.3 · p. 3
- Prior fact-elicitation systems use case-derived schemas, step-by-step LLM and rule assessment, or follow-up questions; this paper evaluates eliciting facts relevant to intake eligibility with intake workers.Examples include,, and [19].Source: §2.3 · p. 3
3. Proposed framework
The proposed housing-intake platform combines formal eligibility rules with LLM assistance in a mobile-friendly application. It stores program-specific criteria and uses follow-up questions before giving applicants a qualified recommendation.
3. Proposed framework
- The user-facing application is built with Docassemble, works on a tenant's smartphone, and presents a typical interaction across five screens in about five minutes.Figure 1 depicts a progression from problem description to follow-up question to a qualified recommendation, with an option to learn how AI is used.Source: §3 · p. 3, Figure 1 · p. 4
3.1. Formal eligibility criteria
- The platform encodes location and statutory eligibility requirements as formal Python rules, while storing unmodified program-specific substantive rules as plain text.Those rules vary in length, formatting, and content; examples include security-deposit disputes, discrimination, and housing subsidies.Source: §3.1 · p. 3, §3.2 · p. 3
3.2. Program-specific substantive criteria
- The study focuses on three of the four programs supported by the platform, and one program's rules include 12 separate grounds for accepting a housing case.Assessing each ground can require multiple facts.Source: §3.2 · pp. 3–4
3.3. Screening with LLM assistance
- Applicants describe their legal problem, after which GPT-4-turbo receives the rules and description and returns qualify, do not qualify, or a follow-up question; the process allows up to 10 follow-ups.The final determination includes a disclaimer that the AI can be wrong and directs applicants to the program for full intake.Source: §3.3 · p. 4
4.1. Dataset
The evaluation uses two datasets to examine initial rule application and follow-up interactions. D1 pairs scenarios with manually coded program outcomes, while D2 captures simulated exchanges in which tenants reveal information gradually.
4.1. Dataset
- The datasets were developed to test LLM-based intake, including initial responses to scenarios and later responses during follow-up interactions.Table 2 presents an example scenario, an excerpt of an intake rule, and the resulting decision.Source: §4.1 · p. 4, §4.1.1 · p. 4, Table 2 · p. 5
4.1.1. D1 - Initial Response Dataset
- D1 contains 48 scenario-jurisdiction pairs, with expected outcomes manually coded for three legal programs; Table 1 summarizes the target-outcome counts by jurisdiction and overall.Scenarios were generated with ChatGPT, then reviewed and reworded to represent common and edge cases in legal aid.Source: §4.1.1 · p. 5, Table 1 · p. 5
4.1.2. D2 - Follow-up Response Dataset
- D2 adds 11 diverse scenarios with simulated tenants sharing small pieces of information at a time, to assess follow-up questions and interaction quality.Source: §4.1.2 · p. 5
4.2. Models
The evaluation covers eight well-known LLMs: two open-source models and six commercial models from Anthropic, Google, and OpenAI, selected to span popularity, price points, and openness.
4.2. Models
- The study tested eight models: two Llama 3.1 Instruct models and six commercial models from Anthropic, Google, and OpenAI.The commercial set includes Claude 3.5 Sonnet, Gemini 1.5 Pro, and four OpenAI models.Source: §4.2 · p. 5
4.3. Prompt design
The prompt was iteratively refined to prioritize avoiding false negatives while keeping the LLM focused on minimum intake eligibility rather than legal advice.
4.3. Prompt design
- The prompt’s main goal was avoiding false negatives, and its iteration excluded scenarios from dataset D1 to prevent overfitting.Tests were run both in ChatGPT and embedded in Docassemble; examples were omitted after they led to hallucinations.Source: §4.3 · p. 6
- After the model gave inappropriate legal advice, the prompt clarified that it should assess minimum intake criteria, not advise tenants.The authors also found that examples prompted hallucinations, so the final prompt omits example replies.Source: §4.3 · p. 6
4.4. Experiments
The experiments evaluated initial-response predictions, analyzed GPT-4-Turbo’s disagreements, and had an expert assess full intake conversations.
4.4.1. E1 - Prediction of the correct initial response
- For initial-response prediction, the pipeline paired each of 48 scenario-jurisdiction cases with each of eight models, producing 384 predictions and explanations.Table 3 reports precision, recall, and F1 for Accept, Deny, Question, and overall weighted average, by model.Source: §4.4.1 · p. 6, Table 3 · p. 7
4.4.2. E2 - Analysis of LLM errors
- For error analysis, the researchers manually examined cases where GPT-4-Turbo disagreed with the manually assigned response, reviewing both error type and explanation.Source: §4.4.2 · p. 6
4.4.3. E3 - Qualitative Analysis of full transcripts
- For transcript quality, one Legal Services of Eastern Missouri expert rated accuracy and overall quality in 11 manually created Docassemble conversations using GPT-4-Turbo.Table 4 reports GPT-4-Turbo precision, recall, and F1 by jurisdiction.Source: §4.4.3 · p. 6, Table 4 · p. 7
5. Results
Across tested models, weighted F1 ranged from 0.56 to 0.82, with GPT-4-Turbo best; analysis also found that errors and transcript quality varied across cases.
5.1. E1 - Prediction of correct initial response
- 0.56–0.82 weighted F1 scores spanned the models, with GPT-4-Turbo performing best on scenario-jurisdiction pairs.Deny-class precision was very high.Source: §5.1 · p. 7
5.2. E2 - Error Analysis
- 76 all-model and 4 GPT-4-Turbo errors involved predicting Question for cases classified as Deny; misclassifying Accept or Question as Deny was very rare.Further review found two annotation mistakes, one ambiguous rule, and six actual LLM errors.Source: §5.2 · p. 7, Figure 2 · p. 8
5.3. E3 - Qualitative analysis
- In 73% of transcripts, the expert rated the overall result correct, while follow-up questions that could improve intake were missed in 63%.The rater gave perfect understandability and satisfaction scores, but identified domain-knowledge gaps such as assessing flimsy doors or locks.Source: §5.3 · p. 8
6. Discussion
The discussion reports strong zero-shot rule application alongside uneven performance across models and jurisdictions, then examines error patterns, follow-up interactions, and study limitations.
6.1. RQ1 - How accurately can large language models apply intake rules to legal scenarios described by laypersons?
- GPT-4-Turbo led the models by an F1-score of .13 over the next best option, while performance varied across jurisdictions from Mid-Missouri (0.69) to Eastern Missouri (0.94).The authors suggest rule format may contribute to this difference; Eastern Missouri's rules are longest and most detailed, while Mid-Missouri's many covered cases may confuse the model.Source: §6.1 · pp. 8–9
6.2. RQ2 - What types of errors do LLMs make, and how do they compare to human intake staff?
- Unnecessary follow-up questions were the most common error (83 cases), compared with 3 incorrect denials and 49 incorrect acceptances; GPT-4-Turbo also exposed two human reasoning errors.The authors report that the model's understanding mistakes were relatively rare and that it appeared to err on the side of caution.Source: §6.2 · p. 9
6.3. RQ3 - Can LLMs effectively elicit missing information through follow-up questions?
- The expert review found missed opportunities to ask additional questions, yet gave the system perfect satisfaction marks; the authors note that the reviewer's goal differed from the tool's minimum-eligibility screening goal.The system was intended to route potentially eligible applicants to a human for full intake efficiently.Source: §6.3 · pp. 9–10
6.4. Limitations
- The authors identify content censorship as a possible barrier to applying the system to other legal topics, including those involving violence or abuse.They also note that prompt refinement on GPT-4-Turbo may have limited other models' performance, and that biased training data may pose risks despite human oversight and other mitigations.Source: §6.4 · p. 10
7. Conclusion and future work
The authors report that LLMs can reduce time spent by applicants and staff, with qualitative review finding thoughtful questions and sound recommendations. They identify improvements and broader eligibility matching as future directions, and conclude that better eligibility information and fewer barriers can help people find legal assistance.
7. Conclusion and future work
- LLMs reduced applicant and staff time; qualitative review found thoughtful questions, sound recommendations, and clarification or acceptance under uncertainty, with minor errors.Observed errors were similar to those a human intake worker might make and sometimes helped catch the authors’ own mistakes.Source: §7 · p. 10
- Future work includes seamless online intake, better analytics, document-based rule updates, prompt and rule refinement, staff-performance and bias evaluation, and multi-provider best-match recommendations.The authors also see potential in semi-structured reasoning to improve performance.Source: §7 · p. 10
- Improved eligibility information and fewer barriers can help more people find legal assistance and support a more accessible, efficient legal system.Source: §7 · p. 10