Source-linked AI summary
Weaving Pathways for Justice with GPT: LLM-driven automated drafting of interactive legal applications
Quinten Steenhuis, David Colarusso, Bryce Willey
TL;DR
Authoring guided interviews for legal forms takes substantial human effort, and the paper explores whether generative AI can reduce it. The authors examine AI-assisted drafting with human review and conclude that this approach may save time without the risks of fully automated form-filling.
Problem
Method
The paper explores GPT-assisted drafting of guided interviews, with human editing and review, alongside the template-driven Assembly Line Weaver.
Source: §3 · p. 3, §6.1 · p. 9, §8 · p. 12Results
The hybrid approach can provide time-saving benefits without the risks of fully automated question-and-answer form-filling.
Source: §8 · p. 12Takeaways & Limitations
The hybrid approach has the potential to expand the number and kinds of forms that can benefit from document automation.
Source: §8 · p. 12On the worst-performing form, 28% of fields remained unidentified; complete documents were most likely when the PDF had few or no checkboxes.
Source: §6.4 · p. 11
Abstract
from arXiv · showhide
Can generative AI help us speed up the authoring of tools to help self-represented litigants? In this paper, we describe 3 approaches to automating the completion of court forms: a generative AI approach that uses GPT-3 to iteratively prompt the user to answer questions, a constrained template-driven approach that uses GPT-4-turbo to generate a draft of questions that are subject to human review, and a hybrid method. We use the open source Docassemble platform in all 3 experiments, together with a tool created at Suffolk University Law School called the Assembly Line Weaver. We conclude that the hybrid model of constrained automated drafting with human review is best suited to the task of authoring guided interviews.
1. Introduction
Authoring guided interviews for legal forms is slow, careful work, and the paper asks whether generative AI can reduce that labor. The authors propose AI-assisted first drafts with human editing, while treating existing forms as a practical starting point rather than necessarily the best way to serve litigants.
1. Introduction
- Creating a guided interview requires adding template markup, contextual prompts, question flow and logic, instructions, and help.These tasks make traditional authoring slow and careful work.Source: §1 · pp. 1–2
- The authors propose GPT-assisted first drafts followed by human editing, rejecting fully automated form-filling except for very simple forms.They suggest that success could lower automation costs enough to make automating hundreds of forms realistic.Source: §1 · p. 2
- The authors characterize their experiments as tentative and early, while acknowledging the promise of the approach.Source: §1 · p. 2
1.1. Is the form the thing?
- Forms encode instructions, fields, and checkboxes that interview authors can interpret, making them a practical starting point even if narratives may ultimately serve litigants better.The paper notes that effective legal claims may be structured better through narratives, while decision makers need facts.Source: §1.1 · p. 2
2. Traditional interactive legal application authoring approaches
Traditional interactive legal application authoring combines template preparation, question design and organization, and conditional logic. The work can take hundreds of hours and requires usability judgments, some of which the lab standardized through the Assembly Line Weaver.
2. Traditional interactive legal application authoring approaches
- A completed application combines a fielded output document, organized interview questions, and rules controlling follow-up questions and included content.Source: §2 · p. 3
- Authors label template fields, create prompts, order fields into screens, and add conditional logic, work that can take hundreds of hours.A representative Massachusetts Defense for Eviction project took about a year and included 1,100 lines of Python, 6,141 lines of YAML, and 13 Word documents.Source: §2 · p. 3
2.1. Interactive legal applications require thoughtful choices that affect usability
- Question order, help, and user-respect require judgment, while the lab reused tested question practices and interview structures through the Weaver.The tool helped apply those choices consistently across forms.Source: §2.1 · p. 3
3. The Assembly Line Weaver
The Assembly Line Weaver is a template-driven tool that produces draft Docassemble guided interviews in YAML. It gathers form and interview details, but does not add interview logic.
3. The Assembly Line Weaver
- The Weaver scans prepared Word or PDF templates for field names and gathers form metadata from the author.Source: §3 · p. 3
- Authors customize branding and templates, provide before-and-after context, write field prompts, and assign fields to screens.Source: §3 · p. 4
- The Weaver outputs a ZIP containing customized YAML, an output template, and a Next Steps template, but does not add interview logic.The files can be further customized.Source: §3 · p. 4
4. A vision for reducing human effort in form automation
The authors explored GPT assistance for labeling fields, drafting prompts, and grouping questions, using different approaches for Word and PDF documents. They did not plan to integrate statute-to-code generation because identifying the relevant statute can require research beyond the form.
4. A vision for reducing human effort in form automation
- The authors prototyped GPT-assisted field labeling, prompt drafting, and question grouping, and tested converting laws into Python code.They do not plan to integrate statute-to-code conversion because identifying a statute may require research beyond the form.Source: §4 · pp. 4–5
- For Word documents, the work focused on identifying and labeling placeholder variables, a gap in the existing Weaver.The authors left open feeding these labels into the Weaver later.Source: §4 · p. 5
- For PDFs, the authors prototyped direct generation of a limited Docassemble interview in YAML to explore model-produced interview flow, with human editing and review still possible.This PDF approach did not use the Weaver’s existing output standardization, and the authors expected to combine the approaches.Source: §4 · p. 5
5. Auto-labeling of Word documents
The section addresses Word documents’ varied placeholder styles while preserving document formatting, then describes tokenizing the document and prompting a model to label placeholders. The revised prompt replaced each formatted run’s text, but the approach lacked a large pre-annotated benchmark.
5. Auto-labeling of Word documents
- Word documents vary in placeholder styles, from brackets to inline text or no marker, making regular-expression labeling unreliable.Unlike PDFs, which usually have lines, boxes, or circles for answers, Word documents are more likely to use placeholder text.Source: §5 · p. 5
- The labeling approach had to preserve formatting, while full uncompressed XML could exceed GPT-4’s 4,096-token reply context window.The authors also sought labels that followed basic rules rather than being entirely free-form.Source: §5 · p. 5
5.1. Tokenizing the input document
- The document was tokenized by extracting paragraphs and formatted runs with python-docx, retaining indices for later modification.The JSON representation did not yet handle tables, which appeared in some dataset documents.Source: §5.1 · p. 6
5.2. Prompting the large language model
- The revised prompt returned each modified run’s full text, allowing the authors to replace that run while retaining its formatting.Because text within a DOCX run shares formatting, replacing the run’s text preserved that formatting.Source: §5.2 · p. 6
- The revised prompt identified placeholder text and replaced it in full, while the earlier positional prompt misplaced Jinja2 variables within placeholders.The earlier approach also returned the full modified document instead of only modified runs in GPT-3.5-turbo; replacing each full run worked with idiosyncratic markup.Source: §5.2 · p. 6
5.3. Using the output to build an interactive legal application
- The approach appeared to work across varied documents, but the authors lacked enough pre-annotated documents to benchmark performance.The labels could be uploaded to the Weaver, where authors could assign questions manually or use its auto-drafting mode, which required substantial human editing.Source: §5.2 · p. 7, §5.3 · p. 7
6. Building interactive legal apps automatically with PDF documents
The PDF workflow generated Docassemble interviews and filled PDFs, but performance depended on placing fields in context, and checkboxes remained difficult. The authors identify human review at multiple stages as a way to address errors and simplify final editing.
6. Building interactive legal apps automatically with PDF documents
- 62% to 69% of fields were automatically recognized and given questions across name change forms from 12 jurisdictions; results ranged from 93% on the best form to 27% on the worst.Unidentified fields were difficult to place in context in the text supplied to GPT.Source: §6 · pp. 8–9
6.1. Finding the fields in context
- Filling fields with placeholders, rendering PDF pages as images, and applying OCR placed answer fields in text context, but small checkboxes often resisted this method.The authors reintroduced checkboxes and other small fields when prompting the model to generate questions.Source: §6.1 · p. 9
6.2. Leveraging context to create variable names, definitions, and questions
- The model generated document and variable names, definitions, questions, guessed data types, and a YAML interview that filled answers into the processed PDF.Checkboxes received special prompting, but were paired with related text only 28% of the time; checkbox fields can represent important legal choices.Source: §6.2 · pp. 9–10
6.3. Validating user interactions
- Interactive GPT validation was abandoned because repeated checks were resource intensive, costly, and risked annoying users or rejecting valid unusual answers.Up-front data-type classification also mishandled phone numbers and ZIP codes, including stripping leading zeros.Source: §6.3 · p. 10
6.4. Additional Limitations & Potential Points of Intervention
- Errors could compound from a missing or mislabeled field into an incorrect question and screen sequence, while early review at multiple stages could simplify final editing.The authors identified review points before finalizing variable names and definitions, and before finalizing questions and data types; they prototyped editing a JSON object at each stage.Source: §6.4 · p. 11
7. Reaching level 1
The experiments indicate that GPT-assisted automation can reach the basic level of a guided interview, while combining it with the Weaver offers a route to pair vetted common questions with drafts for unique fields.
7. Reaching level 1
- The experiments show that GPT with Python can reach level 1: an interview at least as usable as the original PDF.Many PDF-locked forms could benefit from responsive, mobile-friendly design that can integrate with electronic filing.Source: §7 · p. 11
7.1. Weaving the two approaches together
- Word documents yielded better field identification and labeling than PDFs, and the authors expect the full approach to work with labeled Word documents.PDF checkbox context remains a problem for further experimentation.Source: §7.1 · p. 11
- Integrating GPT drafting with the Weaver would pair vetted, usability-tested questions for common fields with model-generated drafts for unique questions.This limits auto-generated definitions and questions to field types absent from the Weaver’s library.Source: §7.1 · p. 11
8. Conclusion
Generative AI can draft interview questions across many forms, but complex legal documents still require human assistance. The authors conclude that hybrid drafting can retain time savings while avoiding risks of fully automated form-filling.
8. Conclusion
- Generative AI can identify user inputs and create interview questions for a wide variety of forms.The authors report that it can work with both DOCX and PDF formats.Source: §8 · p. 11
- For very simple forms with little conditional logic, large language models can draft automations needing almost no human intervention, but complex legal documents require human assistance.Flexible human review at different automation stages lets people intervene as much or as little as needed across varied input documents.Source: §8 · p. 12
- The hybrid approach can combine language-model time savings with avoidance of the risks of fully automated question-and-answer form-filling.The authors say it has potential to expand the number and kinds of forms that benefit from document automation.Source: §8 · p. 12