In With the Old: Enhancing 'Classical' Document Automation with Generative AI

1. Introduction

• Legal document automation has been available since the 1980s and is mature and powerful, though increasingly viewed as “old school” amid newer AI.It is naturally applicable where people without easy lawyer access need to make decisions and produce legally significant documents.Source: §1 · p. 1
• LawHelp Interactive has provided over a million free bespoke document packages each year for several years using mostly HotDocs.Nonprofits and law schools have pursued similar efforts; surveys the field as of several years ago.Source: §1 · p. 1
• The paper explores combining classical document automation with generative AI conceptually and experimentally, including whether LLMs can detect and cure flaws in texts.The authors describe this combination as “neural-symbolic.”Source: §1 · p. 1

2. Related Work

• Prior access-to-justice work includes protection-order advisories, legal information, dispute resolution, and negotiation support.Source: §2 · p. 1
• LLMs have been studied for legal exam questions, statutory reasoning, explaining legal terms [16], document annotation [18], and thematic analysis [19].Access-to-justice applications include legal information, conflict mediation [24], legal intake [25], and support for tool creation and document automation [27].Source: §2 · pp. 1–2
• Earlier document-assembly work explored LLM drafting, symbolic reasoning with LLMs, and extracting form information for automation [30]; this paper tests requirements-based flaw detection, follow-up questions, and correction.Source: §2 · p. 2

3. A Thought Experiment

• Although the interview offers English and Spanish and frequent on-screen help, users are largely left to compose the long narrative response in an empty field.Source: §3 · p. 2
• A proposed help button would launch a chat to guide users through composing an optimal factual statement.The chat could ask about language preference, use interview answers available in session memory, and tailor questions to the user and circumstances.Source: §3 · pp. 2–3
• The chat could gather material through leading questions, stitch it into a draft, and let the user check accuracy and completeness before reassessing edits.Once settled, the text would be entered into the interview so the user could continue assembling documents.Source: §3 · p. 3
• For assisted-pro-se cases, an advocate could receive a machine-generated narrative assessment and suggestions; alternatively, an LLM could conduct the interview from its definition and requirements.Source: §3 · p. 3

4. Some Benefits

• Document assembly elicits information and produces well-formatted, error-free forms but is not suited to long passages; GenAI supports conversation but is not optimal for such forms.Source: §4 · p. 3
• Combining form-like fields with free-form exchanges can serve a wider spectrum of users and needs, while system prompts can operationalize written guidance in interactive conversations.Parametric affordances can provide context about users’ circumstances and goals.Source: §4 · p. 3
• The combined systems can offer multiple languages for input and output, as well as spoken modes.Source: §4 · p. 3

5. Some Challenges

• Advice quality and correctness are concerns, especially for non-English users unfamiliar with legal processes and terminology.Workflows would need to minimize misinformation and excessive suggestiveness.Source: §5 · p. 3
• Providers’ interest in learning from system use can conflict with users’ need for assurance that personal information is not stored or used for training.The passage identifies both interests but does not specify how to resolve the tension.Source: §5 · p. 3
• Without seamless integration of classical and generative-AI interfaces, switching contexts may put off users or cause some to get lost.Source: §5 · p. 4
• Implementation also raises funding, unauthorized-practice, and ethical issues, including clarifying that no lawyer-client relationship exists and disclosing AI assistance.Special-purpose models trained on a situation’s requirements could perhaps reduce session costs.Source: §5 · p. 4

6. Experimental Setup

• The experiment tests whether an LLM can check text against requirements, identify missing elements, ask a user for them, and integrate them into a complete text.Source: §6 · p. 4

6.1. Dataset

• The three scenarios are a landlord letter, a purchase agreement, and an employment-agreement template.The employment template is a simplified HotDocs version with fill-in slots and a conditional section for seminar days.Source: §6.1 · p. 4
• Each scenario has five flawed candidate texts, with flaws such as missing relevant data or misspellings recorded to support evaluation.Source: §6.1 · p. 4
• Table 1 illustrates the instructions, a candidate text, model questions with researcher-provided answers, and the final model-written text.Source: Table 1 · p. 5

6.2. Model

• The experiments used gpt-4o-2024-11-20 at temperature 0 to reduce hallucinations and increase replicability.Source: §6.2 · p. 5

6.3. Pipeline

• The pipeline first elicits missing information and then uses the answers with the initial text to generate a corrected document that follows the instructions.Source: §6.3 · p. 5, §6.3.2 · p. 6

6.3.1. Step 1 - Generate Questions to Obtain Missing Information

• Step 1 prompts the model to generate questions about missing information from the candidate text, using the supplied instructions as the basis.Source: §6.3.1 · pp. 5–6
• To curb over-questioning, the prompt restricted questions to information actually requested by the instructions; prompt tuning otherwise remained minimal.Source: §6.3.1 · p. 6

6.3.2. Step 2 - Producing the corrected document

• Step 2 combines the initial information with answers to the model’s questions to produce a corrected text adhering to the provided instructions.Source: §6.3.2 · p. 6

6.4. Experiments

• The experiments evaluate the LLM’s ability to ask relevant follow-up questions and integrate the answers into a corrected document, using quantitative and qualitative assessment.Source: §6.4 · p. 6

6.5. E1 - Assessing the quality of the questions

• The question-quality assessment asks whether follow-up questions can identify information missing from candidate texts, alongside qualitative notes on the process.Source: §6.5 · p. 6
• The study examines whether LLMs can generate questions that reliably address document flaws, a feature of systems such as Dallma and the legal aid system [25].Source: §6.5 · p. 6

• After users answer the LLM’s questions, the researchers review the generated text for correspondence with the provided instructions.Source: §6.6 · p. 6
• The output is also reviewed for confabulated information, with the exact questions listed in Table 4.Source: §6.6 · p. 7
• A hallucination requires an entirely new fact; details directly stemming from a fact, such as financial damages caused by laptop loss, are excluded.The authors adopt a relatively generous definition because they consider argumentative drafting a potential use for LLMs.Source: §6.6 · p. 7

7.1. Tenant apartment reclamation

• In S1, the model asked about noted flaws in all five cases but often went beyond the instructions with additional questions.In two cases, it asked users to correct text or confirm spelling despite the formal-tone and grammar requirements.Source: §7.1 · p. 7
• In S1, corrected texts included all missing elements but took the form of complete letters rather than requested claim-submission sections.The model also elaborated on underexplained facts, which the authors considered possible embellishment warranting prompt tuning or human review.Source: §7.1 · pp. 7–8

7.2. Purchase agreement

• In S2, questions closely matched identified flaws, although one case missed that sale and delivery dates need not coincide and some questions exceeded instructions.The extra questions addressed potentially relevant details about a bike pickup that the instructions did not request.Source: §7.2 · p. 8
• In S2, every corrected text included the instructed elements and none contained hallucinations, though the model added conventional contract features not requested.These included an introductory contract statement and a signature space, which the authors attributed to the model’s general expectations about sale contracts.Source: §7.2 · p. 8

7.3. Employment Contract

• In S3, the model found missing information in all five examples but sometimes asked about topics already covered or about non-slot template text.S3 used a constrained template with five slots, including one boolean slot controlling whether a section appeared.Source: §7.3 · p. 8
• In S3, the model generally inserted information correctly and mostly applied the conditional, but it failed once and frequently changed details meant to remain fixed.Where protected text must remain intact, the authors suggest considering semi-structured drafting.Source: §7.3 · p. 9

7.4. Overall Discussion

• The experiment suggests LLMs can spot missing facts, obtain them from users, and correct candidate texts, while precision remains difficult.The authors recommend careful prompt specification, preparing outputs for human review, and considering other AI methods in some scenarios.Source: §7.4 · p. 9