Source-linked AI summary
Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice
Hannes Westermann, Jaromir Savelka
TL;DR
Legal-relevant information may sit in complex paper documents, even though AI workflows often expect users to provide it digitally. This study tests multimodal LLM extraction from form images; 73% of fields were correctly extracted on average, with image quality remaining an important constraint.
Problem
Many AI and LLM workflows assume users can provide relevant information digitally, although it may be embedded in complicated paper documents.
Source: §1 · pp. 1–2Method
The paper investigates multimodal LLM extraction of structured information from photographs of forms, including an Ontario residential tenancy form.
Source: §1 · p. 2, §3.1 · p. 3Results
73% of fields were correctly extracted on average across formats and scenarios; accuracy was 89% in S1, 71% in S2, and 59% in S3.
Source: §4.1 · p. 6Takeaways & Limitations
The results indicate potential for multimodal LLMs to help laypeople and self-represented litigants extract and analyze data in printed documents.
Source: §5 · p. 8Good results depend on image quality, and the need for a modern phone and good lighting may exclude some groups from public-facing tools.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Interacting with the legal system and the government requires the assembly and analysis of various pieces of information that can be spread across different (paper) documents, such as forms, certificates and contracts (e.g. leases). This information is required in order to understand one's legal rights, as well as to fill out forms to file claims in court or obtain government benefits. However, finding the right information, locating the correct forms and filling them out can be challenging for laypeople. Large language models (LLMs) have emerged as a powerful technology that has the potential to address this gap, but still rely on the user to provide the correct information, which may be challenging and error-prone if the information is only available in complex paper documents. We present an investigation into utilizing multi-modal LLMs to analyze images of handwritten paper forms, in order to automatically extract relevant information in a structured format. Our initial results are promising, but reveal some limitations (e.g., when the image quality is low). Our work demonstrates the potential of integrating multi-modal LLMs to support laypeople and self-represented litigants in finding and assembling relevant information.
1. Introduction
Laypeople can struggle to complete legal forms, while AI systems often assume users can provide relevant information digitally even when it is buried in complex paper documents. This paper investigates whether multimodal LLMs can extract information from form images for later use.
- Laypeople struggle to choose and complete legal forms and to craft legally sound narratives.Paperwork imposed almost 10 billion hours on people in the United States in 2015.Source: §1 · p. 1
- AI and LLM systems often assume users can provide relevant information digitally, although it may be contained in complicated paper documents that are difficult to interpret.Users may have to locate and type the information themselves.Source: §1 · pp. 1–2
- The paper investigates whether multimodal LLMs can extract information from photographs of forms for use in legal and government-support tasks.Potential uses include filling out forms, informing people about their rights and legal situation, and suggesting court-submission drafts.Source: §1 · p. 2
- The research questions ask how reliably GPT-4o extracts structured data from form photographs and how image quality, complex data, missing fields, and unclear handwriting affect performance.Source: §1 · p. 2
2. Related Work
Prior access-to-justice work includes legal reasoning, document drafting, and decision-support tools, while more recent LLM research spans legal tasks and support for laypeople. This paper extends that work by investigating images as an input for extracting information from paper documents.
AI for Access to Justice
- Access-to-justice systems progressed from formalizing legal reasoning to pro se document drafting, web-based decision support, and JusticeBot.Source: AI for Access to Justice · p. 2
LLMs for Extracting Information from Legal Texts
- Legal-text tools have mapped lay descriptions to legal issues with 93.5% accuracy, improved court-form usability, and used legal-case schemas to identify relevant facts.This paper adds images as an input modality for extracting information.Source: LLMs for Extracting Information from… · p. 2
3. Experimental Design
The study tests structured information extraction from images of an Ontario residential tenancy form using varied scenarios and image formats. GPT-4o receives form images and extraction instructions, and its outputs are compared with gold-standard labels.
3.1. Residential Tenancy Agreement (Standard Form of Lease)
- The dataset uses the first page of Ontario’s required Residential Tenancy Agreement, covering landlord and tenant names, rental-unit address, and condominium status.These fields may be needed when pursuing a claim about apartment deficiencies.Source: §3.1 · p. 3
3.2. Dataset
- The dataset contains 15 samples across three scenarios and five image formats, with 14 expected fields per sample.The scenarios vary names, missing information, tenant count, and other fields; formats range from typed PDFs to handwritten photographs made under different conditions.Source: §3.2 · pp. 3–5
3.3. Model & Prompt
- The pipeline sends form images to GPT-4o with instructions to extract specified fields as textual values in JSON, using temperature 0 to support reproducibility and reduce hallucinations.Images were not cropped or edited, except for rotating sideways images.Source: §3.3 · p. 5
3.4. Experiment
- The experiment compares model outputs with gold-standard labels, counting a field correct only when its text matches exactly, with capitalization treated flexibly.Accuracy is also compared across scenarios and sample versions to examine image imperfections and other task difficulties.Source: §3.4 · p. 5
4. Results & Discussion
Extraction accuracy averaged 73% across formats and scenarios, but varied by scenario, field, and image format. The analysis suggests promise alongside constraints from image quality and model preference for common names.
4.1. General Results
- 73% of fields were correctly extracted on average, with scenario accuracy falling from 89% in S1 to 71% in S2 and 59% in S3.Figure 3 presents accuracy across scenarios and formats.Source: §4.1 · p. 6, Figure 3 · p. 7
- Even incorrect values contained some matching letters or numbers, suggesting the model could locate information despite difficulty reading it accurately.The authors note that form labels could also be hard to read in sloppy versions.Source: §4.1 · p. 6
4.2. Differences between fields
- Accuracy varied by field: town and province were perfectly captured across formats, street names were extracted almost perfectly, and street numbers were correct only 27% of the time.The authors suggest repeated province entries and pretraining may help with location fields; even neat street-number entries could be hard to read.Source: §4.2 · p. 7, Figure 4 · p. 7
- Uncommon names were sometimes changed to more common ones, including Jame to Jane and Douglasson to Donaldson.The authors say such behavior may differ from traditional OCR approaches.Source: §4.2 · p. 7
4.3. Differences in format
- Typed PDFs received almost perfect scores, while accuracy declined for printed forms, especially sloppy forms and formats beyond neat S1 handwriting.Figure 5 shows selected cases where the model still extracted values despite poor data quality.Source: §4.3 · p. 8, Figure 5 · p. 8
4.4. Analysis
- The authors see potential for image-based extraction, especially when a human can verify captured data and correct mistakes.They suggest applications such as extracting fields from documents to fill forms or provide legal information.Source: §4.4 · p. 8
- Poor image quality may constrain results, and reliance on modern phones and good lighting could exclude some groups; common-name preferences may also risk societal bias if unevaluated.The authors note that phone prices and camera quality are improving, which may reduce access barriers over time.Source: §4.4 · p. 8
5. Conclusion & Future Work
The authors find promising potential in using multimodal LLMs to extract structured information from form images in support of laypeople and self-represented litigants. Future work will expand studies and explore integration into systems for legal information or form completion.
5. Conclusion & Future Work
- The study reports promising results and potential to support laypeople and self-represented litigants by extracting and analyzing data in printed documents.The authors frame this as having implications for access to justice.Source: §5 · p. 8
- Future studies will use larger-scale, more varied data and examine the performance effects of changing prompts and models.Source: §5 · p. 9
- The authors plan to investigate integrating the described functionality into a system for providing legal information or filling in forms.Source: §5 · p. 9