PaperlayerView exact revision on arXiv

Source-linked AI summary

ImmigrationReason: A Structured Dataset of U.S. Immigration Appeals for Legal Reasoning Research

Amirhossein Afsharrad, Seyed Shahabeddin Mousavi

arXiv:2608.20391v1cs.CL

TL;DR

Legal NLP lacks resources representing administrative adjudication and the fine-grained evidence weighing and error detection required for legal reasoning. ImmigrationReason addresses this gap with a structured corpus of 12,375 AAO decisions, validated extraction, and analyses spanning legal-regime change and institutional variation.

  • Problem

    Existing legal NLP datasets predominantly use federal case law and coarse-grained classification, while administrative adjudication involves distinct doctrinal standards and complex evidence-based reasoning.

  • Method

    The paper constructs ImmigrationReason from 12,375 non-precedent AAO decisions, adding criterion-level findings, adjudicator-error quotes, legal frameworks, dispositions, and validated transcriptions through a three-pass extraction pipeline.

  • Results

    ImmigrationReason provides structured evidence-sufficiency labels, explicit adjudicator-error annotations, and a natural temporal split across the 2016 Dhanasar rule change, with legally critical-field agreement exceeding 97%.

  • Takeaways & Limitations

    The dataset supports research on legal-reasoning outcomes, adjudicator-error analysis, temporal legal dynamics, and agents for high-stakes regulatory domains.

  • Takeaways & Limitations

    The corpus covers only employment-based immigration appeals and reflects contested appellate denials, limiting generalization to other domains and initial adjudication.

Abstract

from arXiv · show

Most legal NLP resources draw from federal case law and focus on coarse classification, leaving administrative adjudication, where the vast majority of government decisions occur, essentially unaddressed. We introduce ImmigrationReason, a large-scale structured dataset derived from 12,375 non-precedent decisions of the U.S. Citizenship and Immigration Services (USCIS) Administrative Appeals Office (AAO) spanning 2005 to 2026. Each record captures the applicable legal framework, per-criterion evidence-sufficiency findings under a five-category label, verbatim adjudicator-criticism quotes, all citations, and final dispositions, alongside high-quality Claude-transcribed source text. Extraction quality is validated through a three-pass pipeline combining two independent modalities with comparison-prompt adjudication by Opus 4.7, and verified by domain experts on a 500-record sample. The dataset documents nearly 9,000 verbatim instances of AAO-identified legal errors, spans a natural legal-regime transition (the 2016 Dhanasar rule change), and covers 21 years of adjudication. We analyze the dataset in detail and outline research directions it enables, from outcome prediction and adjudicator-error analysis to agent design for high-stakes regulatory domains.

1 Introduction

ImmigrationReason addresses the lack of richly structured administrative-law data for legal reasoning by organizing thousands of USCIS AAO immigration appeals. Its annotations capture criterion-level evidence judgments, adjudicator criticisms, and a built-in legal-regime transition.

  • Existing legal NLP datasets primarily use federal case law and coarse classification, despite legal work requiring multi-step evidence weighing and citation-grounded reasoning.
  • Administrative agencies make far more decisions than federal courts, while USCIS immigration adjudication applies doctrinal standards distinct from common-law case law.
  • ImmigrationReason contains 12,375 non-precedent AAO decisions spanning 2005 to 2026.
  • The dataset records decision metadata, legal-issue reasoning, and five-category criterion findings while separately tracking officer and AAO determinations.
  • Its distinctive resources include senior-tribunal evidence-sufficiency labels, verbatim adjudicator-error criticisms, and a December 2016 Dhanasar regime transition for temporal testing.

2 Related Work

Prior legal NLP resources largely emphasize federal case law, classification, retrieval, or raw text, while immigration-specific resources do not cover U.S. employment-based adjudication. ImmigrationReason adds structured administrative-law data and supports study of legal AI in a deployed context.

  • LegalBench, LexGLUE, LawBench, LEXam, Pile of Law, and CaseHOLD provide benchmarks or corpora but lack per-criterion findings, adjudicator-error annotations, or administrative-law coverage.
  • Existing immigration-law datasets address Canadian refugee-law retrieval or asylum-outcome prediction rather than U.S. employment-based adjudication.
  • Classical OCR has been widely used for scanned legal documents, whereas the paper compares it with Claude-powered transcription across 12,375 paired decisions.
  • ImmigrationReason provides a setting for evaluating citation hallucination and legal argument cogency where AI is already being deployed.

3 Background: AAO Adjudication

The AAO reviews denied employment-based petitions through de novo appellate adjudication under distinct EB-1A and NIW legal frameworks. The corpus also addresses uneven source quality, especially in older scanned decisions.

  • The AAO reviews Form I-140 employment-based petitions after USCIS Service Center denials and issues public non-precedent decisions through de novo review.
  • EB-1A requires at least three of ten evidentiary criteria, followed by a two-step analysis that includes a final-merits determination of sustained acclaim.
  • Since December 2016, NIW adjudications use the three-prong Dhanasar framework, replacing the earlier NYSDOT framework and creating a hard temporal boundary.
  • Claude-powered transcription is compared with legacy OCR to improve recovery of structural content, particularly in pre-2017 scanned documents.
  • The decision body consistently contains an introductory summary, LAW section, ANALYSIS section, and ORDER block.

4 Dataset Construction

The dataset was assembled from public AAO decisions, transcribed into structured records, and extracted through independent model passes with adjudication of disagreements. Its schema preserves nuanced criterion outcomes and reasoning evidence.

  • Source Collection and Filtering: The final corpus contains 12,375 decisions after collecting 13,520 PDFs and filtering O-1 filings, short documents, and duplicates.
  • Preprocessing and transcription: Claude-transcribed Markdown supplements near-lossless PyMuPDF extraction for post-2017 text-native documents and noisy OCR for pre-2017 scans.
  • Structured schema: Each decision is represented as a typed record with decision-level metadata, legal issues, findings, citations, dispositions, reasoning, and extraction confidence.
  • Finding labels: 19.8% of findings are reserved and 3.4% waived, so binary labels would miscategorize 23% of finding instances.
  • Extraction Pipeline: Three extraction passes combine PDF-direct and text-based Sonnet extraction with Opus 4.7 comparison adjudication for disagreements.
  • Extraction Pipeline: 3,401 records, or 27.5%, disagreed across the first two passes, and Opus produced a value unseen in either pass for 10.1% of adjudicated records.

5 Dataset Analysis

ImmigrationReason reveals structured patterns in immigration adjudication, including noisy form-code labels, institutional variation, contested criteria, and a sharp legal-regime transition.

  • California SC has a 54.4% reversal rate versus 36.0% for Vermont SC, revealing substantial institutional variation across originating service centers.
  • For EB-1A, awards and original contributions have the lowest step-1 success rates, while judging is among the highest.
  • Dhanasar Prong 3 has the lowest NIW petitioner success rate and is most commonly reserved in dismissals.
  • The corpus contains 1,280 NYSDOT and 3,484 Dhanasar decisions with essentially zero framework overlap, creating a natural temporal out-of-distribution split.

6 Quality Validation

The dataset’s extraction quality is supported by high agreement on legally critical fields, adjudication of disagreements, and expert review, while outcome figures characterize contested legal criteria.

  • Agreement exceeds 97% for visa category, posture, and dispositions across 12,372 paired records.
  • Dhanasar Prong 3 has the lowest petitioner win rate, while EB-1A awards, published material, and contributions are the most contested criteria.
  • Figure 6 summarizes distributions for posture, visa category, AAO stance, finding states, citations per decision, and director-criticism counts.
  • Issue-type agreement is lower at 83–85% because identifying substantively addressed legal issues is genuinely ambiguous and requires Pass 3 adjudication.
  • Domain experts reviewed a stratified sample of 500 decisions and found all reviewed extractions correct and reasonable.

7 Dataset Contents: A Detailed Description

The release combines transcribed source documents, structured records, reproducible extraction code, OCR comparisons, and adjudication records to support detailed analysis and reuse.

  • Source text uses Claude Sonnet 4.6 vision for pre-2017 documents and PyMuPDF for post-2017 text-native documents, with flagged legacy OCR fallback for 13 decisions.
  • Table 3 reports that vision OCR improves agreement against PDF-direct reference compared with legacy OCR.
  • The corpus contains 12,375 structured records, 45,290 per-criterion findings, approximately 9,000 director-criticism quotes, and more than 150,000 citations.
  • The full codebase covers OCR, structured extraction, three-pass validation, quality reporting, schemas, prompts, batch scripts, and analysis notebooks.
  • OCR comparison records provide per-record evaluation of legacy OCR against Claude-transcribed text across structural features.
  • Adjudication records preserve Opus 4.7 resolutions for 3,401 cases involving disagreements between extraction passes.

8 Proposed Research Directions

ImmigrationReason supports research on prediction, adjudicator errors, legal reasoning, temporal dynamics, model training, fairness, and citation grounding in employment-based immigration.

  • Outcome labels and structured case facts support prediction of overall outcomes, per-criterion findings, and dispositive prongs at varying text-revelation levels.
  • Approximately 9,000 verbatim AAO criticisms provide labeled data for empirical error typologies and models detecting originating-officer mistakes.
  • Per-criterion evidence descriptions and legal standards support evidence-sufficiency classification and reasoning-extraction evaluation against citations and finding conclusions.
  • The Dhanasar transition and 21-year span enable studies of legal-regime tracking and longitudinal changes in evidentiary standards.
  • Paired evidence findings, director reasoning, AAO corrections, and annotated decisions provide training signals for legal-reasoning models and agents.
  • Service-center reversal rates ranging from 36.0% to 54.4% enable aggregate studies of institutional variation, professions, petition types, and national-origin proxies.
  • Citation lists provide ground truth for evaluating hallucinated or correctly grounded references to frequently cited precedents such as Dhanasar and Chawathe.

9 Limitations

The corpus has important scope, selection, privacy, and extraction limitations that constrain how its records should be interpreted and generalized.

  • The corpus covers only employment-based immigration adjudication, excluding asylum, removal, family-based, and non-U.S. systems.Models trained on these data may not generalize to excluded domains.
  • AAO decisions represent denials that petitioners chose to contest and may not reflect initial-adjudication behavior.The appellate selection limits the corpus’s representativeness of the broader adjudication process.
  • Redacted personally identifiable information precludes individual-level demographic analysis.
  • LLM-produced records may contain errors, particularly on ambiguous older documents, despite three-pass validation and 500 expert reviews.Per-record extraction_confidence and extraction_notes fields surface known issues.

10 Conclusion

ImmigrationReason is a released, structured dataset for studying legal reasoning in USCIS immigration appeals, combining criterion-level findings, adjudicator criticisms, legal frameworks, and transcribed decisions. Its extraction process uses multiple passes and explicit distinctions among finding states, while the dataset supports temporal and comparative legal-NLP research.

  • ImmigrationReason contains 12,375 U.S. administrative immigration appeals with deep annotations and high-quality Claude-powered source transcription.The release includes source text, extraction code, and schema under a permissive license.
  • The dataset tracks per-criterion evidence sufficiency, verbatim adjudicator criticisms, and director-versus-AAO findings separately.These annotations support analysis of reasoning and adjudicator error at the criterion level.
  • Extraction quality is validated through a three-pass pipeline combining independent extraction modalities with Opus 4.7 adjudication and 500-record expert review.The comparison pass re-reads the source decision to resolve disagreements and populate all fields.
  • Finding states preserve distinctions among met, not_met, reserved, waived_by_petitioner, and not_addressed rather than collapsing them into not_met.Director and AAO findings are recorded separately for each prong, including verbatim criticism when the AAO faults the director’s reasoning.

G Data Card

The data card describes ImmigrationReason’s composition, intended research uses, limitations, maintenance, licensing, and machine-readable metadata. It emphasizes structured appellate records while cautioning against legal-advice or automated-adjudication use.

  • Dataset composition: ImmigrationReason contains 12,375 non-precedent AAO decisions for EB-1A and NIW petitions spanning 2005–2026.Each record includes legal frameworks, per-criterion findings, citations, final dispositions, and Claude-transcribed decision text.
  • Intended uses: The dataset supports outcome prediction, adjudicator-error analysis, criterion-level LLM evaluation, legal-regime research, and AI-agent development for high-stakes regulatory domains.
  • Out-of-scope uses: ImmigrationReason is not intended as legal advice, guidance for specific pending petitions, or the basis for automated adjudication.It documents historical AAO decisions rather than predicting or prescribing future outcomes.
  • Dataset composition: The corpus comprises 6,981 NIW and 5,394 EB-1A decisions, with 45,290 criterion-level rows in findings_long.parquet.No personally identifiable information is included.
  • Collection and preprocessing: Structured extraction used a three-pass LLM pipeline, while pre-2017 scanned documents were transcribed with Claude Sonnet 4.6 and validated against legacy OCR.
  • Known limitations and biases: The corpus is limited by appellate selection bias, non-precedent decisions, and possible low-frequency extraction errors in unusual or ambiguous cases.
  • Metadata and release: The dataset includes Croissant 1.0 JSON-LD metadata describing its parquet distributions, record sets, license, and version.
Loading 2608.20391v1…