A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

1 INTRODUCTION

• Early expert systems used hand-crafted rules and domain knowledge, but building knowledge bases was slow and experts had imperfect recall.These bottlenecks motivated research into automatic information extraction.Source: §1 · p. 1
• Related challenges include extracting ungrounded knowledge from semi-structured sources and connecting symbolic solvers with large-scale language models.Examples of recent question-answering systems include games and science exams.Source: §1 · p. 1
• Machine Reading and textual-entailment approaches often use dense representations and classifiers, bypassing classic symbolic inference.The introduction contrasts these distributional approaches with explicit structure.Source: §1 · p. 1
• Statutory reasoning applies complex natural-language rules to a case, while the authors report few contemporary capabilities for automatic statutory reasoning.Existing legal tools aid retrieval of relevant documents, but the authors were unaware of strong systems for automatically suggesting legal opinions.Source: §1 · p. 1
• The Prolog system achieves 100% accuracy on the constructed examples, whereas contemporary Machine Reading is insufficient even after legal-domain adaptation.This contrast raises whether to improve semantic parsing into symbolic form or Machine Reading to avoid explicit symbolic solvers.Source: §1 · p. 1

2 DATASET

• SARA combines selected Internal Revenue Code rules with natural-language questions answerable only by referring to those rules.The questions are designed to assess statutory reasoning over rules expressed in natural language.Source: §2 · p. 1

2.2 Key features of the corpus

• Statutory structure includes implicit and previously filled slots, while the law’s open texture makes determining applicability and filling slots challenging.The corpus represents statutes as a tree with 132 leaves and 193 nodes; 62 sections refer to time and 50 formulate exceptions or substitutions.Source: §2 · p. 2, §2.2 · pp. 2–3

2.1 Statutes and test cases

• The statutes cover selected tax topics, retain original numbering, and are simplified to remove highly specific provisions and references outside the selected subset.The dataset assumes each taxable year begins and ends with the corresponding calendar year.Source: §2.1 · p. 2
• For each extracted subsection, the authors created paired natural-language cases where it applies and does not, vetted them for plausibility, and split 176 training and 100 test samples.Positive and negative cases in each pair stay in the same split, while each section follows the overall split proportions.Source: §2.1 · p. 2, Figure 1 · p. 2
• A separate set of 100 cases asks for integer tax owed, requiring the full statutes and basic arithmetic, and is split into 80 training and 20 test samples.The cases were formed by mixing pairs from the first set and manually resolving inconsistencies.Source: §2.1 · p. 2
• Because the statutes were simplified, case answers do not necessarily match the current Internal Revenue Code, whose counterparts may have been repealed, amended, or adjusted for inflation.Source: §2.1 · p. 2

2.2 Key features of the corpus

• The corpus covers numerical reasoning in 55 sections, requiring arithmetic operations to be parsed from natural text, preserves cross-references as shown in Table 1, and leaves four statutory concepts undefined, including kinship and gross-income aggregation.Source: §2.2 · p. 3, Table 1 · p. 3

3 PROLOG SOLVER

• The solver translates statutes into Prolog rules and cases into facts so each case can be answered by a single query.The authors present it as a proof that a carefully crafted reasoning engine with perfect natural-language understanding can solve the dataset, not as a better Prolog system.Source: §3 · p. 3

3.1 Statutes

• Each statutory subsection becomes a rule, with cross-subsection terms represented as arguments that can bind and reuse slots.Unfilled slots remain unbound variables.Source: §3.1 · p. 3
• The solver uses 61 predicates to represent case facts at the granularity of the statutes, including events such as marriage and residence.The predicates follow neo-Davidsonian semantics.Source: §3.1 · pp. 3–4

3.2 Cases

• Case descriptions are manually translated into facts, while entailment prompts and tax questions become queries, including queries with an amount variable.Figure 3 gives example predicates, including a marriage event, its agents, and its start date.Source: §3.2 · p. 4, Figure 3 · p. 4
• The Prolog approach requires domain experts and semantic parsing, and small mistakes can lead to catastrophic failure.The paper contrasts this with machine-learning approaches that can adapt to new legislation and domains automatically.Source: §3.2 · p. 4

4 LEGAL NLP

• The legal NLP work uses unsupervised pretraining on a large legal-text corpus to adapt models for the legal domain.The section describes two unsupervised learning paradigms for this pretraining.Source: §4 · p. 4

4.1 Text corpus

• The 147M-token tax-law corpus combines classified federal cases with IRS private letter rulings and unpublished Tax Court cases.The cases were selected using heuristics for identifying tax-law cases; the rulings and unpublished cases were sourced from IRS and Tax Court websites.Source: §4.1 · p. 4

4.2 Tax vectors

• Tax word2vec preprocessing preserves tax terms of art and statute references as single tokens before training 500-dimensional vectors.A window size of 15 maximized performance on twelve human-constructed analogy tasks.Source: §4.2 · p. 4

4.3 Legal BERT

• Legal BERT keeps Bert-Base-Cased’s architecture but uses parameters further trained on 900M legal-case tokens.Training used masked language modeling on chunks of 510 tokens from selected state and federal cases, excluding cases under 400 characters and those before 1970.Source: §4.3 · p. 4
• Legal BERT’s perplexity was 2.7 versus 14.4 for Bert-Base-Cased; legal-term F1 was 0.44 versus 0.35, while lawyers scored 0.26.The authors attribute the lawyer comparison in part to preprocessing that dropped some nested legal terms lawyers tended to identify.Source: §4.3 · pp. 4–5

5.1 BERT-based models

• BERT-based entailment models encode the relevant statutory subsection with the prompt, while numerical models use an all-statutes input because the full statutes exceed BERT’s length limit.Entailment is evaluated with accuracy; numerical accuracy counts predictions whose normalized tax error is below the substantial-understatement threshold.Source: §5.1 · p. 5

5.2 Feedforward models

• Feedforward models combine embeddings of the statute and context-question strings with their element-wise difference and product before predicting an answer.Ablations remove statute information or both statute and context, and the models also test an identity function in place of the neural network.Source: §5.2 · p. 5

5.3 Results

• Models perform comparably to the baseline; removing statutes or statutes and context leaves performance mostly unchanged, and legal-domain adaptation has no noticeable effect.The paper reports test-set accuracy in Table 5.Source: Table 5 · p. 5, §5.3 · p. 5
• The experiments focus on adapted legal resources and off-the-shelf and historical solutions; specialized reasoning models are left for future work.Source: §5.3 · p. 6

6 RELATED WORK

• Prior work formalized statutes in expert systems, including Prolog translations of nationality and tax law, but the cited projects did not include a dataset or task applied to their programs.The section identifies Sherman’s translation of part of Canada’s Income Tax Act as closest to this work.Source: §6 · p. 6
• Existing legal datasets address statute questions, bar-exam questions, case reasoning, document classification, privacy policies, and dialogue about isolated rules.The paper contrasts its dataset with Saeidi et al. [50], where rules are isolated from the rest of the body of rules and follow-up questions are part of the task.Source: §6 · p. 6
• Science-exam reasoning is related in spirit, but statutory reasoning concerns human-made rules whose truth depends on being written down and agreed to.The paper characterizes statutory reasoning as a particularly pure instance of reasoning with prescriptive language.Source: §6 · p. 6
• Other machine-reading work includes prerequisite toy tasks, numeric reasoning, procedural-knowledge datasets, and instruction-based game-playing agents.The cited examples include DROP [19] for numeric reasoning and Branavan et al. for using instruction-manual text.Source: §6 · p. 6

• The paper introduces a law-statute resource, hand-curated natural-language rules and cases, and a symbolic solver for the challenge task.Source: §7 · p. 6
• Legal-domain NLP has few large-scale systematic resources, and the authors intend this work to contribute to the field.Source: §7 · p. 6
• The dataset is small relative to typical MR resources, and costly legal training data leaves extraction, data-efficient MR, and case-based training-data creation as open directions.Source: §7 · p. 6
• The authors hope the resource will provide a benchmark for legal NLP and machine reasoning and legal-domain NLP models useful to researchers.Source: §7 · p. 6