PaperlayerView exact revision on arXiv

Source-linked AI summary

Classifying Interpretive Canons at the Sentence Level: A Benchmark from the German Federal Constitutional Court

Felix Ringe

arXiv:2609.26945v1cs.CLcs.CY

TL;DR

Prior work identifies interpretive arguments at the paragraph level, not their precise sentence locations, and manual analysis constrains study at scale. This paper operationalizes Larenz’s canons, builds a sentence-level dataset from German Federal Constitutional Court decisions, and evaluates LLM baselines. Its reported finding is that grammatical interpretation is usually easiest to identify; the results are bounded by single-annotator, single-court data and small test sets.

  • Problem

    Prior work models interpretive argument types at the paragraph level rather than identifying their precise locations, while manual effort constrains large-scale study.

    Source: §1 · p. 1

  • Method

    The paper operationalizes Larenz’s interpretive canons as classification criteria, annotates German Federal Constitutional Court decisions at the sentence level, and evaluates four LLMs from three model families.

    Source: §1 · p. 1, §4 · p. 5

  • Results

    The paper reports that grammatical interpretation is usually the easiest canon to identify.

    Source: §9 · p. 8

  • Takeaways & Limitations

    An AI agent reproduced the reported scores and benchmark datasets from input data and methods descriptions without access to the code.

    Source: §1 · p. 1

    Gold labels come from one annotator and one court in one language, and small test sets yield wide confidence intervals that counsel against over-interpreting prompt-condition differences.

    Source: §8 · p. 8

Abstract

from arXiv · show

Judicial reasoning remains challenging for large language models (LLMs) to analyze. This paper contributes a sentence-level benchmark for evaluating the ability of LLMs to classify interpretive canons as articulated by Larenz in the tradition of Savigny. Our contributions are threefold. First, we operationalize this conception of interpretation as classification criteria. Second, we provide a dataset of decisions of the German Federal Constitutional Court annotated at the sentence level. Third, we report baseline evaluations of four LLMs from three model families under expert hand-written prompts, compared against prompts optimized with Genetic-Pareto (GEPA). Mean F1 over the seven binary subtasks clusters between 70.4 and 79.2 across models, with grammatical interpretation usually the easiest canon to identify and systematic interpretation usually the hardest; under the tested configuration, GEPA-optimized prompts do not systematically outperform the hand-written ones, suggesting that the expert prompts provide a meaningful baseline.

1. Introduction

Manual effort has constrained large-scale study of judicial arguments, while prior models identify argument types at paragraph rather than precise sentence level. This paper operationalizes Larenz’s interpretive canons, provides an expert-annotated sentence-level dataset, and evaluates LLM baselines.

1. Introduction

  • Manual effort has constrained empirical legal scholarship’s ability to study interpretive arguments at scale.Reliable LLM classification could reduce analysis costs and support further studies of judicial reasoning.Source: §1 · p. 1
  • Prior work modeled argument types at the paragraph level, leaving their precise locations unidentified.The paper shifts classification to the sentence level and examines grammatical, systematic, historical, and objective-teleological interpretation.Source: §1 · p. 1
  • The paper contributes granular criteria based on Larenz’s conception, an expert-annotated German Federal Constitutional Court dataset, and baseline evaluations of four LLMs from three model families.The evaluations compare expert-written prompts with prompts optimized using GEPA.Source: §1 · p. 1
  • An AI agent reproduced the reported scores and benchmark datasets exactly from the input data and methods descriptions, without access to the code.The paper reports this reproducibility result in Section D.Source: §1 · p. 1

2. Task Methodology

The paper defines interpretation through Larenz’s account of choosing among possible readings on the basis of considerations, then turns that account into criteria for identifying readings and their supporting arguments. It operationalizes four interpretive canons while leaving assertion without a stand-alone classification target.

2. Task Methodology

  • The paper derives its working account of interpretation from Larenz because no standardized definition exists.Interpretation involves choosing among possible readings on the basis of considerations.Source: §2 · p. 2

2.1. Reading

  • A reading is a court-advanced assertion about a statutory provision’s abstract content, and it serves as the reference point for identifying canons.Which canon a consideration invokes can depend on the reading from which it is viewed.Source: §2.1 · p. 2
  • A canon classification can change with the reading at issue: the same statement about one provision is grammatical from one perspective and systematic from another.The paper illustrates this shift by changing the reference point from Section 46 BWahlG to Section 37 BWahlG.Source: §2.1 · p. 2

2.1.1. ABSTRACT

  • A reading must be abstract, applying beyond the decision’s specific facts rather than merely subsuming those facts under a norm.An incidental reference to case-specific facts does not prevent a statement from being read as abstract if it also states a general rule.Source: §2.1.1 · p. 2

2.1.2. ASSERTION

  • An assertion must allow a meaningful counterposition and go beyond merely reproducing statutory wording.Assertion is not operationalized as a stand-alone classification subtask.Source: §2.1.2 · p. 3, §2.1.3 · p. 3

2.1.3. REFERENCE POINT: A SPECIFIC STATUTORY PROVISION WITH DETERMINATE CONTENT

  • The reading must concern a specific statutory provision and determine its content, not merely state how two norms relate.A provision’s content is determinate when the assertion identifies which cases fall under it; compatibility assessments alone do not do so.Source: §2.1.3 · p. 3

2.1.4. SELF-ADVANCED

  • An assertion is self-advanced when both its starting point and endpoint lie in the deciding panel’s reasoning.When an assertion draws on several sources, the paper treats it as self-advanced in the panel’s favor if not all sources are precedents.Source: §2.1.4 · p. 3

2.2. Considerations

  • The four canons concern the considerations supporting a reading: grammatical usage, systematic context and legal structure, legislative intent, and subject-area structures and legal-ethical principles.Objective-teleological interpretation is defined without recourse to the legislator’s regulatory intent.Source: §2.2 · pp. 3–4

3. From Definition to a Sentence-Level Task

The paper uses sentences as measurable components of interpretation and organizes the resulting classification tasks into a staged pipeline. Sentence-level labels can capture multi-sentence support, while reconstructing full arguments and end-to-end evaluation are outside the reported scope.

3.1. Unit of Analysis

  • Free-span extraction could capture exact reading and support boundaries, but those boundaries need not align with sentence or paragraph boundaries.The paper instead uses sentence-level classification as a concise, exhaustive measurement unit.Source: §3.1 · p. 4
  • Classifying every sentence against every applicable criterion fixes the answer set and makes missed arguments count as false negatives.This addresses the risk that models overlook arguments when asked to extract an unbounded number of spans.Source: §3.1 · p. 4
  • A sentence is a measurement unit, not an assumption that an interpretive argument fits within one sentence.Roughly one-third to nearly half of readings, depending on canon, are supported by more than one argument sentence; clustering sentences to reconstruct full arguments is outside scope.Source: §3.1 · p. 4

3.2. Pipeline

  • The pipeline evaluates each sentence on successive reading criteria, then classifies reading–candidate sentence pairs as arguments and, conditionally, by canon.To limit quadratic pair growth, candidates fall within five sentences before and seven after a reading; this window covers about 90% of annotated pairs, while positives outside it are retained.Source: §3.2 · p. 4
  • The benchmark evaluates pipeline stages in isolation; end-to-end runs over full decisions, where upstream errors propagate, are supported by the release but not pursued.The pipeline defines both the annotation scheme and benchmark structure.Source: §3.2 · pp. 4–5

4. Dataset

The dataset applies a sentence-level annotation scheme to decisions of the German Federal Constitutional Court, with labels assigned through a cascade of reading and argument criteria. Its scope and annotation design constrain which sentences count as negatives and which claims the results can support.

Source and scope

  • Annotations cover 28 official German Federal Constitutional Court decisions, but only 15 are exhaustive; the other 13 record positive canon occurrences selectively.Unannotated sentences in the selective decisions are excluded, not treated as negatives.Source: §4 · p. 5

Subtasks

  • Eight subtasks separate three reading criteria, an argument-support gate, four canon classifications, and statutory-reference extraction.Argument labels apply to a candidate sentence paired with the reading it supports; passing the argument gate does not require a positive label for any canon.Source: §4 · p. 5

Class sizes

  • The reading criteria and argument subtasks each contain 400 sentences and 200 sentence pairs, respectively, with 100/300 and 50/150 positive/negative splits.The argument pool is smaller because valid readings are a bottleneck and many receive no explicit surrounding argument; its positive share is deliberately raised for rarer classes.Source: §4 · p. 5

Splits and sentence eligibility

  • All splits are stratified and decision-disjoint, with prompt-example decisions additionally held out from the corresponding test split.The dataset excludes non-evaluable fragments such as bare enumerators, which otherwise could inflate positives for the not-abstract subtask.Source: §4 · pp. 5–6

Annotation limitations

  • A single PhD-level law annotator produced all labels, so the dataset has no inter-annotator agreement measure.Whether a sentence asserts a statutory interpretation rather than restating statutory text is not a standalone target because historical German statutory texts are not available at scale; potential restatements are flagged for later checking.Source: §4 · p. 6

Release

  • The release provides decision-level records for end-to-end evaluation and instance-level records organized by subtask and split, with labels, reasoning, and context.The paper does not pursue end-to-end evaluation of the decision-level format.Source: §4 · p. 6

5. Baseline Evaluations

The baseline evaluation compares four reasoning models under expert-written and GEPA-optimized prompts. Models receive contextualized classification instances, and performance is assessed with task-specific metrics.

Model selection

  • Four reasoning models from three model families are evaluated: DeepSeek-V4 Pro and Flash, Gemini 3.5 Flash Lite, and MiniMax M3.DeepSeek-V4 and MiniMax M3 are open-weight; Gemini 3.5 Flash Lite is proprietary, and all models run at temperature 1.0.Source: §5 · p. 6

Prompt conditions

  • The evaluation compares a non-model-specific expert prompt with model-specific prompts optimized by GEPA, starting from the expert prompt.GEPA uses DeepSeek-V4-Pro as its reflection model for all evaluated models, with static optimizer settings.Source: §5 · p. 6

Prompt optimization

  • GEPA optimizes binary subtasks for accuracy and statutory-reference extraction for sample-averaged F1, using a 20/20/60 search, validation, and test split.The remaining 60% is held out for a single test evaluation; the majority test allocation is intended to maximize evaluation reliability given the annotation budget.Source: §5 · pp. 6–7

Inputs and outputs

  • Each input embeds the target sentence or reading–argument pair in context, and each model returns reasoning plus a structured binary label or statutory-provision list.Context spans two sentences on either side, expanded to ten preceding sentences for statutory references.Source: §5 · p. 7

Evaluation metrics

  • Binary subtasks are scored with positive-class F1, while statutory references use set-overlap F1; a language-model judge matches predicted citations to gold provisions.The judge tolerates formatting variants but requires exact statutory subdivisions; reported confidence intervals are 95% bootstrap intervals from 1,000 instance-level resamples and do not account for within-decision dependence.Source: §5 · p. 7

6. Results

Across the seven binary subtasks, the four models cluster between 70.4 and 79.2 mean F1, with no model family dominating. GEPA does not systematically outperform expert prompts, while grammatical interpretation scores highest and systematic interpretation lowest.

6. Results

  • GEPA mean F1 was 75.4 vs. 72.7 for Gemini 3.5 Flash Lite, 79.2 vs. 75.2 for MiniMax M3, 70.4 vs. 73.5 for DeepSeek-V4-Flash, and 75.1 vs. 75.2 for DeepSeek-V4-Pro.The values are GEPA versus expert-prompt means; under the tested configuration, the expert prompt remains a meaningful baseline.Source: §6 · p. 7
  • Grammatical interpretation had the highest mean F1 (82.9), while systematic interpretation had the lowest (58.4); historical and objective-teleological means were 70.2 and 70.9.Grammatical was best in seven of eight model–prompt conditions, and systematic was lowest in six of eight.Source: §6 · p. 7
  • Four models’ mean F1 across seven binary subtasks spans 70.4–79.2, with no model family consistently leading.The table reports per-subtask positive-class F1 for all four models under expert and GEPA prompts.Source: §6 · p. 7, Table 1 · p. 8
  • Statutory reference is a citation-extraction target scored by per-document set-overlap F1, not binary F1; its span-level F1 is much lower.Source: §6 · p. 8

7. Error Analysis

The error analysis identifies missed multi-sentence arguments and shallow surface cues as major sources of false negatives and false positives.

7. Error Analysis

  • Models miss continuing or concluding sentences in multi-sentence arguments when those sentences lack canon-specific cues, reducing recall for gradual objective-teleological arguments.Source: §7 · p. 8
  • False positives arise when models treat restatements or consequences as grounds and rely on proxies such as “Wortlaut,” any distinct provision, or any section number.Systematic interpretation’s proxy can misfire in both directions: it may react to parallel citations and miss arguments that name no second provision.Source: §7 · p. 8

8. Discussion and Limitations

The results are bounded by annotation, corpus, and evaluation constraints, including small test sets that yield wide confidence intervals.

8. Discussion and Limitations

  • All gold labels come from one annotator, preventing measurement of inter-annotator agreement, and the corpus covers one court in one language.The authors make no claim of transfer to other jurisdictions or legal traditions.Source: §8 · p. 8
  • Wide confidence intervals reflect small test sets, especially for rarer argument subtasks, so prompt-condition differences should not be over-interpreted.The authors identify extending the corpus with multiple annotators as the most direct remedy.Source: §8 · p. 8

9. Conclusion

This section presents a sentence-level framework for classifying interpretive canons in German Federal Constitutional Court decisions, including criteria for identifying the canons, workflow rules for interpretations and supporting arguments, and a prompt for objective-teleological interpretation. It also summarizes the benchmark’s baseline findings on canon classification and GEPA-optimized prompts.

9. Conclusion

  • Grammatical interpretation was usually easiest and systematic interpretation usually hardest to identify, while tested GEPA prompts did not systematically outperform expert-written prompts in mean binary F1.The authors treat the canon-difficulty pattern as plausible in light of legal methodology and the expert prompts as a meaningful baseline.Source: §9 · p. 8
  • The paper operationalizes Larenz’s four classical canons in the Savigny tradition and releases an expert-annotated dataset of German Federal Constitutional Court decisions.Fifteen decisions are exhaustively labeled, with selective extensions for rarer classes.Source: §9 · p. 8
  • The benchmark shifts canon classification to the sentence level, addressing prior work’s reliance on paragraph-level analysis.Source: §9 · p. 8
  • Wide confidence intervals and the dataset’s modest size make these findings preliminary; the authors call for corpus expansion and better evaluations.Source: §9 · p. 8

B.2. Not self-advanced (nicht_selbst_aufgestellt)

  • A claim is not self-advanced when its sentence cites only precedents; without sources, or with any other source type, it is self-advanced.The decision is based on source type, not claim content, and only sources cited in the marked sentence are considered.Source: §B.2 · p. 15

B.3. Statutory reference (konkretes_gesetz)

  • A statutory interpretation must specify the content of a particular provision so that some cases fall within it and others do not; whole-norm validity or compatibility assessments do not qualify.The provision may be identified in context, but general references to a statute, legal principles alone, and proportionality or constitutionality assessments are insufficient.Source: §B.3 · pp. 15, 17

B.4. Argument gate (argument)

  • A potential argument counts as support only when it is directly and logically linked to the interpretation and follows the same line of reasoning; it may argue either for or against it.The argument may appear before or after the interpretation; thematic similarity alone is not enough.Source: §B.4 · p. 18

B.5. Grammatical interpretation (wortlaut)

  • The larger study tracks how interpretive methods in Federal Constitutional Court decisions change over time and defines interpretation as choosing among possible readings of a statutory provision.The workflow aims to record where each decision interprets using four criteria; this part identifies the task as one step in that larger workflow.Source: §B.5 · p. 18
  • Grammatical interpretation concerns the ordinary or specific linguistic meaning of wording in the interpreted statutory provision, not merely the law’s name or another provision.A quotation can qualify when its interpretive conclusion appears in another sentence.Source: §B.5 · p. 19

B.6. Systematic interpretation (systematik)

  • Systematic interpretation uses a specific, distinct statutory provision to explain or deepen the interpretation, while general principles, whole laws, and unspecified provisions do not qualify.The provision may appear in a different sentence and may support a contrary position.Source: §B.6 · pp. 19–20

B.7. Historical interpretation (geschichte)

  • Historical interpretation applies when an argument invokes the regulatory intent of the legislature that enacted the norm; ordinary judicial reasoning about a law’s meaning is insufficient.The text treats legislative materials as evidence of intent, but says references to a norm’s significance alone do not establish that intent.Source: §B.7 · p. 21

B.8. Objective-teleological interpretation (zweck)

  • Objective-teleological interpretation derives a norm’s objective purpose from the current legal order and general principles, without relying on historical legislative intent.The examples distinguish reasoning from current purposes from arguments relying on legislative history or merely relating one tax to another.Source: §B.8 · p. 22
  • The prompt defines the relevant norm as the law being interpreted and the context as the surroundings in which the interpretation and argument occur.It also defines the textual environment as the context from which the interpretation was drawn and which may justify it.Source: §B.8 · p. 22
  • The classifier is instructed to justify its answer before deciding whether the argument sentence matches the criterion’s definition and examples.The supplied text frames the decision as whether an objective-teleological argument is present.Source: §B.8 · pp. 22–23
  • The criterion requires an argument about a norm’s purpose or goal without historical references, especially emphasizing its current purpose in today’s legal order.Source: §B.8 · p. 22

C. Dataset Record Schema

The appendix defines a common JSON record structure and shows how it represents argument pairs and individual readings for different subtasks.

C. Dataset Record Schema

  • Each record is a JSON object with an integer id, a row containing the classified sentence or sentences and context, and observed gold labels.Record identity is recoverable from provenance fields: argument records use decision name, reading id, and candidate start offset; reading records use decision name and sentence id.Source: C. Dataset Record Schema · p. 24

C.1. arguments (example from subtask wortlaut)

  • Argument records pair a reading with a candidate justification, while observed stores the argument gate and canon labels.The example identifies the reading and candidate in the row and gives binary labels and a reasoning string in observed.Source: §C.1 · p. 24

C.2. deutungen (example from subtask konkretes_gesetz)

  • Reading records represent one marked sentence; observed includes the concrete provisions it interprets and the reading-criteria labels.For konkretes_gesetz, the provision list is empty when none is identified, and the reading-criteria subtasks use boolean fields in observed.Source: §C.2 · pp. 24–25

D. Reproducing the Scores and Benchmark Datasets

The authors test whether agents can reconstruct the paper’s scores and benchmark datasets from released inputs and written procedures, without access to the implementation. Agents reproduce both outputs exactly.

D. Reproducing the Scores and Benchmark Datasets

  • Agents independently implement each described procedure without the authors’ implementation, and withheld verifiers compare their outputs with reference results.For score reproduction, the agent receives model predictions, judge decisions, test gold labels, and metric definitions, but no scoring code or reported numbers.Source: D. Reproducing the Scores and… · p. 25
  • All 208 score point estimates are reproduced exactly across four models, two prompt conditions, seven binary subtasks, and four statutory-reference metrics.The scores are determined by the Section 5 protocol and published predictions, judge decisions, and gold labels; bootstrap intervals are excluded from the exact-match check because they depend on the resampling seed.Source: D. Reproducing the Scores and… · p. 25
  • All 2,200 benchmark records across eight subtasks are reproduced exactly, including field values, split membership, and identifiers.The agent uses raw annotation sources and the specified construction procedure, but neither the construction code nor the reference datasets.Source: D. Reproducing the Scores and… · p. 25

E. Benchmark Construction (Methods Appendix)

The appendix specifies a deterministic pipeline for turning annotation exports into benchmark records and splits. It defines the benchmark’s two record families, their labeling rules, and common split constraints.

E. Benchmark Construction (Methods Appendix)

  • A fixed-seed CPython random generator and fixed draw order make the construction reproducible down to record order and identifiers.The appendix states that the same source files yield the same files, fields, and ids; it specifies MT19937 with seed 42 and the standard-library operations used.Source: E. Benchmark Construction (Methods… · p. 26, §E.3.1 · p. 27

E.1. What the benchmark contains

  • The benchmark has eight subtasks, each with train, validation, and test files, organized as five argument tasks and three reading tasks.Argument records are reading–candidate-sentence pairs; reading records represent a single annotated sentence.Source: §E.1 · p. 26

E.2. Source files

  • Construction draws fields from distinct exports, while a separate decision-name list identifies selectively annotated decisions.The structure export supplies sentence text, offsets, context, and reading/argument structure; the raw and merged exports supply designated labels, and the raw export preserves the provision list as separate strings.Source: §E.2 · p. 26, §E.2.1 · p. 26, §E.2.2 · p. 27

E.3. SENTENCE USABILITY (IS_EVALUATED)

  • Sentence fragments are excluded unless they exceed five characters and contain an alphabetic letter; usable sentence text is joined into context windows around marked targets.Window sizes vary by task: reading and argument windows use two sentences before and after, while konkretes_gesetz uses ten before and two after.Source: §E.3.3 · p. 28, §E.3.4 · p. 28

E.4.1. THE K O N KRETES_GESETZ POOL (BUIL D_D E U T U N G_E N T R I E S)

  • For konkretes_gesetz, a reading is positive when its provision list is non-empty; eligible negatives require exhaustive annotation, both reading criteria true, and a determinate Yes/No judgment.Negative kinds distinguish “kein_bestimmter_inhalt” from “keine_konkrete_gesetzesbestimmung.”Source: §E.4.1 · p. 28

E.5. The argument family: the pair pool

  • The argument pool keeps candidate pairs linked to qualifying readings with a non-empty provision list, deduplicates by decision, reading, and candidate offset, and favors individually judged collisions.The argument task labels pairs by the general-argument flag; each canon task considers only argument-positive pairs, excluding null canon labels from negatives.Source: §E.5 · p. 29, §E.5.1 · p. 29

E.6. The reading-criteria family: nicht_abstrakt, nicht_selbst_aufgestellt

  • The two reading-criteria tasks use exhaustively annotated decisions and flip the base criterion polarity, making “No” the positive class.Records retain both base and flipped encodings for abstrakt and selbst_aufgestellt, with null values when the underlying answer is not Yes or No.Source: §E.6 · p. 29, §E.6.2 · p. 30, §E.6.3 · p. 30

E.7. Selecting, splitting, ordering, and numbering

  • All subtasks use decision-disjoint splits and keep decisions quoted in worked-example prompts out of the corresponding test sets.The appendix describes the remaining selection, splitting, ordering, and numbering steps as distinct procedures for the two benchmark families.Source: §E.7 · p. 30

E.7.1. K O N K R E T ES_GESETZ: SPLIT, THEN SAMPLE PER NEGATIVE KIND

The konkretes_gesetz split-and-sample procedure assigns whole decisions to splits, then selects records to meet positive and negative targets while preserving negative-kind proportions. A fixed random stream determines sampling and file order.

E.7.1. K O N K R E T ES_GESETZ: SPLIT, THEN SAMPLE PER NEGATIVE KIND

  • For konkretes_gesetz, targets are 60 positive and 180 negative test records, and 20 positive and 60 negative records each for train and validation.The subtask uses the reading-family targets, not the smaller argument-family targets.Source: §E.7.1 · p. 30
  • Each split preserves the negative pool’s proportions of its two kinds, assigning the rarer kind a target based on its share of all negatives.A fixed 300,000-iteration random-restart search assigns pool decisions to splits, enforcing positive, negative, and per-kind minimum targets and retaining the assignment with greatest minimum slack.Source: §E.7.1 · pp. 30–31

E.7.2. K O N K R E T ES_GESETZ: NUMBERING

  • A seeded sampler processes test, train, then validation, sampling positives and each negative kind before shuffling each split; ids follow that final file order.The search uses one split draw per pool decision, with forbidden-from-test decisions reassigned to train or validation.Source: §E.7.1 · p. 31, §E.7.2 · p. 31

E.7.3. THE ARGUMENT FAMILY (ARGUMENT AND THE FOUR CANONS): SPLIT AND SAMPLE

The argument subtasks use a random-restart split search and a sampler that prioritizes individually judged negatives. The two reading-criteria subtasks instead use exhaustive decision-disjoint splitting, a different sampler, and global numbering.

Argument family split and sample

  • Argument splits use random-restart search, with feasibility requiring every individually judged negative to fit within each split’s negative target.The search uses 300,000 iterations and maximizes minimum slack; the canon subtasks do not impose the individually judged-negative constraint.Source: §E.7.3 · p. 31
  • For argument sampling, individually judged negatives are shuffled first and placed ahead of other negatives before the target count is taken.Positive and negative records are then concatenated and shuffled within each split.Source: §E.7.3 · p. 32
  • Argument records include each selected pair’s reasoning, and records are numbered sequentially within each split file.Argument reasoning comes from general_argument.reasoning; canon reasoning comes from that canon’s reasoning.Source: §E.7.3 · p. 32

Reading-criteria family

  • The two reading-criteria subtasks use exhaustive decision-disjoint split search, with assignments selected to maximize minimum positive and negative slack.Feasible assignments are searched deterministically, with ties resolved by enumeration order.Source: §E.7.3 · p. 32
  • Reading-criteria records are sampled with a single seeded generator and numbered globally across concatenated test, train, and validation lists.Within each split, positive and negative pools are sorted by region id before sampling; the combined list is shuffled before ids are assigned.Source: §E.7.3 · p. 32

E.8. The finished benchmark

The finished benchmark consists of 24 JSON files organized by subtask and split, covering eight subtasks.

E.8. The finished benchmark

  • The finished benchmark comprises 24 train, validation, and test JSON files across eight subtasks.Each file is an array of records containing id, row, and observed fields.Source: §E.8 · p. 32
Loading 2609.26945v1…