Source-linked AI summary
DeonticBench: A Benchmark for Reasoning over Rules
Guangyao Dou, Luis Brena, Akhil Deo, William Jurayj, Jingyu Zhang, Nils Holzenberger, Benjamin Van Durme
TL;DR
LLMs still struggle to reason reliably over complex, context-specific rules in high-stakes domains. DEONTICBENCH addresses this gap with a 6,232-task, multi-domain benchmark supporting direct reasoning and solver-assisted Prolog execution. Across evaluations, hard-subset performance remains weak and training produces limited reliable end-to-end gains.
Problem
Existing benchmarks provide limited evidence about long-context, high-stakes deontic reasoning over obligations, permissions, and prohibitions under explicit rules.
Method
DEONTICBENCH combines 6,232 tasks across four rule domains with direct language evaluation, solver-assisted Prolog execution, and reference programs.
Results
Hard-subset performance remains weak across frontier and coding models, and supervised fine-tuning and reinforcement learning yield limited end-to-end gains.
Takeaways & Limitations
The benchmark supports studying context-grounded rule reasoning and the reliability of executable reasoning structures across symbolic and non-symbolic settings.
Takeaways & Limitations
Current RLHF-based methods remain insufficient for robust rule-grounded reasoning, and the benchmark is released for research rather than operational decision-making.
Abstract
from arXiv · showhide
Reasoning with complex, context-specific rules remains challenging for large language models (LLMs). In legal and policy settings, this manifests as deontic reasoning: reasoning about obligations, permissions, and prohibitions under explicit rules. While many recent benchmarks emphasize short-context mathematical reasoning, fewer focus on long-context, high-stakes deontic reasoning. To address this gap, we introduce DEONTICBENCH, a benchmark of 6,232 tasks across U.S. federal taxes, airline baggage policies, U.S. immigration administration, and U.S. state housing law. These tasks can be approached in multiple ways, including direct reasoning in language or with the aid of symbolic computation. Besides free-form chain-of-thought reasoning, DEONTICBENCH enables an optional solver-based workflow in which models translate statutes and case facts into executable Prolog, leading to formal problem interpretations and an explicit program trace. We release reference Prolog programs for all instances. Across frontier LLMs and coding models, best hard-subset performance reaches only 44.4% on SARA Numeric and 46.6 macro-F1 on Housing. We further study training with supervised fine-tuning and reinforcement learning for symbolic program generation. Although training improves Prolog generation quality, current RL methods still fail to solve these tasks reliably. Overall, DEONTICBENCH provides a benchmark for studying context-grounded rule reasoning in real-world domains under both symbolic and non-symbolic settings.
1 Introduction
DEONTICBENCH targets deontic reasoning over explicit rules in high-stakes, long-context domains. It evaluates direct language reasoning and solver-assisted Prolog workflows, finding persistent difficulty and limited gains from training.
- The benchmark evaluates obligations, permissions, and prohibitions under formal statutes and policies, addressing a gap in high-stakes rule reasoning.
- DEONTICBENCH introduces 6,232 tasks spanning federal taxes, airline baggage policies, immigration administration, and state housing law.
- Its symbolic workflow translates statutes and case facts into executable Prolog, producing formal interpretations and explicit program traces.
- Eight general-purpose and three coding models struggle on hard subsets, while increased reasoning effort does not consistently improve performance.
- Supervised fine-tuning and reinforcement learning improve performance but yield limited end-to-end accuracy gains.
2 DEONTICBENCH Dataset Construction
DEONTICBENCH unifies four real-world rule domains into symbolic and non-symbolic tasks, adds on-the-fly Prolog generation and reference programs, and constructs validated hard subsets for focused evaluation.
- Domains and tasks: The benchmark covers federal taxes, airline baggage fees, state housing law, and immigration appeals with numeric, entailment, and multifactor decision tasks.
- Dataset construction: DEONTICBENCH combines three existing domains with a newly constructed USCIS-AAO dataset and converts them into a unified task format.
- Dataset construction: Unlike original SARA, each instance requires generating Prolog from the statute and case context without a pre-encoded background-rule program.
- USCIS-AAO: USCIS-AAO contains 242 balanced cases, with 121 Accepted and 121 Dismissed labels; remanded cases are classified as accepted.
- Prolog generation pipeline: The pipeline generates statute modules per instance, compile-checks them in SWI-Prolog, and accepts artifacts only when execution and the final answer are correct.
- Hard set construction: Hard subsets are built through automated filtering and iterative human validation, then divided into held-out evaluation and training portions.
- Dataset statistics: Mean Prolog length increases on hard subsets in four domains, including 945 to 1236 for SARA Numeric and 884 to 956 for USCIS-AAO.
3 Experimental Setup
The experiments evaluate frontier and open-source models with direct, zero-shot, and few-shot prompting under domain-specific metrics and repeated sampling on hard subsets.
- Models: The evaluation includes frontier and open-source models such as GPT-4.1, GPT-5.1, GPT-5.2, O3, Claude 4.5 Sonnet, Gemini 2.5 Flash, Kimi K2, and Qwen3-235B.
- Prompting strategies: Direct prompting predicts answers without Prolog, whereas zero-shot and few-shot prompting generate executable Prolog programs under domain-specific constraints.
- Metrics: SARA Numeric and Airline use accuracy with a $1 solver-output tolerance, while SARA Binary, Housing, and USCIS-AAO use macro-F1.
- Evaluation protocol: Hard-subset results use 95% bootstrap confidence intervals and multiple generations per case to capture output stochasticity.
4 Results
Across DEONTICBENCH’s hard subsets, performance varies by domain, model, and prompting strategy, with systematic trade-offs between coverage and reliability. Fine-tuning improves classification more than numeric reasoning, while failures reflect domain-specific rule, fact, and arithmetic bottlenecks.
- Main Results: Prompting effects vary by domain: few-shot helps Airline but not SARA Numeric, while zero-shot and direct solving often outperform Housing on binary tasks.
- Main Results: Stronger frontier models perform best overall, but none is consistently best across domains or settings, and confidence intervals remain wide.
- Main Results: Qwen3-235B rises from 0.7 on SARA Numeric with few-shot prompting to 32.1 under direct prompting, yet open-source models remain less reliable for exact rule-based computation.
- Failure Analysis: Prolog solving produces higher abstention rates, whereas direct prompting reduces abstentions but increases incorrect predictions, exposing a coverage–reliability trade-off.
- Locally Trained Models: Fine-tuning improves classification but not numeric reasoning: few-shot USCIS-AAO rises from 10.3 to over 45, while SARA Numeric remains below 10.
- Locally Trained Models: DPO and Dr. GRPO generally outperform SFT alone, but gains vary across domains and prompting strategies and remain limited on challenging reasoning tasks.
- Failure Modes: Rule-selection errors dominate legally complex domains, fact-extraction errors are prominent in SARA, and arithmetic-precision errors bottleneck quantitative domains.
5 Related Work
DEONTICBENCH builds on rule-reasoning and neuro-symbolic research while targeting context-grounded deontic reasoning in high-stakes domains. Relative to nearby benchmarks, it is larger, broader, and provides executable Prolog artifacts for analysis.
- Prior rule benchmarks emphasize formal structure, representation alignment, multi-step entailment, synthetic first-order reasoning, or reasoning processes.
- RuleArena and CL-bench are the closest prior benchmarks to DEONTICBENCH.
- DEONTICBENCH contains 6,232 tasks versus 816 in RuleArena, covers additional high-stakes legal domains, and releases executable Prolog artifacts.
- Unlike setups relying on pre-encoded background rules, DEONTICBENCH provides statutes and facts per instance and includes reference Prolog for every instance.
- The benchmark follows neuro-symbolic approaches that translate natural language into symbolic programs and execute them with external solvers.
6 Conclusion
DEONTICBENCH evaluates long-context rule reasoning across four high-stakes domains in both direct-language and solver-assisted settings. Results remain weak on difficult subsets, and training yields limited reliable end-to-end gains.
- DEONTICBENCH contains 6,232 high-stakes tasks across federal taxes, airline baggage policies, immigration administration, and state housing law.
- The benchmark supports direct language reasoning and solver-assisted execution with reference Prolog programs.
- Frontier reasoning and coding models perform weakly on difficult subsets, while additional reasoning effort does not consistently improve results.
- Supervised fine-tuning and reinforcement learning improve performance, but reliable end-to-end gains remain limited.
Ethics Statement
The supplied passages describe the benchmark’s research-only safety boundary and its executable-program curation and hard-subset construction procedures. They also detail prompts, pipelines, validation, and training-related implementation choices.
- DEONTICBENCH is released strictly for research and evaluation, not operational decision-making or professional legal, tax, or policy advice.
- The benchmark uses direct, zero-shot, and few-shot prompting, including executable Prolog generation from statutes and case facts.
- One pipeline generates statute modules on the fly, compile-checks them in SWI-Prolog, and then generates question programs.
- Artifacts pass only when they compile, execute cleanly, and match the gold label; failed instances can receive a second attempt before discard.
- Human curation checks rule coverage, fact fidelity, executability, and answer agreement before retaining artifacts.
- Potentially hard instances are identified through frontier-model failures, human-reviewed, and split between evaluation and training.
B.3 Detailed Metrics and Bootstrap Procedure
The appendix defines task-specific scoring, abstention handling, bootstrap uncertainty estimation, prompting formats, and training objectives for SFT, DPO, and Dr. GRPO. Coding-model results show strong prompt sensitivity and unstable Prolog generation.
- Metrics: SARA Numeric and Airline use accuracy with a $1 tolerance, while SARA Binary, Housing, and USCIS-AAO use macro-F1.
- Metrics: Abstentions from unusable outputs are counted as incorrect predictions and mapped to task-specific values or classes.
- Bootstrap Procedure: Bootstrap intervals use 1,000 resamples, sampling cases with replacement and one of K outputs per case to capture stochasticity.
- Prompting: Zero-shot prompting asks models to encode statutes and case facts in runnable Prolog and print the requested answer.
- Training: SFT maximizes likelihood of ground-truth Prolog, DPO uses preferred and dispreferred outputs, and Dr. GRPO uses execution-based rewards.
- Coding Models: GPT-5.2-Codex performs best overall but remains prompt-sensitive, while coding-agent generations show substantial variance and frequent failures.
C.2 Impact of Reasoning Effort on Performance
Higher reasoning effort does not reliably improve SARA Numeric hard-case performance, while GPT-5.1 outperforms GPT-5.2 across most benchmark domains. These findings indicate that performance depends on more than general reasoning effort or model capability.
- Performance does not consistently improve with higher reasoning effort on SARA Numeric hard cases, with diminishing or negative returns in some cases.
- GPT-5.1 consistently outperforms GPT-5.2 across SARA Numeric, SARA Binary, USCIS-AAO, and Housing, while remaining competitive on Airline.
C.3 Additional Failure Mode Analysis
The additional failure-mode analysis presents error-bar charts for SARA Binary, Housing, and USCIS-AAO, but the supplied passages do not state a substantive comparison or outcome.
- Additional error-bar charts are presented for SARA Binary, Housing, and USCIS-AAO in Figure 4.
- The supplied passages identify dataset statistics for USCIS-AAO but do not report the corresponding chart results.
D.1 USCIS-AAO Dataset Scope and Preprocessing
The USCIS-AAO dataset is built from appellate immigration decisions and separates factual narratives from legal analysis to support disposition prediction. Its cases are restricted to 2022–2025 to reduce temporal rule changes, but the extraction boundary and quality remain partly dependent on annotation and review.
- USCIS-AAO Dataset Scope: The AAO independently reviews USCIS decisions de novo across facts, law, policy, and discretion, adjudicating appeals, motions, and certifications.
- Temporal Scope: Cases are restricted to 2022–2025 so the governing legal and policy regime remains approximately stable and temporal rule changes are less likely to confound reasoning evaluation.
- Preprocessing: The dataset preserves governing legal authorities in the Law field while extracting facts from the Analysis section, where factual statements often interweave with legal application.
- Preprocessing: Facts-only extraction uses language-model assistance, human verification, secondary review, and partial rewriting to prevent explicit disclosure of the final resolution.
- Limitations: The factual-analysis boundary is partly protocol-determined, while artifact quality depends on extraction accuracy and consistent human review.
E Dataset Examples
The paper illustrates DEONTICBENCH with representative examples from tax, airline, and USCIS-AAO domains, pairing statutes and case facts with questions, labels, and executable Prolog programs. The examples show how domain rules are encoded into queries for numeric calculation, entailment checking, fee minimization, and case disposition prediction.
- Example Structure: Each representative instance presents statutes, a case description, a question, a ground-truth label, and an abbreviated reference Prolog program.
- SARA Numeric: The SARA Numeric example encodes filing status, tax brackets, agricultural employer tax, and total-tax computation from statutory rules and case facts.
- SARA Binary: The SARA Binary example asks whether Section 152(b)(1) applies to Bob, using qualifying-child and married-joint-return rules to determine entailment or contradiction.
- Airline: The Airline example computes total cost by combining ticket price, checked-bag fees, overweight and oversize penalties, and a choice of two complimentary bags.
- Airline: The airline Prolog workflow minimizes total checked-bag cost across possible selections of two complimentary bags before adding the flight cost.
- USCIS-AAO: The USCIS-AAO example represents a delivery-bond appeal involving nonproduction after a written request, a defective Notice to Appear argument, and an accepted-or-dismissed question.