Source-linked AI summary
DAR: Deontic Reasoning with Agentic Harnesses
Guangyao Dou, William Jurayj, Nils Holzenberger, Benjamin Van Durme
TL;DR
Deontic reasoning applies explicit rules to case-specific facts, but long, cross-referenced statutes make relevant rules difficult to locate. The paper introduces DAR, which lets models inspect statutes on demand through agentic harnesses, and finds that frontier models improve while weaker models degrade and consume more tokens.
Problem
Long, cross-referenced rulesets make it difficult for models to locate the relevant provisions needed for deontic reasoning.
Method
DAR places statutes in a harness environment and lets models examine them on demand through tools, evaluating the setup across multiple harnesses and DeonticBench tasks.
Results
Agentic harnesses improve frontier models but degrade weaker open-source models; on SARA-Numeric, frontier models gain 18–30 points while Qwen3.5-35B falls from 34% to 11%.
Takeaways & Limitations
DAR can push the frontier on hard deontic reasoning tasks for sufficiently capable models, but interactive tool access does not ensure weaker models use it well.
Takeaways & Limitations
The evaluation uses general-purpose harnesses and a single-file statute design, which may not scale to substantially longer rulesets or reflect statute-specific harnesses.
Abstract
from arXiv · showhide
Deontic reasoning is the task of answering questions by applying explicit rules and policies to case-specific facts, for example computing tax liability under a statute or determining the outcome of an immigration appeal. A key technical challenge for LLM-based deontic reasoning is that the relevant ruleset can be long and cross-referenced, so models may still fail to locate the rules needed for a particular reasoning step. We introduce Deontic Agentic Reasoning (DAR), an agentic reasoning setup in which the model interacts with the statutes on demand. We evaluate DAR under multiple harnesses on hard subsets of DeonticBench. Across these settings, we find that agentic harnesses can push the frontier on deontic reasoning tasks, but improvements are not uniform: weaker models often degrade on numerical tasks while consuming far more tokens.
1 Introduction
Deontic reasoning requires applying explicit rules to case-specific facts, but long, cross-referenced statutes make relevant provisions difficult to locate. DAR evaluates whether on-demand statute access through agentic harnesses improves this process, with effects that vary by model capability.
- Deontic reasoning applies explicit rules and policies to case-specific facts in domains including tax computation and policy compliance.
- Long, heavily cross-referenced rulesets contain many irrelevant provisions and qualifications elsewhere, making relevant rules difficult to locate.
- DAR places the statute in a harness file so models can examine it on demand, and evaluates this setup on DeonticBench tasks spanning tax, immigration, and airline policies.
- Agentic harnesses improve frontier models but degrade weaker models, with frontier models recovering from intermediate errors while weaker models spend more tokens on wrong answers.
- On SARA-Numeric, frontier models gain 15–30% under Terminus-KIRA, while open-source models in the same harness degrade by 11–23%.
- Qwen3.5-35B drops from 34% to 11% on SARA-Numeric, while every open-source model collapses to near-zero on Airline and consumes up to 4× more tokens per trial.
2 Deontic Agentic Reasoning
The paper contrasts direct reasoning, which supplies the full statute in one prompt, with DAR, which lets the model query a harness-hosted statute during reasoning. DAR accumulates tool observations as the agent explores the statute.
- Direct reasoning: Direct reasoning gives the model the full statute, case facts, and question in one prompt and requires a one-pass answer.
- Deontic Agentic Reasoning (DAR): DAR stores the statute as statute.txt in a harness while providing the model with case facts, the question, and tool instructions.
- Deontic Agentic Reasoning (DAR): In DAR, the model issues tool calls to read targeted statute portions on demand and may execute Python for numeric computation.
- Deontic Agentic Reasoning (DAR): Each tool call appends an observation to the context, allowing the agent to accumulate observations while exploring the statute.
3 Experimental Setup and Results
The experiments compare direct solving with two agentic harnesses across four hard DeonticBench tasks and nine models. Harnesses improve frontier-model performance but can sharply reduce open-source accuracy while increasing token use.
- Experimental Setup: The evaluation covers SARA-Numeric, SARA-Binary, Airline, and USCIS-AAO across nine open-source and proprietary models.SARA-Numeric and Airline use accuracy, while SARA-Binary and USCIS-AAO use macro-F1.
- Harnesses: The main comparison evaluates Direct Solving against Terminus-2 and Terminus-KIRA, which let models inspect statutes interactively in a sandbox.Terminus-KIRA extends Terminus-2 to target premature submission and poor self-evaluation.
- Frontier Models: GPT-5.2 rises from 30% to 60% on SARA-Numeric under Terminus-KIRA, while Claude Sonnet 4.5 rises from 36% to 54%.GPT-5.1 gains an additional 15 percentage points on SARA-Numeric and remains near 0.86 on Airline.
- Open-Source Models: Qwen3.5-35B drops from 34% to 11% on SARA-Numeric under Terminus-KIRA, and open-source models collapse to near-zero on Airline under either harness.The classification tasks show smaller degradations without a consistent open-source gain.
- Efficiency: Qwen3.5-122B averages 401k tokens per trial and Qwen3-235B 303k under Terminus-2, roughly 4× the frontier-model consumption.Agentic harnesses append each action's output to the next input, increasing inference cost relative to direct solving.
4 Related Work
Related work studies agentic search over corpora and benchmarks for rule-based reasoning. DAR differs by testing reasoning from outside a provided statute while enabling direct statute interaction.
- Harness-based Agentic Search: Prior work interleaves reasoning with search through fixed retriever interfaces, while direct corpus interaction uses general-purpose terminal tools.Direct corpus interaction has been reported to improve agentic search and information-retrieval benchmarks over conventional retrievers.
- Deontic Reasoning Datasets: DeonticBench tests reasoning from outside a specific provided statute, unlike CL-bench's test of operating inside a rule system by following its rules.Earlier benchmarks also focus on multi-step entailment and first-order-logic reasoning.
5 Conclusion
The paper introduces Deontic Agentic Reasoning and finds that harnesses improve frontier models on hard deontic tasks but do not help uniformly. Open-source models degrade while consuming substantially more tokens.
- Conclusion: DAR enables models to access statutes on demand through agentic harnesses rather than receiving them solely in context.The conclusion frames this as a way to expose performance left unused by static long-context prompts for sufficiently capable models.
- Conclusion: Agentic harnesses push the frontier on the hardest deontic reasoning tasks, but frontier models gain while open-source models degrade.The reported pattern is explicitly non-uniform across model capability.
- Conclusion: Open-source models consume up to 4× more tokens under the evaluated harnesses.The token increase accompanies degradation rather than a corresponding performance gain.
Limitations
The evaluation is bounded by statute scale, benchmark and domain coverage, harness coverage, and reasoning-effort settings. These boundaries limit how broadly the findings can be generalized.
- Scalability of DAR: The current DAR implementation may not scale efficiently to substantially longer rulesets because agents must navigate the entire statute file with general-purpose tools.The paper suggests hierarchical statute lookup or learned section-level retrieval as more scalable alternatives.
- Benchmark and domain coverage: The findings come only from DeonticBench’s U.S. tax, immigration, and airline-baggage domains, so replication on larger benchmarks would strengthen generality.Real-world deontic reasoning spans additional domains with different structural properties.
- Harness coverage: The four evaluated harnesses do not include statute-specialized systems with provision-aware navigation or built-in cross-reference tools.Such harnesses might change the observed capability-amplification picture for weaker models.
- Reasoning-effort settings: GPT-5.1 and GPT-5.2 were evaluated with reasoning effort set to none, so other settings may change frontier performance or the frontier–open-source gap.The direction of that change is not established by the paper.
Ethics Statement
The paper emphasizes that deontic reasoning concerns high-stakes decisions and that harnesses are not universal fixes. Current systems can remain inaccurate, while weaker models may become more confident, less accurate, and more computationally expensive.
- High-stakes use: Errors in tax computation and immigration appeal prediction can carry real costs, and frontier models achieve only partial accuracy even with agentic harnesses.The paper cautions against autonomous deployment in legal, tax, and other high-stakes deontic contexts.
- High-stakes use: The evaluated systems are research artifacts rather than substitutes for qualified human professionals.
- Responsible deployment: Agentic harnesses can amplify capabilities rather than uniformly improve performance, with weaker models producing more confident but less accurate outputs.The paper frames this as a reliability concern for responsible deployment.
- Responsible deployment: For weaker models, harnesses can also consume substantially more compute while degrading accuracy, creating both reliability and computational-cost implications.
B.1 Claude Code and Codex CLI
Claude Code provides a strong scaffold for open-source Qwen models, especially on numerical tasks, while Codex CLI adds relatively little structure and often remains near direct-prompting behavior. Terminus-KIRA is strongest for frontier and the largest openweight models.
- Claude Code: Claude Code achieves the highest SARA-Numeric accuracy for three of four evaluated Qwen models, with Qwen3.5-397B as the exception.
- Claude Code: Claude Code is the only evaluated harness recovering non-trivial Airline accuracy on open-source models, reaching 0.050–0.113 across four Qwen models.Codex and Terminus-2 produce near-zero Airline accuracy for those models.
- Claude Code: Direct prompting remains a strong baseline that Claude Code does not uniformly beat for weaker models.
- Codex CLI: Codex CLI generally yields lower SARA-Numeric accuracy than alternative harnesses and near-zero Airline accuracy on open-source models.On classification tasks, Codex is broadly competitive.
- Terminus-KIRA: Terminus-KIRA is strongest for frontier models and the largest openweight models, including GPT-5.2’s 0.600 SARA-Numeric and 0.363 Airline accuracy.
B.2 Recursive Language Models
The Recursive Language Models setup substantially harms performance on numerical and Airline tasks, while remaining comparatively effective on the closed-class SARA-Binary task. Its degradation is strongest where the base model is strongest.
- Overall comparison: RLMs are the weakest setting for every model on SARA-Numeric and Airline, with the most severe effects on stronger base models.
- Numerical and Airline tasks: For GPT-5.1, Airline accuracy falls from 0.863 under direct prompting and 0.889 under Kira to 0.125 under DSPy RLM.
- Numerical and Airline tasks: For GPT-5.1, SARA-Numeric accuracy drops from 0.692 to 0.114 under DSPy RLM.
- Closed-class classification: On SARA-Binary, DSPy RLM can remain competitive: Qwen3-Coder-480B scores 0.697 versus 0.591 under direct prompting.
- Closed-class classification: Qwen3.5-122B is an exception, with DSPy RLM underperforming both other settings on SARA-Binary.
C Error Analysis
Error patterns differ sharply by model category and harness: closed-source models are highly reliable, while open-source failures are dominated by timeouts whose rates vary substantially across harnesses.
- Aggregate reliability: 12.1% aggregate error affects open-source models, compared with 0.7% for closed-source models.Open-source failures are primarily timeouts, while closed-source models have no runtime or parsing failures.
- Failure modes: 10.6% of open-source trials are timeouts, with parsing failures at 1.5% and runtime errors essentially negligible.Timeouts occur when the agent exceeds the ten-minute trial budget without producing an answer.
- Interpretation: Harness loops and timeout budgets interact with model latency more than with model capability in the observed open-source error rates.The authors attribute most observed instability to open-source models exceeding harness time limits rather than intrinsic agent or model failures.