PaperlayerView exact revision on arXiv

Source-linked AI summary

RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications

Jianpeng Zhao, Haihua Xu, Haoyang Zhang, Shuang Qian, Yixiang Tang, Xintao Wang, Kun Sun, Pei Wu, Shuhan Zhong, Pengyang Wang

arXiv:2609.34455v1cs.CL

TL;DR

Rule-governed decisions require models to apply external rules to case facts and justify their conclusions, but answer correctness alone does not assess those grounds. RGDT-Bench provides condition-level warrant supervision and evaluates both completeness and response selection. Warrant supervision improves correct-answer completeness assessment over outcome supervision, while the benchmark also identifies limitations in tested interaction and input settings.

  • Problem

    RGDT reasoning requires applying rules to case facts and justifying decisions, while final-answer accuracy can conceal omissions in those grounds.

    Source: §1 · pp. 1–2

  • Method

    RGDT-Bench uses label-blind claim extraction and deterministic checks against reference warrants to supervise condition-level completeness across four task tracks and eight task–probe combinations.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    10.37 pp: Warrant-RM improves AUROC over Full-response ORM among correct answers, and it leads both outcome-supervised baselines in mean warrant completeness at every candidate count above one.

    Source: §5.4 · p. 9, §5.5 · p. 9

  • Takeaways & Limitations

    RGDT-Bench connects layered failure diagnosis with warrant supervision for reward-model assessment and downstream response selection.

    Source: §6 · p. 9

    The evaluation leaves multi-turn interaction and tool use untested, excludes visual evidence and document layout, and studies sequence-level rather than step-level reward modeling.

    Source: §7 · p. 10

Abstract

from arXiv · show

We study reasoning in Rule-Governed Decision Tasks (RGDTs), where models apply external rules to case facts and justify decisions, as required in policy, contract, and compliance settings. Beyond the deductive capability emphasized by standard mathematical and logical reasoning tasks, RGDTs require interpreting rules and their applicability, assessing conditions from evidence, combining judgments under rules and exceptions, and providing checkable justifications. These demands motivate a benchmark assessing both decisions and their stated grounds. We introduce RGDT-Bench, providing 202.1K condition-level supervision slots across four task tracks and eight supported task-probe combinations that vary access to supporting information. Label-blind extraction and deterministic checks produce labels for warrant completeness: source-referenced coverage and consistency of stated decision grounds. The benchmark attributes failures to four process layers: rule use, condition, evidence, and aggregation, and checks the final outcome. Among evaluable correct responses, warrant incompleteness averages 40.2% across six evaluated LLMs and supported task-probe combinations. Such warrant incompleteness poses potential safety risks and remains difficult to detect: the best of seventeen existing evaluators reaches only 57.69% (random: 50%) task-averaged area under the receiver operating characteristic curve (AUROC). To address this difficulty, we train a simple reward model with warrant supervision. It achieves 69.24% task-averaged AUROC among correct answers, exceeding the matched outcome-supervised baseline by 10.37 pp (percentage points) and the best existing evaluator by 11.55 pp. Beyond completeness assessment, the model outperforms both outcome-supervised baselines across nearly all response-selection comparisons, supporting RGDT-Bench's warrant supervision for RGDT reasoning.

1 INTRODUCTION

Rule-governed decisions require models to apply external criteria to case facts and justify how those facts support an outcome. RGDT-Bench evaluates both decisions and their stated grounds because correct answers can conceal missing or inconsistent justifications.

1 INTRODUCTION

  • RGDTs require interpreting rules, assessing evidence against conditions, aggregating judgments under rules and exceptions, and stating checkable justifications.These demands arise when policy or contract clauses must be linked to differently worded or incomplete case facts.Source: §1 · pp. 1–2
  • 202.1K condition-level supervision slots span four task tracks and eight supported task–probe combinations, with probes varying access to rules and evidence.Label-blind claim extraction and deterministic checks measure stated grounds against source-derived references across rule use, condition, evidence, aggregation, and outcome.Source: §1 · p. 2
  • 24.8%–76.7% of evaluable correct responses have incomplete warrants across six LLMs, and 35.9% of incomplete cases fail at least two process layers.The best of seventeen existing evaluators reaches only 57.69% task-averaged AUROC among correct answers.Source: §1 · p. 2
  • Warrant-RM achieves 69.24% warrant AUROC among correct answers versus 58.87% for a matched outcome-supervised scorer.It also leads both outcome-supervised baselines in mean warrant completeness at every candidate count above one.Source: §1 · p. 2

2 RELATED WORK

Prior work studies rule inference, evidence verification, reasoning-process evaluation, and reward-model assessment, but RGDT-Bench focuses on whether stated decision grounds cover and combine required support.

Rule-based and evidence-grounded reasoning

  • Natural-language rule inference, annotated entailment, evidence verification, and policy or contract tasks address connected components of rule-governed reasoning [Clark et al., 2020; Tafjord et al., 2021; Han et al., 2024; Thorne et al., 2018; Wadden et al., 2020; Sun et al., 2022; Koreeda & Manning, 2021].RuleArena evaluates rule-use precision, recall, and application correctness, while SingGuard applies policy-conditioned rules to multimodal safety moderation [Zhou et al., 2025; SingGuard Team, 2026].Source: §2 · pp. 2–3

Warrants and reasoning coverage

  • Prior warrant and reasoning-coverage work includes implicit-warrant analysis, answer-recovery tests, and detection of synthetic omissions [Emmons et al., 2025].RGDT-Bench specifies reference requirements for rule use, conditions, evidence, and aggregation.Source: §2 · p. 3

Process evaluation and supervision

  • Process-supervision benchmarks and reward models study error localization, step judgments, and trace preferences, whereas RGDT-Bench trains evaluators with response-level completeness labels [Uesato et al., 2022; Lightman et al., 2024; Wang et al., 2024; Zheng et al., 2025; Song et al., 2025; Zeng et al., 2025; Pandit et al., 2025; Zhang et al., 2026].Reward-model evaluations also cover preference, correctness, and reference-based scoring; RGDT-Bench tests warrant assessment and response selection against outcome supervision [Lambert et al., 2025; Liu et al., 2025; Tan et al., 2025; Yan et al., 2026].Source: §2 · p. 3

3 TASK AND WARRANT EVALUATION

RGDT-Bench formalizes tasks in which a decision must be justified by rules, conditions, and evidence. It evaluates whether the stated warrant covers required grounds and supports the decision, including when the answer is correct.

3.1 TASK DEFINITION AND RESPONSE GENERATION

  • RGDT instances pair a question, facts, and natural-language rules; the schema makes rule–condition and condition–evidence links explicit.A generated response contains a reasoning trace and separate terminal decision, while its stated reasoning is the warrant assessed for completeness.Source: §3.1 · p. 3

3.2 REFERENCE WARRANTS

  • A reference warrant records required condition judgments, support, and decision logic, with task-specific requirements and evidence alternatives.For Policy and Transaction, checks assess whether cited evidence supports a condition rather than requiring identification of every supporting passage.Source: §3.2 · p. 4

3.3 WARRANT COMPLETENESS AND CORRECT-ANSWER GAPS

  • Warrant completeness requires passing rule-use, condition, evidence, aggregation, and outcome checks under at least one reference warrant.A warrant may be incomplete even when every stated step is correct.Source: §3.3 · p. 4
  • CWG measures the proportion of evaluable correct responses with incomplete warrants, excluding extraction or parsing failures.It is computed within correct responses, so it is not the difference between overall answer accuracy and warrant completeness.Source: §3.3 · p. 5

4 BENCHMARK CONSTRUCTION AND VALIDATION

RGDT-Bench covers four task families with probes that vary how rules and evidence are presented. It samples model responses, extracts stated claims, checks them against reference warrants, and validates and balances the resulting labels.

4 BENCHMARK CONSTRUCTION AND VALIDATION

  • Four task families cover policy applicability, contract inference, administrative-rule application, and basketball-transaction compliance.The construction preserves source passages, their order, and reference decisions.Source: §4 · p. 5

4.1 WARRANT-ACCESS PROBE DESIGN

  • Eight task–probe combinations use bound evidence, hard negatives, or longer natural-document contexts across 904 prompts.All tasks support P1; Contract and Transaction also support P2/P3 because their source materials provide hard negatives and document context.Source: §4.1 · p. 5

4.2 RESPONSE SAMPLING AND AUTOMATIC ANNOTATION

  • Six generators produce 86,784 trace–decision pairs; an LLM canonicalizer extracts claims and a deterministic verifier checks them against reference warrants.Labels cover 86,435 responses and 202.1K condition judgments.Source: §4.2 · p. 6

4.3 LABEL VALIDATION AND BALANCED SPLITS

  • Manual review of 360 responses yields 84.2% agreement, Krippendorff’s α=0.687, 92.7% precision, and 78.5% recall for complete warrants.Splits group related cases and probe variants together and balance generators, tasks, warrant labels, probe coverage, and response length.Source: §4.3 · p. 6

5 EMPIRICAL EVALUATION

The evaluation traces correct-answer warrant gaps from their frequency and process-layer distribution to evaluator performance, then tests warrant-supervised assessment and response selection.

5.1 RQ1: HOW OFTEN DO CORRECT ANSWERS HAVE INCOMPLETE WARRANTS?

  • Correct-answer warrant gaps range from 24.8% to 76.7%; relaxing the aggregation check lowers the mean from 40.2% to 35.7%.The larger model in each family has higher warrant completeness, but stronger overall performance does not ensure complete grounds for correct decisions.Source: Table 1 · p. 6, §5.1 · pp. 6–7

5.2 RQ2: HOW ARE WARRANT GAPS DISTRIBUTED ACROSS REASONING LAYERS?

  • Rule use is the largest contributor to warrant failures across alternative task weights and P1-only evaluation; aggregation contributes more than evidence.Failures overlap: 35.9% of incomplete warrants fail at least two requirements, so the profiles locate weaknesses rather than isolate causes.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 RQ3: HOW WELL DO EXISTING EVALUATORS ASSESS WARRANT COMPLETENESS?

  • Among correct answers, the best existing evaluator reaches only 57.69% task-averaged AUROC, while GPT-5.6 Sol reaches 53.44%.GPT-5.6 Sol leads the full view with 70.19% AUROC; these results motivate explicit warrant supervision.Source: §5.3 · p. 7, Table 2 · p. 8

5.4 RQ4: CAN WARRANT SUPERVISION IMPROVE COMPLETENESS ASSESSMENT?

  • Among correct answers, warrant supervision raises AUROC over Full-response ORM by 10.37 pp, and Warrant-RM exceeds the best existing evaluator by 11.55 pp.At a score threshold, incomplete-warrant detection rises from 20.69% to 33.56%, while complete-warrant recall falls from 87.02% to 84.16%.Source: §5.4 · p. 9, Figure 5 · p. 9

5.5 RQ5: DO LEARNED WARRANT SCORES IMPROVE RESPONSE SELECTION?

  • At N = 16, Warrant-RM improves completeness and accuracy over both outcome-supervised models across the evaluated response-selection views.On all response groups, it gains 4.40 pp in completeness and 1.69 pp in accuracy over Final-only ORM, and 1.23 pp and 0.46 pp over Full-response ORM; in target-specific groups, it gains 2.18 pp and 1.66 pp over Full-response ORM.Source: §5.5 · p. 9

6 CONCLUSION

The conclusion frames RGDT-Bench as a benchmark for decisions and their grounds, linking layered diagnosis to warrant-supervised models for assessment and response selection.

6 CONCLUSION

  • RGDTs require models to apply external rules to case facts and justify their decisions.RGDT-Bench provides a unified, extensible schema with four task families, controlled probes, and balanced evaluation.Source: §6 · p. 9
  • Layered checks support joint diagnosis of reasoning failures that existing evaluators struggle to recognize among correct answers.The benchmark connects failure diagnosis with learning signals for RGDT reasoning.Source: §6 · p. 9
  • Warrant supervision improves reward-model assessment and downstream response selection over outcome-supervised baselines.The conclusion presents this as support for more reliable RGDT reasoning.Source: §6 · p. 9

7 LIMITATIONS AND FUTURE WORK

The evaluation leaves multi-turn interaction and tool use, visual evidence and document layout, and step-level reward modeling outside its scope.

7 LIMITATIONS AND FUTURE WORK

  • Multi-turn interaction and tool use remain untested because decisions use supplied inputs.Future work can track how warrants evolve as evidence accumulates.Source: §7 · p. 10
  • Text-only inputs exclude visual evidence and document layout, motivating multimodal extensions linked to condition judgments.The proposed extension would connect those sources to condition judgments.Source: §7 · p. 10
  • The study evaluates sequence-level reward models, not step-level process reward models, which require consistent reasoning-step segmentation and condition-level alignment.That alignment is needed to compare process rewards in RGDTs.Source: §7 · p. 10

AI USE STATEMENT

The appendix records AI assistance, maps its supplemental materials, and expands the reference-warrant checks and benchmark construction.

AI USE STATEMENT

  • Generative AI assisted with data synthesis, claim extraction during annotation, language polishing, and typesetting; the authors take responsibility for the final work.Annotation quality was assessed through human review and agreement analysis.Source: AI USE STATEMENT · p. 10

APPENDIX GUIDE

  • The appendix guide follows the paper’s logic from reference requirements and probe construction through annotation, evaluation, and warrant-supervision evidence.Prompt examples connect these stages through one case.Source: APPENDIX GUIDE · p. 14

A REFERENCE WARRANTS AND DIAGNOSTIC CHECKS

  • Reference warrants specify required judgments, their support, and decision logic; completeness requires consistent coverage even when fewer stated grounds suffice for the decision.Rule-use and evidence checks assess support for judgments, while aggregation checks how judgments combine into a decision; unresolved or unspecified support is handled separately.Source: A REFERENCE WARRANTS AND DIAGNOSTIC… · p. 15, Table 5 · p. 15

B TASK COVERAGE, DATA BALANCE, AND WARRANT-ACCESS PROBES

B.1 TASK COVERAGE AND LEAKAGE PREVENTION

The appendix details how data are split and balanced, how probes vary support access, and how extracted claims are verified against reference warrants.

B.1 TASK COVERAGE AND LEAKAGE PREVENTION

  • 90 test cases form 76 source groups, with related cases kept within one split to prevent leakage.All probes from a case share its split.Source: §B.1 · p. 16, Table 6 · p. 16
  • Training contains 51,832 evaluable responses—23,809 complete and 28,023 incomplete warrants—while 200 not-evaluable labels remain outside supervision.Source groups are also the resampling units for uncertainty estimates.Source: §B.1 · p. 16, Table 7 · p. 16

B.2 NATURAL AND BALANCED RESPONSE COLLECTIONS

  • Balanced subsets control task–probe coverage and generator, task, and warrant-label balance; natural responses support prevalence and attribution analyses.Within each task, supported-probe response counts differ by at most one; sequences are retained without truncation.Source: §B.2 · pp. 16–17, Table 8 · p. 17, Figure 6 · p. 17

B.3 CONSTRUCTION OF THE THREE WARRANT-ACCESS PROBES

  • P1 binds evidence to conditions, P2 adds confusable distractors, and P3 restores longer document context while preserving the required support and decision.The probes test different support-access demands, not a predetermined difficulty order.Source: §B.3 · pp. 17–18

C AUTOMATIC ANNOTATION AND HUMAN VALIDATION

  • The annotation pipeline separates label-blind extraction of stated claims from reference-based verification, then checks labels through human review.Generators and evaluators do not receive reference judgments; verification uses the private reference and, for WCV, the separately reported terminal decision.Source: C AUTOMATIC ANNOTATION AND HUMAN… · p. 18, §C.1 · p. 19

C.2 CLAIM EXTRACTION AND WARRANT VERIFICATION

  • Warrant checks assess task-specific judgments, supplied evidence, and how required judgments support the decision; Transaction also checks identification of the posed action.Labels measure warrant completeness, not instruction compliance alone, because prompts do not uniformly request exhaustive coverage.Source: §C.2 · pp. 19–20, Table 9 · p. 20, Table 10 · p. 20

C.3 HUMAN REVIEW OF ANNOTATION QUALITY

Human review measures annotation agreement, while the appendix defines the metrics, weighting, uncertainty procedures, and comparison conditions used for evaluation.

C.3 HUMAN REVIEW OF ANNOTATION QUALITY

  • 84.2% agreement, Krippendorff’s α=0.687, and 92.7%/78.5% complete-warrant precision/recall compare automatic labels with human judgments.Condition judgments reached 87.1% agreement (α=0.729), aggregation 81.5% (α=0.582), and terminal decisions 100%.Source: §C.3 · p. 20, Table 11 · p. 21

D EVALUATION MEASURES AND MATCHED COMPARISONS

  • The evaluation distinguishes answer accuracy, warrant completeness, and the correct-answer warrant gap, with the latter counting incomplete warrants among correct responses with evaluable labels.Not-evaluable warrant labels are excluded from the gap and supervision.Source: §D.1 · p. 21

D.1 METRICS, WEIGHTING, AND UNCERTAINTY

  • Evaluator discrimination, threshold identification, and probability quality use fixed evaluation weights; AUROC ranks complete against incomplete warrants, while Brier and ECE measure probability quality.Equal task averaging and source-group resampling support comparable summaries and paired uncertainty estimates.Source: §D.1 · p. 22
  • Paired comparisons resample linked source groups, reward-model summaries report variation across three fits, and selection gains use a source-group jackknife.All-response-group and target-specific opportunity views use shared eligibility and weights across selectors.Source: §D.1 · pp. 22–23

D.2 EXISTING EVALUATORS

  • Seventeen evaluators receive the prompt, native trace, and terminal decision without reference warrants or diagnostic labels; correct-only scores reflect both evaluator ability and criterion alignment.Native traces lack aligned steps and labels for step-level process reward models, motivating sequence-level scoring.Source: §D.2 · pp. 23–24

D.3 MATCHED REWARD-MODEL TRAINING

The reward-model comparison holds inputs, data splits, weights, and training budget fixed to isolate the supervision target. Supplementary checks find similar correct-only warrant discrimination under shared and warrant-balanced weights.

D.3 MATCHED REWARD-MODEL TRAINING

  • Full-response ORM and Warrant-RM share the same native trace, response weights, and training configuration, so their comparison changes only the supervision target.ORMs learn outcome correctness; Warrant-RM learns warrant completeness.Source: §D.3 · p. 24
  • All three reward models use a Qwen3.5-9B backbone and scalar scoring head, with shared optimization, sample-weighted training, and validation-based checkpoint selection.Inputs remain untruncated within the stated token limits, and test labels are excluded from fitting and model selection.Source: §D.3 · p. 24
  • Shared response weights equalize contributions across models and balance generator, task, probe, and correctness combinations without extra warrant-class weighting.The weights are iteratively adjusted to meet balance targets and normalized to mean one within each split.Source: §D.3 · p. 24
  • Warrant-RM achieves 69.24% correct-only AUROC with shared weights and 69.32% with warrant-balanced weights, supporting discrimination under both schemes.Shared weights are more concentrated: their Top 5% share is 19.84% versus 11.89%, and ESS/n is 45.78% versus 84.32%.Source: §D.3 · p. 24, Table 12 · p. 24

E SUPPLEMENTARY EVIDENCE FOR DIAGNOSIS AND SUPERVISION

E.1 ROBUSTNESS OF CORRECT-ANSWER WARRANT GAPS

The supplementary analyses test the stability of warrant-gap estimates, locate recurring process failures, and compare warrant supervision with outcome supervision. Warrant supervision improves discrimination among correct answers, while the analyses also characterize probe effects and weighting sensitivity.

E.1 ROBUSTNESS OF CORRECT-ANSWER WARRANT GAPS

  • Warrant gaps persist after excluding retried responses, under minimizing or maximizing labels for unavailable originals, and across all six generators.Removing aggregation checks lowers the mean gap from 40.2% to 35.7%, leaving substantial gaps in rule use, condition judgments, or evidence support.Source: §E.1 · p. 25

E.2.1 LAYERED FAILURE ATTRIBUTION

  • Correct responses with incomplete warrants often fail multiple layers: 385 of 391 missing-condition cases and 1,088 of 1,237 incorrect-condition cases also fail another layer.Attribution divides each response’s contribution equally among its failed layers; involvement rates can sum above 100%, while total attribution shares sum to 100% before rounding.Source: §E.2.1 · p. 25, Table 13 · p. 26
  • Rule use remains the largest attribution under all-probe and common-P1 comparisons, while aggregation overtakes evidence in both.P1-only evaluation also changes the response population, comparing 1,895 incomplete-warrant responses with 4,187 in the all-probe view.Source: §E.2.1 · p. 25, Table 13 · p. 26

E.2.2 MATCHED PROBE COMPARISONS

  • P3 reduces warrant completeness in eleven of twelve task–generator comparisons, whereas adding hard negatives in P2 changes it by a mean of only +1.02 pp.P3 also changes support position, density, and structure, so its effect cannot be attributed to input length alone.Source: §E.2.2 · p. 26, Table 14 · p. 26

E.3 PROBABILITY QUALITY AND INCOMPLETE-WARRANT DETECTION

  • Warrant-RM reduces Brier error by 0.0231 and raises incomplete-warrant detection by 9.11 pp in the full view and 12.87 pp among correct answers.These gains accompany a 2.86 pp decrease in complete-warrant recall.Source: §E.3 · pp. 26–27, Table 15 · p. 27

E.4 EFFECTS OF TRACE ACCESS AND WARRANT SUPERVISION

  • With identical full-response inputs and weights, warrant supervision improves warrant AUROC by 6.94 pp, AUPRC by 10.16 pp, and BalAcc by 3.13 pp.Outcome AUROC and AUPRC rise slightly, while outcome BalAcc decreases by 3.27 pp.Source: §E.4 · p. 27, Table 16 · p. 28

E.4.1 WARRANT ASSESSMENT AMONG CORRECT ANSWERS

  • 10.37 pp: Warrant supervision raises correct-only task-averaged AUROC from 58.87% to 69.24% with the same full-response inputs.Gains over Full-response ORM are positive across all four tasks, ranging from 5.08 to 17.37 pp.Source: §E.4.1 · p. 28, Table 18 · p. 28, Table 17 · p. 28
  • Warrant-RM ranks complete warrants higher and incomplete warrants lower than Full-response ORM, and its positive task-level AUROC gains extend across supported task–probe combinations.For complete-warrant median score percentiles, the rank shifts from 49.9 to 55.9; for incomplete warrants, from 46.8 to 38.6.Source: §E.4.1 · p. 29, Figure 7 · p. 29

E.5 RESPONSE SELECTION AND ROBUSTNESS CHECKS

The response-selection analyses compare selectors across groups with all collected responses and groups offering mixed labels for the target. Warrant-RM generally selects responses with higher completeness, with robustness checks preserving the main conclusion; the examples trace one Contract case through the benchmark pipeline.

E.5 RESPONSE SELECTION AND ROBUSTNESS CHECKS

  • The selection analyses distinguish all response groups from target-specific opportunity groups, which exclude groups lacking both positive and negative labels for the evaluated target.All selectors share groups defined independently of their scores.Source: §E.5 · pp. 29–30
  • Warrant-RM has the highest mean completeness across candidate counts N=2–16 in both evaluation views, with gains over Full-response ORM of 0.58–1.23 pp across all groups and 1.99 pp on average in opportunity groups.It also has the highest mean accuracy at N=6–16 for all groups and N=3–16 for target-specific opportunity groups.Source: §E.5 · p. 30, Figure 8 · p. 30, Table 20 · p. 31
  • At N=16, restricting weights to supported combinations changes the accuracy gain over Full-response ORM from 1.66 to 1.77 pp, and an alternative tie-break leaves Warrant-RM ahead in completeness and accuracy.Source: §E.5 · pp. 30–31

F PROMPT EXAMPLES FOR GENERATION, ANNOTATION, AND EVALUATION

  • Figure 9 shows one Contract case under P1 bound support, P2 added hard negatives, and P3 longer natural-document context while preserving the decision.The same return/destruction clause receives different passage identifiers across prompts; R01–R03 name conditions, not reference judgments.Source: F PROMPT EXAMPLES FOR GENERATION,… · p. 32, Figure 9 · p. 32
  • The example trace states three condition judgments but uses only one in its final aggregation, producing an incomplete warrant despite a supported negative decision.Figures 10–12 illustrate generation, reference-based claim extraction and verification, and the LLM-judge input and output format.Source: F PROMPT EXAMPLES FOR GENERATION,… · pp. 32–33, Figure 10 · p. 32, Figure 11 · p. 33

Related papers

Loading 2609.34455v1…